Repository-Level Prompt Generation for Large Language Models of Code
Authors: Disha Shrivastava
Hugo Larochelle
Daniel Tarlow
Venue: Submitted to ICLR 2023 (2023)
Keywords: prompt generation
codex
large language models of code
code-autocompletion
source code
LLM
retrieval

Abstract: With the success of large language models (LLMs) of code and their use as code assistants (e.g.\ Codex used in GitHub Copilot, techniques for introducing domain-specific knowledge in the prompt design process become important. In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using prompt proposals. The prompt proposals take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g.\ imports, parent class files). Our technique doesn't require any access to the weights of the LLM, making it applicable in cases where we only have black-box access to the LLM. We conduct experiments on the task of single-line code-autocompletion using code repositories taken from Google Code archives. We demonstrate that an oracle constructed from our prompt proposals gives a remarkably high relative improvement of 36\% over Codex, showing the quality of these proposals. Further, we show that when we train a model to select the best prompt proposal, we can achieve significant performance gains over Codex and other baselines.

Under review as a conference paper at ICLR 2023
REPOSITORY-LEVEL PROMPT GENERATION FOR LARGE LANGUAGE MODELS OF CODE
Anonymous authors Paper under double-blind review
ABSTRACT
With the success of large language models (LLMs) of code and their use as code assistants (e.g. Codex (Chen et al., 2021 used in GitHub Copilo techniques for introducing domain-specific knowledge in the prompt design process become important. In this work, we propose a framework called Repo-Level Prompt Generator that learns to generate example-specific prompts using prompt proposals. The prompt proposals take context from the entire repository, thereby incorporating both the structure of the repository and the context from other relevant files (e.g. imports, parent class files). Our technique doesn't 't require any access to the weights of the LLM, making it applicable in cases where we only have black-box access to the LLM. We conduct experiments on the task single-line code-autocompletion using code repositories taken from Google Code archives. We demonstrate that an oracle constructed from our prompt proposals gives a remarkably high relative improvement of 36% over Codex, showing the quality of these proposals. Further, we show that when we train a model to predict a prompt proposal, we can achieve significant performance gains over Codex and other baselines.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable performance in natural language processing tasks (Brown et al. 2020 Chowdhery et al. (0022). text-to-image generation (Ramesh ctt al. 2022 Rombach et al. 2021), protein-sequencing (Rives el al. 2019) and even as a generalized agent RRed et al. 2022). As opposed to the rnetrain-finetune paradigm. prompting these LLMs has been found to yield good performance even with few- few-examples (Liu et al.) (021a). A prompt is an input to the LM such that the desired task can be expressed as predictions generated from the LM. Besides providing a mechanism to control and evaluate a L.M, prompts have shown to elicit emergent behaviour as well. Examples of this behavior include GPT-3 (Brown al., 2020)) doing better in tasks it has never seen during training and improved reasoning capabilities with few-shot (Wei et al., 2022) and zero-shot (Kojima et al. (022)) prompts that encourage a chain of thoughts. These factors highlight the importance of designing
effective task task-specific prompt However, currently have limited understanding of how to do this (Reynolds & McDonell, (021).
LLMs have also been used for modeling source code with impressive results (Austin et al. 2021
Fried et al. 2022; Xu et al., (2022a). In particular, one of the best performing I.LM, Codex (Cen al.
2021 has been deployed part of GitHub Copilot 0 state-of-the-art in-IDE code assistant. Despite the growing popularity of L.MSs of code, there is no work that systematically tackles different aspects of prompt generation in relation to source code. One such aspect is that when it comes to code, the relevant context to be put in the prompt can come from not just the current file, but also from outside, such as imports and parent classes. Also, depending on the scenario, the relevant context can be scattered across multiple locations. Since the LLMs have a limited context length available for the prompt, becomes increasing crucial for our -peaifc understanding to guide the selection of relevant context. Currently, it is not clear how to integrate this domain knowledge of what constitutes a relevant context, into the process of creating prompts. Addressing this question has potential benefits in other domains such as question answering (Liu et al. (022) and multi- summarization (Xiao et al. 2022), where domain -spccffcc structured retrieval context can be useful.
Platforms such as PromptBasehttpss// https::/prropttaaee allow huying and selling of prompts.

---

Under review as a conference paper at ICLR 2023
Repository
Current file Alw.sshhhoppppsoo..e jeur
1ode/
seport sempler httpeeeeeeeeetttttttttaeeeeeeeee
Predicted Hole
GabeParsmate,, 2A33
Codex
complen
Nacial eenpeeeeenlllileeeeeeeeoe lampler new
Prompt
public vead
[aat]] al
Contect
pewteeaett clone
Target Hole
juatccaloraad teee:
Predicted
Prompt
Propseal
suport saaples Masiia www.ssaeoooioooooo saaples
Hole Position
Cortent
Codex
Prompt Engineer
Naxtel www.giieeeeeeeeeo pwwciesssogiiin (suataccs )
Prompt Composer
Prompt Proposal
Clatil
p
Classifier
aicessymnttaalooe Zeue yaatscalivaad tces:
Repo-Level Prompt Proposals
Repo-Level Prompt Generator
Import file Macimizingtillas Bamplerijen
Figure I: Repo-Level Prompt Generator: The prompt is generated by combining the context from the predicted prompt proposal p 14, i.e., method names and bodies from the imported file, MaximizingGiobsseeee
java (violet) with the default Codex context (gray).
In this work, we address this problem by proposing Repo-Level Prompt Generator (RLPG), framework that while generating the prompt, incorporates both the structure of the repository as well as the relevant context in all the files of the repository. In RLPG, the choice of where from and what to take from the repository is specified by a set of prompt proposals. For examplc, one of the prompt proposal can be to take all the identifiers used in the first import file. These prompt proposals allow the prompt engineers to induce their domain expertise in the prompt-designing process. With the increasing use of LLMs as assistive agents to humans, demand for transparency and the desire for software engineers to take active part in tailoring prompts to suit their requirements (Jiang et al., 2022: Sun et al. 2022). , this capability becomes important. As suggested in some previous works in NLP Shin et al. 2020 Schick & Schüitze, 2021). our prompt proposals are discrete. However, rather than fixing one particular prompt proposal for each example, we instead predict the best prompt proposal conditioned on the example. We do this by coming up with a neural network called Prompt Proposal Classifter (PPC), that given an example, learns to select a prompt proposal such that the resulting prompt is likely to produce the desired output. Therefore, RLPG allows the introduction of domain expertise, and at the same time facilitates automatic example-specific prompt generation via learned neural network. Note that there are some techniques for automatic prompt generation in NLP (Li & Liang 2021 [Shin et al. 2020 Lester et al., 202T) that require updating some or all of the weights of the LLM. However, the strongest LLMs are not publicly available (e.g. OpenAl provides access only to the generated output from Codex via an API and no access to model weights and data provided), making these techniques less useful under this scenario. RLPG addresses this limitation by generating prompts assuming only black-hox access to the L.M.M
We focus on the task of single-line code- autooopploooonnn in an IDE, where the objective is to predict the blanked-out portion (or target hole) starting from the position of an imagined cursor to the end of line. We operate under the line-level maintenance setting (Shrivastava et al. 2020 Hellendoorn & Devanbu 2017) that reflects the scenario where user is editing existing file. This means that there can be code following the line. Figure providess an illustration of our approach. The prompt proposal classifier takes in the hole position (position of the cursor) in the current file, the repository to which the current file belongs and a set of repo-level prompt proposals as input, and predicts a prompt proposal. In our illustrated example, the predicted prompt proposal corresponds to taking the method names and bodies from Maximi MaximizingGii SaaggeiibbSaammeeee.
java (mg before the hole position indicates that method from the imported file likely to be invoked). The Prompt Composer uses the context from the predicted prompt proposal and combines it with the default Codex context,
2

---

Under review as a conference paper at ICLR 2023
Le., code prior to the position of the hole in the current file. The resulting prompt consists of the method name initializeToAs poosssiiiienttt gnment (from the prompt proposal context) and the method CurrentAssignment
() (from the default Codex context), resulting in successful prediction (brown box on the top) of the target hole. Our key contributions are as follows:
We propose a framework called the Repo-Level Prompt Generator (RLPG) that learns to generate prompts conditioned on the example, without requiring access to the weights of the LLM. To incorporate prompt proposals. These prompt proposals are designed to incorporate both the structure of the repository as well as the relevant context from all files in the repository.
in the prompt design process, RLPG uses a set of repository-level
On the task of single-line proposed prompt proposals gives up 36% relative improvement over Codex. This improvement is pleasantly surprising as Codex has never seen prompts made from these prompt proposals during our prompt proposal classifier to predict the best prompt proposal, we can achieve up to 17% relative improvement over Codex.
aull--auoocomletion, we show that an oracle constructed from our
training. Further, we show that when
REPO-LEVEL PROMPT GENERATOR (RLPG)
In this section, we provide details of our framework. We start by describing our prompt proposals and then discuss our prompt proposal classifier which is followed by a description of prompt composer.
2.1 REPO-LEVEL PROMPT PROPOSALS
The core idea of RLPG consists of substituting part of the default context used by Codex with context coming from somewhere else in the repository. The decision of what to take and from where in the repository to take from is governed by set of prompt proposals. These prompt proposals were decided based on manual inspection of our training data and intend to capture common coding patterns (but more generally can also include prompt proposal can be thought of as function that takes input target hole's ' position and the repository that the hole is a part of, and that returns the prompt proposal context (a string constituted by the context from the prompt proposal). A prompt proposal is specified by prompt source and prompt context type. We mention each of these along with their motivation below.
coding practises). A
Prompt Source: For target hole position, a prompt source determines from where should take code that will be part of the prompt proposal context. We propose ten different prompt sources:
1. Current: take code from the current file excluding the contents of the target hole. The current file is the file that contains the target hole. The code in the current file (e.g. the lines after the hole position) can be very useful in predicting the target hole.
2. Parent Class: take code from the file that contains the parent of the class to which the target hole belongs. The intuition behind this is to account for cases where method present in the parent class is invoked in the current file (i.c. the child class).
3. Import: take code from the import files used in the current file. The dependencies specified via imports can provide useful cues to predict the target hole.
4. Sibling: take code from the files that are in the same directory the current file. Files in the same directory tend to share code variables (e.g. identifiers).
5. Similar Name: take code from files that have similar name as the current file. Similar names are determined by doing a splitting of the file name based on underscore or camel-case formatting and then matching parts of the filename, If one or more parts matches, two files are considered to have similar names. The intuition behind this is that software developers tend to name files based on the functionality of the code written in that file. Therefore, a similar name file might contain some portion of the code that is common with the current
file and hence might be useful for predicting the target hole.
6. Child Class: take code from files that have the current file their parent class file.
7. Import of Parent Class: take code from the import files used in the parent class files.
8. Import of Sibling: take code from the import files used in the sibling files.
9. Import of Similar Name: take code from the import files used in the similar name files.
10. Import of Child Class: take code from the import files used in the child class files.
3

---

Under review as a conference paper at ICLR 2023
The last four prompt sources are useful when the target hole occurs at the very beginning of the current file. In these cases, there would be less context coming from other prompt sources. For each prompt source, we can get either single file ranked list of files (see Appendix [B.I). In the latter case, we will take context from these files until we exhaust the maximum context length allocated to the prompt proposal.
Prompt Context Type: The prompt context type determines what code to take from the prompt source. We propose seven different prompt context types (Appendix B.2 has examples of each type):
1. Post Lines (PL): Take all the lines after the target hole line till we reach the end of the file. This context type is applicable only when prompt source is the current file
2. Identifiers (I): Take all the identifiers used in the prompt source.
3. Type Identifiers (TT): Take all the type identifiers used in the prompt source.
4. Field Declarations (FD): Take all the field declarations used in the prompt source.
5. String Literals (SL): Take all the string literals used in the prompt source.
6. Method Names (MN): Take all the method names along with their signatures that are used
in the prompt source.
7. Method Names and Bodies (MNB); Take all the method names along with their signatures and corresponding bodies used in the prompt source.
By combining prompt sources with prompt context types, we get a total of 63 prompt proposals (see Appendix B.4 for details). Note that depending on the target hole, not all prompt proposals would be applicable (e.g. if there are no parent classes in the current file, prompt proposals with prompt source as parent class file won't 't be applicable). In Figure]] [ the predicted prompt proposal corresponds to taking prompt source Import and prompt context type MNB. We aimed for set of prompt proposals that offer more diversity rather than a set of prompt proposals that are all good. This in turn ensures that for any hole position, significant number of prompt proposals are applicable.
2.2 PROMPT PROPOSAL CLASSIFIER (PPC)
Given hole position, the goal of the prompt proposal classifier is to predict the prompt proposal p that will lead to success, where success happens when the predicted hole h exactly matches the target hole h. This task is formulated as a multi-label binary classification problem since for a given target hole, more than one prompt proposals can lead success. In this formulation, we treat the default Codex context as one of the prompt proposals. Next, we describe the training procedure for PPC.
For each target hole h, we generate a ground-true -tuuh vector Y = [ which is a multi-hot vector of size M, where M is the total number of prompt proposals. This vector is obtained by feeding the prompt generated from prompt proposal p into Codex and then seeing whether h = h. there is a match, we say that the prompt proposal p is successful. For hole h, if a prompt proposal p is applicable and leads to success, 2/ and will be zero otherwise. For each hole h, we obtain mask Th where Th 1 when p is applicable or zero otherwise. The overall training loss L can be expressed as the sum of individual hole losses 2 as follows:
Training:
N
N
M Σ
BCE
1/) To where M
Σ Tm
M
A=1 1=1
In the above equation, N is the total number of holes encountered while training. hh denotes the total number of applicable prompt proposals for i and BCE corresponds to the binary cross entropy loss. Masking ensures that we consider only the prompt proposals that are applicable. Next, we describe our two variants of PPC that be used obtain the prediction i/.
Let H be the hole window that includes code present around the hole h excluding the hole itself. In our work, we take two lines before the hole position, the code up to the hole position and two lines after the hole position. We use a pretrained model F& to obtain a context representation vector of size Z, where Z is the dimension of the hidden state of the model. Specifically, we take the hidden state at the first position, i.c. the representation of the [CLS] token. To make training of PPC
RLPG-H:
We also conducted experiments (Appendix D.2 where we take lines starting from the 4th line after the hole.
4

---

Under review as a conference paper at ICLR 2023
computationally efficient, the parameters ó are frozen during training. The RLPG-H model takes the context representation of the hole window and projects it to the prompt proposal space of size M via two dense layers with non non-linenn linearity in between (see Equation 2) Taking the sigmoid of this output gives the prediction of the prompt proposal.
up = P(y = 1 = (g (F,(H))) + +
(2)
The motivation behind this variant is to use the similarity of the hole window and the prompt proposal context determine which prompt proposal can be useful. Given a particular hole h, let C denote the prompt proposal context from prompt proposal p Intuitively, if the hole window contains variables (e.g. identifiers) that are similar to the variables in the prompt proposal context, then there are chances that might occur somewhere in Cp The similarity is modeled using multiheaded attention mechanism (Vaswaní et al. 2017). by treating the projected hole window representation as query Q" and the projected prompt proposal context representation K as key (Equation 3
RLPG-R:
The value V is the same the key.
Q"= (H
K =
V F
(3)
Qh Kh.
Att(Q", MM("",K), V/h) = 1,, softmax
(4)
ddx
MultiHead(Q' K V W concat (head,, headz, head,)
(5)
where head; = AU(W? Q", W. "
1/ P 111 ch) = sigmoid (W,.(Mulliffeadd (Q", KD, v/h) + bn)
(6)
In the equations above, dg is the dimension of the key, W.O, W.K. WV are the query, key and value projection matrices, t is the number of heads and W the linear projection that combines the heads. The output from Equation 5 is fed to module G' consisting of two-layers of feedforward network with relu activation in between (see Appendix for more details). The resulting output is then linearly projected and sigmoid is applied get the predicted prompt proposal (Equation 6
2.3 PROMPT COMPOSER
The prompt composer combines the context from the selected prompt proposal (given by PPC) with the context normally used by Codex (default Codex context) to generate the prompt. Since the total length that can be used for prompt is fixed, we adopted dynamic context allocation strategy where the prompt proposal context is shorter than its allocated length, we assign the remaining portion from the prompt proposal context to the default Codex context. The prompt proposal context is always added before the default Codex context. For all prompt proposals, we assign half of the total context length to the prompt proposal context and the remaining the default Codex context. For post lines, in addition, we also assign one-fourth and three-fourths of the total context length to the prompt proposal context. If the prompt proposal context the default Codex context is greater than the context length allocated to it, we truncate it (see Appendix B.3 for our truncation strategies).
3 EXPERIMENTS AND RESULTS
In this section, we describe our process of dataset creation, details of experiments along with their results and interesting ablation studies.
3.1 DATASET CREATION
To mitigate the effects caused by potential memorization of the code present in the dataset used for
training Codex, we avoided code repositories from GitHub (Chen et al., (021).. Instead, we scraped Google Code for repositories in Java (removing the ones that matched with repository GitHub
$ http:://codeeggggll..oommaa
5

---

Under review as a conference paper at ICLR 2023
with the same name). We selected the repositories that had a permissive license giving a total of 47 repositories. We divided the repositories into train, validation and test splits, where each repository in its entirety is part of a split. In each file within repository, comments, and set the hole position to be the middle character in the line. All the characters from the middle position the end of the line constitute the target hole.
remove lines that are blank and
Since code duplication has been shown to have adverse effects (Allamanis 2018), within a repository, we look for files that are exact replica of each other, but placed in a different folder. We mark all such copies as duplicates and omit all of them when creating target holes for our dataset. Note that the prompt proposal context can still come from the duplicate files. We felt comfortable with this choice since we wouldn't it want to predict target hole in duplicate file, but we can still use the context from the duplicate file to predict the hole in a file that is not its duplicate (e.g. in a sibling file). Further, we found that the repositories were quite uneven in terms of their size. To avoid large repositories dominating the training of PPC, we capped the maximum contribution of holes from a repository to 10,000, i.e. if the total number of holes in the repository exceeded 10,000, we selected 10,000 holes randomly from the total holes.
Please see the left part of Figure for statistics of our dataset. The #Holes represents the holes after deduplication and capping. For some of our prompt proposals, we require semantic information that that
can be obtained with a parse tree. We used the tree-sitter API for Java that enables us to get the AST of a file and query it. Since our prompt proposals need information at a repository level, we stored some extra information that allowed to collate the information from individual files according to the directory structure inside the repository (see Appendix A for more details).
3.2 EXPERIMENTAL DETAILS
We used the OpenAI Codex Completions API for generating the predicted hole from the Codex model. In particular, we used the code seade-ddvinni--00 engine with temperature set to 1.0 and stop criteria as newline. The completion length was kept to be 24 and the maximum prompt length was 4072. Tokenization was done using the suggested tokenizer To allow for fast computation, we used simple models like CodeBERT (Feng et al. (2020] and GraphCodeBERT (Guo) al.) 2020) as our pretrained models. One of the limitations of these pretrained models is that the maximum context length that can be taken as input by these models is much smaller than the maximum context length allowed by Codex. Therefore, when getting the representation of the prompt proposal context that is used by PPC, we need to truncate the prompt proposal context that might lead to omitting important parts of the prompt proposal context in certain cases. Using pretrained models that allow larger context length or models that augment the context (Wu et al. 2022) offer avenues for future work. See Appendix D.4 for results when using a smaller context length from Codex.
Prompt Generation:
Computational Complexity and Scalability of RLPG:
To collect the ground-truth data for training our prompt proposal classifier, we queried the Codex API for each applicable prompt proposal per hole (maximum rate limit of 400 holes per minute). The computational complexity of training our larger RLPG-R variant (3.6M parameters, 141269 holes and 9.19 minutes per epoch on a single Tesla V100 GPU) is much smaller than finetuning all or some part of Codex (12B parameters). During inference, we need to calculate the repo-level statistics just once and all the subsequent hole completions in the repo can utilize this cached information, incurring additional computational complexity. Besides training the PPC, all our experiments were performed on CPU with 8GB RAM. Our prompt proposals are based on concepts such as post lines, imports, similar name files, method names and identifiers that are quite general and are applicable to other programming languages. In addition to the existing prompt proposals, our framework provides the flexibility to incorporate new prompt proposals. Since the cost of retraining RLPG with the extended prompt proposals is extremely low (much lower than finetuning Codex with the new prompt proposals), our framework can be used to make interventions on the LLM to address observed weaknesses as long as the intervention can be expressed as a prompt proposal that adds the missing context to the LLM. As opposed to techniques that perform prompt engineering in the latent space and require access to the weights of the LLM such as Li & Liang 2021), h, RLPG facilitates expressing intent in the form of prompt proposals that are intuitive for humans, easy understand and do not require access to the weights of the LLM.

---

Under review as a conference paper at ICLR 2023
Data
SR
SR Oracle(%
Rel. f over Codex(9) (%)
Feature
Train
Val
Test
Total
Split
Codex(% (%)
# Repositories
19
14
14
47
Train
59.78
80.29
34.31
Files
2655
1060
1308
4757 189557
Val
62.10
79.05
27.28
Holes
92721
48548
48288
Test
58.73
79.63
35.58
Figure 2: (Left) Statistics of our dataset; (Right) Performance of the oracle relative to Codex.
Methods: We experimented with the following methods for generating the prompt:
1. Codex: Using the default context from Codex the entire prompt.
2. Oracle:
Using the ground-truth vector Y (mentioned in Section 2.2 The prompt generated corresponds to using any of the successful prompt proposals (i.e., v 1). Since this information is not available at inference, the oracle performance represents an upper bound.
3. Fixed Prompt Proposal: Using the most successful prompt proposal for target holes. This was chosen based on the performance on the validation set and corresponded to taking 75% of the total context length from post lines in the current file.
4. RLPG-H and RLPG-R: Using the prompt proposal predicted by the RLPG-H and RLPG-H
varients of PPC. The selected prompt proposal corresponds to taking the argmax of the predicted probabilities over different prompt proposals.
5. RLPG-BM25: Instead of using PPC to rank prompt proposals, using the scores obtained by BM25 (Jones et al., 2000)) to select the best prompt proposal. The scores calculated with the hole window being the query and prompt proposal contexts being the search documents.
This serves non-learned retrieval method that makes use of our prompt proposals.
6. File-level BM25: Same as above, except that instead of using our prompt proposal contexts, search documents consist of full context from other files in the repository.
7. Random: For each target hole, select a context randomly from anywhere in the repository.
8. Random NN: Same as Random, except that amongst the randomly chosen contexts, we
take the nearest neighbours of the hole window in the representation space of pretrained model. This analogous to the technique used Liu et al. (2022).
9. Identifier Usage: For each target hole, take the closest identifier and take usage windows that identifier from everywhere in the repository. We take two lines above, two lines below and the usage line as the usage window. We can rank the usage windows either randomly (random) or based on the nearest neighbour distance to the hole window in the representation space (NN).
The last four methods help us understand the performance when a context other than the prompt proposal context is used. To generate prompt using these methods, we take 50% of the context from these followed by the default Codex context that takes up the remaining context length. For the NN baselines,
CodeBERT (Feng et al.) 2020) as the pretrained model. The contexts are taken in the increasing order of the nearest neighbour distances, until we exhaust the allocated context length. RLPG-BM25 helps us understand the role of PPC. See Appendix C.3 for more details on the implementation of these methods.
Evaluation Metrie:
As mentioned in Section 2.2 the predicted hole string generated by Codex and the target hole string, In our experiments, we report the percentage of successful holes divided by the total number of holes for each split. We will call this success rate (SR) going forward.
measure success, used exact match between
3.3 RESULTS
In this section, we present the results of the following two research questions explored in this paper:
[RQ1] Is it useful to generate a prompt that is composed of code context that is different from the default Codex context? If yes, what context can be useful?
[RQ2] For each target hole, is there a way of automatically selecting the prompt? If yes, how does this system perform relative to Codex?
RQ1 Performance of Prompt Proposals: We found that combining the prompt proposal context (context from other files in the repository) with the default Codex context led to substantial improve-
7

---

Under review as a conference paper at ICLR 2023
Table 1: Success Rate (SR) of different methods on the test data when averaged across all holes (hole-wise) and across individual repositories (repo-wise)
Success Rate( (6
Rel. 7(%) (hole-wise)
Success Rate( (repo-wise)
Rel. 7(%)
Method
(hole-wise)
(repo-wise)
Codex CChen et al.
2021
58.73
60.64
Oracle
79.63
35.58
80.24
32.31
Random
13
-1.02
58.95
-2.79
Random NN
58.98
0.43
60.04
-0.99
File-Ievel BM25
63.14
7.51
64.28
6.00
Identifier Usage (Random)
64.93
10,55
67.83
11.85
Identifier Usage (NN)
64.91
10.52
67.94
12.03
Fixed Prompt Proposal
65.78
12.00
68.01
12.15
RLPG-BM25
66.41
13.07
68.15
12.39
RLPG-H
68.51
16.65
69.26
14.21
REPG-R
67.80
15.44
69.28
14.26
ment in performance. The right part of Figure shows the performance of an oracle constructed from our prompt proposals. We see that across all data splits, the prompt proposals contribute to significantly large improvements over Codex (upto 36% for test split). These results might seem surprising as Codex has not been trained on prompts that consist of context other than the default Codex context. What makes this result more surprising is that in most of the cases, the prompt consists of mashed up context without logical ordering that may not even look like semantically meaningful chunk of code (e.g. list of string literals from a sibling file followed by the default Codex context post lines placed before the default Codex context opposed to after). These results might suggest that as long as the relevant context (in our case repo-level knowledge in the form of prompt proposals) is present in any form in the prompt, it can be quite effective.
RQ2 Performance of PPC: Having seen promise in our prompt proposals, next we present the results of RLPG, which for each target hole predicts single best prompt proposal. Table I presents the success rates along with the percentage of relative improvements for the test data. The second and third columns correspond to the averages across all holes in the data. The last two columns correspond to the average success rate of individual repositories. The latter metric doesn't 't account for the size of the repository. As can be seen from the table, all the RLPG variants as well as the fixed prompt proposal improve the performance significantly over Codex. The random baselines are either worse or on par with Codex. Identifier usage is good baseline but still performs worse than either the fixed prompt proposal or RLPG. The improved performance of HLPGGBM25 as compared to fixed prompt proposal shows the value of generating example-specific prompts using RLPG. However, both the learned variants of RLPG, i... RLPG-H and RLPG-R outperform the RLPG-BM25, highlighting the importance of learning PPC. See Appendix D for performance of all methods on individual repositories. Note that even though consider identifier usage separate baseline, one could consider it as one of the prompt proposal leading to further improved performance of RLPG.
Despite our efforts of avoiding overlap, since the training data for Codex is not exactly known, there might be a slight possibility that part of our Google Code data is part of the training data for Codex. Even if there were an overlap, we want to point out that since Codex has seen the default Codex context during training, it would be more beneficial to use the default Codex context in the prompt rather than the context from the prompt proposals or any other context from other baselines. Therefore, under this scenario, our evaluation would be more generous to the Codex baseline with results biased more in favour of the Codex baseline than other methods we have used.
Variation with #attempts: Imagine scenario where we have a human-in -omman-i--the-loop who has been given ie attempts to prompt the L.M and then can choose one of the & hole predictions. We wanted to see how does the performance of our framework varies with #attempts under this setting. This corresponds to using & prompts generated with top-k prompt proposals (one prompt per proposal) and marking success if any of the k prompts lead to success. The left part of Figure 3 shows the variation of SR over the validation data with the value of k. For RLPG, the top-k prompt proposals were chosen based on the decreasing order of probabilities given by PPC. For the fixed prompt proposal, the top-k prompt proposals were decided based on decreasing order of success rate of the individual prompt proposals on the validation dataset. From the figure, notice that we increase the value
8

---

Under review as a conference paper at ICLR 2023
Out
78
200 5b
Vil Suscees Rate (53
5imh
76
Inplimh
Imo5ib
72
Proposal (repo-sise)
70
RLPG (nole weal
Flawd Promost Praposal (holensis))
(rcpo
10
ImpChel
10 1617
Figure 3: (Left) Variation of RLPG and Fixed Prompt Proposal with #attempts (k); (Right) Mean success rates of different prompt sources when they are applicable.
of k. the performance increases gradually at first and then saturates towards the oracle performance (79.05% for val data). This behaviour is observed for both fixed prompt proposal well RLPG. However, we see that for the same value of k. the success rate for RLPG is higher indicating that PPC learns useful ranking of the prompt proposal contexts that scale well with the #attempts.
Performance based on Prompt Proposals: The right part of Figure 3 shows mean success rate of prompt sources when count success only when the corresponding prompt source is applicable. From the figure, we see that the current file is the most important prompt source. Closely following are sibling files and similar name files. We see that all prompt sources have non-zero chances of success, highlighting the usefulness of each prompt source. See Appendix [.I on prompt context type and Appendix [ for analysis of successful and failed sample cases.
2.1]for for a similar breakdown based
4 RELATED WORK
Recently, there has been lot of work around large language models of code. One class of models are the decoder-only models that correspond to generating code from left-to-right.
LLMs for Code:
Codex (Chen al. (00TT) Google's model (Austin al. (02T), GPT-J-6B (Wang & Komatsuzaki
2021b, GPT-Neo Black et al. 2021b). GPT-Neo-X (Black et al.
2021a). CodeParrot (Tunstall et al.
2022), ) PolyCoder (Xu et al. 2022a) and InCoder (Fried et al.)
2022) are some examples. We also
have some encoder-only models that use masked language modelling objective. CodeBERT (Feng) ct al. 2020), GraphcodeBBRRT (Guo et al. 2020) and CuBERT (Kanade al., (020) are examples of such models. Lastly, we have the class of encoder-decodes models that generally use bidirectional encoding of a context to decode a series of masked tokens. Code-T5 (Wang et al. 2021) and
AlphaCode (Li et al. 2022) are examples of such models.
Fewer works use information from outside the current file. Hellendoorn Devanbu (2017) propose nested n-gram model that utilizes locality- based cache where the locality consists of all directories from the root of the project (inclusive of the current file). Zhang et al. (2021) uses the parent class generate the comments for the child class. Paashkkaanloo al. 2022b capture the structure and semantics of the repository by converting it into a relational database and propose graph-walk based mechanism for pruning the unrelated context. Lyu et al. (2021) incorporates the API-dependency graph in a L.STM-based Seq2Seq model to assist in code generation. Xu et al. (2022b) incorporate three types of structural locality features while training the kNN-LM (Khandelwa wwal et al. 2020). These features are binary variables that correspond to the presence or absence of similar hierarchy. The three levels of hierarchy are (a) sibling file, (b) file in the same repo (c) no hierarchy. In contrast we have a much richer set of prompt proposals incorporating the semantics and structure of the repository. Also, we assume black-box access to the actual LM and restrict ourselves to generating prompt for the LLM without performing any finetuning of the LLM.
Repo-Level Info:
There have been promising works around prompt generation techniques in NLP. Broadly, there are two categories of automatic prompt generation techniques. The first category corresponds to producing continuous/soft prompts where the prompt is described in the latent space of language model (Li & Liang| 2021 Qin & Eisner 2021 Bragg et al.) 2021] Lester al.) 2021 Liu et al. 2021b). For example, Prefix- Tuning (Li & Liang (021)) adds a prefix to the LM that can be learned by finetuning examples from the downstream task. The second category produces discrete
Prompt Generation:

---

Under review as a conference paper at ICLR 2023
prompts where the prompt is text string that can be interpreted by human (Shin et al. 2020 Gao al. 2021 Schick & Schüize 202T). For example, Autoprompt (Shin et al. 2020) generates prompt using fixed template consisting of trigger tokens. The trigger tokens are shared across all inputs and determined by mandeett-gudded search involving the L.M. Our work falls in the category of discrete prompt generation techniques we produce prompt consisting of code tokens that can be easily interpreted by a human. However, in contrast to prior works that use set of fixed templates for all examples, we learn to produce prompts conditioned on each example. Another important distinction is that we do not require access to the weights of the LM. A concurrent work as ours (Wang et al. 2022) studies the role of prompt-tuning when compared to fine-tuning for code translation, defect localization and code summarization. However, their technique requires access to the weights of the LLM and they perform experiments over models that are much smaller scale than Codex. To the best of our knowledge, our work is the first to explore automatic prompt generation in a black-box access setting in the domain of source code.
5 CONCLUSIONS AND FUTURE DIRECTIONS
We present RLPG, a framework that learns to automatically generate prompts conditioned on the example, without requiring access to the weights of the LLM. RLPG utilizes the structure of the repository as well as the context from other files in the repository using a set of easy to understand prompt proposals. Note that even though we have scoped and worded our prompt proposals to be repository-level. the idea of RLPG and prompt proposals in itself is quite universal and need not be scoped to repository. Taking context from other repositories well as external knowledge such as API dependencies offers an interesting direction to explore in the future. In this work, we are taking context from only one prompt proposal. For future work, want learn model that can automatically compose a prompt from multiple prompt proposals (see Appendix D.3 for promising initial results). Other interesting directions include incorporating the user's feedback in RLPG and extending RLPG to multi-line code autocompletion.
REFERENCES
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code, 2018. /tL/Lttps.//rxiv.orgg//////////
06469
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint (0Xiv:2/08.077::: 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016
Sid Black, Stella Biderman, Alex Andonian, Quentin Anthony, Preetham Gali, Leo Gao, Eric Halla- han, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Jason Phang, Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. GPT-NcoX: Large scale autore- gressive language modeling in pytorch, 2021a. URL http::tttttttttte
com/ hooo/eleuue eleutherai
pypt-neox
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfiow, March 2021b. URL /htpps:/ 5297715 If you use this software, please cite it using these
doi. htt.loogg....211//// metadata.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. FLEX: Unifying evaluation for few-shot NLP. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htRL:hhtt:://// ptttts::/oppnrreiiee. www./eee//eee. forum? id= WnGcwXI.YO3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vors Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and
10

---

Under review as a conference paper at ICLR 2023
Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems. volume 33. pp. 1877-1901. Curran Associates, Inc., 2020. URL https: http::::/ppoceeeenng...eeee CC/ pwtpp//a////.1lee///// 1etell/e1455000bo11 cb4 967 18 4
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.088
2022.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjum Shou, Bing Qin, Ting Liu, Daxin Jiang, al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 2020.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis, 2022. URL htttps://axxi..or///// abs/ 225p//9909220999999
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 381 6-3830, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 acl-long.299 long. 295
URL https:// https::/aclantthooll/ /tt
acc
Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng. Duyu Tang, Shujic Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:20099 083366 2020.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Vincent J. Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pp. 763-773, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450351058, doi: 10.1145/3100233.3100222 URL
3106290
Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. Discovering the syntax and strategies of natural language programming with generative language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22 New York, NY, USA, 2022. Association for Computing Machinery. /tllhtttss///do.rrgggr
ISBN 978145039 doi: 0121144//49111225501510 3491102.350182
Karen Sparek Jones, Steve Walker, and Stephen E. Robertson. A probabilistic model of information retrieval: development and comparative experiments part 1. Inf. Process. Manag., 36(6):779-808 2000. doi: 10. 10.1011//0000 (30)
URL (ttttt::: /tttt::://ooo //oi org 10. 101 6/50306
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 12-18 July 2020, Proceedings of Machine Learning Research. PMLR, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learn- ing Representations, 2020. htt.hhtttt////peeeeeeet
formm 3ttp://ooouummmm111333333

---

Under review as a conference paper at ICLR 2023
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /1911 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https: attppss/aaclemtiioooggy... orrg2/22211 emr ma mm.in.244 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 0982-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ acl--...l-oon3355
httt:t/:: www.aacnnnnnoooooyooooooooooooot org acc-ooooo.......
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022. URL CRLhttps:: //arxiv..vvvvao/aaa/.ai org/ abs/2203 012203.07811 07814
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLlO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114 14, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10. 18653//112022.deelioo 1.10. URL https: http:///caaaaata/o wwwpatthhh.oyyo.oo/2222222 deelio-1.10
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, 2021a.
13586
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv:2103.10385, 2021b.
Chen Lyu, Ruyun Wang, Hongyu Zhang, Hanwen Zhang, and Songlin Hu. Embedding api dependency graph for neural code generation. Empirical Softw. Engg. 26(4), jul 2021. ISSN 1382-3256. doi: 10. 10.10077/1100644-22--99998 09968-2
URL htttps://da org/l0 10. 1007/s10664 021
Pardis Pashakhanloo, Aaditya Naik, Hanjun Dai, Petros Maniatis, and Mayur Naik. Learning to walk over relational graphs of source code. In Deep Learning for Code Workshop, 2022a. URL
htttps://ppenrev/ net htt////orumrddeeeee (orumm?.d=-uuGGGOOW...
Pardis Pashakhanloo, Aaditya Naik, Yuepeng Wang, Hanjun Dai, Petros Maniatis, and Mayur Naik. Codetrek: Flexible modeling of code using extensible relational representation. In International Conference on Learning Representations, 2022b. URL
met//ooune 2idd-did--wwss fmBmf
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202
URL https
7/aclanthen wowhologyy.oooo orgy..rrg//0 2021.
410
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. arXiv e-prints, art. arXiv:2204.06125, April 2022.
12

---

Under review as a conference paper at ICLR 2023
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards. Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent, 2022. URL https
T/arxiiv.rr//bs//2220 06175
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the www.xii..org/baa.. abbeg//bss/ii1 07350
few-shot paradigm, 2021. URL httpp::/aa.iv..rr//pp/.o
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, 2019, doi: 10.1101/622803. httph:ttt///w/t http://wwwbbooxx..ooo/ www.voorg.oooo.o wttp//ontee//..1110.001. 102.1120//222
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255 269, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 eacl-mainn.0. URL
www.nntwwoooooooooooooo wgg.ooog....../22.. .tt.221eeallmmmrr -ma 20
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. On-the-fly adaptation of source code models. In NeurIPS 2020 Workshop on Cosputer-Assistd Programming, 2020. https/
//openreview.oo net forum? I
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, jan 2014. ISSN 1532-4435.
Jiao Sun, Q. Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D. Weisz. Investigating explainability of generative ai for code through scenario-based design. In 27th International Conference on Intelligent User Interfaces, IUI 222 pp. 212-228, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781 450391443. doi: 10.1 00.11455/4900999351111
URL.https://dda www.oogg//o.... 11../ /tttt//s//oaaoooro0..11445/////0 /004///44999999 290.44999999911199999
Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Trans- formers. O'Reilly Media, Inc.", 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, pp. 6000-6010 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860966
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http:///w....oooooooo//////eaaaa transformer jax May 2021.
Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael Lyu. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In Proceedings of the 30th ACM Joint European Sofwware Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, nov 2022. doi: 10. 10.11455754425033449113 URL http:httt///tott //oo. .org/ 1ttpe/11..1142233540050 3549113
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, 2021.
13

---

Under review as a conference paper at ICLR 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https
//openreview.net wet.:e/oooommmmaii
Wen Xiao, Beltagy, Giuseppe Carenini, and Arman Cohan. PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume Long Pupers), pp. 5245- 5263, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.acl- 1ong.l30..0336.. URL wttph::p:///mmm
/tRs:ttps: http:::://aalaaatolooooo/oo www.:a/www....a....///.2......a. www.ooo.......n......... 360
Frank Xu. Uri Alon, Graham Neubig, and Vincent J Hellendoom. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202.13110 2022a.
Frank Xu, Junxian He, Graham Neubig, and Vincent Josua Hellendoom. Capturing structural local-
ity in non- parametric language models. In International Conference on Learning Representations, 2022b. URL httt:ss://peerreiiiieee do
Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Raymond J. Mooney, Junyi Jessy Li, and Milos Gligoric. Learning to generate code comments from class hierarchies, 2021. URL https
7/arxiv. ottp//rg///a/ooooo/ 13426
14

---

Under review as a conference paper at ICLR 2023
A DATASET CREATION DETAILS
A.1 CREATION OF HOLE COMPLETION DATA
To collect the hole completion data, we scraped Google Code for repositories tagged with the language "Java". Then we deduplicated repositories by searching for a matching repository with the same name on GitHub. For those repositories with zero matching names on GitHub, we downloaded the archive and extracted the source code (preserving the directory structure). Next, we tried to determine the licenses of all repositories by either looking for a LICENSE file or matching with keywords "license", "copyright", "mit", etc. For repos for which our process was able to come up with a known license, we selected the ones having a permissive license, i.e., MIT, Apache and BSD. This was followed by removing files that are exact duplicates of each other within repo. One of the reasons we found this reter-repos duplication may be because sometimes developers adopt lousy practises where instead of declaring a package and importing functions, they simply copy-pas the desired file in the current folder. The target holes coming from any of the duplicate files do not form part of the hole completion dataset. However, these files might be used to contribute to prompt proposal context for completing a target hole in a non-duplicate file. For the remaining files, we took each line that is not blanked line or comment, and chose the middle character as the hole position, i..., all the characters from the middle of the line to the end of the line form target hole. To avoid large repos having strong bias on our prompt proposal classifier, we capped the contribution from each repo to be a maximum of 10000 holes. If the number of holes in the repo exceeds 10000, we randomly select 10000 holes.
A.2 CREATION OF DATA FOR REPO-LEVEI. PROMPT PROPOSALS
We used the tree-sitter API for Java to get the parse-tree of an individual file in a repo. To get information at a repo-level, for each file in the repo, we stored the following information:
1. list of all class names in the file. This helped us to get the parent or child class file corresponding to given parent or child class.
2. the file corresponding to each import statement.
3. for each import statement in the file, the position in the file where the import is used. This is used for ranking the files based on the heuristics mentioned in Table 2
4. list of sibling files
5. list of similar name files. This was done by splitting the filenames based on either camel-case underscore. If the sub-parts of two files match, then they are said to have similar name.
The above meta-data was calculated only once for each repo The subsequent hole completions can use the same cached information. In practise, we can efficiently. For prompt proposal, given the prompt source, we first obtain single file or ranked list of files (see Table 2 using the info in the parse tree in conjugation with the above repo-level meta-data. All the prompt proposal context type information (MN, MNB, SL, I, TI, FD) can then be obtained by querying the parse tree of the selected file.
hash to store and retrieve this info
B PROMPT PROPOSAL. DETAILS
B. RANKING OF FILES BASED ON PROMPT SOURCE
In 2 we provide details of how we select files for given prompt source. Depending on the prompt proposal, we get either single file or a list of files ranked based on some criteria. For example, if the prompt source is Import, we take all the import statements used in the current file and identify the location the current file where the corresponding imports have been used. According to our heuristic, the closer is the import usage to the hole position, the more likely it is for the prompt proposal context coming from the corresponding import file to be more relevant (to predict the target hole). We get a ranked list of import files sorted based on increasing order of distance (i.e., number of lines) between the import usage and the hole position. We start by taking all of the prompt proposal
"https://code www.o//odgggoggttcocoo//cccceee cow/archiv/// http:://gghhnaa.ooommmeeesseeees
15

---

Under review as a conference paper at ICLR 2023
context from the first file in the ranked list and then keep iterating the ranked list until either the total context length allocated to the prompt proposal gets exhausted or we reach the end of the ranked list.
Table 2: Selecting files for prompt source
Prompt Source
File Ranking
Current
file with the target hole. Returns single file.
Parent Class
file that contains the parent class that occurs closest to the target hole. Returns a single file.
Import
files with the corresponding import usage ranked based on the proximity to the hole. Returns ranked list of files.
Sibling
files with import usage common to the current file and the sihling file, ranked based on the proximity to the hole. The total number common imports between the current and the sibling file is used as a tie- tie-breaker. Returns a ranked list of files.
Similar Name
files with import usage common to the current file and the similar name file, ranked based the proximity the hole. The total number of common imports between current and similar name file used as a tie-bricee breaker. Returns a ranked list files.
Child Class
files with import usage common to the current file and the child file, ranked based on the proximity to the hole. The total number of common imports between the current and the child class file is used a tie-breaker. Returns a ranked list files.
Import of Sibling
import files ranked based on the frequency usage all the sibling files. Returns a ranked list of files. import file ranked on the basis of frequency of usage in all the similar namc files. Returns ranked list of files.
Import of Similar Name
Import of Parent Class
import file ranked the basis of frequency of usage in all the parent class files. Returns a ranked import file ranked on the basis of frequency of usage in all the child class files. Returns ranked list of files.
of files.
Import of Child Class
B.2 EXAMPLES OF PROMPT CONTEXT TYPE
We provide examples of each of our prompt context type below:
1. Post Lines (PL) For the example shown in Figure of the main paper, post lines will take all the lines after the line mg. www.taalizeTossiignee the end of the file (AffinityPropagall java).
()) till we reach
2. Identifiers (I): Identifiers are the names of variables used in the code. For example, for the prompt proposal context taken from the imported file shown in Figure I in the main paper (highlighted in violet), identifiers are InitializeToAss simmensniimmnttt (line
1), a (line 1), currentAssignaeee
(line 2), a line 2), clone (line 2).
a IreadyInitillssn _ed_ (line 3), justOneRound (line 4).
3. Type Identifiers (TT): Type Identifiers define the type of an identifier. For For example, in the code snippet class DPAffinityPropagett on extends Snffinityrropageee
labeled type identifier. Similarly in the snippet
DPAPParameters parameters_i. DPAPFarameter is a type identifier.
4. Field Declarations (FD): The variables of class type are introduced by field declarations. For example, double [] mHi jMj and MessageaaluePair(11 sortedMHi jMaiiHHHMM are examples of field declarations.
5. String Literals (SL): A string literal is the sequence of characters enclosed in double- quotes. For example, in the code snippet, System.err, enttemmmrrrprintln (DDPAP load Warning: unknown parameter have two string literals: (a)
entries [0] +
value (ntries(1]);
"DPAP load Warning: unknown parameter (b) ", va value
16

---

Under review as a conference paper at ICLR 2023
6. Method Names (MN): For the example shown in Figure of the main paper, public void InitialiieTosssignnnnne prompt context type.
(int [] a) is the method name
7. Method Names and Bodies (MNB): For the example shown in Figure 1 of the main paper, the part highlighted in violet represents the method names and bodies.
B.3 TRUNCATION STRATEGIES FOR PROMPT PROPOSAL CONTEXT
If the prompt proposal context is greater than the context length allocated to it, then need to truncate the prompt proposal context. We followed the below two schemes for truncating context:
front: We truncate the context from the front. This is used for all prompt sources except Parent
Class and when we take PI. from Current.
back: We truncate the context from the back. This is used when the prompt source is Parent Class and when we take prompt context types other than PL from Current.
The truncation strategies for each case were selected based on results on small validation set. For the prompt source Current, except when the prompt context type is PL, we always start by taking code of prompt context type from after the hole position. This makes sense as the default Codex context will anyways contain code before the hole. Only if this turns out to be blank, we will use the code of context type from before the hole.
B.4 LIST OF PROMPT PROPOSAL.
Table 3: List of our proposed repo-level prompt proposals
Prompt Proposal ID
Prompt Source
Prompt Context Type
0, 2, 3,4
Current
MN, I, TI, SL, FD
6. 7
Current
PL. (laking 25%, 50% and 75% contribution to the total context length)
8,9, 10,
11,
12, 13
Parent Class
MNB, MN, I, TI, SL, FD
14. 15.
16,
17,
18, 19
Import
MNB, MN, I, TI, SI., FD
20.
21,
22
23,
25
Sibling
MNB, MN, I, TI, SI., FD
26,
27,
28,
29,
30
Similar Name
MNB, MN, TI,
FD
32.
34,
35,
Child Class
MNB, MN,
SL., FD
38,
39,
40,
41, 42,43
Import of Sibling
MNB, MN, I, TI, ST FD)
44. 45,
46
47,
48, 49
Import of Similar Name
MNB, MN, I, TI, SL, FD
50,0
51.
52,
53,
55,
Import of Parent Class
MNB, MN, I, Tl, SL, FD
56.
57,
58,
59,
61
Import of Child Class
MNB, MN, I, TI, ST,
ST., FD
Codex
B.5 OTHER PROMPT PROPOSAL VARIATIONS
We experimented with other variations that include: (a) appending class names at the beginning of the prompt proposal context, (b) using newline or space to join the prompt proposal context and the default Codex context, (c) taking all or the top-k of the prompt context types, (d) ordering of top-k.
Context Separator:
This defines how we join the prompt proposal context string to the default Codex context string. We experimented with space and newline as context separators.
Prompt Proposal Context Formatting: We can format the prompt proposal context before giving it to the Prompt Composer. We experimented with the following options:
1. _ass_name: append [class name of the file] at the beginning of the prompt proposal context taken from each file that is part of the prompt source. For example, if we are taking prompt proposal context from two import files fl and f2, the prompt proposal context will be formatted as: [class name of f1] prompt proposal context from fl space + [class name of f2] prompt proposal context from f2. We use this when the prompt proposal context types are MN, I. TI, FD and SL.
2. class method name: we apply this only when the prompt proposal context type is MNB. We append method names at the beginning of each of the corresponding method bodies. We also
17

---

Under review as a conference paper at ICLR 2023
append the prompt proposal context from file with the name of the class as described in the previous item.
3. comment: Adding in the prompt proposal context comment, i.c., formatting it as: / prompt proposal context */. This wasn't found to be much useful.
4. none: passing the prompt proposal context is. We use this when the prompt proposal context type is PL.
For each of the prompt proposal context types, except PL, we experimented with taking the (a) first (b) last and (c) all of the prompt proposal context types, i.c., we can take first- 10 identifiers. We found 'all' be the best among all. Top-k:
Top-k Type:
We experiment with k values of (a) 10 (b) 20 and (c) all. We found 'all" to work best for all prompt context types.
C IMPLEMENTATION DETAILS
C.I RLPG-H
We used Adam (Kingma & Ba 2015) optimizer with a learning rate of 3e-4 and batch size of 64. We used CodeBERT (Feng et al. 2020) as our pretrained model Fg to obtain the representation of hole window. The size of the representation (corresponding the hidden dimension of the [CLS]] token) is 768. W E g612x768 b² 512, W E 163x512 63.
C.2 RLPG-R
optimizer with a learning rate of 3e-4 and batch size of 64. We used CodeBERT (Feng al. (2020) as our pretrained model F6 obtain the representation of hole window and prompt proposal context. The size of the representation (corresponding to the hidden dimension of the [CLS] token) is 768. equations 1, 2 and Section 3.2, the projection matrices W E Xdmodede W.K Rd
We used Adam (Kingma & Ba 2015)
WV Rdd Xdmadel Wo E Rdmodet xTd. For the multihead attention, we used dg = dg d, 32,
and dmode! = 768, W, E 66x and bp 63. For each head, we perform scaled dot-product attention (Equation 4). G module consists of dropout (Srivastava al. 2014) layer, residual connection (He al. 2016). layernorm (Ba et al.] 2016). followed by a sequence of (a) dense layer of weights=2043 x 768, bias=768, (b) relu activation, (c) dense layer of weights=768 x 2048, bias=2048, (d) dropout layer, (e) residual connection, (f) layernorm. A dropout value of 0.25 was used while training. Our model resembles one layer of the transformer encoder block (Vaswani et al. 2017.
C.3 BASELINES
Random baseline first selects file randomly from the current repository followed by selecting random line within that file. We choose all the lines starting from that line to the end line of the chosen file context (excluding the hole window if the chosen file is the current file). The nearest neighbour similarity is based on the dot product between the representation of the hole window and the representation of the context, where we use pretrained CodeBERT (Feng al., 2020) model to obtain the representations. For the Identifier Usage baseline, if the nearest identifier to the hole doesn't return any usage window, we proceed the next nearest identifier. For faster computation and to avoid memory issues when rumning our hardware, for NN baselines, collect 64 random neighbours and then rank based on the nearest neighbour distance. The BM25-based baselines the Okapi BM25 implementation with default parameters given by the pip package rank-bm2 0. 0.2.2 10 For file-level BM25, if the file context exceeds the allocated context length, we truncate from the back.
18

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL RESULTS
D.1 ABLATION ON PERFORMANCE BASED ON PROMPT PROPOSAL
Figure 4 shows the mean success rate of prompt context types when success counted only for the cases when these prompt contexts are applicable. As can be scen from the figure, post lines is the most useful prompt context type on an average. The contribution from other prompt context types though smaller than post lines is still significant highlighting the importance of each prompt context type.
70
PL
60
TI
50
MN
MNB
40
SL
FD
30
20
10
Figure Mean success rate validation data based prompt context type when they are applicable.
25
Cur Sib
20
Imp5mmN
15
ImpSib
12
ImpPaCI
ChCl
(Left) Normalized success rate of prompt sources when applicable, (Right) Normalized success rate of prompt context types when applicable
Figure 5:
Figure shows the normalized success rates where the normalization is performed across the prompt proposals. This helps us understand the relative performance of prompt proposal sources and context types. The left part of the figure breaks down the performance based on prompt sources and the right part breaks down based on prompt context types. One thing to note from the plot of prompt context types is that when we consider relative performance, post lines is no longer the most dominant context type. This is because post lines is tied to only when the prompt source corresponds the current file, thereby contributing to lower numbers when compared to most of the other context types that are tied to all prompt sources.
19

---

Under review as a conference paper at ICLR 2023
D.2 PERFORMANCE ON NON-IMMEDIATE POST LINES
Table shows the performance of post lines when starting the fourth line after the target hole line (i.e., skipping three lines after the target hole) as opposed to starting from the line that immediately follows the target hole line. This experiment helps us understand the performance when we are interested in doing much harder task of multi-line code autocompletion, wherein the objective is to predict not just the blanked out portion in the current line but also say the next three lines. This can correspond to completing a block of code like a function body. As can seen from the table, when starting from the fourth line, away we move from the target hole, the less relevant the post lines context would be. However, the performance drop is not significant suggesting that post lines is still very useful prompt context type that can be used under the setting of multi-line aode-autocompleton. Equivalently, we can include this one of the prompt proposals in framework along with the current version of post lines.
very slight deterioration in performance. This is expected because the farther
Table 4: Success Rate (SR) when taking different versions of post lines.
Success Rate( %)
Rel. 1%%) (hole-wise)
Success Rate( %))
Rel. 1%%) (repo-wise)
Method
(hole-wise)
(repo-wise)
Codex (Chen et al. 2021
58.73
60.64
Post Lines (immediate lime after the hole)
65.78
12.00 10.86
68.01
12.15
Post Lines (skipping three lines after the hole)
65.11
66.42
9.53
D.3 COMPOSITION OF PROMPT PROPOSALS
Table shows the performance of the two versions of RLPG when we compose the prompt proposal context from prompt proposals. We take the top-/ prompt proposals given by RLPG based on decreasing order of probability. To decide how much context should be used for each prompt proposal, we divide the total context length in proportion to the normalized probabilities of the top-/ prompt proposals. As be seen from the table, even though PPC is explicitly trained perform composition (both the ground-truth vector and the representation of prompt proposal context involve single prompt proposal), all the compositions lead significant improvements over Codex. However, as expected the best results correspond to taking context from a single prompt proposal (i.e., the training setting). The drop in success rate with that explicitly training RLPG to learn to compose contexts from different prompt proposals can lead promising results and hence offers an interesting future direction.
and I is not that significant, which suggests
Table 5: Success Rate (SR) of different compositions of the prompt proposals on the test set.
Success Rate( %)
Rel. 1(%) (hole-wise)
Success Rate %)
Rel. 1(%) (repo-wise)
Method
(hole-wise)
(repo-wise)
Codex (Chen et al.
2021
58.73
60.64
(
68.51
16.65
69.26
14.21
RLPG-R (1
67.80
15.44
69.28
14.26
RLPG-H (1
67.07
14.20
67.87
11.91
RLPG-R
66.57
13.35
67.88
11.94
RLPG-H 5)
66.60
13.40
67.91
11.98
RLPG-R 5)
65,78
12.01
67.69
11.62
RLPG-H (1 10)
65.53
11.58
67.24
10.88
RLPG-R (1 10)
63.59
8.27
65.98
8.79
D.4 EFFECT OF CONTEXT LENGTII
To understand the effect of context length on the performance of our prompt proposals, we took half of the context length available for prompt in Codex and observed the performance of the oracle and fixed prompt proposal. As before, saw that oracle constructed from our prompt proposals shows remarkable improvement over Codex highlighting the value of our prompt proposals. However, when compared larger context length, the relative gains smaller. This is expected smaller
20

---

Under review as a conference paper at ICLR 2023
context length means that the relevant context coming from a prompt proposal needs to be truncated to make inside the prompt, thereby leading to loss of information.
Table 6: Success Rate (SR) of Codex and oracle over the test set when the total context length 2048.
Success Rate( %) (hole-wisc)
Rel. 1(%) (hole-wise)
Success Rate( %)
Rel. 1(%)
Method
(repo-wise)
(repo-wise)
Codex CCen 2021
57.77
58.90
Oracle
61.90
7.15
67.18
14.07
D.S PERFORMANCE ON INDIVIDUAL REPOSITORIES
Table Success Rate of different methods on training data
Repo name
#Total Holes
Oracle
Codex
Fixed prompt proposal
RLPG-H
RLPG-R
largemail
1653
75.38
55.11
62.73
63.94
63.28
fipserverremoteddmin
86.44
66.11
76.09
76.21
76.76
mytSlib
91.65
53.58
61.34
73.51
74.46
seamlets
92.74
62.25
62.72
71.55
74.27
gloodb
10000
91.07
57.50
57.50
70.32
72.31
jjskit
9043
80.36
65.61
72.18
72.00
72.44
mobileexpensetrucker
2298
75.94
57.88
67.28
66.84
66.97
gfsfa
10000
80.55
$7.33
57.33
59.28
65.24
swc574-group3
2029
76.79
54.46
66.19
65.16
64.91
strudem-sicsa
6131
77.83
64.96
72.55
73.25
73.32
soap-dte
1370
81.24
64.82
70.73
71.61
72.70
openprocesslogger
7191
81.06
62.19
71.77
72.22
72.62
lupestry-sesang
397
72.54
45.84
61.21
60.71
63,98
exogdx
735
84.76
63.81
75.51
75.92
76.60
designpatternjavapccoo
1069
78.30
54.82
64.36
63.99
68.57
quidsee
3020
81.66
60.79
69.50
70.36
70.26
healpix-rangeset
4734
63.54
48.71
$4.67
$4,94
55,07
sol-agent-platform
10000
73.76
S8.22
65.72
65.65
65.94
rsbotownversion
10000
75.23
$7.89
65.58
66.22
66.31
Table 8: Success Rate of different methods validation data
Repo name
#Total Holes
Oracle
Codex
Fixed prompt proposal
REPG-H
RLPG-R
lyrond math- math-mech-ere mech-eshopp
721
83.91
60.33
71.15
71.57
72.68
2225
83.46
62.20
72.76
73.53
73.17
infinispan- mnfnispan-storage service
373
82.31
71.85
78.55
76.94
77.75
teammates-shane shakthi
7665
82.02
63.74
72.38
72.47
72.46
javsummertramework
10000
79.27
55.92
65.30
65.74
65.55
tinwiki
10000
73.67
69.27
69.27
69.12
69.58
jloogle
3145
84.55
73.16
77.87
77.17
77.36
jcontenedor
5464
81.26
58.99
67.77
67.95
68.32
sohocms
772
76.68
57.90
67.10
67.49
67.62
affinity_propageeee
1466
79.54
59.14
70.33
70.26
70.26
jatu4test
1921
71.06
44.09
54.92
55.91
57.47
swinagile
2595
79.69
63.01
72.29
72.49
72.68
navigablep2p
1322
75.72
59.76
65.43
65.13
65.28
springlime
879
83.50
62,34
74.18
74.86
74.40
21

---

Under review as a conference paper at ICLR 2023
Table 9: Success Rate of different methods on test data
Name
Repo
#Total Holes
Oracke
Codex
Fixed
RLPG-H
RLPG-R
Kandom
Random NN
Iden Usage (Randon))
Iden Usage (NN)
File-Lvee BM225
RLPG-
BM25
dovetaildb
10000
76.89
57.12
66.15
66.06
66.25
57.45
57.58
61.39
60.77
59.39
66.05
projes-pt-dline
10000
52.67
52.93
56.21
realtinage
2513
57.78
quikiedlice jana
1138
70.21
69.36
46.13
48.15
60.37
62.92
61.41
elperandsiman
1766
78.05
53.57
62.51
02.4
61.66
55.66
57.76
69.42
08.4
09.14
61.55
sisonei-jonn-ape
830
73.42
57.57
62.1
62.69
63.29
57 00
$7.21
71.28
72.35
63.77
63
incrpghot
6391
83.67
89.67
77.24
76.71
76.05
09.55
70.54
74,68
74.43
09.32
75.75
obotsimulato 200P
7514
75.43
56.28
67.45
07.55
67.55
$6.4
56.
64.61
64.71
62.96
66.
gut-plugimietest
73
84,93
60.27
68.49
65.75
68.49
58,
57.53
63.01
63.01
50.68
75.34
apiitiieeds
1385
70.11
wickethits
79.3
64.48
hucourses
390
70
xfiane
3055
84.00
62.82
73.62
72.73
63.67
65.17
77.25
75.97
77.32
74.01
7 Table and T the success rates of different methods over individual repositories in the training, validation and test splits, respectively. The repo-wise averages in Table in the main paper were calculated by taking the average of numbers corresponding to each column. The hole-wise averages correspond to multiplying the repo-wise numbers of each method by the total holes in the repo to get the total number of successful holes by that method for that repo. We then add the total number successful holes across repos and divide it by the total number of holes in the entire data split to get the hole-wise averages.
E ANALYSIS OF SAMPLE CASES
In Figure RLPG selects the prompt proposal that corresponds to taking method names and bodies from the imported file (i.e. Maximi z ingii woaggglbbsaamaer........ java ). Note that mg. before the hole position indicates that a method used in the imported file is likely to be in- voked. In this case, the prompt proposal context (highlighted in violet) contains the method name Init i ToAAss grment (part of target hole). This in conjunction with the default Codex context which contains the method
() (part of target hole) leads to gen- eration of successful prompt. On the other hand, the prompt created from the default Codex context fails to predict the target hole in this case. In general, we observed that in the absence of a strong signal, Codex has tendency to give preference to natural language comments occurring before the hole position, e.g. naming the method based on the comment. This in certain cases might hurt. We provide insatnces of positive and negative samples cases for RLPG below:
SntrrentAssiigmmeett
E. POSITIVE CASES
We provide some examples of cases where RLPG led to the correct prediction and Codex failed.
1. Cases where part of the target hole found exactly in the prompt proposal context.
RLPG Propagation (int numVars) vs Codex
P Propagation ()
RLPG =
tersFromPine le (String filename)
vs
Codex
ters (String filename)
RLPG als ingginngaaacrr) { vs Codex als ("numVars")
RLPG ", value
entries ries [1]); Codex });
RIPG stem. exit (1); vs Codex = st
em err. pwr..pintt 1 ("DPAP load error:
ex ex.get
2. Cases where Codex takes strong hint from the preceding natural language comment, thereby producing incorrect predictions.
RLPG= d PassMessages () vs Codex d DoCheRRoundd ()
RLPG
teger> CurrentExemplaarc ()
vs
Codex
teger> ChooseFxemplate ()
22

---

Under review as a conference paper at ICLR 2023
RLPG
ring FileName ()
{
Codex
ring GetAlgorithm"llenan () {
E.2
NEGATIVE CASES
In certain cases, extra information from prompt proposal-context might lead to confusion and produce incorrect predictions.
RLPG = an hasConverged vs Codex = an converged_;
RLPG
[i] [i] -Double, MAX VALUE; vs Codex [[i] 0
23


Reviews:
[{"review_id": "hhWb5DvQ_i_", "reviewer": "Reviewer_9is8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose prompt generator based on the whole code repository for the code completion tasks.\nAuthors show a significant improvement over baseline Codex results by using their prompt generator.", "summary_of_the_review": "I think that the paper presents a novel approach for improving results of tasks such as code completion by generating prompts from whole repository using a trained framework. This is interesting work and should be accepted to ICLR.", "strength_and_weaknesses": "Strengths:\n- Addresses an important and novel area of good prompt generation for LLM tasks\n- Approach does not require access to the weights of LLM\n- Suggests and implements prompt generator that uses information from the whole repository. The prompt generator also uses repository level prompt \"proposals\" (rules/suggestions).\n- Using generated prompts significantly improves Codex results\n\nWeaknesses:\n\n- It is not clear whether prompt proposals are really per-repository. It seems that they will be pretty universal and should not vary much. So the value of repository-specific prompt proposals is not really established or proven. This is a minor issue though.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ddhYN3Isbv", "reviewer": "Reviewer_VAEL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a repository-level prompt generation (RLPG) approach for language models of source code. Without having access to the weight of LLMs, RLPG can improve the performance of LM in code completion task. Through training the prompt proposal classifier, the prompt composer can generate high quality prompts for LM, increasing the accuracy of code completion.", "summary_of_the_review": "I appreciate that the proposed technique can generate prompts in a black box manner. Different from previous work about line-level code completion, this paper considers the completion from the cursor to the end of the line. It involves a wide range of prompt sources and prompt context types. The proposed approach is well ablated and extensive experiments are conducted. RLPG is also efficient for both training and inference.\n\nThe proposed work learns to generate example-specific prompts using prompt proposals, which are taken from the structure of the repository and the context from other relevant files (e.g., imports, parent class files). The authors may also consider other code information such as API. See, for example:\n\nLyu et al., Embedding API dependency graph for neural code generation. Empirical Software Engineering, 26(4):61, 2021. \n\nFor RLPG-H, the used code context is two lines before and after the hole. Are two lines (or four) enough? Maybe the target hole is related to some statements far away from the target (e.g. some global variable definition).\n\nThe Identifier Usage (Random) also achieves promising results, why?\n\nFigure 1 is not easy to understand.\n\nRecently, the following work also applies prompt learning to pre-trained code models: Wang et al., No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence, https://arxiv.org/abs/2207.11680.\n\nThe proposed technique is more about software engineering/programming language, the novel technical contribution to neural networks/language models is a bit limited. \n", "strength_and_weaknesses": "Strengths:\n\n- A novel way of generating prompts.\n\n- Extensive experiments.\n\n- RLPG is efficient for both training and inference.\n\nWeaknesses:\n\n- The paper is better for a software engineering/programming language conference.\n\n- Only the structure of the repository and the context from other relevant files are considered in the prompt proposals.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bmtUB-Rhu_", "reviewer": "Reviewer_oQyE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to make prompt proposals from repository-level context and add concatenate the prompt proposals with normal context to LLM in order to feed more related information to achieve better generations. The paper explores different sources of repository level prompts, as well as prompt context types. Experiment results show that additional repository level prompt help improve the performance significantly.\n", "summary_of_the_review": "Overall, I think this is a decent paper with novel method proposed and well-performed experiments. I have some concerns on the paper, and would want to see more experiments and results to make the paper more complete.", "strength_and_weaknesses": "Strength:\n- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.\n- The paper explores several variants on the prompt sources and prompt types, and made a nice figure to demonstrate which prompt source works the best, though only one is used in experiment at a time.\n\n\nWeakness:\n- Taking the whole file from the whole import file seems unnecessarily large context. Rather, if the import statement only imports one class or one function from the module, it makes sense to only consider the function implementation, rather than the whole file where the function resides. This especially applies to 7) Method Names and Bodies prompt type.\n- The paper only considers one prompt proposal, which is a large limitation of the paper. Why don't the authors take top-k prompt proposals and put into context instead of just one? Besides, I'm curious whether there is a better choice of prompt proposal, for example, intuitively the function names, docstrings, and variable names of the imported function could help inform the model the functionality of the imported function.\n- It seems that, RLPG-H is more like classification, while RLPG-R is more like retrieval. Would a lexical search method like BM25 could retrieve relevant prompts already compared? That way no training for the PPC is needed. And I would expect it works reasonably well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ksgVN5wz7R", "reviewer": "Reviewer_xrrm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is part of the stream of papers that try and generate code given a prompt. This paper tries to assess that if additional context is given as part of the prompt, whether the performance can improve. In particular, they predefine different types of contexts and then train a classifier to predict which context should be prepended to the prompt. To get the training data, they given different types of contexts to Codex and if a particular context is able to make the completion match the ground truth completion, they label that context as 1. They then train a multi-label classifier and use its predictions at inference time to decide what context to add on to the prompt.\n\nThey compare against several baselines - no additional context, oracle context (using the ground truth correct contexts) which is the upper bound, random context, nearest neighbors in representation space from the random contexts, lines using the closest identifier throughout the repo, and context to the right of the line to be completed.\n\nThe strongest baseline is the right context baseline against which they show a 1-4% relative improvement and a 14-16% relative improvement over no additional context (Table 1)", "summary_of_the_review": "While I believe the ideas in the paper are interesting and novel, the fact that we do not know the pretraining data which could easily overlap with the test data + the deduping with respect to the the known part of pretraining data being imperfect (based on repo name match instead of file content match or suffix array deduping), the results are not reliable.", "strength_and_weaknesses": "Strengths:\n\n(1) The idea of generating training labels for the correct contexts and training a classifier to predict what the correct contexts should be is novel and interesting\n\n(2) They investigate multiple possible baselines \n\nWeaknesses:\n\n(1) Overlap with pretraining data -- Github is not the only source of training data for Codex (https://openai.com/blog/openai-codex/ -- \"OpenAI Codex is a descendant of GPT-3; its training data contains both natural language and billions of lines of source code from publicly available sources, including code in public GitHub repositories\"). Thus it is very possible that Google Code -- which they use for their test data -- is part of the training data for Codex. Furthermore, deduping based on just repo name match is highly imperfect. A much better way would be file level deduping or suffix array based deduping. Unless there's a better understanding of what the pretraining data consists of, the results might be invalid. The authors could consider using a model like CodeGen for which the pretraining dataset is known. Alternatively they could use code published after June 2021 (the training data cutoff for the Codex davinci-002 model), dedupe it against the code data available upto June 2021 and use that as their test data.\n\n(2) The improvements over both right context as well as identifier usage baselines are modest.\n\n(3) They don't explore how different prompt proposals could be combined with each other", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject