Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to make prompt proposals from repository-level context and add concatenate the prompt proposals with normal context to LLM in order to feed more related information to achieve better generations. The paper explores different sources of repository level prompts, as well as prompt context types. Experiment results show that additional repository level prompt help improve the performance significantly.

strength_and_weaknesses: Strength:
- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.
- The paper explores several variants on the prompt sources and prompt types, and made a nice figure to demonstrate which prompt source works the best, though only one is used in experiment at a time.


Weakness:
- Taking the whole file from the whole import file seems unnecessarily large context. Rather, if the import statement only imports one class or one function from the module, it makes sense to only consider the function implementation, rather than the whole file where the function resides. This especially applies to 7) Method Names and Bodies prompt type.
- The paper only considers one prompt proposal, which is a large limitation of the paper. Why don't the authors take top-k prompt proposals and put into context instead of just one? Besides, I'm curious whether there is a better choice of prompt proposal, for example, intuitively the function names, docstrings, and variable names of the imported function could help inform the model the functionality of the imported function.
- It seems that, RLPG-H is more like classification, while RLPG-R is more like retrieval. Would a lexical search method like BM25 could retrieve relevant prompts already compared? That way no training for the PPC is needed. And I would expect it works reasonably well.

clarity,_quality,_novelty_and_reproducibility: This paper is well-writtern and is of good quality. The proposed method is somewhat novel. And I expect that the results should be easy to reproduce.

summary_of_the_review: Overall, I think this is a decent paper with novel method proposed and well-performed experiments. I have some concerns on the paper, and would want to see more experiments and results to make the paper more complete.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is part of the stream of papers that try and generate code given a prompt. This paper tries to assess that if additional context is given as part of the prompt, whether the performance can improve. In particular, they predefine different types of contexts and then train a classifier to predict which context should be prepended to the prompt. To get the training data, they given different types of contexts to Codex and if a particular context is able to make the completion match the ground truth completion, they label that context as 1. They then train a multi-label classifier and use its predictions at inference time to decide what context to add on to the prompt.

They compare against several baselines - no additional context, oracle context (using the ground truth correct contexts) which is the upper bound, random context, nearest neighbors in representation space from the random contexts, lines using the closest identifier throughout the repo, and context to the right of the line to be completed.

The strongest baseline is the right context baseline against which they show a 1-4% relative improvement and a 14-16% relative improvement over no additional context (Table 1)
strength_and_weaknesses: Strengths:

(1) The idea of generating training labels for the correct contexts and training a classifier to predict what the correct contexts should be is novel and interesting

(2) They investigate multiple possible baselines 

Weaknesses:

(1) Overlap with pretraining data -- Github is not the only source of training data for Codex (https://openai.com/blog/openai-codex/ -- "OpenAI Codex is a descendant of GPT-3; its training data contains both natural language and billions of lines of source code from publicly available sources, including code in public GitHub repositories"). Thus it is very possible that Google Code -- which they use for their test data -- is part of the training data for Codex. Furthermore, deduping based on just repo name match is highly imperfect. A much better way would be file level deduping or suffix array based deduping. Unless there's a better understanding of what the pretraining data consists of, the results might be invalid. The authors could consider using a model like CodeGen for which the pretraining dataset is known. Alternatively they could use code published after June 2021 (the training data cutoff for the Codex davinci-002 model), dedupe it against the code data available upto June 2021 and use that as their test data.

(2) The improvements over both right context as well as identifier usage baselines are modest.

(3) They don't explore how different prompt proposals could be combined with each other
clarity,_quality,_novelty_and_reproducibility: Clarity:
It's not clear how they are selecting the prompt proposal at inference time. Are they taking the argmax over the probabilities for the different prompt proposals?

Quality:
Error bars are not given. It's unclear how much overlap the test set has with the pretraining data.

summary_of_the_review: While I believe the ideas in the paper are interesting and novel, the fact that we do not know the pretraining data which could easily overlap with the test data + the deduping with respect to the the known part of pretraining data being imperfect (based on repo name match instead of file content match or suffix array deduping), the results are not reliable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a repository-level prompt generation (RLPG) approach for language models of source code. Without having access to the weight of LLMs, RLPG can improve the performance of LM in code completion task. Through training the prompt proposal classifier, the prompt composer can generate high quality prompts for LM, increasing the accuracy of code completion.
strength_and_weaknesses: Strengths:

- A novel way of generating prompts.

- Extensive experiments.

- RLPG is efficient for both training and inference.

Weaknesses:

- The paper is better for a software engineering/programming language conference.

- Only the structure of the repository and the context from other relevant files are considered in the prompt proposals.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The implementation of the model is not available for replication study. The proposed work appears to be novel.
summary_of_the_review: I appreciate that the proposed technique can generate prompts in a black box manner. Different from previous work about line-level code completion, this paper considers the completion from the cursor to the end of the line. It involves a wide range of prompt sources and prompt context types. The proposed approach is well ablated and extensive experiments are conducted. RLPG is also efficient for both training and inference.

The proposed work learns to generate example-specific prompts using prompt proposals, which are taken from the structure of the repository and the context from other relevant files (e.g., imports, parent class files). The authors may also consider other code information such as API. See, for example:

Lyu et al., Embedding API dependency graph for neural code generation. Empirical Software Engineering, 26(4):61, 2021. 

For RLPG-H, the used code context is two lines before and after the hole. Are two lines (or four) enough? Maybe the target hole is related to some statements far away from the target (e.g. some global variable definition).

The Identifier Usage (Random) also achieves promising results, why?

Figure 1 is not easy to understand.

Recently, the following work also applies prompt learning to pre-trained code models: Wang et al., No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence, https://arxiv.org/abs/2207.11680.

The proposed technique is more about software engineering/programming language, the novel technical contribution to neural networks/language models is a bit limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors propose prompt generator based on the whole code repository for the code completion tasks.
Authors show a significant improvement over baseline Codex results by using their prompt generator.
strength_and_weaknesses: Strengths:
- Addresses an important and novel area of good prompt generation for LLM tasks
- Approach does not require access to the weights of LLM
- Suggests and implements prompt generator that uses information from the whole repository. The prompt generator also uses repository level prompt "proposals" (rules/suggestions).
- Using generated prompts significantly improves Codex results

Weaknesses:

- It is not clear whether prompt proposals are really per-repository. It seems that they will be pretty universal and should not vary much. So the value of repository-specific prompt proposals is not really established or proven. This is a minor issue though.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
I think that the paper presents a novel approach for improving results of tasks such as code completion by generating prompts from whole repository using a trained framework.
The results of the paper should be reproducible assuming that the dataset used by authors to train the PPC is available.


summary_of_the_review: I think that the paper presents a novel approach for improving results of tasks such as code completion by generating prompts from whole repository using a trained framework. This is interesting work and should be accepted to ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The prompt source and context are important contribution that was found by corpus analysis. The overall idea is also interesting that could inspire future research in prompting code LLMs. However, after reading the paper carefully, some of my concerns share with those of reviewers oQyE and xrrm, in particular the dataset and using only one prompt proposal. 
justification_for_why_not_higher_score: Experiments might have some issues.
justification_for_why_not_lower_score: NA
Decision: Reject