Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper observes that existing models can only generate seen libraries and function calls at test time. However, new functions and libraries are introduced all the time, and even a seen function call can have arguments that were not used in the training data. To address such unseen functions in code generation, the authors propose a simple and effective approach for code generation by retrieving the relevant documentation, dubbed DocPrompting. The extensive experiments demonstrate that DocPrompting consistently improves NL->code models in two tasks, in two PLs, and across multiple strong base models. DocPrompting improves strong base models such as CodeT5 by 2.85% in pass@1 (52% relative gain) in execution-based evaluation on the popular Python CoNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo-1.3B by up to 6.9% exact match, and Codex by 6.78 charBLEU score.
strength_and_weaknesses: Strengths:
1. The idea of leveraging the newly added manuals and documentation to generate code containing unseen and unused functions and libraries is interesting and novel. The work demonstrates an exciting endeavor direction to generate high-quality code.
2. The main results from Tables 1 & 3 demonstrate that the improvement of the proposed DocPrompting is significant across various pre-trained models. Conducting experiments on more than one benchmark and thorough ablations make the conclusion of this work convincing.
3. The presentation of this work is excellent. The rich use of diagrams and tables makes this paper easy to understand.

Weaknesses: 
1. Although the usage of documentation for code generation is novel, it is somewhat simple from a technical point of view.
2. I examined the supplementary material provided by the authors and only found the data files. It would be better if there were codes to prove its reproducibility.
clarity,_quality,_novelty_and_reproducibility: Please see the Strength And Weaknesses
summary_of_the_review: In my opinion, the proposed DocPrompting method is interesting and novel, and the experimental results successfully demonstrated the effectiveness of the proposed method. Considering the technical simplicity and reproducibility, I give a weak accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose DocPrompting for the problem of generating code from natural language instructions. DocPrompting supplies the model with in-context documentation and uses retrieval to find relevant documentation for a given input. The authors experiment with using BM25 and training dense encoders for retrieving documentation, and find consistent improvements with DocPrompting. The authors focus on evaluation settings where functions in the test set are not seen during training, and also introduce the new code generation dataset “tldr”.
strength_and_weaknesses: Strengths:
* The paper has a clear motivation and is easy to follow.
* The idea of document prompting is tested across a range of models and retrievers with promising results. The authors also compare DocPrompting to a baseline with in-context examples (ExPrompting) instead of in-context documentation (DocPrompting).
* The n-gram analysis provides a simple account of how DocPrompting may bring benefits.

Weaknesses:
* I am unsure how fair the evaluation setting with unseen functions is for the baseline models, especially since the ExPrompting baseline cannot retrieve examples of unseen functions from the training set.
clarity,_quality,_novelty_and_reproducibility: The presentation is of high quality and the paper is well written. The idea of retrieving documentation for prompting seems interesting and novel to me.
summary_of_the_review: This paper proposes the simple and intuitive idea of incorporating documentation into code generation tasks and finds that it brings empirical benefits, especially when generalizing to unseen functions. The paper is well executed and will likely lead to future work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes an approach which uses documentation to generate code conditioned on a natural-language (NL) intent. The approach is essentially a standard retrieval-conditioned generation strategy, but where the set of documents being retrieved are constructed based on documentation.

The paper describes a set of practical instantiations of this technique, exploring a variety of retrieval models (sparse heuristic-based retrievers, and "dense" learned retrieval models trained to recover "correct" docs), and a few variants of generator architectures based on transformers, with either fine-tuning or few-shot prompting. The paper also introduces two datasets: a new benchmark for NL->bash prediction based on the `tldr` docs, and a variant of the CoNaLa benchmark with a different test split combined with a new reference corpus of documentation based on `devdocs.io`.

The authors show that retrieving documentation improves performance across all model types, and also study the effects of retrieving documentation vs code, providing some ground-truth knowledge, and using different retriever strategies.
strength_and_weaknesses: Strengths:

- Using documentation as an extra input to generating code is a good idea for ensuring models generalize to new types of functions, and this paper confirms that this generalization is indeed enabled by this approach.
- The authors compare their approach with a variety of alternatives and find a consistent improvement across experiment settings.
- The new datasets/benchmarks are interesting and seem like a reasonable way to evaluate generalization to unseen types of function.

Weaknesses:

- ~~The novelty is primarily empirical in nature, and~~ the technical approach is essentially an application of known techniques. *(update: After reflection and discussion with the authors, I admit that there is novelty in the problem formulation as well, which goes beyond the setting considered by previous work.)*
- ~~The introduction makes bold claims about "all existing code generation models" which I do not think are justified.~~ *(update: The unsupported claims have been qualified in the new revision.)*
- ~~The notion of "ground truth docs" is a bit confusing.~~ *(update: The paper has been reworded to make this more clear.)*
- It's possible there is some function-knowledge leakage due to the use of pretrained code models for retrieval (perhaps the authors can clarify this) *(update: I am still curious whether this influences the results, but it is a bit orthogonal to the main point of the paper, since it would also affect baselines. The authors have added a brief discussion of this potential leakage to the appendix.)*

clarity,_quality,_novelty_and_reproducibility: ## Novelty

### Overly-strong claims about existing models not being able to generalize *(addressed by new revision)*
The introduction makes very strong claims:
- "all existing code generation models either learn directly from input-output pairs provided as training data ... or learn the mapping between input and output implicitly from naturally occuring corpora"
- "all these works assume that all libraries and function calls were seen in the training data; and that at test time, the trained model will need to generate only seen libraries and function calls"
- "Thus, existing models inherently cannot generalize"

These claims seem much too strong. Although there has been a lot of work in the setting the authors discuss, there have also been a number of works that do not fit neatly into this categorization, and that allow models to generate code using new functions and libraries. For instance:

- [Hashimoto et al. (2018)](https://proceedings.neurips.cc/paper/2018/file/cd17d3ce3b64f227987cd92cd701cc58-Paper.pdf) and [Lu et al. (2017)](https://arxiv.org/pdf/1705.09042.pdf) learn to retrieve from a dataset of examples at inference time, which could be distinct from those used at training time. Some of the other works the authors cite also do something similar (e.g. Parvez et al. (2021) and Pasupat et al. (2021))
- [Shrivastava et al. (2021)](https://arxiv.org/abs/2206.12839) show that code generation can be improved by conditioning on context extracted from other relevant files, which may include new functions
-  [Wu et al. (2022)](https://arxiv.org/pdf/2203.08913.pdf) use an external memory to allow transformers to memorize new data at training time and recall it later, and show that this allows it to predict newly-defined functions at test time.

### Novelty of the approach and findings
I am not aware of prior work which specifically retrieves documentation to generate code from natural language, although there has been prior work on both *retrieving relevant documentation* (e.g. section 5.6 of [this survey](https://dl.acm.org/doi/pdf/10.1145/3212695)) and on *retrieving existing code to generate code* (e.g. [Hashimoto et al. (2018)](https://proceedings.neurips.cc/paper/2018/file/cd17d3ce3b64f227987cd92cd701cc58-Paper.pdf)). 
As such, the empirical results provided by the authors do provide new evidence that documentation helps for code generation across a number of model architectures and tasks (as one might expect).
Additionally, the tldr task, the resplit of CoNaLa, and the documentation pool from DevDocs are new contributions of this work, and might be useful for evaluating improved documentation-retrieval models in the future.

Beyond this, the technique itself appears to be essentially the same as prior work: they use standard retrieval techniques based on existing algorithms and pre-trained models, and their generation models are also models that have been used for code or text generation before.

## Clarity
Overall, the paper is clearly written.

One thing I thought could be improved: Section 3.1 starts from the assumption that we are given triplets $(n, c, \mathcal{D}_n^*)$ where $n$ is a natural-language intent, $c$ is the target code, and $\mathcal{D}_n^*$ is the "ground truth docs" for $n$. I didn't think this was very well explained, since it's not obvious why $n$ would have ground truth documentation associated with it. From the rest of the paper and the appendix, it appears that $\mathcal{D}_n^*$ is actually selected based on a heuristic that uses $c$, not $n$. I think it would be better to explicitly describe this process in section 3.1, e.g. by stating that we are given a pair $(n, c)$ and then we infer a set of reference docs $\mathcal{D}_n^*$ from $c$. (I also wonder whether "ground truth docs" is the best terminology to use, since it's still being inferred heuristically and not manually observed or labeled.)  *(update: this has been addressed by the new revision.)*

Other minor suggestions:  *(update: addressed by the new revision.)*

- Section 4.2 states that ground-truth docs are constructed by "matching the function name with its documentation". However, there may be multiple function names in an example, which makes this statement ambiguous.
- I found page 7 hard to read, since it has a table at the top and a figure in the middle, with ordinary text interleaved around it.
- Figure 3 has overlapping text, and the axes are not labelled (I assume the x-axis is supposed to be $k$?)

## Quality
The dataset setup and experiments, which make up the bulk of the contribution, appear to be high quality overall. The  tldr dataset is an interesting source of documentation for bash code, and re-splitting the CoNaLa dataset based on use of new functions is a good idea for assessing generalization ability.

Experimentally, the authors do a good job of evaluating DocPrompting across a variety of metrics and for a variety of base models. They also conduct a series of interesting ablations and extensions, including using oracle access to the appropriate documentation, comparing to example-based prompting, and investigating different retrievers.

One potential concern is that the strongest results on CoNaLa are given by CodeT5, but CodeT5 was pretrained on a dataset of code that might include some of the "held-out" Python functions described in 4.2. This might be a source of knowledge leakage, since CodeT5-based retrievers might just pick documentation that includes functions that accomplish the task by simply memorizing the functions. It would be helpful to analyze whether this leakage might have occurred, and if so, how much of an effect it might have on the results in Tables 3 and 4. (Am I right in assuming all of the results in Table 3 used a CodeT5-based retriever?)  *(update: Authors have added some discussion of this in the appendix.)*

Additionally, I wasn't convinced by the argument in section 6.1. If I understand correctly, this section compares n-gram overlap between the natural language $n$ and code $c$, and also n-gram overlap between the "ground-truth" docs $\mathcal{D}_n^*$ (along with $n$) and the code $c$. But, from the appendix, $\mathcal{D}_n^*$ is selected exactly by finding the documentation with the highest string overlap with $c$! So it seems obvious that $\mathcal{D}_n^*$ and $c$ would have high n-gram overlap, regardless of how much documentation does or doesn't "ease the mapping between NL and code". (A better experiment might be to look at n-gram overlap between $c$ and the *retrieved* examples from the model, using only information from $n$ instead of the oracle. It would also be interesting to look at overlap between the NL description and the documentation to make sure the gap is actually being "bridged" on both sides.)  *(update: The authors have taken my suggestion here and presented results using the retrieved documents.)*

## Reproducibility
The technique is fairly straightforward, and the authors release their new benchmarks and datasets, so it seems likely this work will be reproducible.
summary_of_the_review: *Original review:* Using documentation as additional context for generating code is a sensible idea, and it is good to have a thorough empirical confirmation that this does in fact yield higher accuracy in the NL -> code setting. Thus, although the novelty of the technique itself is limited, and the findings aren't particularly surprising, I tend to lean toward acceptance (assuming the issues I mention regarding strength of claims, clarity, dataset leakage, and n-gram overlap can be resolved).

*Updated review:* The authors have resolved my concerns, and upon reflection I do think that the problem formulation is interesting regardless of the novelty in the technique itself. I have increased my score from 6 to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents the first approach for retrieving code documentation for code generation as a way of addressing the fact that APIs are constantly changing. They present a general strategy entailing a retriever (sparse-BM25 or dense-large pretrained model) which first retrieves a set of relevant documents from a pool of documents (which does not need to be fixed). Then, the retrieved documents and concatenated with the NL intent to form the prompt that is fed into a generation model for code generation. They evaluate a number of different generation models, including T5, CodeT5, GPT-Neo, and Codex. For evaluation, they curate two different datasets, by ensuring disjoint training/validation/test splits, spanning two different programming languages: bash commands (tldr) and Python (re-splitting CoNaLa). Their results show that in general, their DocPrompting approach tends to achieve superior performance over simply prompting with the NL intent alone. 

Contributions:
- First paper to consider retrieving documentation for code generation
- Two curated datasets that can be useful for future work
- Extensive empirical experiments and ablation studies demonstrating the utility of their approach

strength_and_weaknesses: Strengths:
- The motivation and approach is well-grounded
- There are extensive results, spanning many different metrics. The Recall_unseen metric is particularly useful in demonstrating how much the prompting could help in a realistic scenario in which models are tested when new API changes are made on unseen functions.
- The analysis in Section 6 is very thorough, providing possible explanations for a number of different observations that are made.

Weaknesses:
- Adding retrieval will likely increase latency. The authors do not discuss/report how this is affected. As the size of the document pool increases, it seems that model latency will also increase.
- While the reason for splitting paragraphs into individual documents is explained in the Appendix, it seems odd to treat them as independent documents. For example, in Figure 2, the two bottom documents cannot standalone without the top one.
- It is not clear how k is selected for the number of documents to retrieve. Is it always 3?
- The annotation process for writing the test cases is not clear. In Appendix B, it is written that the two first authors did this. It is not clear whether each example was annotated by both of them and if so, what the agreement on that was.
- Missing reference: https://arxiv.org/pdf/1808.09588.pdf

clarity,_quality,_novelty_and_reproducibility: - Clarity: Most things are explained clearly either in the main paper or the appendix, except for the few points that are raised in the Weaknesses above.
- Quality: Well-designed and experiments and well-written work.
- Novelty: There is novelty in being the first paper to retrieve documentation for code generation and also in terms of the two newly curated datasets. However, in terms of technical novelty, already existing models and techniques are used for retrieval and generation.
- Reproducibility: Model training details are provided in the Appendix, which should be sufficient for reproducing these results, once the datasets are made available.

summary_of_the_review: While the technical novelty may be limited, there is novelty in terms of application and benchmark creation which will be useful for future work. The experiments and empirical results are thorough. Overall, I feel that this is a good paper contributing useful ideas and artifacts to the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a method to generate code from natural language prompts by retrieving snippets of documentation that is relevant to the prompts and augmenting the codegen model prompt with those snippets. It is evaluated extensively on several existing benchmark, also introduces a new benchmark (for bash), and is compared to adequate baselines (ROBERTA, CodeT5) that it can augment itself (adding retrieval). It improves performance across the board. Possible improvements to the submission include a more thorough data leakage / retrieval advantage analysis in the results (even though no reviewer expects it would change the experimental conclusions), and more details for reproducibility. Overall, it is a simple method with a direct, visible effect on tasks of the codegen community.
justification_for_why_not_higher_score: The scope is limited to codegen tasks.
justification_for_why_not_lower_score: This is a valuable addition to the ICLR line-up, and the method is simple and applicable enough that it would be of interest to enough people.
Decision: Accept: notable-top-25%