DocPrompting: Generating Code by Retrieving the Docs
Authors: Shuyan Zhou
Uri Alon
Frank F. Xu
Zhengbao Jiang
Graham Neubig
Venue: ICLR 2023 notable top 25% (2023)
Keywords: code generation
retrieval-conditioned generation

Abstract: Publicly available source-code libraries are continuously growing and changing. This makes it impossible for models of code
to keep current with all available APIs by simply training these models on existing code repositories. Thus, existing models inherently cannot generalize to using unseen functions and libraries, because these would never appear in the training data. In contrast, when human programmers use functions and libraries for the first time, they frequently refer to textual resources such as code manuals and documentation, to explore and understand the available functionality. Inspired by this observation, we introduce DocPrompting: a natural-language-to-code generation approach that explicitly leverages documentation by (1) retrieving the relevant documentation pieces given an NL intent, and (2) generating code based on the NL intent and the retrieved documentation. DocPrompting is general: it can be applied to any programming language and is agnostic to the underlying neural model. We demonstrate that DocPrompting consistently improves NL-to-code models: DocPrompting improves strong base models such as CodeT5 by 2.85% in pass@1 (52% relative gain) and 4.39% in pass@10 (30% relative gain) in execution-based evaluation on the popular Python CoNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo1.3B by up to absolute 6.9% exact match.

Published as a conference paper at ICLR 2023
DocPrompting: GENERATING CODE BY RETRIEVING THE Docs
Shuyan Zhou', Uri Alon'
Frank F. Xu', Zhiruo Wang', Zhengbao Jiang', Graham Neubig
'Language Technologies Institute, Carnegie Mellon University. 'Inspired Cognition
{shuyanzh, ualon, fangzhex, zhiruow, zhengbaj eeeiggesssssa..........eeeee.. cmu edu
ABSTRACT
Publicly available source-code libraries are continuously growing and changing. This makes it impossible for models of code to keep current with all available APIs by simply training these models on existing code repositories. Thus, exist- ing models inherently cannol generalize to using unseen functions and libraries, because these would never appear in their training data. In contrast, when human programmers use functions and libraries for the first time, they frequently refer to textual resources such as code manuals and documentation, to explore and understand the available functionality. Inspired by this observation, we introduce generation approach that explicitly leverages code documentation by (1) retrieving the relevant documentation pieces given a natural language (NL) intent, and (2) generating code based on the NL intent and the retrieved documentation. DocP Proprrmpging general: it can be ap- plied to any programming language, and is agnostic to the underlying neural model. We demonstrate that DocPromp rompting consistently improves NL-to-code models: DlocPrompting improves strong base models such as CodeT5 by 2.85% in pass @ (52% relative gain) and 4.39% in pass@ 10 (30% relative gain) in execution based evaluation on the popular Python CaNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo-1.33 by up to absolute 6.9% exact match.
DocPrompting: latural-l
INTRODUCTION
We address the task of natural language to code generation (NL-<cdde): generating a code snippet, written in a general-purpose programming language such Python Bash, given a natural language intent. This task has seen sharply growing popularity recently due to the emergence of large language models trained on vast amounts of natural language and code (Chen al.) 2021 Xu et al. 2022 Fried et al., (002)). NL-+code models facilitate programming for both professional and inexperienced programmers, by allowing programmers to write code by only expressing their higher-level intent.
Many existing code generation models either learn directly from input-output pairs provided as training data (Allamanis et al. [2155] Yin and Neubig. 2017 [yer et al.) 2018 BBockkshmidd et al. 2019: Xu et al., [2000] Alon al. 22200] Wang al. 2021) learn the mapping between input and output implicitly from naturally occurring corpora of intertwined natural language and code (Austin et al. 2021 Nijkamp et al. (022). Nevertheless, all these works assume that all libraries function calls were seen in the training data; and that at test time, the trained model will need to generate only seen libraries and function calls. However, new functions and libraries are introduced all the time, and even a seen function call can have unseen arguments. Thus, these existing models inherently cannot generalize to generate such unseen usages.
In contrast to these existing models, human programmers frequently refer to manuals and docu- mentation when writing code (Nykaza et al. 2002 Lethbridge et al. 2003). This allows humans to casily use functions and libranies they have never seen nor used before. Inspired by this ability,
Data and code available http://///gggth..././///aaaaaa//

---

Published as a conference paper at ICLR 2023
from pygments import code "print("reading docs")' highlight PythonLexer() HtmiFormatter())
Generate HTML with pythou syntax highlighting for "print(readding does') "
Retriever
Generator
Pygment generie syntax highlighter
A formatter takes the token stream and writes to an output file
A lexer splits the source into tokens, fragments ...
class Htm/Formatter
Format tokens as HTML <span> tags with..
class Lexer For Python source code
Figure 1: DocPrompting: given an NL intent 2 the retriever retrieves set of relevant documenta- tion { @,., 2 from a documentation pool D Then, the generator generates the code @ based on the NI and retrieved docs. DocPrompting allows the model to generalize to previously unseen usages by reading those docs. Italic blue highlights the shared tokens between NL and docs; Bold shows shared tokens between docs and the code snippet.
we propose DocPrompting: a code generation approach that learns to retrieve code documentation before generating the code. An overview of our approach is illustrated in Figure First, a document retriever uses the NL intent @ to retrieve relevant code documentation (..). 2G from documen- tation pool ④ Then, a code generator uses these does in its prompt to generate the corresponding code C. The documentation pool serves with new contents (c.g., documentation of newly released libraries), without re-training any model component. This way, DDccPrompting can leverage newly added documentation, and it can generate code containing unseen and unused functions and libraries. DocPrompting is general and applicable to any programming language and underlying base architecture. To the best of our knowledge, this is the first demonstration of leveraging documentation in models of code explicitly and effectively.
external data store that be updated frequently
We demonstrate the effectiveness DocPrompting on two NL-xcode benchmarks and tasks, across two programming languages, and using several base models: GPT-Neo (Black et al. (202T). T5 (Raffel al. 2020), CodeT5 (Wang et al., 2021), Eesioonin-Decodee (Izacard and Grave 2021 and Codex (Chen et al. 2021). Further, experiment with both sparse retrievers such as BM25 (Robertson and Jones 1976) and dense retrieval models such as SimCSE (Gao et al, 2021), Finally, we introduce two new benchmarks for retrieval-basee code generation: (a) in Bash, we curate new benchmark by crawling the tldr repository, and constructing the training/developmen//ess splits without overlapping commands; (b) in Python, we re-split the popular CoNaLa benchmark (Yin al. 2018) by making every test example contain at least one Python function that is not seen in the training data. Models that DocPrompt ing consistently outperform their base models that generate code solely based on the NL intents. Using DocPrompting improves strong base models such as CodeTT by 2.85% in pass @ 1 (52% relative gain) and 4.39% pass @ 10 (30% relative gain) in exccution- based evaluation in CoNa on the new tldr dataset, Doc Doomromg improves CodeTTS and GPT-Neo- 1.3B by up to absolute 6.9% exact match. We release our new benchmarks, including annotation of oracle documents for each example and pools of documentation, to serve test-bed for future retrieval-based code generation models.
2 CODE GENERATION BY READING THE DOCS
Our underlying assumption is that code documentation is the most exhaustive yet succinct resource for most libraries and programming languages (Roehm et al. 2012)) and that documentation allows to effectively generalize to unseen libraries and functions (Forward and Lethbridge (2022). We follow the retrieve-then-gane generrte paradigm (Lewis et al., 2020 Guu et al.) 2020). focusing on retrieving documentation. In this section, describe the general approach of Doc DocPrompting: in Bnd 6.2 we elaborate and experiment with practical implementations of DocPrompting.
Formulation Given NL intent n, our goal is generate corresponding code snippet written in some programming language (PL) such as Python. We assume that model has access to collection of code documentation D. Each document d, D describes the usage of library, function, or
2

---

Published as a conference paper at ICLR 2023
argument in that PL. The construction of D is flexible: it can either be comprehensive set of all available libraries and functions a PL, or customized subset for the scope of a specific project.
2.1 BACKGROUND: EETRIEVAL-CONDITION
GENERATION
Although model may use the entire collection of documents D, only a few documents in D relevant for any particular intent. Further, it is usually computationally infeasible to directly condition on the entire, unbounded, collection of documents while making predictions. Thus, we first let the model select subset of documents D, = {d1. .2. d2 = D that are potentially relevant given n, and refer to this subset while generating c.
Overall, we decompose the probability of generating into the probability of choosing a particular subset of documents P (Dn D. n), and the probability of generating the code conditioned on the intent and the selected documents P (cl n); finally, we marginalizing over all D, S D:
Dn) - (c nn, P (Dr.) D,
(1)
assuming that is independent of given D, (that is, (c D D.)). Since enumerating all possible subsets D, is computationally infeasible, we follow the common practice and approximate the marginalization over D, in Equation (I) by taking the most probable subset of retrieved documents D,,, and then conditioning the prediction of e on these most likely documents:
Dr argmax =D (D,, D.
P(e D, P(cl Da,n) P(D. D,
2.2 DocPrompting: GENERATING CODE BY RETRIEVING THE DoCS
Equation 2] implies that DocPrompting relies of two main components: A retriever R. retrieves relevant documents Dr, given the intent n; and a generator 9 generates the code snippet c conditioned on the retrieved documents D,, and the intent n, which compose a new prompt. Specifically, R. computes similarity score (d, n) between intent n and every document dę e D. Thus, the subset D,, D the top-k documents with the highest similarity scores: D,, top-kd, (s (d),
An overview of our approach is illustrated in Figure 1 given the intent Generate HTML with python syntax highlighting for "print(' 'reading docs") '"", the retriever R retrieves three relevant documents: d describes the syntax highlighting library pygnenss, d2 describes the class F sythonLexee and class. Given these docs and the intent, the generator generates from the pygment library.
d3 describes the FommFormm's the code snippet which uses PythonLexer and fttmlFormate
3 PRACTICAL INSTANTIATIONS OF DocPrompting
DocPrompting is general approach that is not bound to any specific model choices, and it can be instantiated with any base retriever and generator. This section presents the concrete instantiations of R and G that we found to provide the best performance in our experiments.
3.1 RETRIEVER INSTANTIATION
experiment with two main types retrievers: sparse retrievers and dense retrievers. As our sparse retriever, Elasticscaref with the standard BM25 (Robertson and Jones 1976). This retriever represents documents using sparse features that rely on word frequencies, such as BM25 and TF-IDF
As our our dense retriever, we follow prior work (Chen et al. 2020; Karpukhin et al. 2020 Gao et al. 2021): given a triplet (n, c, D,,, where DD,,, are the oracle docs for n, each dj Dj and form positive pair (n,d)), d)), while each dł D", and form a negative pair (nz, dř). We train the retriever in contrastive fashion where the similarity score of a positive pair is maximized while that of in-batch negative pairs is minimized. For pair (n,, d,)), the loss function is defined as:
exp exp( (sim(mn,,, hay))
L log exp simm(hn; +En exp sim(hnnh):;)
(3)

---

Published as a conference paper at ICLR 2023
where fgg is the representation of computed by neural encoder, and B are positive does for other examples in the batch. We define (mm, hy) as the cosine similarity between h2 and hy.
We use all (n,, d,)) in the training set as supervised training dataset. Additionally, we use all sentences in the documentation pool for weak supervision: Following Chen et al. (2020) Gnd positive
Gao et al.
(202T), representations of the same sentence with different dropout masks are treated example. Instead of using either supervised or weakly supervised training as in Gio et al. (2021), > we simply mix the two resulting supervision signals, and examples are randomly distributed into batches. This mixture of tasks not only facilitates the learning process (. but also reduces the engineering effort required to store and reload models for separate supervised and unsupervised training phases. We initialize the retriever encoder with either the best model of Gao et al. (2021) or the encoder of
CodeT5- -aaaee (Wang et al. 2021). Additional training details are provided Appendix C
3.2 GENERATOR INSTANTIATION
We experimented with variety of generator models. We used GPT-Neo- MeNNNNo-15M... 125M, GPT-Neo- 1.3B (Black ct al. 202T) and Codex (Chen et al. 202I), where we concatenate the retrieved documents and the NI intent single, long, prompt. T5-base (Raffel et al. 2019) and CodeTT5-base (Wang al. 2021) have a shorter input size of 512 tokens, which is sometimes too short for the concatenation of multiple docs. Thus, for T5 and CodeT5 we apply the fusion-in-decoder approach (FiD; Izacard and Grave 2021): we first concatenate the intent with each retrieved d, é 'D, and encode each (n, d,) pair independently. Then, the decoder attends to all encoded NL-document pairs. We finetune the generator to maximize the log-likelihood of the reference code given n and Dn,.
With Codex (Chen al., 202T we performed few-shot learning rather than finetuning because the model parameters are not publicly available. We constructed the prompt with three static examples, each of which is concatenation snippet. We then appended the test example and its retrieved documentation to the few-shot examples. We used the code vade-davinci-011 version because we suspect potential Icakage of the test set into the training set of uade-davinci-02. See more details in Appendix H Training details, param-e settings and example prompts can be found in Appendices and
retrieved documentation, an NL intent and the reference code
4 EXPERIMENTAL SETUP
We evaluate DocPrompting on two NL-xcode tasks: shell scripting ($4.1 in which we generate complex shell commands given intent, and Python programming ($4.2) where generate answers in Python for NL questions. In this section, we first introduce a newly curated benchmark tldr; we then describe our re-split of the popular CoNaLa benchmark (Yin al. 2018). For each benchmark, we provide a global documentation pool D that is shared for all examples and oracle documents D; which we use to train the retriever. We release our newly curated benchmarks to serve as best-bed for future retrieval based code generation models.
4.1 SHELL SCRIPTING
project that maintains easily- readable help pages with examples for over 2.5k Bash com- mands in over 25 natural languages" We collected pairs of English intents and Bash command lines. The NL intents written by human users, and the Bash commands range from popular ones like cat and tar, to uncommon commands such as toilet and faket tme. Our resulting t clr benchmark contains 1,879 unique Bash commands and 9,187 NL-Bash pairs. We constructed the training, development and the test set with completely disjoint commands to test the generalizability of code generation model. The shared documentation pool D is made up of the 400k: paragraphs from the 1,879 Bash manuals. Each paragraph describes single concept such as an
tidr is a community-dee
NL
Show slurm jobs queued by a user 'xyz' every seconds
Code squeue -u xyz -1
squeue is used to view job and job step for Slurm jobs
Request jobs or job steps from list of users.
Repeatedly report the information at the interval specified
Oracle docs
Figure 2: An example NL-code pair from tldr, along with three oracle documentation items.
https://githube Ottt:////ggttotoooooo0/0000-0000
4

---

Published as a conference paper at ICLR 2023
argument flag. We further curated the oracle documents D for each example using simple string matching. An example from tidr is shown in Figure 2 first work to leverage t tldr an NL-cc benchmark. Detailed statistics and additional details are provided in Appendix A In trd each NL intent results in a single Bash command with a combination of argument flags. We therefore first retrieve entire Bash manual; then, take the lop manual and retrieve the top-10 paragraphs from that manual.
To the best of our knowledge, this is the
Evaluation metries We measure: (a) command name accuracy (CMD Acc) whether the command name (e.g., cat) is an exact match; (b) exact match (EM) exact match between the reference and the generation; (c) token-level Fl; and (d) character-level BLEU (charBLEU; [in et al. 2018; Shi et al., (022). In all metrics, we disregard user-specific variable names in the references and the models outputs. For example, "mycli -u [user] -h [host] [database) se] evaluated as "mycli -u $1 -h $2 $3".
4.2 PYTHON PROGRAMMING
CoNaLa (in et al. 2018 a popular benchmark for NL-Python generation. NL intents are StackOverflow questions, and code snippets are their answers. Both intents and code snippets rewritten by human annotators. We re-split the dataset to test models' generalization to unseen Python functions. In re-split, verifed that every example in the development the test set uses at least one Python function (e.g., plt. plot) that was not seen in the training data. In addition, we make sure that the examples from the same StackOverflow posts are in the same prevent leakage. This re-split results in 2,135/201/545 1/344 examples in the
sets, respectively.
The CoNaLa documentation pool DD contains 35, 56,63 documents, each describing single function, from all Python libraries available on DevDocs (https: http:::/ddvvdcaeo io). These include built-in libraries and other popular libraries such as numpy. We constructed the oracle docs D; for each example by matching all function names in the target code e with docs. More details in Appendix B
Evaluation metries We follow Yin et al. 2018)) and measure BLEU-4. Since we focus on generaliza- tion to unseen functions, we additionally report function name recall (recall) and unseen function recall (recallunsoen) which measures recall among function calls that do not appear in the training set. Finally, following Chen et al. (202T); Austin et al. (2021) we used the manually written unit tests from Wang et al. (2022) for 100 examples from CoNaLa's 's test set and measure pass @ k. We followed Chen et al. (2221) and performed nucleus sampling (Holtzman et al. 2019) with p 0.95. For each k we searched for the best temperature for each model from (0.2, 0.4, 0.6, 0.8, 1.0). On average, each example has 2.03 tests. The concatenation of multiple Python docs often exceeded the length limit of GPT-Neo, we hence experimented this dataset with FiD, which allows longer inputs. Additional details are provided in Appendix B
RESULTS
In all following results, all models with DocPrompting use the top-10 retrieved docs from the best retriever on that dataset (Table 4). Every baseline uses the exact same setup as its "+DocProme DoDopppomppng"" version, except for not using the documentation.
5.1 SHELL SCRIPTING RESULTS
Results for t cldr are shown in Table DocPrompting consistently improves the base models. For example, T5+DocPrompling achieves more than twice higher accuracy in predicting the command name, more than 16 charBLEU points on the entire prediction, and almost 9% of absolute exact match gain, compared to the vanilla T5. In the few-shot learning setting with Codex, DocPrompting brings gains of 6.7 charBLEU points, and consistent improvement across all metrics over the baseline that observes only NL-code pairs in its prompt. These results show that retrieving documentation also benefits strong models such Codex, and with only few examples in the context.
Code generation with oracle command names In realistic settings, human programmer may know the command name they need to use (e.g., awk), but not know the exact usage and flags. In fact, better understanding of the usage of known commands is the purpose of Unix pages and the
5

---

Published as a conference paper at ICLR 2023
Table Results shell scripting, using BM25 retriever with top-10 retrieved docs, the test set of t lld. For the "oracle command name" experiments, we selected the best model of each type.
Model
CMD Acc (%)
EM (%)
Token Fl
charBLEU
11.96
1.94
28.75
19.99
GPT-Neo-125M
+DocPrompling
25.32
3,56
31.23
24.43
14.55
3.12
32.46
24.70
GPT-Nco-1.3B
+DocPrompting
27.59
9.05
37.24
30.57
10.02
0.76
19.90
25.48
T5
+DocPrompting
30.28
9.16
37.58
31.97
14.60
2.18
30.00
21.50
CodeT5
+DocPrompting
30.72
9.15
36.71
33.83
27.48
8.94
36.04
16.94
Codex 3-shots
+DocPrompting
31.21
9.29
36.77
23.72
With the oracle command name
12.96
59.36
45.05
T5
+ +DocPrompting
22.55
64.84
54.28
22.44 32.43
62.26 69.73
50.29
Codex 3-shots
+ DDocPrompting
55.21
Table 2: Comparison to approaches that retrieve examples
(Parvez et al.
2021
Pasupat et al.
2021
Model
CMD Acc (%)
EM (%)
Token Fl
charBLEU
+ExP rempt ing
6.68
0.32
20.49
11.15
GPT-Neo-125M 125M
DocPrompting
25.32
3.56
31.23
24.43
+ExP rempt ing
14.01
2.8
30.07
22.11
GPT-Nco-1.3R
+DocPrompting 27.59
9.05
37.24
30.57
t cldr project. We conducted an oracle experiment where we provided T5 (which was the strongest model using DocPrompting) and Codex with the oracle command name (e.g., awk). This oracle information is provided to both the baseline and the model that uses DocPrompting. The results are shown on the bottom part Table When the oracle command is given, DocPrompting further improves over the base models. For example, when providing Codex with the ground truth command name, DocPrompting improves exact match from 22.44% to 32.43%.
Should retrieve documentation examples? All existing retrieval-based models of code retrieve NL-code pairs or code snippets. rather than documentation. To simulate this scenario, we followed Parvez et al. (202T) and Pasupat et al. (202I) to retrieve NL-code pairs from the training set of tldr, and refer to this baseline as ExP r rompting We finetuned the best retriever RoBERTa and two generators, and retrieved the top-30 NL-code pairs for every example. As shown in Table retrieving documentation (DocPrompiigg) provides much higher gains than retrieving examples (ExPrompt rompt ing). Theoretically, adding examples of unseen commands can help ExP rompt ing generalize to them as well. However, new libraries and functions may not have available examples on the web yet, while documentation often does becomes available when the library is released.
5.2 PYTHON PROGRAMMING RESULTS
Table 3 shows the results on CoNaLa. CodeTS5DooPPomppiin vields a 1.65 BLEU improvement over the mfnte-of-the-ate art baseline that was initialized with CodeT5 When measuring the recall of the generated function names, the benefit of DocPrompting especially higher for unseen functions (recallunwen). For example, promptingg achieves 18.30 compared to only 9.03 of the base Code'T5 unseen functions. Additionally, Pooprompgigg improves in-context learning setting with Codex.
*In separate experiment the original split of CoNa l.a. this baseline achieved BLEU score of 39.12 12, which outperforms the previous state-of-the-art (Beau and Crabhé 2022 by 4.92 BLEU points.

---

Published as a conference paper at ICLR 2023
Table Results on CoNaLa, using a CodeT5 retriever with top-10 retrieved docs. Function recall (Recall) measures how many functions in the reference code are correctly predicted, and unseen function recall (Recallanseen) only considers the subset held out from the training data.
Model
BLEU
Recall
Recallanwn
43.16
39.52
Codex 3-shots
DocProomping
43.47
39.87
+ Docl roopplong oracle docs
50.59
57.84
28.07
14.36
2.57
T5
+ DocPrompting
30.04
21.34
8.24
34.57
24.24
9.03
CodeT5
+ DocPrompting
36.22
27.80
18.30
+ DocPrompping oracle docs
49.04
72.20
63.91
tldr
CoNaLa
40
35.46
100%
35
31:87
NLe-C
30
27.54
80%
(NL+Doca) Code
pass @
25
20
18.70
23.38
25.54
27.08
Recall
60%
52%
15
14.31
40%
24%
6044
28%
10
8.26
+DocPrompting
20%
:24
14
0/3
10%
7.
5
5.41
CodeT5
0%
016259
0100%
0
110
50
100
200
2
5
k
D-gram
n-gram
Figure 4: Using documentation significantly in- creases the n-gram overlap recall between the input and the output, in tldr and CoNaLa.
Figure 3: @ of CodeT5 with and without DocPrompting on 100 CoNaLa examples.
We hypothesis that the minor gain is mainly due the potential data Ieakage Codex, which violates the split of seen and unseen functions. Another reason is that strong generator such as Codex may require an equally strong retriever as well. We find that Codex can achieve even higher results with an oracle retriever, which shows the potential further improvement by improving the retrievers. Finally, CodeT5 performs better than T5, with and without using DocPrompting. This emphasizes the importance of using code-specific pretrained models.
Execution-based evaluation The results are shown Figure Using DocPrompting consistently outperforms the baseline CodeT5 for all values of pass k. For example, DocPrompting yields 2.85% improvement on pass@1 and 4.45% improvement on pass@5, which are realistic numbers of completions that can be suggested in IDE. When l 200, DocPrompting widens the gap to 8.38% These results demonstrate that DocP DocPrompting does not only improve the quality of the generated code in its surface form, but also increase its functional correctness. Additional details and results are provided in Appendix
ANALYSIS
6.1
WHY DOES READING THE DOCUMENTATION HELP GENERATING MORE ACCURATE CODE?
We believe that one the major reasons is that documentation eases the mapping between NL intents and code, since the documentation contains both NL descriptions and function signatures. We calculated the n-gram overlap between the NL intents and their corresponding code snippets (NL+code)), and the overlap between the NL intents with their top-10 retrieved documents and their code snippets (NL+tdocs+<code) As shown in Figure 4, adding documentation significantly increases the overlap across n-grams, and increase, for example, the unigram overlap from 1 12% to
7

---

Published as a conference paper at ICLR 2023
Table 4: Retrieval performance of multiple models the dev set of tldr (top) and CoNaLa (bottom).. RoBERTa is the best model taken from from Gao et al. (2021), and CodeTTS is the encoder of C5 base (Wang al. 202T). Models with the subscript "off-shelf" are the off-the-shelff models, and the other models were finetuned with the objective in [Equation 3 The last column is the best model (RoBERTa for tldr and CodeT5 for CoNaLa) trained without the weak supervision corpus.
BM25
RoBERTa shif
RoBERTa
CodeTSoutthhel
CodeT5
Best w/o weak sup.
32.81
17.53
30.03
10.45
18.10
28.30
5
51.73
37.89
52.50
20.26
38.52
50.50
tldr
59.86
46.80
60.33
25.73
51.03
59.84
20
62.01
56.11
64.30
33.65
57.26
62.30
3.01
4.46
13.49
4.60
16.54
10.51
5
7.16
7.58
26.38
8.63
42.35
21.15
CoNaLa
10
9.73
10.93
34.86
12.25
55.81
29.34
20
11.46
13.89
45.46
18.46
66.79
42.21
24% in tldr. That is, one of the reasons that retrieving documentation helps generating accurate code that documentation bridges the gap between the "intent terminology" and the "code terminology".
6.2 ABLATION STUDY
We compared different configurations of the retriever, to gather more insights for effective DocPrompting. Table shows comparison between different retrievers and their setups. First, the performance of BM25 varies among datasets: In tldr, BM25 matches the recall of trained dense retrievers; however in CoNaLa, BM25 achieves only recall @ 10 of 9.73%, and strong dense retrievers such as the encoder of CodeT5 achieve recall @ 10 of 55.81. We hypothesize that this difference between datasets stems from the ways these datasets were created: tidr intents were written based existing Bash commands and manuals; while CoNaLa examples were mined from StackOverflow posts, where users ask questions with limited or no context. Thus, NL intents CoNaLa require better semantic alignment with the documents, and thus benefit from dense retrievers. The gap resulting from different data curation processes was also observed by RRodriuuez and Boyd-Graber (2021) in open-domain question answering (QA).
Second, retrievers that were pretrained on the target programming language are generally stronger. For example in CoNaLa, CodeT5 which was pretrained Python, is both better off-the-shelf retriever and a better finetuned-retriever than RoBERTa, which was pretrained mainly on text. In contrast, tld based on Bash. which neither CodeT5 nor RoBERTa were explicitly pretrained on. Thus, tldr benefits mostly from BM25 and RoBERTa rather than CodeT5 as retrievers.
Finally, training the retriever using weak supervision on the documentation pool (Section 3.1 dramatically improves the retriever. The recall of the best retrievers of each dataset without this corpus shown the last column of Table ("Best w/o weak sup."). On CoNaLa, removing this corpus results in severe performance degradation. One possible explanation is that this weak supervision helps the retriever perform domain adaptation more effectively.
6.3 CASE STUDY
We examine the models' outputs and show two representative examples in Table 5 example, Image. open was not seen in the training set, and the baseline CodeT5 incorrectly predicts os open. In contrast, using DocPrompting allows to retrieve the docs and to correctly predict Image. open. In the second example, df.to to csv was not seen in training, and the baseline CodeT5 fails to correctly predict it. In contrast, DocPrompting does predict most of the df.to_sst call correctly, thanks to the retrieved docs. Nevertheless, DocPrompting generates an incorrect argument skiprows=1, instead of header=False. The reason is that along with the retrieved documentation of df. to.tt.coo to. the retriever also retrieved the documentation of df www..erassss... ccvv which has skiprows argument. That is, the generator the function df to csv. Further improving the retrievers and the generators, and based on the validity of argument names, may mitigate such mistakes.
In the first
argument of df read.csv with
8

---

Published as a conference paper at ICLR 2023
Table 5: Examples of predictions from CoNaLa, of the base CodeT5 compared to CodeT3+DocPromen Unseen functions are underscored.
NL Intent: Open image "picture.jpg"
Ground truth:
img Image. ('picture. jpg'') \n Img.show os op.o (' ('picture. jintturre.ppp,... 'r')
CodeT5:
CodeT5 Prompiing
image opene picture. ipg', 'rb')
NL Intent: Exclude column names when writing dataframe 'df" a file 'illenaee.cs''
Ground truth:
df.to (' ('filename. csv', header=False)
CodeT5: MaleTT+DDccPompppa
df drop co'c'l coll"), axis=l, inplace-True) secto filename. cav', skiprows-1)
RELATED WORK
Code generation The most common practice in NL-code generation is training model on a dataset of NL-code pairs (Allamanis et al. [015] Yin and Neubig 2017 Rabinovich et al. 2017 Iyer et al. 2018). Nevertheless, all these works assume that their training corpus covers all required libraries and functions, and their models are inherently incapable of generating libraries and functions that were not seen in the training data. On the contrary, DocPrompting allows models to generate calls unseen function, by retrieving these functions" documentation and reading them at test time. Hayati et al. (2018): Parvez et al. (202T): Hashimoto et al. (2018) and Lu et al. (017) learn to retrieve examples at test time; Pasupat al. (222) also considered settings where the test data has distribution shift from the training data. However, when new libraries are released they often come with documentation, and thus assume that documentation for new libraries is much
likely be available than concrete natural language intent and code snippet pairs (n, e) that use these libraries already. The models of (Sriivastava al.) and Wu al.) (202T) retrieve code snippets from relevant files in the same project; contrarily, when predicting new libraries and functions that are external to the user's project, documentation is the source that the most likely be available.
Retrieval augmented generation The paradigm of retrieve-then-generae has gained popularity in the field of open- open-domain question answering (Guu al. 2020 Lewis et al., 2020; Karpukhin et al., 2020), where the answer for an open-domain question exists in only few documents out of much larger pool. Although DocPrompting takes a similar approach, documentation retrieval in code generation is even more valuable, since code libraries are updated constantly, and new libraries are introduced daily. Thus, DocPrompting allows updating the documentation pool frequently with new contents, without re-training any model components.
Documentation conditioned generation The model of Zhong et al. (2019) reads documents to understand environment dynamics in gorl- game, and Branavan al. (2011) controls situated agents in a game (Civilization II) by reading the game's manual. However, all their models were tailored to specific games; in contrast, DocPrompting is general and is applicable for variety of programming languages and datasets.
CONCLUSION
We propose DocPrompting, simple and effective approach for code generation by retrieving the relevant documentation. DocPrompting consistently improves NL-code models in two tasks, in two PLs, and across multiple strong base models. DocPrompting improves strong base models such as CodeT5 by 2.85% in pass @ 1 (52% relative gain) in execution-based evaluation on the popular Python CoNa La benchmark; on a new Bash dataset 1 dr. DocPrompting improves CodeT5 and GPT-Neo-1.3B by up to 6.9% exact match, and Codex by 6.78 charBLEU score.
These results open a promising direction for NL-ccode generation. We believe that our results can be further improved using more clever encoding of the structured nature of long documents, and using joint training of the retriever and the generator, which hopefully will avoid cascading errors. Further, we believe that the principles and the methods presented in this paper applicable to additional code-related tasks, and other documentation like resources such as tutorials and blog posts. To these ends, make all code, data, and models publicly available.

---

Published as a conference paper at ICLR 2023
9 ACKNOWLEDGEMENT
We thanks the anonymous reviewers for their useful comments and suggestions. This work is supported by a gift from Amazon AI and contract from the Air Force Research Laboratory under agreement number FA8750-19-22 19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
REFERENCES
Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. Bimodal modelling of source code and natural language. In Francis R. Bach and David M. Blei, editors, Proceedings the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015. volume 37 JMLR Workshop and Conference Proceedings, pages 2123-21322 JMLR.org, 2015. URL http: http::///occeeeeegeeee mir press/ hatps/////aaaaaaaaaaaamam 15. h
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. International conference on machine learning, pages 245 256. PMLR, 2020.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et Program synthesis with large language models, ArXiv preprint, abs/2108.07732, 2021. URL httph::tt///rtttt org abw.g///////// 22002211.1.. 2111107738 07732
Nathanaël Beau and Benoît Crabbé. The impact of lexical and grammatical processing on generating code from natural language. ArXiv preprint, abs/2202.13991 2022.
13972
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregres- sive Language Modeling with Tensorfowo,, 2021. htt.tttss/:.o.org/////pr oro//o.../.....11 zenodo 5297715 If you use this software, please cite it using these metadata.
S.R.K. Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in Monte-Carie framework. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 268-277, Portland, Oregon, USA, USA,2011, Association for Computational Linguistics. URI.https:: http:::/aacaaththii ogy. crg/p11 1028
Marc Brockschmidt, Miltiadis Allamanis, Alexander Gaunt, and Oleksandr Polozov. Generative code modeling with graphs In International Conference on Learning Representations, 2019. URL https
//openrevies eew net poet/forrun
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2 03374, 2021. URL //tp/v/axvvvv/vvo//////.../..vv/ org/ /bs//b///////
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1607. PMLR, 2020. URLLhttp://proccedaee
mlr prwss / v1 19/ chen20 html
Andrew Forward and Timothy C Lethbridge. The relevance of software documentation, tools and technologies: survey. In Proceedings of the 2002 ACM symposium on Document engineering, pages 26-33, 2002.
Daniel Fried, Armen Aghajanyan, Jessy Lin. Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. ArXiv preprint, abs/2204. Abs99204.05999, 2022. httltttsp:///:i..ccom
abs 2204. 05999
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE:
Simple contrastive learning sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.1 10.18653/v1/221 (t065///12222...mmpmmmmm... m55522. URL https: :/acl anthol wwwccholoo..rro. /202 www.lmmnppe....m..... m-mminn
552
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In International Conference on Machine I Learning,
2020. URL htt.tttsss////// / wttp:/wwwwwoog/ot//////ae/// www.///g///aooooooooo/eeeeeeeeeo 10396
10

---

Published as a conference paper at ICLR 2023
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wee Wei Chang. Realm: Retreeval-uuggmeteed language model pre pre-training. ArXiv preprint, abs/2002.08909, 2020. UR/jotpsss/arxx...ory//.a
2002. 2002.08909 08909
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. A retrieve-and-edii framework for predicting structured outputs. Advonces in Neural Information Processing Systems, 31, 2018.
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Authony Tomasic, and Graham Neubig. Retrieval based neural code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 925 930, Brussels, Belgium. 2018. Association for Computational Linguistics. doi: 10. (0.18666 1/65/////D11---// 1111. URL https:htss://aaee http:////caaaaath/ooaooa ogy www.orgg/ll...
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degencration. arXiv preprint arXiv: 09755, 2019.
Srinivasan Iyer. Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in pro- grammatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643 1652, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18 1192. http:/ttscccaaattaiiaee wwtp:/ooooo....1--/oaeea 11192
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain ques- tion answering. In Proceedings of the 16th Conference of the Europaa Chapter of the Association for Compu- national Linguistics: Main Volume, pages 874-880, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 1/252/11/22211..1 leacl-main.723 74. URL httttpss///caaaattoooot//o
/wtg/222l.ccmomm www.e-cccaannn... -main. 74
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs, IEEE Transactions on Big Data, 733):535-547, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retricval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, 2020. Association for Computational Linguistics. doi: 10. 0018665551//22200mmmmmmmmmm...5 URL atttps://aclanthollog...o
emnlp main.555
Timothy C Lethbridge, Janice Singer, and Andrew Forward. How software engineers use documentation: The state of the practice. IEEE software, 20(6):35-38 39, 2003.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Schastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensivg NL. tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL ba httpp:://ppocecninng......../p/e www.nnuui..... wwt..ips...pppppr///////pp wwtp:/pppppp/p//oeaeepp 00043323020517880c
7481e Awstract........................ .html
Xi Victoria Lin, Chenglong Wang. Luke Zettlemoyer, and Michael D. Ernst. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language
Resources Association (ELRA). URL.ttts. //cc///aaa///// anthhn www.ttchcooo... ogy. org/ /.1l 8- 1491
Yanxin Lu, Swarat Chaudhuri, Chris Jermaine, and David Melski. Data-driven program completion. arXiv preprint 17Xx::1705.09042. 2017.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong, A conversational paradigm for program synthesis. arXiv preprint, 2022.
Janet Nykaza, Rhonda Messinger, Fran Bochme, Cherie L Norman, Matthew Mace, and Manuel Gordon. What programmers really want: results of a needs assessment for sdk documentation. In Proceedings of the 20th annual international conference on Computer documentation, pages 133-141, 2002.
Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval aug- mented code generation and summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2734,73, Punta Cana, Dominican Republic, 2021, Association for Computational Linguisties. doi: 10.18653/v1/22 102333/1/222..// windinngs-ennlp...... URI. https: : / //acl anthol 1 ogy ogy.org/2021 emn Fi ndi ems 1p.232
11

---

Published as a conference paper at ICLR 2023
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7683-7698, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 1/2186651112221.coom .emplp main 607. URL hlttpp://aclantholooyy... org 002/ 2021. emnlp-main 607
Maxim Rabinovich, Mitchell Stern, and Dan Klein, Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1139- 1 1449, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/vL/P177 1105. URL
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv preprint, 1tt//990.11083.. 2019. htLttttts
org/ abs 1910. 10683
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter Liu. Exploring the limits of transfer learning with a unified text-to-texi transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Stephen E Robertson and K Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information science, 27( 1976.
Pedro Rodriguez and Jordan Boyd-Graber. Evaluation paradigms in question answering. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9630-9642, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 1/253/ emnlp-main.75$6
wtt///2g1122// emnip-mmmme main. 758
Tobias Rochm, Rehecca Tiarks, Rainer Koschke, and Walid Maalej. How do professional developers comprehend software? In 2012 34th International Conference on Software Engineering (ICSE), pages 255-265. IEFE, 2012.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution, 2022. URL https:
2204 11454
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow, Repository-lewe prompt generation for large language models of code. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Yue Wang, Weishi Wang. Shafiq Joty, and Steven C.H. Hoi. CodeTS: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 105553/112221.emm/ eemnpp 685n.685. URL htttss:/accLanthooogg.
2021. emnlp maamnlp-mann...mmmm 685
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution- evaluation for open-domain code generation. arXiv preprint arXiv:2212.1 2022.
Yuhuai Wu. Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2021.
Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig, Incorporating external knowledge through pre-training for natural language code generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 6045-6052. Online, 2020. Association for Computational Linguistics. doi: 10. 1/230v aww.:/200000.mmmmm......5.-3. 538, URL UtK//ttppp::/acaattoologg /rr/ 2020. acl macc--maai0 ma main. 538
Frank F Xu, Uri Alon. Graham Neubig, and Vincent Hellendoom. systematic evaluation of large lan- guage models code. ArXiv preprint, abs/2202.13169, 2022. URL http:ttss///ccv..orggg./ /abs ab2/ 2202 13169
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10. wwt.:/ooooyyy.oo11111110101110
18653/v1/P17- 1041.
Pengcheng Yin. Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning mine aligned code and natural language pairs from stack overflow. 2018 IEEE/ACM 15th international conference on mining software repositories (MSR), pages 476-486. IEEE, 2018
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. ArXiv preprint, abs/1 900019900,,,10, 2019. URI. httpsss//axxii..og///////1/..1.
12

---

Published as a conference paper at ICLR 2023
A TEDR: A NEWLY CURATED SHELL SCRIPTING BENCHMARK
NL- Bash pairs For each command (e.g., cat), users contribute examples of pairs of NL descriptions and bash code (mainly one one-liners), including various flags and arguments, which cover the common usages of that command. An example is shown in Figure 2)
We crawl No-c pairs from the markdown file in the linux and common folders. We discard Bash commands whose manual is unavailahle (discussed below). The detailed statistics are shown in Table 6 On average, each command has 4.84 Na-+Bash pairs and there is total of 9187 NI-code pairs. To test the generalizability of model, construct the training, development and the test with completely different commands.
Table 6: The statistics of the tldr shell scripting benchmark
# Commands
NL-Bash pairs
train 1315
6414
dev
376
1845
test
188
928
total
1879
9187
Documentation pool D We take the bash manual of the 1897 bash commands in tldr to construct documentation pool. We search each command name at manned. ore pages (the same the Unix 'man <command> command), and then extract the text contents from the returned manual page. We further break each manual into multiple paragraphs by line breaks so that each paragraph delicately describes a single concept such as a command functionality or a flag usage. We make this decision due to the large volume of content each manual has, which is too long to fit the length limitation of neural model, and too noisy and distracts the model with irrelevant information. This results in 400k individual entries in the pool in total.
[ website which archives Unix manual
Oracle manual D: We find the ground truth documentation for each (n,c) pair through command and flag matching heuristics. For instance, given code snippet toilet "input text' -f font_fi Lename' we constrain our search the documentation from toilet manual page and select documentation that starts with -f flag as an oracle paragraph. Along with the first paragraph that commonly summarizes a command, these paragraphs forms D;,-
Evaluation metrics We use four evaluation metrics to measure the quality of the generated code: (a) com-
mand name accuracy (CMD Acc) measures whether the command name (e.g.. cat) is predicted correctly; (b) token-level F1 converts the reference code and the generated code to bag of words and measures the token-level precision, recall, and Fl overlap: (c) exact match (EM)- - measures the exact match between the reference and the generation; and (d) character-level BLEU (charBI.EU) [in et al. 2018 Shi et al. 2022). .. For token level F1, exact match, and charBLEU, we disregard all user-specilic variables in the references and the system outputs. For example, "mycli -u [user] -h [host] ] [database |" is converted into "myeli $1 -h $2 $3". This is mainly because the variables
not instantiated tidz and the style of the placeholder varies among contributors. For example, some contributors might write [user] as rnse] ] or [your_name]. Therefore, measuring the surface form of user-specific variable names is less meaningful.
B RE-SPLITTING CONALA
NL-Python pairs We adapt the popular CoNaLa benchmark and re-split the dataset to test the generaliza- tion scenario. This re-split makes every example in the development and the test have at least one Python function (e.g., plt. . plot) that was not seen the training data. There are 2135, 201, and 543 examples in the training, development and test sets, respectively, We follow the original work Yin et al. (2018) to evaluate the system outputs with BLEU-4, Since we focus on the generalization setting, we additionally report unseen function accuracy, which measures the percentage of correctly predieted held-out out functions that do not appear in the training
work|Yin
hhttps://manedd.org
13

---

Published as a conference paper at ICLR 2023
Human-annotated unit tests Following Chen al. 2021 and Austin et al. (2021). we conduct execution- based evaluation on CoNaLa to mcasure the functional correctness the generated code. We randomly selected 100 examples from the test set and manually annotated unit test for each example, For example, we wrote tests such as assert gen_code ("abeds", 2) == 4 and assert gen.code ( ("abde", 2) == -1 to verify whether the function gen code could perform "find the index of sub string 's' in string 'str' starting done by authors of the paper who program with Python daily. On average, we annotate 2.03 unit tests for each example.
from index 2". Each example was annotated by single annotator. The annotation
Documentation pool D Our documentation pool contains 35763 manuals. These functions are from all Python librarics that are available on DevDoca" These libraries contains the Python built-in library, and popular librarics like numpy and pandas. The documentation on DevDocs are curated and further transformed and indexed to allow for quick searching of APIs. We then extract cach API signature and the corresponding documentation in every library, remove any content in the documentation that is not text, and segment the documentation into multiple paragraphs based the <p> HTML tags. The documentation pool then contains pairs of the API signature and a single paragraph in the correspondigg documentation. Although the documentation pool is not comprehensive to cover all Python libraries and functions, we find it has a high coverage rate on the CoNaLa dataset. This choice reflects the flexibility of our approach upon the characteristics of target scenario.
Oracle manual D To find the oracle documents (n,c) c) example, we first index the function names with absolute path (e.g. plot is indexed with matplot pypplot plot) with Elasticsearch. Then we query the search engine with clean version of where variable name removed. The top-5 functions after de-duplication treated oracle manuals D.
given NL intent DJ from the original
Natural language and code associations during pretraining Despite our efforts, it is possible that some of the held-out functions in the test retriever and a generator. Since the generators were initialized from the same checkpoint both the baselines and DocPrompting models, such a possible association is expected to equally help both models. In the retriever, such possible association did not cause the retriever to see the exact NL intents together with the corresponding documentation, and thus the matching between NL was not leaked. However, it is possible that there had been semantically similar intents seen along with the code snippets of the held-out functions. Nevertheless, such co-occurrence "indirect" and "unsupervised".
seen associate with NL contexts (e.g., comments) during
the pretraining
C DENSE RETRIEVER TRAINING
We finetune the model for 10 epochs with batch size of 512 and learning rate of le - 5. Since CodeT5 does not [CLS] token, alternatvelyy take the average of the hidden state of the last layer as the text representation. For CoNaLa, we also use the first 100k "mined" examples provided as part of CoNaLa as supervised corpus. For CoNalia, we only apply a single search step because each code snippet commonly contains more than one function. Wc also observed that using the first sentence that normally summarizes the usage of function achieve the best retrieval performance than other alternatives such as using the first paragraph, or simply truncating to the maximum token length. The training takes up 15 hours single A6000 GPU.
D GENERATOR TRAINING
We train our single-source generators for 20 epochs with learning rate 4e 5. We train our FiD-based generators for 10000 steps. The doc length is set to 200, any further content will be truncated. We follow (Izacard and Grave 202T) to learning rate to 5e-5 5 with 2000 steps warmup and linear learning rate decay. The batch size is set to 8. The best model is selected based on the token-level score on the development set for t.lrr and BLEU score for CoNaLa. The training takes 8 hours on a single A6000 GPU.
E CODEX PROMPTS
For the baseline, prompt Codex with three NL-code pairs and append the test query the end. An example on :idr is shown on top of Table On bottom, we list the prompt with DocPrompting where documentation is provided along too. In the oracle command name setting, we prepend the command name before each NL
https:://evddae
14

---

Published as a conference paper at ICLR 2023
CoNaLa
Retrieval Recal @
70
36.2
Generation BLEU
60
35.9
60.4
35.6
50
40
30
20
Recall
BLEU
3
5
10
15
20
25
30
Retrieved Does
Figure 5: The recall@k (%) and the corresponding BLEU score by using these top-k docs on
CoNaTa dataset (using Code
intent for the baseline prompt. For DocPrompting prompt, we replace the potential does with the retrieved does from the oracle manual.
F ADDITIONAL ANALYSIS
Parameter efficiency As shown in Table under given parameter budget. we find that Dompromn mostly benefits from parallel encoding (FiD). For example, the parallel encoding T5+DocPrompting (220M parameters) significantly outperforms scaling up Goo+DocPromppe
125M parameters joint encoding Neo- 225M+DocPrompting Only to 1.3B parameters manages to match the 220M parameter 15+DocPrompting A possible explanation is that although the base Nco-1.3B (without DocPrompting generally performs better than the base T5 (without DocPrompttee since documents encoded independently the encoder side.
parallel encoding allows to utilize the retrieved documents better,
The impact of the number of documents Figure shows recall and the BLEU score compared to k, the number of retrieved documents. Increasing & consistently yields a higher recall; however, as more irrelevant documents are retrieved, the generator cannot effectively distinguish them from the relevant ones and the overall performance remain similar. For example, CodeT5 achieves the highest BLEU score using 5 < 10. In contrast, when the generator provided with the oracle docs only, its BLEU score reaches 49.04 (Table 3). This suggests that both precision and recall docs are important, and the henefit of using larger values in open domain QA [Izacard and Grave 202T does not necessarily hold in code generation.
Full n-gram overlap Table shows that using documentation significantly increases the n-gram overlap recall between the input and the output, in tidr and CoNaLa. Since used BM25 to retrieve does in tidr, the MLe~Retrievee docs overlap high by construction. CoNaLa, the NLe-Retrieved does unigram overlap high as well, but since we used a dense retriever, the general n-gram overlap does not have to be high for DocPrompting to work well.
Retrieval latency Although retrieving does results in additional test-time computation, the increase in latency not prohibitive. First, encoding the input for the retrieval step "costs" single forward pass through the retriever's encoder, which is significantly less expensive than generation (which requires multiple time steps of the decoder). All the documentation in the retrieval pool can be encoded in advance, and finding the top-k results can be performed quickly using libraries such as FAISS Johnson et al. (2019) on the GPU or ScaNN Guo al. (2020) CPU. The cost of this top-k search is sub-linear the size of the document pool. Second. the additional imput the generator results an increased memory consumption, but only a small increase latency since the tokens a given input can encoded in parallel. this difference is crucial in practical settings, we can decrease the number of retrieved documents. Figure 5 shows that retrieving as few as five docs may be sufficient in many cases.
G FULL PASS @ k PLOTS
In the main execution- based evaluation, pass @ results in Section 5.2 and Figure 3 took the best temperature for every model and value of k. Here, show all the @ le plots with different temperatures in Figure 6
15

---

Published as a conference paper at ICLR 2023
get the label fat32 partition fatlabel /dev/sdal
# END
display information without including the login, jepu and pepu columns
W -soot
END
file column 9 csvsort -c 9 data. csv
END
# search for package in your current sources
Potential document O: fatlabel will display or change the volume label or volume ID on the MS- DOS filesystem located on DEVICE...
get the label fat32 partition fatlabel /dev/sdal
END
Potential document w displays information about the users currently on the machine, and their processes. The header shows, in this order
Potential document 1: -s, -hort Use the short format. Don't print the login time, JCPU or PCPU times.
display information without including the login, jepu and pepu columns # END
--short
Potential document O: Sort CSV files. Like Unix "sort" command, for tabular data
Potential document I: usage: csvsort [-h] [-d DELIMITERJ [-t] l-q QUOTECHAR] [-u 0.1,2.3) [-b] [-p ESCAPECHAR)...
Potential document 2: optional arguments: -h, -hel show this help message and exit -n, names Display column names and indices from the input CSV and exit. -c COLUMNS
Potential document 3: csvsort exampleifrealdan//YY0... EDU Recipients by State.csv
Potential document 4; esveut -c 1,9 exampls/iealldna//Y000 hwwp:///ww........eeeeeeenneeese Statecc.s -aite.ss--svvsee csvsort -r-c 2 head -n 5
file column 9 csvsort -c 9 data. csv END
Potential document 1:
Potential document 2:
search for a package your current sources
Table 7: Top: baseline Codex prompt with three NL-code pairs and a test intent. Bottom: DocPrompt- ing prompt for Codex. In each in-context learning example, the oracle does. the NL intent and the corresponding bash command are provided. We use up to five oracle docs for these examples. For a test example, the top-5 paragraphs from the retriever are represented with the NI. intent. The documents' contents were omitted ("...") to save space.
16

---

Published as a conference paper at ICLR 2023
Table n-gram overlap between different contents (%). Using documentation significantly increases the n-gram overlap recall between the input and the output, in t tldr and CoNaLa.
tldr
2
3
CoNaLa
2
3
5
NL Code
Code
30
14
11
7
(NL Aretrrieeedd docs)--Code
24
2
0
(NL+rctriee docs)==Code
91
52
28
16
11
NL Retrieved docs
39
8
NL Retrieved does
72
14
3
1
temperature-DD. =0.2
temperature-00.4
0.18
Code15
+0xcPomping
0.175
0305004
0.150
0.10
0.103
0.075
Codets
+uuchomating
0.059
50
100
125
150
175
25
75
100
125
150
175
230
temperature-0.6
temperatune-O.s
0.33
Codd75
Code75
1Dcchamp0ng
:Dochomping
0.30
0.25
0.20
00000
0.10
0.13
temperature-10
0.35
0.30
0.25
$4508k
0.20
125
175
Figure 6: Pass on 100 examples on the test set with different temperatures.
17

---

Published as a conference paper at ICLR 2023
Table Results on tlär and CoNaLa with code-davinci-002.
tldr
Model
CMD Ace (%)
(%)
Token
charBLEU
Codex 3-shots
39.01
14.55
44.89
33,93
+DocPrompting
36.10
13.97
42.55
32.93
With the oracle command name
20.22 33.15
59.22
38.14
Doopprong
68.59
44.76
CoNa la
BLEU
Recall
48.39
43.35
+ Propriomg
47.21
44.70
DocPrimpinng oracle docs
54.67
59.68
H EXPERIMENTS WITH code-davinci-002
The results with code-davinci-002 under few-shot learning setting is shown in Table 9
the non-oracle settings, Modex+DocPMnne Prompting did not improve over the base Codex; one explanation might be that the datasets leaked into the training corpus of the Codex. For example, CoNala included in the large CommonCrawe corpus that was used to train GPT-3, and possibly Codex. Therefore, Codex might have memorized the target code, and thus did not need the additional documentation. Although the data leakage issue might have happened in code uade-davinci-001 stronger 002 version. Regardless, we believe that large capacity of Codex requires an equally strong retriever to improve over the base model. With an oracle retriever, DocPrromping yields significant improvement on both datasets. Thus, the non-oracle results could be further improved using stronger non -rrccll retriever.
extracted from StackOverflow, which
well, suspect that this issue has worsened in the
I EXAMPLES
1.1
top three cases, baseline could not generate correct bash command while T5+DocPromptingg retrieves the correct bash documentation and use the correct arguments. In the bottom two cases, T5 also predicts the bash command correctly. However, in the puppet case, it misses it halluciate the argument flags example and -rggumentss which lixirr does not support at On other hand, T5+DocPrompting does ground its generation to the retrieved does and therefore it generates more accurate code.
Examples on tldr are in Table 10
concrete usages of executing the code. In the elixir
I.2 CoNALA
More examples on CoNaLa are shown Table 11
httppss//cooooorralorrrro hwwp://wwwoooe/ee/oe/eeeeeeeeeee data/
18

---

Published as a conference paper at ICLR 2023
Table 10: Examples of predictions from t.ldr, of the base T5 compared to T3+DocPromme Dropproopp.ngg.... In tldr, the bash commands in the test set are completely different from those in the training set.
play midi file specific key (0= majorla minor, majorld minor, +1 major'e minor, etc.) Ground truth:
timidity force-keysig-' -flats +sharps' 'path/to/ file.mid' midiplay 'key'
TS:
T5+DocPrompling
timidity -eysigg 'ElEIE' 'path/toofiilemidd:
clean up all temporary tex files in the current directory
Ground truth:
latexmk tex clean latexmk -c
IS+DocPrompting
specify the hash algorithm for the index file (experimental)
Ground truth:
git show-index - sha256' 'path/to/file' hash e=exprimeentl 'path/to/indes http///edex.hhhhs hashes git show-index --object- -ormatt- /pahh/oo/las--algorttt
T5:
T5+Doc Prompting:
pptthtto/experiientall
execute puppet cnde Ground truth:
puppet apply - execute 'code' puppet
IS:
TS+DocP 1S+DocPrompting rompting:
puppet apply -xexcute 'path/too/file.pppaa
evaluate elixir code by passing as an argument
Ground truth:
elixir -e 'code'
T5:
elixir --example --arguments 'path/to/file.elixxt
DocDocP Prompling:
elixir path/to/fill.eliix
Table 1I: Examples predictions from CoNaLa, of base CodeT5 compared to CodeT5+DocPrompphe Unseen functions are underscored.
set the current working directory to c: :e:eussessuuuae (nseerss bpythnn
Ground truth:
os chdir e: (Users ( (uname desktop (python') (dstktolpythhon os system (' c: (Users (Users\unamee (naae op desktopppyython' (python')
CodeT5:
CodeTT5-DDocPrompiing os. chdin c: (Userss uname desktop (ytthnn)
convert dataframe 'df" to integer-type sparse object Ground truth:
of.to.sparse_(1 np ip.isisstane (df, np. np. intege)) to.roes i'))
CodeT5:
SaalTT--DocPrompinnng
19


Reviews:
[{"review_id": "yDIFtSzPJ5G", "reviewer": "Reviewer_6cUe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an approach which uses documentation to generate code conditioned on a natural-language (NL) intent. The approach is essentially a standard retrieval-conditioned generation strategy, but where the set of documents being retrieved are constructed based on documentation.\n\nThe paper describes a set of practical instantiations of this technique, exploring a variety of retrieval models (sparse heuristic-based retrievers, and \"dense\" learned retrieval models trained to recover \"correct\" docs), and a few variants of generator architectures based on transformers, with either fine-tuning or few-shot prompting. The paper also introduces two datasets: a new benchmark for NL->bash prediction based on the `tldr` docs, and a variant of the CoNaLa benchmark with a different test split combined with a new reference corpus of documentation based on `devdocs.io`.\n\nThe authors show that retrieving documentation improves performance across all model types, and also study the effects of retrieving documentation vs code, providing some ground-truth knowledge, and using different retriever strategies.", "summary_of_the_review": "*Original review:* Using documentation as additional context for generating code is a sensible idea, and it is good to have a thorough empirical confirmation that this does in fact yield higher accuracy in the NL -> code setting. Thus, although the novelty of the technique itself is limited, and the findings aren't particularly surprising, I tend to lean toward acceptance (assuming the issues I mention regarding strength of claims, clarity, dataset leakage, and n-gram overlap can be resolved).\n\n*Updated review:* The authors have resolved my concerns, and upon reflection I do think that the problem formulation is interesting regardless of the novelty in the technique itself. I have increased my score from 6 to 8.", "strength_and_weaknesses": "Strengths:\n\n- Using documentation as an extra input to generating code is a good idea for ensuring models generalize to new types of functions, and this paper confirms that this generalization is indeed enabled by this approach.\n- The authors compare their approach with a variety of alternatives and find a consistent improvement across experiment settings.\n- The new datasets/benchmarks are interesting and seem like a reasonable way to evaluate generalization to unseen types of function.\n\nWeaknesses:\n\n- ~~The novelty is primarily empirical in nature, and~~ the technical approach is essentially an application of known techniques. *(update: After reflection and discussion with the authors, I admit that there is novelty in the problem formulation as well, which goes beyond the setting considered by previous work.)*\n- ~~The introduction makes bold claims about \"all existing code generation models\" which I do not think are justified.~~ *(update: The unsupported claims have been qualified in the new revision.)*\n- ~~The notion of \"ground truth docs\" is a bit confusing.~~ *(update: The paper has been reworded to make this more clear.)*\n- It's possible there is some function-knowledge leakage due to the use of pretrained code models for retrieval (perhaps the authors can clarify this) *(update: I am still curious whether this influences the results, but it is a bit orthogonal to the main point of the paper, since it would also affect baselines. The authors have added a brief discussion of this potential leakage to the appendix.)*\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "njHPw3iKxt1", "reviewer": "Reviewer_145E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose DocPrompting for the problem of generating code from natural language instructions. DocPrompting supplies the model with in-context documentation and uses retrieval to find relevant documentation for a given input. The authors experiment with using BM25 and training dense encoders for retrieving documentation, and find consistent improvements with DocPrompting. The authors focus on evaluation settings where functions in the test set are not seen during training, and also introduce the new code generation dataset “tldr”.", "summary_of_the_review": "This paper proposes the simple and intuitive idea of incorporating documentation into code generation tasks and finds that it brings empirical benefits, especially when generalizing to unseen functions. The paper is well executed and will likely lead to future work.", "strength_and_weaknesses": "Strengths:\n* The paper has a clear motivation and is easy to follow.\n* The idea of document prompting is tested across a range of models and retrievers with promising results. The authors also compare DocPrompting to a baseline with in-context examples (ExPrompting) instead of in-context documentation (DocPrompting).\n* The n-gram analysis provides a simple account of how DocPrompting may bring benefits.\n\nWeaknesses:\n* I am unsure how fair the evaluation setting with unseen functions is for the baseline models, especially since the ExPrompting baseline cannot retrieve examples of unseen functions from the training set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jKAKfiCKDd", "reviewer": "Reviewer_k6bL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the first approach for retrieving code documentation for code generation as a way of addressing the fact that APIs are constantly changing. They present a general strategy entailing a retriever (sparse-BM25 or dense-large pretrained model) which first retrieves a set of relevant documents from a pool of documents (which does not need to be fixed). Then, the retrieved documents and concatenated with the NL intent to form the prompt that is fed into a generation model for code generation. They evaluate a number of different generation models, including T5, CodeT5, GPT-Neo, and Codex. For evaluation, they curate two different datasets, by ensuring disjoint training/validation/test splits, spanning two different programming languages: bash commands (tldr) and Python (re-splitting CoNaLa). Their results show that in general, their DocPrompting approach tends to achieve superior performance over simply prompting with the NL intent alone. \n\nContributions:\n- First paper to consider retrieving documentation for code generation\n- Two curated datasets that can be useful for future work\n- Extensive empirical experiments and ablation studies demonstrating the utility of their approach\n", "summary_of_the_review": "While the technical novelty may be limited, there is novelty in terms of application and benchmark creation which will be useful for future work. The experiments and empirical results are thorough. Overall, I feel that this is a good paper contributing useful ideas and artifacts to the community.", "strength_and_weaknesses": "Strengths:\n- The motivation and approach is well-grounded\n- There are extensive results, spanning many different metrics. The Recall_unseen metric is particularly useful in demonstrating how much the prompting could help in a realistic scenario in which models are tested when new API changes are made on unseen functions.\n- The analysis in Section 6 is very thorough, providing possible explanations for a number of different observations that are made.\n\nWeaknesses:\n- Adding retrieval will likely increase latency. The authors do not discuss/report how this is affected. As the size of the document pool increases, it seems that model latency will also increase.\n- While the reason for splitting paragraphs into individual documents is explained in the Appendix, it seems odd to treat them as independent documents. For example, in Figure 2, the two bottom documents cannot standalone without the top one.\n- It is not clear how k is selected for the number of documents to retrieve. Is it always 3?\n- The annotation process for writing the test cases is not clear. In Appendix B, it is written that the two first authors did this. It is not clear whether each example was annotated by both of them and if so, what the agreement on that was.\n- Missing reference: https://arxiv.org/pdf/1808.09588.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ng06Gpx119-", "reviewer": "Reviewer_xC2U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper observes that existing models can only generate seen libraries and function calls at test time. However, new functions and libraries are introduced all the time, and even a seen function call can have arguments that were not used in the training data. To address such unseen functions in code generation, the authors propose a simple and effective approach for code generation by retrieving the relevant documentation, dubbed DocPrompting. The extensive experiments demonstrate that DocPrompting consistently improves NL->code models in two tasks, in two PLs, and across multiple strong base models. DocPrompting improves strong base models such as CodeT5 by 2.85% in pass@1 (52% relative gain) in execution-based evaluation on the popular Python CoNaLa benchmark; on a new Bash dataset tldr, DocPrompting improves CodeT5 and GPT-Neo-1.3B by up to 6.9% exact match, and Codex by 6.78 charBLEU score.", "summary_of_the_review": "In my opinion, the proposed DocPrompting method is interesting and novel, and the experimental results successfully demonstrated the effectiveness of the proposed method. Considering the technical simplicity and reproducibility, I give a weak accept.", "strength_and_weaknesses": "Strengths:\n1. The idea of leveraging the newly added manuals and documentation to generate code containing unseen and unused functions and libraries is interesting and novel. The work demonstrates an exciting endeavor direction to generate high-quality code.\n2. The main results from Tables 1 & 3 demonstrate that the improvement of the proposed DocPrompting is significant across various pre-trained models. Conducting experiments on more than one benchmark and thorough ablations make the conclusion of this work convincing.\n3. The presentation of this work is excellent. The rich use of diagrams and tables makes this paper easy to understand.\n\nWeaknesses: \n1. Although the usage of documentation for code generation is novel, it is somewhat simple from a technical point of view.\n2. I examined the supplementary material provided by the authors and only found the data files. It would be better if there were codes to prove its reproducibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%