Python Code Generation by Asking Clarification Questions
Year: 2023

Abstract:
Code generation from text requires understanding the user’s intent from a natural language description and generating an executable code snippet that satisfies this intent. While recent pretrained language models demonstrate remarkable performance for this task, these models fail when the given natural language description is under-specified. In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the underspecification of a natural language description can be resolved by asking clarification questions. Therefore, we collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers. The empirical results of our evaluation of pretrained language model performance on code generation show that clarifications result in more precisely generated code, as shown by the substantial improvement of model performance in all evaluation metrics. Alongside this, our task and dataset introduce new challenges to the community, including when and what clarification questions should be asked. Our code and dataset are available on GitHub.

heading: 1 Introduction
text: Text-to-code generation aims to understand a user’s intention represented by a natural language description (NLD) to generate a code that satisfies the user’s intention. Models for this task are a crucial component of digital pair-programmers, which assist data scientists (Agashe et al., 2019; Liu et al., 2021), software developers (Chen et al., 2021; Xu et al., 2022), and computer programming educators (Li et al., 2022).
Recent work addresses this task using pretrained language models (PLMs) fine-tuned on large-scale
∗ Work done while being a postdoc at UKP Lab. 1 https://github.com/UKPLab/codeclarqa
code data in general-purpose programming languages, such as Python and Java (Chen et al., 2021; Li et al., 2022; Nijkamp et al., 2022; Chowdhery et al., 2022; Xu et al., 2022; Lahiri et al., 2022).
Although these models are successful, they fail to resolve the case of an NLD that lacks enough specifications. Figure 1a depicts an example of under-specified NLD (shown in yellow). The problem of missing specifications in NLDs not only widely occurs in real-world use cases (Lahiri et al., 2022; Chaurasia and Mooney, 2017) but is also important for training text-to-code generation models. Although important, alleviating the underspecification of NLDs is challenging for two rea-
14287
sons. First, missing specifications can happen at various levels, including individual operations, argument values of the operations, and sub-tasks consisting of several operations decomposed from the entire source code file as the task. Second, it is not obvious how to identify if an NLD carries information about specifications at any level mentioned above or not.
In this paper, we introduce interactivity into textto-code generation for specifications on the level of individual operation calls in Python. We hypothesize that by gathering more specifications using these interactions, we can alleviate the incompleteness of NLD’s specifications and thus generate a more precise code (Figure 1a). To train and evaluate such models, we introduce the CodeClarQA dataset collected through a novel method to synthetically generate clarification questions and answers (CQAs) for an NLD-Code pair. To map the operations to natural language, we retrieve the API documentation. If there is a low similarity between the NLD and the operation’s documentation, we identify the operation as a missing specification and generate a clarification question (CQ). The answers to CQs are selected from the given code. Furthermore, we propose a pipeline to demonstrate the use case of our dataset for developing NLD-Code generation models. Our pipeline consists of three modules – a clarification need predictor, a CQ generator, and a code generator. For each module, we introduce models that can serve as baselines.
To evaluate the quality of our dataset, we conduct a human evaluation. We also evaluate the models we proposed for each component of our pipeline. Our empirical results show that by conditioning PLM-based code generation models on CQAs in our dataset, the performance of these models increases, indicating the correctness of our hypothesis and our collected dataset. Alongside,
our experimental results show that advanced PLMs (e.g., RoBERTa, BART, T5, and CodeT5) struggle to achieve high performance under the interactive code generation pipeline. This important observation demonstrates the difficulty of our dataset for the recent PLMs, introducing new challenges for these models.

heading: 2 Creating the CodeClarQA Dataset
text: We aim to make the process of code generation interactive such that by asking CQs about a given NLD, we resolve NLD under-specification before generating any code. To do so, we design a novel method to synthetically collect CQAs for a given NLD-Code pair, leading to the new dataset, which we call CodeClarQA. Figure 2 shows a general view of our data creation method.

heading: 2.1 Identifying Key Operations
text: Key operations correspond to sub-tasks decomposed from the code snippet as the task. For instance, the code in Figure 1 can be decomposed into three sub-tasks: call the logistic model, use grid search to fit different logistic models, and save the best model. The corresponding key operations are sklearn.LogisticRegression, sklearn.GridSearchCV, and joblib.dump. Ideally, an NLD should provide enough information about key operations in a code. If the NLD does not do so, it lacks sufficient specifications. Thus, our first step for generating CQAs for a given NLD-code pair is to identify the key operations required to generate the code from NLD.
To identify key operations, we represent the code as a graph. Data flow graphs are an effective structural representation of code semantics in a code. Given this fact, we use the graph defined by the GraphGen4Code toolkit (Abdelaziz et al., 2021), the state-of-the-art toolkit for generating a graph
from a source code, including API-related operations and the data flow. This makes it easy for us to identify key operations. Figure 1a shows the graph representation. Non-leaf nodes represent key operations. Edges indicate the data flow of the operations. For each NLD-Code pair, we parse the code to generate a graph.
Given a graph, we identify nodes with the following properties as key operations: (i) For operations that are one object/function and its methods/fields, we treat the object/function as a key operation. This is coherent with one hypothesis behind the design of GraphGen4Code, where an object/function is first initiated before fields/methods thereof are applied. For instance, sklearn.GridSearchCV is the key operation among all operations related to it, as other operations apply a method (.fit) or read a field (.best_estimator_) of it (Figure 1b). (ii) For a multiple-operation line of code, the last operation on the data flow path is a key operation. For instance, sklearn.GridSearchCV and numpy.linspace are in the same line. sklearn.GridSearchCV is a key operation since sklearn.GridSearchCV is the line’s highest-level operation (Figure 1b). See Appendix A for details of the procedure of identifying key operations.

heading: 2.2 Is a Key Operation Missing in NLD?
text: Given a set of key operations required to generate a code, we should identify if the given NLD provides any information about these operations. To do so, for each key operation, we propose to align the schema of textual documentation of a key operation with the schema of a given NLD. A schema (Majumder et al., 2021) is defined as a set of important elements of a document. Every schema element is either in the form of (verb, key-phrase, relation) or (key-phrase), where key-phrase is extracted using YAKE (Campos et al., 2020), and verb and relation are obtained by searching through the closest verb and its dependency relation using the dependency tree (Qi et al., 2020). An example of (verb, key-phrase, relation) is (transforms, final estimator, obl), and an example of (key-phrase) is (pipeline).
For each key operation required to generate a code, we compute similarity scores for all schema element tuples using elements from the NLD and the documentation. For each pair of schema elements, we use a pretrained text encoder (Reimers and Gurevych, 2019) to compute similarity scores
between these phrases as key information. Note that we combine verb and key-phrase if the schema element is in the triplet form before computing the similarity score. Eventually, we identify a key operation is missing in the NLD if the highest similarity score of all schema element pairs is lower than the threshold t. Each key operation is then labeled as aligned or missing. We perform a grid search to find the best t on a validation set, maximizing the F1 score. See Appendix B for an example.

heading: 2.3 Generating CQAs for Missing Key Operations
text: We formulate CQs as multiple-choice questions and yes/no questions. The former needs an answer with yes/no following a choice of an API call. The latter requires only an answer with yes/no.
Multiple-choice. We collect all extracted key operations from the dataset, mentioned or missing, that contain 1023 different API sub-modules, methods, and fields. We then extract the last tokens from each operation name, filter out all stop words, and keep operations that share the same last token of their names. For instance, sklearn.partial_fit and sklearn.fit share the same last token as fit. Note that we hypothesize that for operations with the same name but from a different library, e.g., keras.fit and sklearn.fit, they refer to the same operation. We generate multiple-choice questions for these key operations if they are missing. To do so, we use the template Do you want to call anything related to LAST_TOKEN? If yes, which one?.
Yes/No. For operations that do not belong to multiple-choice questions, we generate a yes/no question using the template Do you want to call OPERATION_NAME documented as DOC?. For instance, a CQ about numpy.logspace is generated as “Do you want to call numpy.logspace documented as Return numbers spaced evenly on a log scale?”

heading: 2.4 Dataset
text: We use NLD-Code pairs in the notebookCDG dataset (Liu et al., 2021) to create CQAs because of the high code quality ensured by votes of the Jupyter Notebooks and the high NLD quality ensured by the preprocessing method based on the study of markdown documentation (Wang et al., 2021a). We first identify key operations (§2.1) and label them as either aligned or missing (§2.2). Finally, we select NLD-Code pairs with at most five
missing key operations, duplicate missing key operations, and create CQAs (§2.3). Table 1 shows dataset statistics.

heading: 3 Pipeline for CQ-driven Code Generation
text: Our system generates precise code by asking CQs before generating. To do so, it uses an interactive code generation pipeline that includes three modules: (i) a clarification need predictor, (ii) a CQ ranker, and (iii) a code generator.
Given an NLD, the clarification need predictor predicts the need to ask CQs, with labels Need and No Need. If there is a need for asking CQs, the CQ ranker selects n CQs. We set n as five to push these models to choose CQs with the most information gains. Given the NLD, CQs and corresponding answers, the code generator generates a code.

heading: 4 Experiments
text: Having proposed our dataset (§2) and a pipeline for interactive code generation (§3), we next evaluate the quality of the dataset creation method by focusing on §2.2 and results of use our dataset to evaluate recent PLM-based models for each pipeline module for interactive code generation, before assessing the quality of the pipeline. The dataset evaluation analyzes the effectiveness of identifying key operations, while experiments on the pipeline aim to validate our hypothesis that interactiveness helps code generation and evaluate task difficulty.

heading: 4.1 Dataset Evaluation
text: To evaluate our dataset creation method, we randomly split our dataset into train/validation/test sets. We asked two Ph.D. students in computer science to annotate each NLD-Code pair in the validation and test sets. The annotation for each NLDCode pair is a binary label, indicating if the NLD
misses any key operation from the code. These annotations let us (i) study the properties of our dataset and (ii) evaluate the quality of our method for finding missing key operations using different text encoders. See Appendix §D for more details.
Setting. The validation and test set consist of 100 NLD-Code pairs respectively. The Fleiss Kappa is 0.74 (0.83 for the validation and 0.66 for the test set). We randomly chose one annotator’s annotation as reference labels. See Appendix §E for more analysis on annotation results.

heading: 4.2 Clarification Need Predictor
text: In order to label when CQs were needed, we learned a binary classifier. This classifier predicts, for an NLD, whether it needs further clarification. The classifier was trained on the NLD-Code pairs in the training portion of the CodeClarQA dataset.
Setting. We fine-tune baseline pretrained transformer classifiers, including BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and the encoder of BART (Lewis et al., 2020). To include models trained on NLD data, we also fine-tune the encoder of PLBART (Ahmad et al., 2021). Models are fine-tuned on the training set with NLDs as the input. We fine-tune each model for 10 epochs with learning rate 5×10−5 and pick the best-performing model on accuracy. We compare the models on the test set using accuracy, precision, recall, and F1.

heading: 4.3 CQ Ranker
text: Given an NLD, a CQ ranker should recommend potential key operations by asking CQs. We formulate this as a ranking task, where we select a subset of CQs from a universal set of CQs. We use all created CQs using our method mentioned in §2 as the universal set.
Setting. We follow Aliannejadi et al. (2021) and fine-tune cross-encoders on all NLD-CQ pairs and experiment with models used in §4.2. Given an NLD-CQ pair, each model is trained to do binary classification. At inference time, all CQs in the universal set are paired with a given NLD and ranked by model score. Given an NLD, positive samples CQs created in the dataset. To create negative samples, we experiment with random negative sampling and BM25 (Robertson et al., 1995). The number of negative samples selected is the average number of positive samples. Each model is trained for 10 epochs with learning rate 5 × 10−5.
We evaluate model performance with the test set on R@k, k ∈ {1, 3, 5, 10}.

heading: 4.4 Code Generator
text: The key hypothesis of our work is that interactive code generation systems outperform noninteractive ones. In this experiment, we conduct a proof-of-concept experiment to validate this hypothesis, assuming a perfect interactive system with perfectly asked CQs and answers. We finetune models with and without oracle CQAs from our dataset. Note that for both yes/no and multiplechoice questions, we have only positive answers in our dataset.
Setting. We experiment with models mentioned by Zhou et al. (2022) for fine-tuning, including GPT-Neo-{125M, 1.3B} (Black et al., 2021), T5 (Raffel et al., 2020), and CodeT5 (Wang et al., 2021b). We include CodeParrot-{110M,1.5B} (Tunstall et al., 2022). Note that for CodeParrot110M, we use the model fine-tuned on text-to-code generation.2 Moreover, we finetune PLBART-base (Ahmad et al., 2021). We train each model for 40 epochs with learning rate 5 × 10−5. Each experiment takes up to 6 hours on a single A100 GPU. We evaluate models on BLEU score (Papineni et al., 2002), CodeBLEU score (Ren et al., 2020), and Exact Match (EM). Note that we don’t include stateof-the-art execution-based metrics (Huang et al., 2022; Yin et al., 2022), since it requires us to include code context into the dataset, which leverages the difficulty of dataset construction. As we don’t
2 https://huggingface.co/datasets/codeparrot/
github-jupyter-parsed
include code context into the dataset, code predictions are more likely to fail on e.g. variable naming, which affects the execution results but does not necessarily lead to poor code quality.

heading: 4.5 End-to-end Pipeline Evaluation
text: To assess the performance of the entire pipeline (§3), we use the best-performing models for each module. We pass an NLD to the clarification need predictor. Given a positive prediction, we pass the NLD to the CQ ranker. For each NLD, we select the top-k (k ∈ {1, 3, 5}) ranked CQs by the CQ ranker. We compare them to CQs created using our approach and select overlapping CQs. Finally, we concatenate the NLD and all selected CQs with corresponding answers and feed them to the code generator.

heading: 5 Results

heading: 5.1 Dataset Evaluation
text: We first evaluate the effect of different text encoders on the performance of our method for identifying missing operations. Table 2 shows the results. We achieve the best performance using MPNetbaseqa-cos text encoder. We then use our annotations to analyze the predictions of this model. Table 3 shows the results of this analysis in terms of False Positive (FP) and False Negative (FN) errors. For the sake of brevity, we report the full list in Appendix §D.
The “Taxonomy” and “Element Pair” error types take up to 7.32% and 8.57% of all operations predicted as aligned in the validation/test sets, respectively.
The rare case of FP predictions suggests that our approach to generating CQAs effectively creates CQAs for missing key operations. The Taxonomy error relates the differences related to the taxonomy of terms that could not be identified, taking up to about 8.57%. The Element Pair error relates to
the cases where non-relevant schema elements are aligned, taking up to about 8.57%. The Argument error represents the alignment between arguments, taking up only 4.08%/4.35% of all negative predictions from the validation/test set. Table 4 shows examples of these errors.
For the taxonomy error, our method identifies a schema element match of linear models but fails to predict the difference between a lasso linear model and a linear model in the taxonomy of machine learning terms. This finding shows a potential direction of future work, in which aligned operations might require clarification to be distinguished from operations with similar names. The example of Argument error reflects the case where a complete semantics of the operation needs both the documentation and the argument values. As we proposed to compare documentation and the NLD, we miss out on arguments that can complement the semantics of the operation. The corresponding example shows that the operation .apply ’s semantics is incomplete without the argument str. This is reflected in the design of our method, as we use API documentation which reflects the semantics of the API call, while argument values are not documented.
The Element Pair error example shows that (make, index, obj) from the documentation’s schema is aligned with (index) from NLD’s schema.
In contrast, the key operation from the documentation should be either drop or deleted.

heading: 5.2 Clarification Need Predictor Evaluation
text: Table 5 summarizes the results of different classifiers. Most tested models obtain relatively high performances except for RoBERTabase, which overfits the imbalanced data where 63.71% samples have positive labels, as shown by the high recall but low precision. Moreover, BERTbase has the best performance on both accuracy and F1 score.

heading: 5.3 CQ Ranker Evaluation
text: We report the results of our experiments on CQ generation in Table 6. The results confirm that our design of selecting CQs is reasonable, with the best-performing model showing similar results to the “Question Relevance” task designed by Aliannejadi et al. (2021). However, we hypothesize that our task is more challenging, as the lexical overlap between the NLD and the correctly selected CQs is low due to our design of dataset creation
which looks for key operations with documentation that has no keyword matches to the NLD. This requires the model to utilize the under-specified NLD and infer the topic of the task and the user’s intent before providing suggestions by asking CQs.
Our hypothesis is strongly supported by the low recall of the BM25 ranker, which ranks CQs based on their lexical similarities with NLD. Moreover, we find that models trained with the BM25 negative sampler always perform lower than the ones trained with the random sampler, which also supports our hypothesis because the BM25 negative sample is expected not to select CQs that have high lexical overlap with the NLD as negative samples, while they have a higher chance of asking key operations that are “mentioned”.

heading: 5.4 Code Generator Evaluation
text: We train recent models using only the NLDCode pairs or with NLD, Code, and CQAs in the CodeClarQA dataset. The experimental setup aims to test our hypothesis that interactiveness helps code generation by running code generation models with “perfect” clarifications. Note that this only serves as proof of concept, as CQAs contain operation names in the target source code, leading to data leakage because the names of the API calls exist in the CQs.
Table 7 shows that all models fine-tuned with CQs have substantially better performance, with the largest gap of 14.28 in BLEU, 10.5 in CodeBLEU, and 6.03 in EM reached by PLBARTbase, which supports our hypothesis that interactions help code generation. Moreover, all models pre-
trained on code data have better performances, with CodeT5base and PLBARTbase as the bestperforming models we tested.

heading: 5.5 Pipeline Evaluation
text: We use BERTbase clarification need predictor, BARTbase CQ ranker with random negative sampling, and PLBARTbase trained with CQAs. Given the question ranker’s predictions, we select CQAs from the test sample with CQ included in the top-k (k ∈ {1, 3, 5}) list yielded by the CQ ranker. Besides concatenating selected CQs to NLDs, we also concatenate CQs without selecting them, treating them as “unanswered clarifications”.
We report the results of pipeline evaluation in Table 8. We find that model performances on all evaluation metrics substantially increased with more highly-ranked CQs being included and “answered” by comparing highly-ranked CQs and the CQAs in the dataset. Moreover, we also find the opposite trend for “un-answered clarifications” where models perform worse with more highly-ranked CQs included (but not “answered”). This aligns with the challenge of asking CQs mentioned in §5.3.
Last but not least, we compare the pipeline inference results in Table 8 to the results in Table 7. Notably, our pipeline underperforms models trained on data with only NLDs and code. This is expected, as we use code generators that are fine-tuned on all CQAs, and the results of ranking CQs suggest that the task of asking CQs is challenging (§5.3).

heading: 6 Analysis
text: Intuitively, asking CQs helps code generation because it provides more specifications, thus aligning model generations to desired and better-quality outputs. To test if this hypothesis stands under the context of our proposed task and pipeline, we analyze
model generations quantitatively and qualitatively.
Recall of identified missing key operations. Table 9 shows the recall of missing key operations from predictions. We find that training with clarifications includes substantially more missing key operations, while the pipeline still does not outperform models trained on data with only NLDs and code, similar to Table 8. Furthermore, we report Pearson correlation between the recall of missing key operations and code generation results (See Table 10), finding high and positive correlations which support our hypothesis that asking CQs helps code generation through clarified key operations.
Case study. We examine predictions and provide an example in Table 11. We find that training with oracle CQAs leads to predictions close to the ground truth, especially on operations, with only differences at argument-level specifications, which is expected as we focus on clarifications on operations. However, the task is challenging as the top 5 ranked CQs do not include CQs in the reference CQAs, leading to the pipeline prediction including a call of confusion matrix but missing AdaBoostClassifier and cross_val_predict.

heading: 7 Related Work
text: CQ generation. Aliannejadi et al. (2019, 2021) define CQs based on facets/aspects of the text input’s topic, guiding annotators to write CQs based on the facet information. Eberhart and McMillan (2022) ask CQs for query refinement based on facets/aspects from existing NLDs in a dataset. Our
work is distinguished from the above works as our method does not require a predefined collection of facets/aspects of the text inputs. The advantage of our method is that we collect NLDs as specifications from code.
More generally, two main focuses of work on CQ generation are (i) disambiguation of terms (Xu et al., 2019; Guo et al., 2021) and (ii) providing more information (Rao and Daumé III, 2018; Guo et al., 2021; Majumder et al., 2021; Nakano et al., 2022). With the goal of disambiguation of terms, Xu et al. (2019) utilize the knowledge base to create CQs that disambiguate different entities that share the same entity names. Guo et al. (2021) included CQs of coreference resolution that disambiguate pronouns. Rao and Daumé III (2018); Guo et al. (2021) define CQs to gather information missing from textual input. Majumder et al. (2021) ask CQs on missing information from the item description but existing in similar items, defined as missing schema. Nakano et al. (2022) construct pseudo-CQs by eliminating a part of a sentence and transforming it into a CQ and a corresponding answer. Our work adopts the definition of CQs as asking for new information and is distinguished from these works by defining a new type of information as key operations for a code, which are challenging to be defined and identified if they are included in the original text query.
Text-to-Code generation. Text-to-code generation was first defined through learning on the parallel corpus of NLD-Code pairs (Allamanis et al., 2015; Miceli Barone and Sennrich, 2017; Yin et al., 2018). To study programming in practice with dependency between different code snippets, Iyer et al. (2018) introduced a more challenging task that studies generation based on NLD and programming context. Agashe et al. (2019) address the task of generating code cells on Jupyter Notebook given previous markdown and code cells.
Our work also sources NL-Code pairs collected from Jupyter Notebooks (Liu et al., 2021). We do not consider dependency between different code/markdown cells when creating CQA, because including previous cells will change the necessity of asking some CQs and make our CQA creation method less controllable.
Recent research also focuses on generating code utilizing API knowledge or existing source code. Xu et al. (2020) augment data with samples created by documentation. Parvez et al. (2021) retrieve samples from the training set or an archived code database. Zhou et al. (2022) use retrievalaugmented generation approach by retrieving documentation from source code API usage. In contrast, we design the task of retrieving CQs and consider interactivity between the model and the user.

heading: 8 Conclusion and Future Work
text: In this paper, we introduced a new challenge of asking clarification questions for code generation for Python, along with a method to generate a dataset to create clarification questions and answers that do not require human annotations over the whole dataset. We release our collected dataset CodeClarQA, which consists of clarification questions and answers on API usage. We further proposed a pipeline system implemented by recent text
and code encoders to evaluate model performances on this challenge. Our experimental results confirm that clarification questions and answers are strong information-gathering methods for better generation of code while deciding when to ask clarification questions and what questions to ask remains challenging. Future works include improving clarification questions for higher user engagement and question diversity; studying the lack of user intent completeness beyond the level of operations, e.g., lack of user intent completeness in arguments; and introducing conversational relations between clarification questions.
Limitations
Our method primarily focuses on operation-level specifications, while there are real-world use cases with other specifications. Moreover, our method of creating CQAs can only be scaled to all Python codes that involve heavy API usage. However, if a similar code knowledge graph generator of another language is developed, our method can also be scaled to the corresponding language. Our method is also limited in identifying specifications missing from the NLD, suggesting potential future work to create CQs about specifications “mentioned but not specified enough” in the NLD.
Ethical Concerns
One concern about the data is the issue of copyright. Liu et al. (2021) have checked the data policy of all 20 Kaggle competitions, in which none has copyright issues. Furthermore, they have contacted Kaggle’s administrator and have made sure that the dataset collection procedure did not violate the platform’s policy. We also check the license of open-source APIs when collecting documentation and make sure that there is no concern about copyright issues. Another concern about the data is that it might include privacy data. Again, we think that our data has a minimum risk of leakage of data with privacy concerns since we only collect data from the 20 Kaggle competitions where there is no concern of privacy data. The API documentation also has the minimum risk of containing data with privacy concerns.

heading: Acknowledgements
text: We thank Xuye Liu and Dakuo Wang for providing the original dataset and source code for dataset preprocessing. We thank Nico Daheim, Ben Peters, Mert Tiftikci, Kexin Wang, Imbesat Hassan Rizvi, Dominic Petrak for their valuable feedback and suggestions on a draft of this paper. We thank the anonymous reviewers for their detailed and insightful comments.
This work has been funded by the LOEWE Distinguished Chair “Ubiquitous Knowledge Processing” (LOEWE initiative, Hesse, Germany), by EU’s Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), and by the Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020.

heading: A Procedure of Identifying Key Operation.
text: We present our procedure for identifying key operations in Algorithm 1 as a detailed description of §2.1. Given an NLD-Code pair and all source codes from its corresponding notebook, our method first extracts operations for the entire notebook and selects operations corresponding to the code from the NLD-Code pair. We then identify key operations by keeping (i) operations from the same API submodule that have the shortest data flow path and (ii) operations that correspond to the last operation within the same line. Note that we also filter out operations that (i) are print functions, (ii) are numerical operations, and (iii) have no corresponding documentation.

heading: B Preliminary Experiments on Identifying Missing Key Operations
text: We also considered code/documentation-trained models for computing similarities preliminarily. We experimented with RFLHU-BERTOverflow (Abdelaziz et al., 2022), which is trained on documentation-StackOverflowPosts pairs and performs similarly to the publicly unavailable RFLHUCodeBERT in Abdelaziz et al. (2022). We obtained 75.59, 57.14, 55.56, and 56.34 in accuracy, precision, recall, and F1. This is substantially lower than all the results from Table 2.

heading: C Example of Identifying if an Key Operation is Missing
text: We present an example of identifying if a key operation is missing figure 3. Given the key operations we have extracted (Figure 1b), we identify if a key operation is missing by comparing all its schema elements with schema elements of the NLD.

heading: D Examples of Error Types
text: We analyzed predictions of MPNetbaseqa-cos text encoder using our annotations. Table 12 shows examples of all types of FP and FN predictions we categorize. We also present in Table 13 the statistics of all FP and FN predictions.

heading: E Annotation
text: We asked two Ph.D. students to annotate 200 NLDCode pairs, respectively. It takes a volunteer about
Algorithm 1 Procedure of Extracting Key Operations 1: function EXTRACT KEY OPERATIONS AND DOCUMENTS(code_sample, code_notebook)
▷Step 1: Parse the code to build the graph and get operations for the notebook. 2: ops_notebook ← GraphGen4Code(code_notebook). ▷Step 2: Find all operations corresponding to code from the NLD-Code pair 3: ops_sample ← get_ops_sample(code_sample, code_notebook, ops_notebook) ▷Step 3: Pop operations of either import functions or numerical operations 4: for op ∈ ops_sample do 5: if type(ops_sample) ∈ {import_function, numerical_expression} then 6: ops_sample.pop(op) ▷Step 4: Keep operations that are the last call of the same line of code 7: for op ∈ ops_sample do 8: if op is not the last operation of the line then 9: ops_sample.pop(op)
▷Step 5: Pop operations of print functions 10: for op ∈ ops_sample do 11: if type(ops_sample) ≠ print_function then 12: ops_sample.pop(op) ▷Step 6: Keep only operations that do not have other operations in its data flow path 13: key_ops ← [] 14: for op ∈ ops_sample do 15: if {n ∣ n ∈ ops_sample & n ∈ op.path} = ∅ then 16: key_ops.append(op)
2 hours to annotate. We show the guide in figure 4 and an example of annotation figure 5.
Discrepancy of annotation between development and test set. We noticed the discrepancy of Fleiss Kappa between the development and test set. We then asked annotators to provide reasons for different annotations. As a result, subjectivity is the main reason for differences between annotations. An example is shown in figure 5, where fitting the model is not directly mentioned yet can be inferred from the NLD. We also find that the test set contains
more examples like this one, leading to a discrepancy of Fleiss Kappa between the development and the test set. We accept this difference as subjectivity is part of deciding whether an operation is mentioned.

heading: F Examples of CodeClarQA Dataset
text: We present examples from our dataset in Table 14.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: No section number, "limitations" section (page 8 and 9)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 5.1

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: No section number, "abstract" (page 1) and "introduction" (page 1,2)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2.1, 2.4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2.1, 2.4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 2.1, 2.4, "Ethical Concerns" section (page 9)

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 2.1, 2.4

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No section number, "Ethical Concerns" section (page 9)

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 2.1, 2.4, "Ethical Concerns" section (page 9)
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2.4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.1

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix D

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4.1

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 4.1
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. The only requirement for annotators is that they are experts in coding python.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? There is no need to do it. The only requirement is that they are experts in coding python.

heading: Future Work
text: Future works include improving clarification questions for higher user engagement and question diversity; studying the lack of user intent completeness beyond the level of operations, e. g. , lack of user intent completeness in arguments; and introducing conversational relations between clarification questions. Limitations
Our method primarily focuses on operation-level specifications, while there are real-world use cases with other specifications. Moreover, our method of creating CQAs can only be scaled to all Python codes that involve heavy API usage. However, if a similar code knowledge graph generator of another language is developed, our method can also be scaled to the corresponding language. Our method is also limited in identifying specifications missing from the NLD, suggesting potential future work to create CQs about specifications “mentioned but not specified enough” in the NLD. Ethical Concerns
One concern about the data is the issue of copyright. Liu et al. (2021) have checked the data policy of all 20 Kaggle competitions, in which none has copyright issues. Furthermore, they have contacted Kaggle’s administrator and have made sure that the dataset collection procedure did not violate the platform’s policy. We also check the license of open-source APIs when collecting documentation and make sure that there is no concern about copyright issues. Another concern about the data is that it might include privacy data. Again, we think that our data has a minimum risk of leakage of data with privacy concerns since we only collect data from the 20 Kaggle competitions where there is no concern of privacy data. The API documentation also has the minimum risk of containing data with privacy concerns