SlowGuess/NAACL_2022_OCR huggingface dataset:
Changyoon Lee, Yeon Seonwoo, Alice Oh
School of Computing, KAIST
changyoon.lee@kaist.ac.kr, yeon.seonwoo@kaist.ac.kr
We introduce CS1QA, a dataset for code-based question answering in the programming education domain. CS1QA consists of 9,237 question-answer pairs gathered from chat logs in an introductory programming class using Python, and 17,698 unannotated chat data with code<sup>1</sup>. Each question is accompanied with the student's code, and the portion of the code relevant to answering the question. We carefully design the annotation process to construct CS1QA, and analyze the collected dataset in detail. The tasks for CS1QA are to predict the question type, the relevant code snippet given the question and the code and retrieving an answer from the annotated corpus. Results for the experiments on several baseline models are reported and thoroughly analyzed. The tasks for CS1QA challenge models to understand both the code and natural language. This unique dataset can be used as a benchmark for source code comprehension and question answering in the educational setting.
Question answering (QA) studies systems that understand questions and the relevant context to provide answers. Question forms include single document QA (Rajpurkar et al., 2016), multi-hop QA (Yang et al., 2018), conversational QA (Reddy et al., 2019), and open domain QA (Kwiatkowski et al., 2019). Questions about specific domains are asked in NewsQA (Trischler et al., 2016) and TechQA (Castelli et al., 2020), and images are provided with the question in visual QA (Antol et al., 2015). Another interesting field of QA asks questions about source code (Liu and Wan, 2021).
A useful application of QA is in the educational domain. Asking questions and getting the answer is an essential and efficient means of learning. In this paper, we focus on QA for programming education,
Figure 1: An example of our data tuple. Each data tuple consists of {question, answer, question type, code, relevant code lines}. We annotate the type of each question and the code lines (orange) relevant to the question.
where both the input modes and the domain pose interesting challenges. Answering these questions requires reading and understanding both source code and natural language questions. In addition, students' questions are often complex, demanding thorough understanding of the context such as the intention and the educational goal to answer them.
Recently, models that understand programming languages (PL) have been studied, and show promising results in diverse code comprehension tasks (Alon et al., 2018; Feng et al., 2020; Guo et al., 2021). However, these models have limitations to support question answering. They are not trained on datasets containing questions about the code and are not designed for QA tasks. Also, many assume fully functional code as input, while students' code contains diverse syntax and logical errors and is often incomplete.
To address this issue, we introduce CS1QA, a new dataset with tasks for code-based question answering in programming education. Questions and answers about programming are collected from the
1The code and the data used in this paper can be found at https://github.com/cyoon47/CS1QA.
Human Language Technologies, pages 2026 - 2040
naturally occurring chat messages between students and TAs. The question type and the code snippet relevant to answering the question are also collected. The final CS1QA dataset consists of question, question type, answer, and code annotated with relevant lines. The data is collected mostly in Korean and then machine-translated into English and quality-checked for easy application on models pretrained in English. Figure 1 shows an example of our data. We also include two-semesters' worth of TA-student chat log data consisting of 17,698 chat sessions and the corresponding code.
We design three tasks for the CS1QA dataset. Type classification task asks the model to predict the question type. Code line selection task asks the model to select lines of code that are relevant to answering the given question. Answer retrieval task finds a similar question already answered, and uses its answer as the answer to the given question. The outputs for these tasks can help the students debug their code and the TAs spend less time and effort when answering the students' questions.
Finally, we implement and test baseline models, RoBERTa (Liu et al., 2019), CodeBERT (Feng et al., 2020) and XLM-RoBERTa (Conneau et al., 2020), on the type classification and code line selection tasks. The finetuned models achieve accuracies up to  $76.65\%$  for the type classification task. The relatively low F1 scores of  $57.57\%$  for the line selection task suggest that the task is challenging for current language models. We use DPR (Karpukhin et al., 2020) to retrieve the most similar question and its answer. We compare the retrieved answer with the gold label answer, and achieve a BLEU-1 score of 13.07, which shows incompetent performance of answer retrieval on CS1QA dataset. We show with a qualitative evaluation the model behavior with different inputs for the first two tasks. Our contributions are as follows:
- We present CS1QA, a dataset containing 9,237 question-answer-code triples from a programming course, annotated with question types and relevant code lines. The dataset's contribution includes student-TA chat logs in a live classroom.
- We introduce three tasks, question type classification, code line selection and answer retrieval, that require models to comprehend the text and provide useful output for TAs and students when answering questions.
- We present the results of baseline models on the tasks. Models find the tasks in CS1QA challenging, and have much room for improvement in performance.
Code-based Datasets Recently, research dealing with large amounts of source code data has gained attention. Often, the source code data is collected ad hoc for the purpose of the research (Allamanis et al., 2018; Brockschmidt et al., 2018; Clement et al., 2020). Several datasets have been released to aid research in source code comprehension, and avoid repeated crawling and processing of source code data. These datasets serve as benchmarks for different tasks that test the ability to understand code. Such datasets include: ETH Py150 corpus (Raychev et al., 2016), CodeNN (Iyer et al., 2016), CodeSearchNet (Husain et al., 2020) and CodeQA (Liu and Wan, 2021). We compare these datasets with CS1QA in Table 1.
In an educational setting, students' code presents different characteristics from code in these datasets: 1) students' code is often incomplete, 2) there are many errors in the code, 3) students' code is generally longer than code used in existing datasets, and 4) questions and answers from students and TAs provide important additional information. In CS1QA, we present a dataset more suited for the programming education context.
Source Code Comprehension In the domain of machine learning and software engineering, understanding and representing source code using neural networks has become an important approach. Different approaches make use of different characteristics present in programming languages. One such characteristic is the rich syntactic information found in the source code's abstract syntax tree (AST). Code2seq (Alon et al., 2018) passes paths in the AST through an encoder-decoder network to represent code. The graph structure of AST has been exploited in other research for source code representation on downstream tasks such as variable misuse detection, code generation, natural language code search and program repair (Allamanis et al., 2018; Brockschmidt et al., 2018; Guo et al., 2021; Yasunaga and Liang, 2020). Source code text itself is used in models such as CodeBERT (Feng et al., 2020), CuBERT (Kanade et al., 2020) and DeepFix (Gupta et al., 2017) for use in tasks such
Dataset	Programming Language	Data Format	Dataset Size	Data Source
ETH Py150	Python	Parsing AST	7.4M files	GitHub
CodeNN	C#, SQL	Title, question, answer	~187,000 pairs	StackOverflow
CodeSearchNet	Go, Java, JavaScript, PHP, Python, Ruby	Comment, code	~2M pairs	GitHub
CodeQA	Java, Python	Question, answer, code	~190,000 pairs	GitHub
CS1QA	Python	Chat log, question, answer, type, code	9,237 pairs	Real-world classroom
Table 1: Comparison between different code-based datasets and CS1QA.
as natural language code search, finding function-docstring mismatch and program repair.
The tasks that these methods are trained on target expert software engineers and programmers who can gain significant benefit with support by the model. On the other hand, students learning programming have different objectives and require fitting support by the models. Rather than getting an answer quickly, students seek to Students ask lots of questions while learning, and thus question answering for code is needed. CS1QA focuses on code-based question answering and can be used as training data and a benchmark for neural models in an education setting. The CS1QA data can also be used for other tasks than QA, such as program repair and code search.
3 CS1QA Dataset
The data for CS1QA is collected from an introductory programming course conducted online. Students complete lab sessions consisting of several programming tasks and students and TAs ask questions to each other using a synchronous chat feature. We make use of the chat logs as the source for the natural question and the corresponding answer. These chat logs are either in Korean or in English. The student's code history is also stored for each programming task for every keystroke the student makes. This allows us to extract the code status at the exact time the question is asked, which provides valuable context for the question. We take this code as the context for the given question. The thorough code history and the student-TA chat logs are a unique and important contribution of CS1QA. CS1QA also contributes with data from multiple students working on the same set of problems.
3.2 Question Type Categorization
Answering different types of questions requires understanding the different intentions and information - answering questions about errors requires identi
fying the erroneous code and answering questions about algorithms requires understanding the overall program flow. As the different question types affect the answering approach and location of code to look at, knowing them in advance can be beneficial in the QA and code selection tasks.
Allamanis and Sutton (2013) have categorized questions asking for help in coding on Stack Overflow into five types. We adapt these types to students' questions. In addition, we define the "Task" type that asks about the requirements of the task. TAs' question types are derived from the official instructions by the course instructors given in the beginning of the semester. TAs were instructed to ask questions that gauge students' understanding of their implementation, for example by asking the meaning of the code and reasoning behind the implementation. TAs' probing questions are categorized into five types: Comparison, Reasoning, Explanation, Meaning, and Guiding. Examples for the question types can be found in Table 2. We present intentions of the question types in Table 3.
3.3 Collecting Question-Answer Pairs with Question Types
We collected a total of 5,565 chat logs over the course of one semester from 474 students and 47 TAs. After removing the logs where the TA did not participate in the chat, 4,883 chat logs remained.
We employed crowdworkers with self-reported skill in Python of three or higher on a 5-point Likert Scale to collect the questions. Each worker first selected messages in the chat log corresponding to the question and the answer, then selected the question type. Workers were provided with descriptions of the question types with examples before working on the task. Workers were asked to divide the message into individual questions when there were multiple questions or answers in the message. They were instructed to only choose programming related questions, for which the answer is obvious in the chat from the question alone. This ensures that the questions and answers are independent from the
Question Type	Allamanis' Type	Question	Answer
Code Understanding	How/why something works	Why is the print cards function at the bottom of the check function? Can I not have it?	This is because if two cards match through a check, you have to show them two cards.
Logical Error	Do not work	Now, the file is created, but when I go inside and look at the value, it seems to be a little different from the one requested in the problem.	You seem to have forgotten the line break in the middle I think you can add \n
Error	Do not work	I don't know where the task 2 error came from...	When creating image There is a negative number in the image size Please do something like absolute value
Function/Syntax Usage	Way of using	And I forgot how to make a blank image	Blank images can be created with new_img= create_picture(width,height)!
Algorithm	How to implement	So, what if there is any other way to count the number including the number not included in the randomly created list?	Parameter: a list which is returned from drawing_integers Count integers function is supposed to take that as input
Task	-	Isn't it a task in which the number of iteration steps changes according to the input value?	Yes, but the value of x must also change according to the input value!
Comparison	-	How was the method of reading and writing differ-ent in task1?	There was a difference between read mode and write mode, open(file name, r) and open (file name, w)
Reasoning	-	I've read the task1 code. What is the intention of using continue on line 55?	This is to go back to the beginning without execut- ing the next print function.
Explanation	-	How do you create new_img when 'horizontal' is input as Direction in Task2?	I did it like I did with vertical, but since the y value is changing, when I change the y value and run the loop, I did x first among x and y.
Meaning	-	Can you explain the role of the global keyword in Task 1?	If you use a variable specified only within a function, it cannot be used in other functions, so I used it for global variable processing!
Guiding	-	Is there a simpler way to change average_integers using a function already defined in the python list??	In average_integers, it would be more convenient to use the len function when counting the total number of elements.
Table 2: Examples of translated and untranslated question and answer texts for each question type in CS1QA. First column shows our type classification, and second column shows the classification by Allamanis and Sutton (2013). The first six rows in the top part are student question types, the last five rows in the bottom part are TA's probing question types.
chat history. Every chat log was annotated by two workers to ensure the quality of annotation. A total of 20,403 question-answer pairs were collected.
The question and answer texts are machine-translated using Google's Neural Machine Translation model (Wu et al., 2016) from Korean to English to form the dataset in two languages. The translation allows for easy integration of CS1QA data to models pretrained in English, which make up a huge portion of NLP models.
3.4 Selecting Code Lines
Providing relevant code snippets allows the answerer to identify the problem more quickly and easily. We annotate the lines of code that provide information necessary to answer the question for use in the code line selection task.
We collected code for all questions asked by students. For the TA probing questions, we collected code for all Reasoning and Meaning types, and 472 randomly selected Explanation questions, for a total of 4677 questions. This keeps a balance in the number of questions for each type. Comparison questions were left out as they require comparison of code across different tasks, making the annotation and the tasks too complicated. We exclude Guiding questions as answering them requires more
than just understanding code; the answer is often new algorithms not based on the current code.
We employed crowdworkers who have worked as TAs for the programming course to select the code that the questions refer to. We provided the workers with the collected questions, answers, and the student's code for each task at the time the question was asked. The workers selected the code file for the question and the relevant code lines to answer the question. When reading the code was not necessary to answer the question, the workers were asked to choose Not Applicable (NA) for the code selection. For every question, two workers made code annotations.
A total of 9,359 code selections were made by the workers. Some of the selections with empty code or incorrectly extracted code were removed from the dataset. The remaining 9,237 questions annotated with type, lab and task numbers, code, code lines and answer make up the final CS1QA dataset. Every code selection made by the workers is used as gold labels even if the two workers choose different lines. Thus, every question can have up to two correct code selections in different parts of code. An example of the data is found in Appendix A.
Q Type	Intention	#Q	#Code	NA (%)	Span (%)
S	Code Understanding	Understanding the functionality of the code	105	209	33.9	11.0
Logical Error	Investigating the cause of the unexpected outputs of the code	541	1060	16.7	21.6
Error	Resolving syntax errors and exceptions	488	959	10.1	13.0
Function/ Syntax Usage	Learning correct usage of a function or syntax	411	811	55.4	11.8
Algorithm	Learning the underlying algorithm for the task	603	1194	50.0	19.3
Task	Confirming the goal and requirement of the task	677	1322	82.3	15.3
T	Reasoning	Understanding the reasoning behind student's implementation	402	799	5.6	21.8
Explanation	Checking the validity of student's explanation of their code	472	940	2.6	42.8
Meaning	Checking the meaning of a function or a variable in the code	978	1943	2.1	24.2
Table 3: Types of questions asked by students (S) and TAs (T) in a programming class. Questions are categorized by different intention and information required to answer them. The number of questions and code snippets collected from the annotators, percentage of Not applicable code selections and selected code lines are reported.
3.5 Quality Control and Validation
As the workers worked independently, there were some differences in the annotated data even when they correspond to the same question. There were some questions that were selected by only one worker as well. These questions are further reviewed to ensure the quality of the collected dataset.
Out of 20,403 collected questions, 3,556 questions were selected by only one worker, and 4,787 pairs of questions had some differences between the workers' selections. The remaining questions had perfect agreement between the workers. The authors reviewed questions selected by only one worker, and those without perfect agreement. Unnecessary words present in only one text were removed and crucial words missing in the question were added to the text while preserving the meaning to make the two texts equal. The conflicts in question types were resolved with the authors' additional vote that made a clear majority in the type selection.
We calculate the inter-rater reliability score with Cohen's Kappa (Cohen, 1960) for the question type selection. The Kappa value is calculated between every pair of workers who selected the same question-answer pairs. The mean of the Kappa values is 0.657, which suggests substantial agreement for type classification between the annotators.
Out of 9,237 questions with code line selection, 2,197 pairs had perfect agreement (100% overlap), while 1,225 pairs had  $0\%$  overlap. We compute the mean line F1 as the measure for agreement of spans,
considering one annotator's span selection as the ground truth and the other annotator's selection as prediction. The resulting F1 score is 0.6482. The disagreements are largely due to selecting different but relevant code and selecting different amounts of surrounding context in the code.
We design three tasks for the CS1QA dataset that identify important information that leads to the answer.
The type classification task is to predict the question's type. We use nine types of questions that we categorized as the candidates for classification, each question belonging to a single type. We use the accuracy and macro F1 score as the measure of performance. The code line selection task is to select lines of code that give relevant information to answer the question. The code is a strong supporting context to answering the given question, and this task tests the model's ability to retrieve this critical information.
For the code line selection task, we use the Exact Match (EM) and line F1 score as the measure of performance, same as the metrics used for supporting fact selection task in HotpotQA (Yang et al., 2018). The EM score measures the proportion of selections that exactly match the ground truth. The line F1 score measures the average overlap between the selected lines and the ground truth selections. The score is computed by treating the selections as bags of lines and calculating their F1 with the annotated lines. These two tasks take as inputs the lab
and task numbers of the question, the questioner (student or TA), question and the code texts.
The answer retrieval task retrieves a similar question given an unseen question, and uses the retrieved question's answer as the answer to the unseen question. BLEU score is calculated between the retrieved answer and the gold label answer.
Answer generation task given the question and the code context is possible with CS1QA dataset. However, meaningfully generating the answer demands a model that understands long and erroneous code, and the natural language question. This poses a significant challenge, and we leave the generation task as future work.
644 out of 9,237 questions are originally asked in English, while the rest are asked in Korean. The CS1QA dataset is split into train, development and test sets in the ratio of 0.6, 0.2 and 0.2 respectively, keeping the ratio of question types in each set the same to ensure equal distribution in all three sets.
5.1 Text Lengths
Table 4 shows the statistics of question and answer token lengths, for data translated to English (EN) and the original (ORIG) data, and the number of lines of code.
Data	Min	Max	Mean	Median
Question	EN	1	119	15.7	13
ORIG	1	79	10.9	9
Answer	EN	1	272	27.2	22
ORIG	1	166	17.6	14
Code	-	1	655	76.0	52
Table 4: Statistics of question, answer lengths in tokens and code length in number of lines in CS1QA.
The lengths of questions and answers lie mostly between 10 to 30 tokens. The distributions show long tails for both questions and answers, but answers are more evenly distributed. The distribution of token lengths for questions and answers can be found in Appendix B.
The number of lines of code shows a peak between 12.5 and 50, as shown in Figure 2. Code snippets have a wider distribution in length. This can be the result of varying difficulties of tasks, with more difficult tasks requiring longer code snippets to solve. The number of lines of code in CS1QA
Figure 2: The total number of lines in code and the proportion of selected code lines in code (color coded). The last bin contains all code longer than 250 lines.
is larger than those in other code-based datasets, which can present interesting challenges.
5.2 Question Type Distribution
We present the distribution of question types in Table 3 in number of questions collected and number of code snippets collected.
The CS1QA dataset contains similar number of questions for each student type of questions, except for Code Understanding type, which contains significantly fewer questions. One plausible reason for this is that most of the tasks require writing the program from scratch, thus students ask fewer questions about the skeleton code.
There are more Meaning questions than other types of TAs' probing questions. This can be because TAs often ask the students about the meanings of functions and variables to make sure that the students understand the code they wrote for each task.
5.3 Code Line Distribution