# Evaluation of a System for Generating Programming Problems Using Form Services

Takumi Daimon, Kensuke Onishi

► To cite this version:

Takumi Daimon, Kensuke Onishi. Evaluation of a System for Generating Programming Problems Using Form Services. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.491-503, 10.1007/978-3-031-43393-1_45. hal-05533348

HAL Id: hal-05533348

https://inria.hal.science/hal-05533348v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Evaluation of a System for Generating Programming Problems Using Form Services

Takumi Daimon<fnref n="1" /> and Kensuke Onishi<fnref n="2" />

**Abstract.** Programming must be evaluated as a subject when taught in high schools. As a result, increasing the efficiency of question generation and grading has become a pressing concern. Waquema is a system for automatically generating and grading programming problems that we created. Waquema is available in the following two versions: web ( generates web pages) and cloud (generates Google Forms). At our university, we used the questions generated by Waquema’s cloud version for exercises and exams in lectures. The system was then evaluated using a student questionnaire survey. We also conducted a questionnaire survey of the teachers who used the system and evaluated the time required for the exercises and examinations. The results showed that the difficulty level of the generated problems was appropriate for the students, and that the system could reduce the time required to prepare for the exercises.

**Keywords:** Programming education · problem generation · automatic grading

## 1 Introduction

In Japan, with the revision of the Courses of Study, the previous information will be reorganized, and “Information I” and “Information II” will be taught as high school subjects beginning in 2022. Students will learn programming in the “Computers and Programming” unit of “Information I,” a required subject. Programming must be evaluated before when it can be taught as a subject. Therefore, programming exercises and examinations are required to assess students’ understanding.

Teachers can conduct written exams or use learning management systems when developing exercises and exams. Exams that can be administered on a computer can be created with a learning management system. Written exams have the advantage of being administered without the use of a computer; however, they necessitate teachers spending time for not only preparing the questions

<footnote n="1">¹ Graduate School of Science, Tokai University, 4-1-1, Kitakanme, Hiratsuka, Kanagawa 259-1292, Japan
daimon@fuji.tokai-u.jp</footnote>

<footnote n="2">² Tokai University, 4-1-1, Kitakanme, Hiratsuka, Kanagawa 259-1292, Japan
onishi@tokai-u.jp</footnote>
to be asked but also grading them. Improving the efficiency of question preparation and grading is an urgent issue to promote the reform of teachers' working styles.

The online judging system has the advantage of making it simple to judge the correctness of a question by submitting a program. Conversely, majority of the questions in the online judging system necessitate the implementation of a program whose output matches the example answers. It is preferable for beginners and novices who are trying to improve their understanding to have a variety of randomly generated problems based on the topic being studied. Several systems, primarily in the Java language, automatically generate and grade programming problems. Systems created by university students and teachers [1] are mostly used in universities, but there are no systems that generate randomized problems based on the unit being studied.

Waquema [2] is a system that automatically generates and grades programming problems in Python, JavaScript, and C++, which are educational programming languages for high school Information I and II. Waquema is available in the following two versions: web (generates web pages) and cloud (generates Google Forms). The development and evaluation of the cloud version are described in this paper. The following is the structure of this paper: Sect. 2 introduces related research in this field, Sect. 3 provides an overview of the system, Sect. 4 describes the questionnaire items that the system can generate, and Sect. 5 describes the details of the questionnaire items and their data. The questionnaire results are discussed in Sect. 6, and Sect. 7 is a summary of this paper.

# 2 Related Research

## 2.1 Automatic Generation of Programming Problems

Kitamura et al. [3] proposed a method for automatically generating programming problems by analyzing and processing example answers with user-defined tags. They demonstrated that this method can be used in conjunction with an existing learning system, MAX/JAVA [1].

Uchida [4] developed “JavaDrill,” a system for automatically generating fill-in-the-blank problems in the Java language. This system generates fill-in-the-blank problems by inputting the problem number, problem name, problem text, and source code, and setting blanks for reserved words and operators. JavaDrill is designed for programming beginners, and aims to improve their programming skills by teaching them the grammar and reserved words of the Java language.

Ariyasu et al. [5] created a system that automatically generates fill-in-the-blank questions and their answers in C language. This system accepts as input the program code of the answer and data describing the tendency of the question. The system can place blanks according to the intention of the author by inputting variables, type declarations, functions, and conditional judgments into the question trends. This system aims to improve the efficiency of the process of creating a large number of exercises.
Takeuchi et al. [6] developed a system that generates fill-in-the-blank prob-
lems in the Haskell language. This system can automatically generate fill-in-
the-blank questions and determine their correctness. They have also proposed
an effective method for generating fill-in-the-blank questions that considers the
significance of programming elements, such as functions and recursive structures.

## 2.2 Research on Automatic Grading of Programming Problems

Kogure et al. [7] created a monitoring system for learning programming. This
system allows you to track the types of tasks that your students are working on.
Furthermore, the system automatically analyzes the learner's program so that
the teacher can provide appropriate instructions to the learner.

Ito et al. [8] developed a plug-in for judging the correctness of assignment
programs running on Moodle. In this system, a server for judging correct and
incorrect answers is prepared separately from the server on which Moodle runs.
The system then checks whether there is a difference between the correct answer
program prepared on that server and the result of running the assignment pro-
gram, and returns the result to Moodle. This allows the user to immediately
know if the submitted assignment program is correct or not.

Kitaya et al. [9] developed a Web-based system for submitting, grading, and
returning the results of Java programming assignments. The system evaluates
the correctness of the programs using compiler checks, JUnit tests, and output
result tests, and it is distinguished by its extremely high grading accuracy.

Saikkonen et al. [10] developed Scheme-robo, an evaluation system for Scheme
programming exercises that evaluate the procedure, structure, execution time
and so on. Scheme-robo examines programs from an algorithmic standpoint and
determines their correctness.

Aizu Online Judge [11] is a popular online judging system. Aizu Online Judge
is public online judging system developed at the University of Aizu. Users can
solve problems, submit their source code, and be judged in Aizu Online Judge.

# 3 Programming Problem Generation System: Waquema

## 3.1 Overview

Cloud Waquema's website is depicted in Fig. 1. The generated problems in the
cloud version can be exported to the teacher's Google Drive as Google Forms. To
create a question, click "Generate." To run the program created in the editor,
click "Run." Clicking "Settings" brings up the settings screen (Fig. 2), where
you can choose the language to be used and the questions to be generated.

Figure 3 shows how this system outputs the questions as a Google Form. The
answers and scores are contained in the output Google Form, and the user can
edit the questions on the Form and process the results using Google Spreadsheet.
Supported programming languages are JavaScript, Python, and C++.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Programming problem generation system.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Configuration.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Problem outputted as Google Form.</figcaption></figure>

## 3.2 Generable Problems

**Questions that Answer the Output Results:** Figure 4 depicts a question that answers the output result. This problem requires the user to select one appropriate output result from the options obtained by running the program in the problem statement. It is generated only if the output instruction is included in the input source code. If the output result can be interpreted as a numerical value, then the value closest to the value is selected. If the output result is separated by commas, then choices with different comma separators can be generated as shown in Fig. 4. Otherwise, a choice consisting of a substring of the output result will be generated.
**Question to Answer the Value of a Variable:** Figure 5 shows a question to determine the value of a variable. In this question, you must select one of the options in the question text as the value of the variable at the specified location in the program.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Questions that answer the output results.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5. Question to answer the value of variable.</figcaption></figure>

**Questions on the Types of Variables:** Figure 6 shows a question that asks about the types of variables. This question requires you to choose one of the options as the appropriate variable type at the indicated location in the program in the question text.

**Error Finding Problem:** Figure 7 shows an error finding problem. This problem is to choose one line number from the program in the question text that contains a spelling error. The misspelling is associated with one of the reserved words chosen at random from the input source code. The spelling error is assigned to a randomly selected reserved word from the input source code, and the question is generated only if the reserved word is present in the input source code.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Questions on the types of variables.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 7. Error finding problem.</figcaption></figure>
**Sorting Problem:** Figure 8 shows a sorting issue. The goal of this problem is to select all the program fragments that do not cause errors when shuffled by separating them by one line each and sorting them in the order of the alternatives.

**Reserved Word Selection Problem:** Figure 9 depicts the problem of reserved word selection. The goal of this problem is to find all the reserved words that do not cause errors when entered into the blank set for the program in the problem statement.

<figure><img src="image_10.png" /><figcaption>Fig. 8. Sorting problem.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 9. Reserved word selection problem.</figcaption></figure>

# 4 Evaluation Experiment

## 4.1 Evaluation by Students

A questionnaire was distributed to the students to evaluate the exercises and tests developed using the system's generated problems. Students were only asked to complete the questionnaire if they agreed to the data usage policy, which stated that the collected data would only be used for system improvement and paper writing. As a result, 61 of the 75 students who attended the lecture responded. Items and results are shown in Table 1. Table 1 displays the items and results of the 4-point scale, with 4 representing the highest and 1 representing the lowest. The average in Table 1 was calculated by multiplying each rating by the number of respondents in Table 3, then dividing the total by the number of respondents (N = 61). The same questionnaire was also distributed for the same lecture given in the fall semester of 2021. The questionnaire included a 4-point evaluation and a free writing section. Sixty-three students responded out of the 94 who attended the lecture. Table 2 summarizes the items and results of the questionnaire.
<table><caption>**Table 1.** Assessment items and ratings on the problem (2020, N = 61).</caption><thead><tr><th rowspan="2">Item Number</th><th rowspan="2">Item</th><th rowspan="2">Average</th><th colspan="4">Value / %</th></tr><tr><th>4</th><th>3</th><th>2</th><th>1</th></tr></thead><tbody><tr><td>1</td><td>Was the difficulty of the question appropriate?</td><td>3.27</td><td>37.0</td><td>54.1</td><td>6.6</td><td>1.6</td></tr><tr><td>2</td><td>Were the instructions in the question easy to understand?</td><td>3.34</td><td>42.6</td><td>49.2</td><td>8.2</td><td>0.0</td></tr></tbody></table>

<table><caption>**Table 2.** Assessment items and ratings on the problem (2021, N = 63).</caption><thead><tr><th rowspan="2">Item Number</th><th rowspan="2">Item</th><th rowspan="2">Average</th><th colspan="4">Value / %</th></tr><tr><th>4</th><th>3</th><th>2</th><th>1</th></tr></thead><tbody><tr><td>1</td><td>Was the difficulty of the question appropriate?</td><td>3.12</td><td>25.3</td><td>65.0</td><td>6.3</td><td>3.1</td></tr><tr><td>2</td><td>Were the instructions in the question easy to understand?</td><td>3.31</td><td>42.8</td><td>49.2</td><td>4.7</td><td>3.1</td></tr></tbody></table>

<table><caption>**Table 3.** Evaluation value of Table 1 and Table 2.</caption><thead><tr><th>Item</th><th>Value</th></tr></thead><tbody><tr><td>Very true</td><td>4</td></tr><tr><td>Fit the bill a little</td><td>3</td></tr><tr><td>Not very applicable</td><td>2</td></tr><tr><td>Not applicable at all</td><td>1</td></tr></tbody></table>

## 4.2 Checking Comprehension Using the Output of the System

We used the questions generated by this system in JavaScript lectures for second-year university students to conduct for comprehension checks from November 12 to December 3, 2020. Table 5 displays the results of the comprehension checks for each lecture. The comprehension check was graded on a four-point scale to determine the level of understanding of the lecture material. Students could submit their answers as many times as they wanted. Figure 10 depicts the distribution of scores at the time of the first submission, while Fig. 11 depicts the distribution of scores at the time of the final submission for students who submitted more than once. Figure 12 also displays the frequency with which each response is submitted.

## 4.3 Evaluation of the System by Teachers

A questionnaire for teachers was used to assess the system. It was explicitly stated that the data gathered would only be used for the dissertation and not for any other purpose. Respondents were only asked to answer the questionnaire if and only if when they agreed with the data usage policy. As a result, we
<table><caption>**Table 4.** Multiple-choice survey questions and results($N = 61$).</caption><thead><tr><th colspan="2">How did you feel after answering the questions?</th></tr><tr><th>Item</th><th>Percentage</th></tr></thead><tbody><tr><td>I was able to monitor my understanding</td><td>73.8%</td></tr><tr><td>I found it difficult</td><td>31.1%</td></tr><tr><td>I sensed that the teacher had hand-crafted the problem</td><td>14.8%</td></tr><tr><td>I felt that the questions presented were not suitable for review</td><td>6.6%</td></tr></tbody></table>

<table><caption>**Table 5.** Results of the comprehension check for each lecture.</caption><thead><tr><th>Item</th><th>11/12</th><th>11/19</th><th>11/26</th><th>12/3</th></tr></thead><tbody><tr><td>Average score (maximum 4 points)</td><td>2.68</td><td>2.39</td><td>2.16</td><td>2.84</td></tr><tr><td>Average number of answers</td><td>2.22</td><td>1.41</td><td>1.46</td><td>1.38</td></tr><tr><td>Correlation between score at first answer and number of answers</td><td>-0.13</td><td>-0.38</td><td>-0.37</td><td>-0.26</td></tr></tbody></table>

<figure><img src="image_12.png" /><figcaption>**Fig. 10.** Distribution of scores when answering for the first time.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 11.** Distribution of scores at the time of the final answer for students who answered the questionnaire more than once.</figcaption></figure>

were able to obtain questionnaires from five teachers, one for the JavaScript version, three for the Python version, and one for the C++ version of the system, respectively. The questionnaire for the teachers included four options, a multiple-choice question, and a question asking them to describe any other problems or opinions they would like to see resolved. Table 9 shows the results of the multiple-choice questions. Table 6 shows the results of the four-choice questions, and Table 7 shows the choices for the questions in Table 6.

# 5 Analysis of Evaluation Data

## 5.1 Analysis of Students Evaluation

Tables 1, 2, and 4 are discussed further below. Item 1 in Table 1 had an average of 3.27. In Table 2, the average for Item 1 was 3.12. More than 90% students
<figure><img src="image_14.png" /><figcaption>**Fig. 12.** Distribution of the number of times students answered the questionnaire.</figcaption></figure>

<table><caption>**Table 6.** Evaluation by teachers. ($N = 1,3$ and $1$ for JavaScript, Python, and C++, respectively).</caption><thead><tr><th rowspan="2">Item No.</th><th rowspan="2">Item</th><th colspan="3">JavaScript</th><th colspan="3">Python</th><th colspan="3">C++</th></tr><tr><th>A</th><th>B</th><th>C</th><th>D</th><th>A</th><th>B</th><th>C</th><th>D</th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><td>1</td><td>How easy was it to use the cloud version of Waquema?</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>2</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>2</td><td>On an average, how long did it take you to complete the creation of an exercise from the source code?</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>3</td><td>On an average, how long did it take you to complete an exercise creation from the system's output?</td><td>0</td><td>0</td><td>1</td><td>0</td><td>2</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>4</td><td>What percentage of the questions in the system's output were appropriate for practice?</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>3</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>5</td><td>On an average, how long did it take you to complete the creation of tests from the source code?</td><td>0</td><td>0</td><td>0</td><td>1</td><td>2</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>6</td><td>On an average, how long did it take you to complete the creation of tests from the system's output?</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>7</td><td>What percentage of the problems in the system's output were appropriate for testing?</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>1</td><td>0</td></tr></tbody></table>

indicated that the difficulty level of the problems generated by the system was appropriate, in both questionnaires. Furthermore, Table 4 shows that more than 70% of the students indicated that they were able to use the system to check their understanding. These findings imply that the system-generated questions
<table><caption>**Table 7.** Question items of Table 6.</caption><thead><tr><th>Item No.</th><th>A</th><th>B</th><th>C</th><th>D</th></tr></thead><tbody><tr><td>1</td><td>I think so very much.</td><td>I think so.</td><td>I don't think so.</td><td>I don't think so at all</td></tr><tr><td>2, 3, 5, 6</td><td>Less than<br>10 min</td><td>10 to 20<br>minutes</td><td>20 to 30<br>minutes</td><td>More than<br>30 min</td></tr><tr><td>4, 7</td><td>80% or more</td><td>60% to 80%</td><td>40% to 60%</td><td>Less than 40%</td></tr></tbody></table>

<table><caption>**Table 8.** Change in time to generate problems using Waquema.</caption><thead><tr><th></th><th>Teacher 1</th><th>Teacher 2</th><th>Teacher 3</th><th>Teacher 4</th><th>Teacher 5</th></tr></thead><tbody><tr><th>Language</th><td>JavaScript</td><td>C++</td><td>Python</td><td>Python</td><td>Python</td></tr><tr><th>Exercises</th><td>Faster</td><td>Later</td><td>Faster</td><td>Faster</td><td>Same</td></tr><tr><th>Tests</th><td>Same</td><td>Faster</td><td>Later</td><td>Same</td><td>Same</td></tr></tbody></table>

<table><caption>**Table 9.** Multiple-choice questionnaire on the issue.</caption><caption>Which types of questions did you want to use?</caption><thead><tr><th rowspan="2">Item</th><th colspan="2">JavaScript</th><th colspan="2">Python</th><th colspan="2">C++</th></tr><tr><th>Excurs.</th><th>Tests</th><th>Excurs.</th><th>Tests</th><th>Excurs.</th><th>Tests</th></tr></thead><tbody><tr><td>1. Questions that answer the output results</td><td>1</td><td>1</td><td>3</td><td>3</td><td>1</td><td>1</td></tr><tr><td>2. Question to answer the value of a variable</td><td>1</td><td>1</td><td>3</td><td>3</td><td>1</td><td>1</td></tr><tr><td>3. Questions on the types of variables</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td></tr><tr><td>4. Sorting problem</td><td>0</td><td>0</td><td>2</td><td>2</td><td>0</td><td>0</td></tr><tr><td>5. Error finding problem</td><td>1</td><td>0</td><td>2</td><td>1</td><td>0</td><td>0</td></tr><tr><td>6. Reserved word selection problem</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

were appropriate for assessing the students' level of comprehension. This implies that the system-generated questions, as well as the teacher's question selection, were effective. In Table 1, Item 2 had a weighted average of 3.34. More than 90% students said “I think so very much.” or “I think so.”, with no one saying “I don't think so at all.” In Table 2, Item 2 had an average of 3.31. More than 90% students said “I think so very much.” or “I think so.” while the number of students who said “I don't think so at all” increased from 2020. The percentage of students who said “I don't think so.” decreased. Five students had difficulty understanding the meaning of the questions in both questionnaires, but the majority of the other students were able to understand and answer them (Table 8).

## 5.2 Analysis of Confirmation of Understanding

Table 5 will be discussed. The score of the first answer and the number of answers had a consistent negative correlation (−0.38 to −0.13). This indicates that students who received low scores on their first attempt answered the questions mul-
tiple times. The scores of the last answer are concentrated at 4 points in Fig. 11. It can be seen that several students who answered the question more than twice, did so until they received four points. Figure 10 shows the number of students who scored 4 points on the first try as the number of lectures increased. The gradual increase in the number of students who could get full marks in a single attempt can be attributed to students understanding the patterns and methods of answering the questions and becoming accustomed to checking the lecture content. In the first lecture (11/12), the number of answers was distributed over a wide range (Fig. 12). This may be due to the fact that the students were not yet accustomed to checking the lecture contents. A student submitted the questionnaire six times when checking the lecture content in one session (Fig. 12).

## 5.3 Analysis of Teachers Evaluation

Table 6 will be discussed. For Item 1, one respondent answered “I think so very much”, and four answered “I think so.” The simple operation method, in which problems are generated by entering the source code and pressing a button, received high marks. There were opinions in the free description column, such as “the policy that problems cannot be generated without entering the program is a problem” and “it would be great if there were templates.” As a result, we believe that the difficulty in operating this system stems from the process of writing a program in the editor and testing its functionality. We further discuss about Items 2 and 3 (creating practice problems). The four teachers spent the same amount or less time on writing exercises, while one teacher spent more time. Using this system, however, all of the teachers completed the exercises less than 30 min. We can observe that the time required to create the problems has been reduced, as has the burden placed on the teachers. Concerning the percentage of questions that can be used for practice (Item 4), four teachers said it ranged from 40% to 60%, while one said it ranged from 60% to 80%. Based on these results, we can conclude that our system is capable of generating questions suitable for exercises, although the selection of questions is necessary.

Items 5 and 6 will be discussed in the following section. One teacher needed less time to create test questions, while three teachers needed the same amount of time. Using this system, it took more than 30 min for two teachers and less than 30 min for three teachers to create test questions. It is clear from the preceding data that the time required to create test questions has not changed considerably, nor has the burden on teachers. Concerning the percentage of questions that can be used for the exam (Item 7), one teacher responded “60% - 80%,” two responded “40% - 60%,” and two responded “less than 40%.” Based on these findings, we can conclude that this system’s generation of test questions has room for improvement. Furthermore, some of the system’s questions were of a type that we had not previously considered. For instance, in a question about reserved words, multiple choices were the correct answer (*let* and *var* in JavaScript). As a result, we can expand the types of questions that can be asked in a problem by searching for appropriate questions in the system output.
We will further discuss Table 9 (Questions to be used for exercises and tests). First, let us discuss the exercise questions. All the teachers said they would use Items 1 and 2. Three faculty members said they would use Item 5 if it were available. Less than two teachers indicated an interest in using the other items. Nobody expressed an interest in using Item 6. The output results of the program, problems related to variable values, and problems finding errors in reserved words are the items that we would like to use as practice problems from the list above. Following that, we will discuss the exam questions. For Items 1 and 2, all the teachers indicated an interest in using the software. Only two or fewer teachers indicated an interest in using the remaining items. No one expressed an interest in using Item 6. Based on the above findings, we would like to use this as a practice exercise for questions about program output and variable values. Because questions about output results and variables values are important factors in assessing students’ understanding of programming structures and algorithms, it is safe to assume that all the teachers indicated a desire to use it.

# 6 Conclusion

The questions generated automatically by the proposed system, Waquema, were used to assess students’ comprehension during actual lectures. Following that, we distributed a survey to the students to assess the system. We also distributed a questionnaire to the teachers who used the system and assessed the amount of time required for the exercises and examinations. The results of the student questionnaire revealed that the difficulty level of the system-generated questions as well as the instructions in the questions were appropriate. Based on the results of the questionnaire sent to the teachers who used the system, it was discovered that using the system could reduce the time required to prepare the exercises. The proposed system would not be able to reduce the amount of time needed to prepare examination questions. We hope to use Waquema at more sites in the future to collect knowledge and data with the intent to make the system more efficient in preparing examination questions.

# References

1. Yamashita, M.: Construction of max/java, a web-based java programming learning support system. In: Bachelor Thesis of the Meiji University Department of Computer Science (2011). in Japanese

2. Daimon, T., Onishi, K.: Development of a system for generating programming problems in a web browser. In: IPSJ Symposium on Information Education, pp. 75–80 (2020). in Japanese

3. Kitamura, K., Tamaki, H.: Automatic generation of programming problem content from tagged sample programs. In: IPSJ Technical Report CE(122), pp. 1–8 (2013). in Japanese

4. Uchida, Y.: An automatic drill-producing system for elementary programming learning. In: IPSJ Technical Report CE(92), pp. 109–113 (2007). in Japanese
5. Ariyasu, K., Ikeda, E., Okamoto, T., Kunishima, T., Yokota, K.: Automatic generation of fill-in-the-blank exercises in adaptive c language learning system. In: DEIM Forum 2009, pp. 1–5 (2009). in Japanese

6. Takeuchi, R., Ohkubo, H., Kasuya, H., Yamamoto, S.: An learning support environment for Haskell programming by automatic generation of cloze question. In: IPSJ Technical Report SE(171), pp. 1–8 (2011). in Japanese

7. Kogure, S., Nakamura, R., Makino, K., Yamashita, K., Konishi, T., Itoh, Y.: Monitoring system for the effective instruction based on the semiautomatic evaluation of programs during programming classroom lectures. Res. Pract. Technol. Enhanced Learn. 10, 1–12 (2005)

8. Itou, K., Mima, Y., Ohnishi, A.: A linkage mechanism between course-management-system and coursework-checking-functions over web services. IPSJ J. 52(12), 3121–3134 (2011). in Japanese

9. Kitaya, H., Inoue, U.: An online automated scoring system for java programming assignments. Int. J. Inf. Edu. Technol. 6, 275–279 (2016)

10. Saikkonen, R., Malmi, L., Korhonen: Fully automatic assessment of programming exercises. In: ACM Sigcse Bulletin, pp. 133–136 (2001)

11. AIZU ONLINE JUDGE. https://judge.u-aizu.ac.jp/onlinejudge/. Accessed 24 Jan 2022