# Improvement of Fill-in-the-Blank Questions for Object-Oriented Programming Education

Miyuki Murata, Naoko Kato, Tetsuro Kakeshita

► To cite this version:

Miyuki Murata, Naoko Kato, Tetsuro Kakeshita. Improvement of Fill-in-the-Blank Questions for Object-Oriented Programming Education. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.530-541, 10.1007/978-3-031-43393-1_48. hal-05533291

HAL Id: hal-05533291

https://inria.hal.science/hal-05533291v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Improvement of Fill-in-the-Blank Questions for Object-Oriented Programming Education

Miyuki Murata<fnref n="1" />(✉) , Naoko Kato<fnref n="2" />(✉) , and Tetsuro Kakeshita<fnref n="3" />(✉)

**Abstract.** We have developed a programming education tool, named ‘pgtracer’, which provides fill-in-the-blank questions in C programming. Pgtracer provides programs and trace tables with blanks. A trace table represents the execution sequence of the target C program. As a result of our research, we could estimate the achievement level of the students and clarify the answering process by analyzing the logs collected by pgtracer. To improve software quality and reusability, object-oriented technology is important from various perspectives, and there is an urgent need to train engineers proficient in object-oriented programming. Thus, we are extending pgtracer to the Java program. Trace tables are extended to represent individual instances and message sending among the instances. In this paper, we create fill-in-the-blank questions for the Java program and attempt to have students solve some of the fill-in-the-blank questions using the “embedded answer (Cloze)” question format of Moodle’s question function. The results of the trial are discussed, and improvements are made to the questions and the user interface. We also introduce the notion of ignorable blanks which students do not need to fill. This type of blank is useful to increase the variety of questions.

**Keywords:** Learning Analytics (LA) · programming education · object-oriented programming · Java · fill-in-the-blank question

## 1 Introduction

The development of information technology in recent years has led to the provision of IT-based services via many application domains. As these services become more sophisticated and complex, object-oriented programming becomes crucial to improving the quality and efficiency of these services.

Although object-oriented programming education is provided at universities and institutes of technology to software engineering students, insufficient training time and lack of teaching staff hinder sufficient programming training. Considering this situation,

<footnote n="1">¹ National Institute of Technology, Kumamoto College, Koshi, Japan
m-murata@kumamoto-nct.ac.jp</footnote>

<footnote n="2">² National Institute of Technology, Ariake College, Omuta, Japan
naoko@ariake-nct.ac.jp</footnote>

<footnote n="3">³ Saga University, Saga, Japan
kake@is.saga-u.ac.jp</footnote>
we are developing pgtracer, a programming education support tool that provides fill-in-the-blank questions for Java programming. This project is an extension of pgtracer for C programming [1, 2]. Since the pgtracer log contains every activity of the students, we can obtain valuable knowledge of the student understanding level and characteristic process of the students by analyzing the logs collected by pgtracer using learning analytics (LA) methods [3, 4]. The knowledge obtained from these LA functions can be used to improve the effectiveness of programming education.

In this paper, we developed fill-in-the-blank questions for Java programs. We proposed fill-in-the-blank questions for pgtracer utilizing a Java program and trace tables representing the behavior of the Java program [5]. This is the advantage of our approach. A trace table corresponds to an instance and represents the values of the variables and outputs of the instance as well as the message sending between instances. It is necessary for students to correctly trace changes in variable values and messages sending to understand and write a program. We extended a trace table for the Java program to represent message sending. Furthermore, we introduced a special type of blank that students do not need to fill. This type of blank allows teachers to provide more flexibility in controlling the difficulty level of the questions while hiding hints.

In this paper, we perform a trial using our initial fill-in-the-blank questions in an actual class using a quiz, one of the Moodle activities, to evaluate the validity of the questions. We also analyze the results of the trials and improve the fill-in-the-blank questions and the user interface. These considerations will be useful for the implementation of pgtracer for Java programs.

This paper is organized as follows. In the next section, we describe related research on object-oriented programming education and learning analytics. We introduce pgtracer functions in Sect. 3. In Sects. 4 and 5, we explain fill-in-the-blank questions for a Java program and the creation of the fill-in-the-blank questions respectively. In Sect. 6, we describe the lecture and the trial, and in Sects. 6.2 and 6.3 we analyze the evaluation results. In Sect. 7, we discuss the improvement of the fill-in-the-blank questions and the user interface of pgtracer. We conclude this paper in the last section.

## 2 Related Works

Hsiao et al. proposed web-based parameterized questions for Java [6]. The questions examined the final value of a variable or predicted the text being printed. Pgtracer allows blanks to be set not only in the final value but also in the variable values during program execution. This is useful to facilitate students’ understanding of the program.

Truong et al. introduced a static analysis framework for a Java program [7]. The student answers the entire program for an assigned question. The system uses the model answers and predefined gaps for analysis. In pgtracer, the same kind of problem can be created by defining the entire program as a blank. Furthermore, by analyzing the answer logs of all students, pgtracer can discover trends in errors other than the expected ones.

Hauswirth et al. proposed an Informa clicker system to train Java programming [8]. Informa provides several types of questions related to program code such as multiple-choice questions but does not address the tracing of variables.

Funabiki’s research group proposed several programming education systems for Java that utilize fill-in-the-blank problems such as [9]. However, they do not address message
sending or variable values for each step of program execution. Our proposed question provides more flexibility because it can also define the blanks for these parts.

There have been studies on the use of LA in programming education. Fu et al. have developed a system that provides a learning dashboard by analyzing error message output at compilation for the C programming language [10]. Carter et al. estimate students’ abilities by comprehensively considering the results of analyzing log data in programming activities and students’ learning behaviors [11]. The log data used in these studies include programs submitted by students and error messages output at compilation, etc. Grover et al. used logs of submitted programs, students’ solution behavior, and operations (execution, insertion, editing, and deletion) for block-based programming to estimate computational thinking estimation [12]. Although research results in C language and block programming have been published, research on object-oriented programming languages, which is the subject of this paper, is new. Pgtracer logs every time each blank is filled in. This allows us to collect candidate answers not only the final answer but also the process leading up to the answer. This makes it possible to analyze the process of students’ answers.

## 3 Programming Education Support Tool Pgtracer Utilizing Fill-in-the-Blank Questions

Pgtracer for the C language provides fill-in-the-blank questions for a student and automatic evaluations of the student’s answer [1]. Pgtracer also automatically collects student logs, such as the student’s answer, the correct answer, and the required time, immediately after filling in each blank. Pgtracer provides analysis functions for the collected log from various viewpoints such as analysis functions of each student, each question, each blank, and the detailed learning process of a student [2]. The analysis functions are useful for analyzing student achievement and learning processes. Figure 1 shows the programming education process utilizing pgtracer.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Programming Education Process utilizing pgtracer.</figcaption></figure>

## 4 Fill-in-the-Blank Questions of the Java Program

### 4.1 Blanks Within a Program

In this section, we describe the fill-in-the-blank questions of the Java program. Figure 2 shows an example.
Since the high-level component of a Java program is a class, there is one source file for each class. Thus, the fill-in-the-blank question of a Java program generally contains multiple source files. We also assign a step number to each statement, including the instance variable and local variable definitions, because they may contain initialization statements. Step numbers are assigned according to the following rules and correspond to the step numbers in the trace table described in Sect. 4.2.

* Assign a sequential step number to each statement defined in the Java program, such as assignment statement, method call, and control statement, such as if, else, switch, case, default, and return as well as an instance variable definition.

* The step number starts from 1 in each method definition.

* Assign step numbers such as “x.y” to compound statements having nested structures.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Example of a Program with Blanks (Template Method, Main Class)</figcaption></figure>

A token is a string that cannot be further subdivided, such as a variable name, class name, operator, or keyword. A comment is defined as a token. Pgtracer allows defining of a blank at an arbitrary sequence of tokens. Thus, part of a statement, an entire statement, and a sequence of consecutive statements can be defined as a blank.

There are two types of blanks. One requires an answer and the other does not. We call a blank of the latter type an ignorable blank. By defining the ignorable blank, it is possible to hide codes and comments that are clues to the blanks that are required answers, such as when there are similar codes before and after the blank. The difficulty level of a blank can be controlled more flexibly by introducing ignorable blanks.

Because it is also possible to hide comments using the ignorable blank, we can also control the difficulty level of a question. To distinguish between the blanks and ignorable blanks, the backgrounds of these blanks are filled with different colors. The blanks without a number represent ignorable blanks in Fig. 2.

## 4.2 Blanks Within a Trace Table

A Java program is executed through a message sending between the objects. If the trace table is expressed in the order of the execution steps, then the table becomes complicated. Therefore, a trace table was defined for each object. Each instance is expressed as
“ClassName#X” to distinguish between the instances of a class. For example, the identifier of the first generated instance of the class “IDCardFactory” is “idCardFactory#1”, which will be the name of the trace table. The proposed trace table is defined as follows. Figure 3 shows a part of this for space limitation.

In the “Caller of the Method” column, the object from which the method is called, the method name, and the step number are displayed. The upper row in the sub-item of the “Argument of the Method”, “Instance Value”, “Local Variable of the Method” and “External Object” columns shows the data type or class name. The bottom row in the sub-item is the variable name. In each cell, the stored value is displayed as the basic data type. An instance identifier is displayed for a class. If no area is allocated to the variable in the program, nothing is displayed. When the instance creates another instance or sends a message to an instance, all instances are listed as sub-items in the “External objects” column.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Example of a Trace Table with Blanks (Factory Method, Instance ID = idCardFactory#1)</figcaption></figure>

An object can be called multiple times. A double line is drawn in the trace table to distinguish the series of processes in each method call. Specifically, a double line is drawn under the row that corresponds to the return statement.

A single line of code can contain multiple operations, such as assigning the result of a function call to a variable. The purpose is to represent the order in which each operation is performed in a trace table. This allows the student to trace the program completely. In this case, the steps of the two operations are identical. The operation represented by a row in the trace table can be recognized by referring to the column of instance variables and external objects.

In a trace table, the value of each cell is defined as a blank. Specifically, a blank can be defined on a variable value, output value, step number, object identifier, class name, and method name. Moreover, a blank can also be defined for a data type, class name, variable name, or object name. As in the case of the program, it is possible to define ignorable blanks that do not require answers in a trace table.

## 5 Creation of Fill-in-the-Blank Questions

In this section, we propose a development strategy for the fill-in-the-blank questions. We plan to provide the created problems in “Exercise in Programming III” in the first semester of the 2021 academic year. The class was conducted in the third academic year of computing courses at our university.
## 5.1 Proving Fill-in-a-Blank Questions

The students learned Python and C++ in their first and second years, respectively. They began learning object-oriented programming using Java in the target class. The class intends to learn the purpose and use of various tools used in the practical field of software development using the Java language. The textbook [13] explains the 23 design patterns introduced by Gamma et al. [14]. In the lecture, detailed explanations of the language were not provided to the students. To support them in mastering Java programming, we provide the fill-in-the-blank questions proposed in this study.

The fill-in-the-blank questions were created by selecting 12 topics from the textbook and using sample programs. The topics were selected in consideration of the contents to be covered in the class, and the levels of the problems were set as beginner, intermediate, and advanced according to the contents of the topics and the order in which they were taught (Table 1).

<table><caption>**Table 1.** The Topic of the Providing Fill-in-the-Blank Questions</caption><thead><tr><th>Level</th><th>Beginner</th><th>Intermediate</th><th>Advanced</th></tr></thead><tbody><tr><td rowspan="3">Design Pattern (The section in the Textbook)</td><td>TemplateMethod (3)</td><td>Decorator (12)</td><td>Adapter (2)</td></tr><tr><td>FactoryMethod (4)</td><td>Strategy (10)</td><td>Builder (7)</td></tr><tr><td>Iterator (1)</td><td>AbstractFactory (8)</td><td>Command (22)</td></tr></tbody></table>

## 5.2 Development Policy of the Fill-in-the-Blank Questions

Since the students had already learned the basics of programming through the previous classes, we designed questions for the students to learn the basics of object-oriented programming and design patterns.

The blanks within the program are defined mainly to check the understanding of Java-specific grammar such as class definitions and to check the execution flow of message sending. Furthermore, the blanks within the trace table are mainly defined for topics related to the caller objects, such as classes, method names, step numbers; items related to the called object; and values that change as messages are transmitted.

Considering these factors, we developed the following policies to create questions at appropriate levels on the topic.

1. Define approximately ten blanks that required an answer per question.

2. Create two problems for one topic with different difficulty levels.

3. Clarify the educational objective of each question.

4. Clarify the intent for each blank.

5. To define blanks within a program, we used individual tokens as the basis at the beginner level. We used a longer sequence of tokens for intermediate and advanced levels.
6. To define blanks within a trace table, we mainly used variable values at the beginner level. At the intermediate level, we used more blanks, such as instance identifiers and methods.

According to our previous experience [3, 4], we found that if the number of blanks in the question was too many, students' motivation would decrease, which would prevent them from continuously using pgtracer. In our previous experiment, five blanks were defined per question because the target students were beginners in the C language and the total number of lines of the program was approximately 20. In this study, however, approximately ten blanks are defined for each question because the level of programming proficiency of the students is expected to be higher than that of the previous students. The total number of lines in the program is larger for the same reason.

The second policy is defined to investigate the difference in difficulty between the types of questions. Furthermore, the educational objective of the questions and the intention of each blank, clarified by Policies 3 and 4, are used to analyze the learning logs. Policies 5 and 6 are defined to clarify the differences among beginner, intermediate, and advanced levels.

# 6 Primary Trials of the Questions at Lecture

## 6.1 Trial Using Moodle

Because the Java version of pgtracer is still under development, we used the embedded answer (cloze) feature of Moodle to represent the answer field. The programs and trace tables for the fill-in-the-blank questions were manually created and displayed as images. Since a fill-in-the-blank question contains several Java programs and trace tables, each was displayed in a separate tab on the browser. Representative screenshots of a question are available on the web due to space limitations.<fnref n="1" />

Each week, we conducted trials with the following problems, as indicated in Table 2. Suffixes such as PR and TR in the problem ID indicate that the problem type is a program or trace table, respectively. The column on the right shows the number of examinees on the first attempt. Ex01_PR, Ex01_TR, and Ex08_SP are sample problems that explain how to answer or the concept of a trace table. Therefore, we have excluded these problems from the discussion.

## 6.2 Answer Result

Table 3 represents the number of blanks and the average of correct answer ratio for each of the fill-in-the-blank questions whose problem type is “program”, categorizing the blanks into two types that are composed of one or more tokens. At both beginner and intermediate levels, the average of correct answer ratio for the blanks containing multiple tokens was lower than for the blanks containing a token. This indicates that it is more difficult for students to answer a blank containing multiple tokens. For both blanks containing one or more tokens, the average of correct answer ratio is lower

<footnote n="1">¹ http://y-page.y.kumamoto-nct.ac.jp/u/m-murata/WCCE2022/.</footnote>
<table><caption>**Table 2.** Problems and Number of Examinees</caption><thead><tr><th>Week</th><th>Design Pattern</th><th>level</th><th>Problem Type</th><th>Problem ID</th><th># of Blanks</th><th># of Examinees</th></tr></thead><tbody><tr><td rowspan="2">1</td><td rowspan="2">Exercise to understand fill-in-the-blank questions and trace tables</td><td rowspan="2"></td><td>Program</td><td>Ex01_PR</td><td>6</td><td rowspan="2">67</td></tr><tr><td>Trace table</td><td>Ex01_TR</td><td>4</td></tr><tr><td rowspan="2">6</td><td rowspan="2">Template Method</td><td>Beginner</td><td>Trace table</td><td>Ex06_PR</td><td>12</td><td rowspan="2">71</td></tr><tr><td>Beginner</td><td>Program</td><td>Ex06_TR</td><td>12</td></tr><tr><td>7</td><td>Factory Method</td><td>Beginner</td><td>Trace table</td><td>Ex07_PR</td><td>9</td><td>58</td></tr><tr><td rowspan="2">8</td><td>Simple Example</td><td>Beginner</td><td>Program</td><td>Ex08_SP</td><td>2</td><td rowspan="2">67</td></tr><tr><td>Iterator pattern</td><td>Beginner</td><td>Trace table</td><td>Ex08_TR</td><td>10</td></tr><tr><td>10</td><td>Factory Method</td><td>Beginner</td><td>Trace table</td><td>Ex10_TR</td><td>9</td><td>61</td></tr><tr><td>11</td><td>Composite</td><td>Beginner</td><td>Trace table</td><td>Ex11_PR</td><td>11</td><td>69</td></tr><tr><td>13</td><td>Strategy</td><td>Intermediate</td><td>Program</td><td>Ex13_PR</td><td>6</td><td>66</td></tr><tr><td>14</td><td>Observer</td><td>Intermediate</td><td>Program</td><td>Ex14_PR</td><td>10</td><td>65</td></tr></tbody></table>

at the intermediate level than at the beginner level. At the intermediate level, we set blanks that required an understanding of the processing flow of the program to derive the correct answer. Therefore, we were able to reflect on the difficulty level intended by the contestants, even if it was a blank containing a token.

<table><caption>**Table 3.** Number of blanks and right answer correct for problems whose type is “program”</caption><thead><tr><th rowspan="2">Level of Problem</th><th rowspan="2"># of problem</th><th colspan="2">Blanks containing a Token</th><th colspan="2">Blanks containing Multiple Tokens</th></tr><tr><th># of Blanks</th><th>Average of Correct Answer Ratio (%)</th><th># of Blanks</th><th>Average of Correct Answer Ratio (%)</th></tr></thead><tbody><tr><td>Beginner</td><td>3</td><td>22</td><td>76.5</td><td>10</td><td>57.7</td></tr><tr><td>Intermediate</td><td>2</td><td>11</td><td>67.2</td><td>9</td><td>41.9</td></tr></tbody></table>

Table 4 represents the number of blanks and the average correct answer ratio for the fill-in-the-blank questions whose problem type is “Trace table” categorized by the type of blanks. Table 4 indicates that the average correct answer ratio for the step number of the method caller and the variable value that answers the instance is low. This suggests
that problems related to object-oriented specific method invocations and instances are difficult for students.

<table><caption>**Table 4.** Number of blanks and correct answer ratio for problems which type is “Trace table”</caption><thead><tr><td>Type of Blanks</td><td># of Blanks</td><td>Average of Correct Answer Ratio (%)</td></tr></thead><tbody><tr><td>Call for constructor or method</td><td>16</td><td>72.5</td></tr><tr><td>Step number of the calling method</td><td>3</td><td>46.0</td></tr><tr><td>Called Class</td><td>2</td><td>61.7</td></tr><tr><td>Return value (Instance)</td><td>5</td><td>64.0</td></tr><tr><td>Variable value (Instance)</td><td>1</td><td>38.8</td></tr><tr><td>Variable value (Basic type)</td><td>4</td><td>71.1</td></tr></tbody></table>

## 6.3 Questionnaire Result

On the last day of the lecture, we administered a questionnaire regarding the trial. The total number of responses received was 69. Regarding fill-in-the-blank questions, 63.8% answered that they were rather difficult. This suggests that the level of difficulty of the problem is appropriate. The number of fill-in-the-blank questions was appropriate for 87.0% of the respondents. We consider that ignorable blanks allowed us to set an appropriate number of blanks to be answered while hiding codes and variable values that could be used as hints.

Furthermore, we obtained 14 comments from the students. There are five responses regarding the method of displaying the programs and trace tables. In this experiment, the programs and trace tables were presented as images in multiple tabs, but it was necessary to switch tabs when understanding the program and trace messages sent and received, which may hinder students' understanding. In addition, three responses are related to writing answers. They complained about incorrect answers because of differences in capitalization and lowercase letters when typing. However, in actual programming, the difference between lowercase and uppercase letters can be fatal; therefore, we should consider it educationally reasonable and require the correct input.

# 7 Improvement of Fill-in-the-Blank Questions

In this section, we describe the improvement of the fill-in-the-blank questions. The programs and trace tables for the fill-in-the-blank questions described in Sect. 4 are referred to as old programs or old trace tables.

## 7.1 Improvement of Programs

In the old programs, step numbers were also added to the instance variable-definition statements. However, in Java, instance variables are initialized with default values when
the constructor is called and values are assigned by the statements written in the constructor. This default value is implicitly set and not explicitly stated in the program. For this reason, we decided against assigning a step number to the statement of an instance variable definition and describing the default value in the trace table because they would confuse students. In addition, explicitly stating the value of the instance variable in the first-instance variable assignment statement is consistent with the policy of Java programs that there is no indefinite state, such as variable values.

The sample programs referenced to create the old programs were from a textbook published in 2004 and did not correspond to the evolution of the Java language, such as support for generic terms. The third edition of the textbook was published in 2021, and the sample programs have been updated since 2004 to reflect the evolution of the Java language. Using the sample programs in the third edition, it is now possible to provide problems that correspond to the latest evolution of Java.

When the incorrect answers were carefully examined, we found that there were other answers in addition to the set of correct answers and that there were answers that were incorrect because they filled the blank with other correct answers. For example, when “i < 5” was the correct answer, students answered “5 > i” or “i ≤ 4”, resulting in an incorrect answer. To avoid this, we adjusted the location of the blanks so that the correct answer was unique. As an improvement on the previous example, we can eliminate another correct answer by not setting a blank in part “i <”. Instead, we set it in “5”.

In addition, many answers were incorrect because of the presence or absence of whitespace, such as “i < 5” when “i < 5” was correct. According to Java coding conventions [15], a space should be inserted before and after the comparison operator. In system development, the reusability of programs is required, and it is important to be able to code according to the guidelines, not just to have a program that works correctly. Therefore, we decided to treat this as an incorrect answer.

## 7.2 Improvement of Trace Tables

The trace tables were created by manually tracing the program; however, now that the automatic trace table generation function has been completed with a few minor adjustments [16], we are using the trace tables created by this function. This function outputs a trace table in an Excel file format by providing a Java program and input data. Each sheet corresponds to each instance in the output Excel file. With the completion of this function, the development cost of a trace table is significantly reduced.

In the old trace table, the definitions of instance variables are also shown in the trace table, and the values were assumed to be indefinite; however, for the reasons mentioned in Sect. 7.1, we decided not to include them in the trace table as well.

## 7.3 User Interface

In the old fill-in-the-blank questions, programs and trace tables were presented as images. The images were displayed in a new browser tab. According to the questionnaires from the students, several answers were including that the switching tabs were troublesome and made it difficult to understand.
Therefore, as shown in Fig. 4, when asking fill-in-the-blank questions, the frame function of the browser is used to display a list of programs and trace tables. By selecting an item from the list, the program or trace table concerned is displayed. In addition, for instance, the collapsing function is used to switch between showing and hiding instances of the same class. This user interface can also be applied to pgtracer.

<figure><img src="image_7.png" /><figcaption>Fig. 4. Improvement of the User Interface</figcaption></figure>

## 8 Conclusions and Future Works

In this paper, we improve the fill-in-the-blank questions for pgtracer, a Java programming education support tool using fill-in-the-blank questions. The improvement is made based on the results of the trial. We also improve pgtracer user interface to reduce the difficulty of solving problems for students. These improvements are expected to help students understand the program and trace tables, and promote continuous learning. In pgtracer for the C language, the program answered by the student is compiled and executed, and if the output is the same, it is judged to be the correct answer. By incorporating this module into the Java language pgtracer, we believe that it will be possible to control the difficulty level of the problem by making the setting of blank removal more flexible. In this study, we did not analyze the required time as analytical data, but we believe that an analysis including the required time is necessary to estimate students' programming ability. The analysis of response time is an issue to be addressed in the future.

In the future, we plan to complete pgtracer and conduct experiments using the problems we created. Using the collected data, we will be able to extract the tendency of students' answers and the tendency of questions that are prone to mistakes using LA techniques, and we believe that the obtained knowledge will contribute to Java programming education.

**Acknowledgment.** This research is supported by JSPS KAKENHI under grant numbers #20K03265 and #20K03233.
References

1. Murata, M., Kato, N., Kakeshita, T.: Development and evaluation of programming education support tool pgtracer utilizing fill-in-the-blank question. IPSJ Trans. Comput. Educ. 2(2), 20–36 (2016). (in Japanese)

2. Kakeshita, T., Ohta, K.: Student log analysis functions for web-based programming education support tool pgtracer. IPSJ Trans. Comput. Educ. 5(2), 456–468 (2019)

3. Kakeshita, T., Murata, M.: Application of programming education support tool pgtracer for homework assignment. Int. J. Learn. Technol. Learn. Environ. 1(1), 40–61 (2018)

4. Murata, M., Kakeshita, T.: Analysis method of student achievement level utilizing web-based programming education support tool pgtracer. In: 5th International Conference on Learning Technologies and Learning Environment (LTLE 2016), Kumamoto, Japan, pp. 316–321 (2016)

5. Murata, M., Kato, N., Kakeshita, T.: Fill-in-the-blank questions for objected-oriented programming education. In: 10th International Congress on Advanced Applied Informatics (IIAI-AAI), pp. 116–122 (2021)

6. Hsiao, I., Brusilovsky, P., Sosnovsky, S.: Web-based parameterized questions for object-oriented programming. In: E-Learn’08: World Conference on E-Learning (2008)

7. Truong, N., Roe, P., Bancroft, P.: Static analysis of students’ Java programs. In: Sixth Australasian Computing Education Conference (ACE 2004) (2004)

8. Hauswirth, M., Adamoli, A.: Teaching Java programming with the Informa clicker system. Sci. Comput. Program. 78(5), 499–520 (2013)

9. Funabiki, N., Matsushima, Y., Nakanishi, T., et al.: A Java programming learning assistant system using test-driven development method. IAENG Int. J. Comput. Sci. 40(1), 38–46 (2013)

10. Fu, X., Shimada, A., Ogata, H., Taniguchi, Y., Suehiro, D.: Real-time learning analytics for C programming language courses. In: 7th International Conference on Learning Analytics and Knowledge (LAK 2017), pp. 280–288 (2017)

11. Carter, A.S., Hundhausen, C.D., Adesope, O.: The normalized programming state model: predicting student performance in computing courses based on programming behavior. In: ICER 2015 - Proceedings of the 2015 ACM Conference on International Computing Education Research, pp. 141–150 (2015)

12. Grover, S., Basu, S., Bienkowski, M., Eagle, M., Diana, N., Stamper, J.: A framework for using hypothesis-driven approaches to support data-driven learning analytics in measuring computational thinking in block-based programming environments. ACM Trans. Comput. Educ. 17(3), Article no. 12 (2017)

13. Yuki, H.: An Introduction to Design Patterns using Java Programming Language, 3rd edn. SoftBank Creative, Tokyo (2021). (in Japanese)

14. Gamma, J., Helm, E., Johnson, R., Vlissides, R.: Design Patterns Elements of Reusable Object Oriented Software. Addison-Wesley Professional, Boston (1994)

15. Code Conventions for the Java Programming Language. https://www.oracle.com/java/techno logiesjavase/codeconventions-contents.html. Accessed 25 Feb 2021

16. Kishikawa, M., Ohtsuki, M., Kakeshita, T.: Generating trace table for Java programs. In: Proceedings of the 11th International Congress on Advanced Applied Informatics (IIAI-AAI), pp. 200–207 (2022)