# Cycles in State Transition as Trial-and-Errors in Solving Programming Exercises

Taku Yamaguchi, Yoshiaki Matsuzawa, Ayahiko Niimi, Michiko Oba

► To cite this version:

Taku Yamaguchi, Yoshiaki Matsuzawa, Ayahiko Niimi, Michiko Oba. Cycles in State Transition as Trial-and-Errors in Solving Programming Exercises. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.542-553, 10.1007/978-3-031-43393-1_49. hal-05533301

HAL Id: hal-05533301

https://inria.hal.science/hal-05533301v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Cycles in State Transition as Trial-and-Errors in Solving Programming Exercises

Taku Yamaguchi<fnref n="1" />(✉), Yoshiaki Matsuzawa<fnref n="2" />(✉), Ayahiko Niimi<fnref n="3" />(✉), and Michiko Oba<fnref n="3" />(✉)

**Abstract.** In recent years, Japan's education system has started regarding trial-and-error favorably and is accordingly evaluating its strategy. Evaluation of trial-and-error has become crucial to learning analytics. Jigsaw Code is a programming exercise that gives students shuffled code blocks of a solution program, including fake ones, and asks them to rearrange the blocks in the correct order. It logs the solving actions of each student. This paper shows that the cycles in the state transition of the solution indicate the students' trial-and-error process. Two hundred and thirty undergraduates solved 11 puzzles (problems) of Jigsaw Code. We analyzed the transition of the solution sequence at each solving action and found cycles in the state transition. For 97% of the plays, including the frequently appearing cycles, the first state of each cycle was the subsequence of the final solution. A cycle in the solution state transition shows that the student returned to the cycle's first state after searching for other solutions, as they probably regarded the cycle's first state as leading to the correct final solution.

**Keywords:** Trial-and-error · Programming Exercises · Cycles in the State Transition

## 1 Introduction

The education system in Japan regards trial-and-error favorably and evaluates the strategy of trial-and-error. It is a human advantage to find a satisfactory solution to a problem that seemingly has no correct solution. Accordingly, trial-and-error evaluation is becoming crucial in learning analytics.

However, there have been few mechanical ways to estimate students' trial-and-error processes other than to find backtracking. Some researchers are evaluating the solving processes of programming exercises where students rearrange the shuffled code fragments of a solution program into the correct form instead of writing codes from scratch.

<footnote n="1">¹ Graduate School of Future University Hakodate, Hakodate, Hokkaido, Japan g3221002@fun.ac.jp</footnote>

<footnote n="2">² Aoyama Gakuin University, Sagamihara, Kanagawa, Japan matsuzawa@si.aoyama.ac.jp</footnote>

<footnote n="3">³ Future University Hakodate, Hakodate, Hokkaido, Japan niimi@fun.ac.jp, oba@tachibana-u.ac.jp</footnote>
However, those researchers exclude trial-and-error or depend on the existence of the correct solution.

In this paper, we propose estimating students’ trial-and-error processes in solving such exercises independently from the existence of the correct solution by detecting cycles in the state transition of the process.

## 1.1 Trial-and-Errors in Education

In recent years, the education system of Japan has started regarding trial-and-error favorably and is evaluating its strategy (described in the next paragraph); in other words, trial-and-error processes can have strategies. The Central Council for Education of Japan has stated that, “unlike Artificial Intelligence, humans can set their own goals and find satisfactory solutions to problems which have no solutions [emphasis added], even in a complex environment of diverse contexts. Humans develop these abilities through learning.” [1, p. 10]. The Council remarked that it is necessary to help children “go beyond the efficiency of solving problems with predetermined solutions and the efficiency of completing predetermined procedures” toward engaging in trial-and-error [1, p. 10]. Hence, the solving process of problems with no solution can be considered the subject of learning analytics.

The Curriculum Research Center in the National Institute for Educational Policy Research published reference materials explaining learning assessment [2]. In these materials, as examples of trial-and-error strategies, they mentioned that, while determining the parameters of a mathematical expression, “trial-and-error without perspective” is defined as “inputting a randomly selected number and then making adjustments”; additionally, “trial-and-error with perspective” is defined as “making adjustments after obtaining a rough estimate.” [2, p. 59]. Moreover, they explained that trial-and-error processes with and without perspective are “fully satisfied” and “generally satisfied,” [2, p. 59] respectively. Therefore, there are different types of trial-and-errors, which they evaluate positively.

Evaluation of trial-and-error is becoming increasingly crucial in learning analytics. It may not necessarily be similar to evaluating the edit distance from the optimal process of the correct solution, or analyzing the correlation with the percentage of correct solutions, because sometimes there are no correct solutions.

## 1.2 Research Ignoring Trial-and-Error

Parsons problems (also referred to such as Parson’s problem and Parsons puzzles) are a programming exercise [3–5] that shuffles the code blocks and asks students to rearrange them in the correct position. They are similar to our Jigsaw Code described in Sect. 3.1 (see Fig. 1). As a representation of students’ puzzle-solving behavior, Kumar [3] proposed a solution sequence, defined as a sequence of blocks in the temporal order when placed in the correct position. The block’s temporal order is when it is finally placed in that position by the student, even though the block may be placed in the correct position (and removed from there) multiple times in the solving process.

If a student moves blocks in the order s3 (the ID of a code block), s5, s4, s2, s1, s5, s2, s4, and those blocks are in the correct position, the solution sequence will be [s3, s1,
s5, s2, s4]. In this case, the solution sequence does not show that the student moved s5 a second time, s4 a third time, and s2 a fourth time.

As Kumar stated, this approach loses trial-and-error information. He regarded trial-and-error—such as backtracking and looping behavior—as unproductive and as a sign of students’ misconception, not as part of a problem-solving strategy. The concept of “misconception” implies that a correct solution exists. This view of trial-and-error may be due to the usage of the Parsons puzzle for rote learning the syntax of a programming language.

## 1.3 Research Identifying Backtracking as an Intentional Trial-and-Error

Helminen et al. [8] proposed representing the solving processes in Parsons problems as state transitions of the solution and visualized them as diagrams (defined later in the Method section). In individual transitions, there were loops (we call these “cycles” in this paper) returning to states visited earlier in the same transition path. They identified different kinds of loops. Backtracking is a kind of loop returning to “an earlier state by undoing the operations exactly in the reverse order.” A circular loop is a kind of loop “returning to an earlier state via different intermediate states.” Helminen et al. stated that it is difficult to say whether a circular loop is an intentional trial-and-error or an accident, while one might assume backtracking is intentional.

This paper reveals that also a circular loop points out intentional trial-and-error.

## 1.4 Research Depending on the Correct Solution

Through split-paper testing [6], Kuno [7] proposed evaluating thinking, judgment, and expression abilities, not rote learning. For example, he proposed to evaluate the ability to argue in a well-reasoned order. Split-paper (“Tanzaku” in Japanese) testing belongs to the same type of programming exercise as Parsons problems and our Jigsaw Code (described in Sect. 3.1).

Ueno et al. [9] proposed representing the solving processes in split-paper testing as state transitions of the solution and visualizing them as diagrams (defined later in the Method section). They represent the solution as a sequence of split papers. Unlike the process described in the Method section, their method aggregates all of the students’ transitions into a single diagram. Using such a diagram, they have found, for example, that students tend to construct a program from outer parts, such as def and end. However, they pointed out that the diagram contains too much information to identify the stumbling points (trial-and-errors). They defined trial-and-error depending on the transitions of students providing both correct and incorrect solutions. Hence, it may be challenging to apply their method to problems with no solutions.

# 2 Objective

This paper aims to show how cycles in the state transition of the solution indicate the students’ trial-and-errors process. A cycle refers to a clue to a student’s knowledge rather than their misconception. The students who formed the cycle believed that the first state would help them reach the correct solution, and used it as a branch point.
# 3 Method

This paper defines the state transition of the solution and the cycle (described in Sect. 3.2) in the solving process of Jigsaw Code puzzles (described in Sect. 3.1) and elucidates how a cycle can help infer a player's trial-and-error process (described in Sect. 3.3). Subsequently, we verify this by confirming that the first state in the cycle is a subsequence of the player's final solution (described in Sect. 3.4). We analyze data from 11 Jigsaw Code puzzles, played by 230 undergraduates in an introductory programming course for first-year undergraduate students (described in Sect. 3.5).

## 3.1 Jigsaw Code

We developed and used the Jigsaw Code—a programming exercise—in class, which gives students shuffled code blocks of a solution program, including fake ones, and asks them to rearrange these blocks in the correct order. Accordingly, it logs the solving actions taken by each student [10].

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** “2-2 Drawing Sunflower”: A sunflower drawing (left) and puzzle on a programming exercise Jigsaw Code (right). Students select pieces from the left side, deselect and sort pieces on the right side, and complete the correct pieces on the right side to draw a sunflower. “`var t = createTurtle();`” is placed above the right side box in advance and could not be moved.</figcaption></figure>

Figure 1 illustrates Jigsaw Code on the screen. Similar to a jigsaw puzzle, Jigsaw Code breaks the whole code into code fragments or *pieces*. When a student, *player*, starts solving a puzzle, it shuffles some pieces and places them in the choice box, the frame on the left. Sometimes, it places some correct pieces above, inside, or below the solution box at the beginning, which is the frame on the right. Unlike the original jigsaw puzzle, there may be *fake pieces* mixed in. A player then drags a correct piece from the choice box and drops it into the solution box. When a piece turns out to be incorrect, the player can drag and drop it back into the choice box. A player can sort pieces by dragging and dropping pieces inside both boxes. By repeating these actions, the player completes the puzzle. As well as the jigsaw classroom [11], Jigsaw Code was named after a jigsaw puzzle because students are processing pieces (portions of a picture) of their answer—the complete picture—.

Jigsaw Code collects the player's actions in the solving process:
* type of the action

* piece ID of the target of the action

* sequence of IDs of the pieces in the solution box

* a timestamp.

Jigsaw Code collects these actions associated with a Document ID generated uniquely for the play.

## 3.2 State Transition of Solution Sequence and Cycles

**State of the Solution.** The state of the solution refers to the sequence of the pieces in the solution box. Jigsaw Code records the state when a player makes an action in the solving process. A piece has an ID that is usually hidden from the players. We represent the state as a sequence of IDs of the pieces. Figure 2 illustrates a state in the process of solving the puzzle “2-2 Sunflower.” S1, s2, and ... are the pieces’ IDs, respectively. The instruction “Draw a sunflower using a for statement” is a piece and has an ID as well. The sequence of IDs “s8, s3, s5, s6” in the solution box is the state of the solution at the point.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** In the middle of the solution process. The sequence of the piece IDs “s8, s3, s5, s6” is the state at this time and is not correct as a final solution.</figcaption></figure>

**State Transition and Diagram.** The state of the solution will change with progression. We represent the state transition in the diagram shown in Fig. 3. In the diagram, a circle represents a state. A label in the circle is the sequence of the pieces representing the state. An arrow represents a transition from one state to another state. In this paper, the state diagram represents the real-life transition, while the state diagram used in software engineering represents possible transitions. The diagram does not aggregate all students’ transitions, but corresponds to a single one. Figure 3 includes three examples of the state transitions diagram. A number on the arrow counts the times the player has transitioned. In the state transition diagram on the right, the player transitioned from state “s2, s3” to “s2, s3, s10” twice in one solution process. We draw these diagrams by “d3-graphviz”¹ (and D3.js) on a web browser.
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Examples of state transition diagrams. The left includes one cycle, the middle includes no cycles, and the right includes two cycles. The ellipses are the normal states, the rounded box is the first and the last state in the cycle, the diamond is the following state of the first in the cycle, and the octagon is the following state of the last state out of the cycle.</figcaption></figure>

**Cycle.** A cycle in a state transition graph is a non-empty closed sequence of states in which all states (vertices), except the first and the last state, are distinct. In Fig. 3, the left diagram has one cycle (Helminen et al. called it a circular loop [8]), and the middle has none. The right has two cycles: ("s2, s3," "s2, s3, s10," "s2, s3") and ("s2, s3, s10," "s2, s3," "s2, s3, s10") (Helminen et al. called them backtracking [8]). The shape indicates the type of state. The rounded box represents the cycle's first and last state; the diamond represents the following state of the first state; the octagon represents the next state of the last one out of the cycle. A state is not precisely shape-coded because it sometimes belongs to more than one of these three types (rounded box, diamond, and octagon.) Fig. 4 extracts three cycles from Fig. 3: the left cycle is from the left state transition diagram, and the rest two are from the right.

<footnote n="1">¹ GitHub - magjac/d3-graphviz: Graphviz DOT rendering and animated transitions using D3
https://github.com/magjac/d3-graphviz.</footnote>
**The First Piece Moved in the Cycle.** The piece moved first in the cycle represents the difference between the first state (rounded box) and the next state (diamond). In the left cycle in Fig. 4, s9 is the first piece moved. In the righthand cycle, the first action is a removal: s10 is removed from the solution box and is the first piece moved in the cycle.

**The First Piece Moved After the Escape from the Cycle.** The first piece moved after the escape from the cycle represents the difference between the last state (the first state, too, rounded box) and the next state of the last state (octagon). In the left cycle in Fig. 4, s2 is moved right after the escape. When the last state (rounded box) is the final solution of the player, no piece is moved after the escape from the cycle.

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Start and endpoint of the cycle, the first state after the start, the first state after escape from the cycle.</figcaption></figure>

## 3.3 Trial-and-Error

Cycles in a state transition provide a clue to inferring a player's trial-and-error. A cycle indicates that the player has returned to the first state in the cycle. They moved from the first state to explore the correct solution (trial), reached some state and realized it is a wrong state (error), returned to the first state (the last state); this state transition forms a cycle. At the time, the player must have regarded the cycle's first state, their current state, as leading to the correct final solution, which is why they returned to the first state.

In the state transition on the left in Fig. 5, the player reached “s3, s4.” Then, they took a detour to “s3, s4, s8,” instead of going straight to “s3, s4, s5” from “s3, s4.” They tried “s3, s4, s8”, realized that it was wrong, and returned to “s3, s4.”

## 3.4 Verification

If the cycle's first state is the subsequence of the player's final solution, it is more likely that the player regarded the first state as leading to the correct final solution (Fig. 5).
It does not matter whether the player's final solution is correct. Here, a subsequence of a given state (sequence of pieces) is one that can be derived from the given state by deleting some or no pieces, without changing the order of the remaining pieces.

In the transition diagram on the left in Fig. 5, the player did not go from “s3, s4, s8” back to “s4, s3” possibly because they did not regard “s4, s3” as leading to the correct solution. In other words, it was not a coincidence that the player did not go back to “s4, s3” because it was not the subsequence of their final solution “s3, s4, s5, s6.”

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** State transitions include cycles whose first state (rounded box) is the subsequence of the final solution. “S3, s5, s4, s6” is the subsequence of “s3, s5, s4, s8, s6” in the right hand diagram.</figcaption></figure>

We now evaluate the data collected by Jigsaw Code to verify our estimation method for trial-and-error. We take the frequently appearing cycles and examine if the first state in the cycle is the subsequence of the final solution. Here, “frequently appearing” means that the cycle appears over average + two standard deviation times. If a cycle appears frequently, it is not a coincidence; there must be a reason. Furthermore, if the frequently appearing cycle’s first state is frequently the subsequence of the final solution, that is not a coincidence either.

## 3.5 Data

We evaluate the data collected by Jigsaw Code used in short tests in an introductory programming course for first-year undergraduate students. Two hundred and thirty students took the course online and solved eleven puzzles in total in 2020. The course covers
variables, conditional branch, loop, and function with JavaScript, HTML, and Turtle Graphics Library.

The students solved the short tests at the beginning of the class. The purpose of the puzzles was to review the previous lesson. The teacher designed the puzzles to include fake pieces to let students (players) carefully select and deselected correct/fake pieces. The puzzles covered various topics, such as drawing a sunflower using the Turtle Graphics Library (Fig. 1), drawing an hourglass, and calculating the discounted rate for travel.

## 4 Result

For 97% of the plays that include the frequently appearing cycles, their first states of the cycles are the subsequence of their final solution (Table 1).

<table><caption>**Table 1.** The last column is the ratio of plays whose cycle's first state is the subsequence of their final solution. A: frequently appearing cycles, B: plays (state transitions) that include the frequently appearing cycles (A), C: plays that include frequently appearing cycles (A), and the first state of the cycle is the subsequence of the final solution.</caption><thead><tr><th>puzzle</th><th>No. of A</th><th>No. of B</th><th>No. of C</th><th>C/B (%)</th></tr></thead><tbody><tr><td>1-1 Drawing hourglass</td><td>3</td><td>69</td><td>68</td><td>99</td></tr><tr><td>1-2 Determining concentration</td><td>2</td><td>20</td><td>20</td><td>100</td></tr><tr><td>1-3 Replacing digits</td><td>4</td><td>52</td><td>45</td><td>87</td></tr><tr><td>2-1 PCR</td><td>4</td><td>30</td><td>29</td><td>97</td></tr><tr><td>2-2 Drawing sunflower</td><td>2</td><td>8</td><td>8</td><td>100</td></tr><tr><td>3-2 Calculating the square</td><td>1</td><td>9</td><td>9</td><td>100</td></tr><tr><td>3-3 Multiplication table</td><td>2</td><td>31</td><td>31</td><td>100</td></tr><tr><td>4-1 Drawing donut</td><td>2</td><td>27</td><td>23</td><td>85</td></tr><tr><td>4-2 Determining leap year</td><td>1</td><td>11</td><td>11</td><td>100</td></tr><tr><td>5-1 Go To Travel discount rate #1</td><td>1</td><td>22</td><td>22</td><td>100</td></tr><tr><td>5-2 Go To Travel discount rate #2</td><td>4</td><td>22</td><td>22</td><td>100</td></tr><tr><td>Average</td><td></td><td></td><td></td><td>97</td></tr></tbody></table>

## 5 Discussion

97% is high. For example, “S3, s5, s4, s6” is the subsequence of “s3, s5, s4, s8, s6” and has 4! − 1 = 23 different anagrams that are not the subsequence. It is not an accident but intentional that the cycle’s first state is the subsequence of the final solution.

What happened in the process, according to the cycle in Fig. 5? Let us observe the pairs of the first piece moved in the cycle (diamond) and the first piece moved after the
escape from the cycle (octagon). Figure 6 shows the final solution of the player for “2-2 Drawing Sunflower”. Half of the pairs derived from the data were (s8, s5). The state transition diagram on the left of Fig. 5 is an example for the pair (s8, s5).

The piece s5 “t.rt(108);” commands the turtle to turn right by 108°, whereas s8 “t.rt(90);” commands it to turn right by 90°. The diagram on the left of Fig. 5 shows that the players selected s8 at first (trial) but deselected it later (after realizing their error); they went back to “s3, s4,” and this time, selected s5 instead of s8. They figured out the angle of turning to the right through trial-and-error. The teacher mixed the fake piece s8 to test students’ comprehension.

They did not go back to “s4, s3” further because they believed “s3, s4” to be correct. We infer their belief because “s3, s4” is a subsequence of their final solution “s3, s4, s5, s8,” shown in Fig. 6.

Furthermore, it seems that s5 and s8 were the players’ alternatives [12]. We will leave that verification to future research.

A cycle in the solution state transition shows that the student returned to the cycle’s first state after searching other solutions, possibly because they regarded the cycle’s first state as leading to the correct final solution. When they returned to the cycle’s first state exactly in the reverse order—backtracking—it seemed that they thought computationally [13]. When they did not return exactly in the reverse order, it is unclear whether they thought computationally; for example, the entire cycle may be one step in backtracking. We will leave it to future research.

s1 Draw a sunflower using a for statement

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 6.** The final solution for the “2-2 Drawing Sunflower” example</figcaption></figure>

Trial-and-error does not always begin with selecting a new piece. Some cycles begin with the removal of pieces. In Fig. 7, the player probably thought s5 unnecessary in the rounded box state, removed it, and transitioned to diamond state; however, they retrieved it later and transitioned to the rounded box state again.

There may be a trial-and-error process that does not form a cycle. In Fig. 8, the player added s6 to the state “s10, s4, s5,” removed s10 from the state “s10, s4, s5, s6,” and added s3 to the state “s4, s5, s6.” This process does not form a cycle. The state “s10, s4, s5, s6” may be an error for the player, but it is unclear when their trial began. We will leave this type of trial-and-error for future research.
<figure><img src="image_11.png" /><figcaption>**Fig. 7.** A state transition diagram (inverted counterclockwise) for “2-2 Drawing Sunflower.” It contains a cycle that begins with deselecting of a piece s5.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 8.** A state transition diagram (inverted counterclockwise) for “2-2 Drawing Sunflower.” It does not contain any cycles but may represent trial-and-error because of selecting and deselecting piece s10.</figcaption></figure>

# 6 Conclusions

The cycle is a clue to the player’s knowledge, rather than their misconception. It mechanically points out more trial-and-errors than backtracking does. This finding meets the demands of Japanese education regarding trial-and-error favorably and evaluates trial-and-error strategy.

In future research, we expect cycles to provide a clue as to what the players are trying to do, such as their alternatives they attempt. We expect to apply this method to problems that have no solutions because the above-discussed results and discussions are independent of whether the player’s final solution is correct.

A limitation of this study is that the trial-and-error process does not always form a cycle. Furthermore, we elucidated that cycles in state transitions indicate trial-and-errors only with regard to programming exercises. It is not clear whether this is true for other problem-solving processes.

**Acknowledgments.** This work was supported by JSPS KAKENHI Grant Number 20H01728.

# References

1. Central Council for Education: Youchi-en, Shou-gakkou, Chuu-gakkou, Koutou-gakkou oyobi Tokubetsu-Shien-gakkou no Gakushu-shidou-youryou tou no Kaizen oyobi Hitsuyou- na Housaku ni tsuite (Japanese) [Improvement of the Curriculum Guideline for Kindergartens, Elementary Schools, Junior High Schools, High Schools, and Special Needs Schools and Necessary Measures (Report)]. https://www.mext.go.jp/b menu/shingi/chukyo/chukyo0/tou shin/1380731.htm
2. National Institute for Educational Policy Research: Shidou to Hyouka no Ipponka no tameno Gakushu Hyouka ni kansuru Sankou Shiryou (Koutou-gakkou hen) Jouhou (Japanese) [Reference Materials on Learning Evaluation for the “Integration of Teaching and Evaluation” (High School Part), Information]. https://www.nier.go.jp/kaihatsu/shidousiryou.html

3. Kumar, A.: Representing and evaluating strategies for solving parsons puzzles. In: Coy, A., Hayashi, Y., Chang, M. (eds.) ITS 2019. LNCS, vol. 11528, pp. 193–203. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22244-4_24

4. Du, Y., Luxton-Reilly, A., Denny, P.: A review of research on parsons problems. In: Proceedings of ACE 2020: Twenty-Second Australasian Computing Education Conference, pp. 195–202 (2020)

5. Maharjan, S., Kumar, A.: Using edit distance trails to analyze path solutions of parsons puzzles. In: Rafferty, A.N., Whitehill, J., Cavalli-Sforza, V., Romero, C. (eds.) The 13th International Conference on Educational Data Mining (EDM 2020), pp. 638–642 (2020)

6. Nakayama, Y., Kuno, Y., Kakuda, H.: Split-paper testing: a novel approach to evaluate programming performance. J. Inf. Process. 28, 733–743 (2020)

7. Kuno, Y.: Examination questions construction manual for evaluating thinking/judgement/expression abilities (Japanese). In: Jouhou Kyouiku Symposium (Japanese) [Information Education Symposium], vol. 2018, no. 1, pp. 1–8 (2018)

8. Helminen, J., Ihantola, P., Karavirta, V., Malmi, L.: How do students solve parsons programming problems?: an analysis of interaction traces. In: Proceedings of the Ninth Annual International Conference on International Computing Education Research (ICER 2012), pp. 119–126. Association for Computing Machinery, New York, NY, USA (2012)

9. Ueno, S., Terui, Y., Imamura, R., Kuno, Y., Egi, H.: Analysis of the answering processes in split-paper testing to promote instruction. In: Rodrigo, M.M.T., et al. (eds.) Proceedings of the 29th International Conference on Computers in Education. Asia-Pacific Society for Computers in Education (2021)

10. Yamaguchi, T., Oba, M.: Measurable interactive application to find out user recognition and strategy when problem solving. J. Softw. 15(1), 12–22 (2020)

11. Aronson, E.: Jigsaw Basics (n.d.). https://www.jigsaw.org/pdf/JigsawBasics.pdf

12. Yamaguchi, T., Matsuzawa, Y., Niimi, A., Oba, M.: Detecting hesitation in a programming process by analyzing temporal co-occurrence of selection and discard operations. Computer in Education Technical reports, 2021-CE-160(8), pp. 1–6 (2021). (in Japanese)

13. Jeannette, W.: Computational thinking. Commun. ACM 49(3), 33–35 (2006)