# Evaluation of a Data Structure Viewer for Educational Practice

Kensuke Onishi

► To cite this version:

Kensuke Onishi. Evaluation of a Data Structure Viewer for Educational Practice. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.504-516, 10.1007/978-3-031-43393-1_46. hal-05533290

HAL Id: hal-05533290

https://inria.hal.science/hal-05533290v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Evaluation of a Data Structure Viewer for Educational Practice

Kensuke Onishi(✉) 🌐

Tokai University, 4-1-1, Kitakanme, Hiratsuka, Kanagawa 259-1292, Japan

onishi@tokai-u.jp

**Abstract.** Data structures and algorithms are fundamental subjects in the curricula of information science faculties and departments. Recently, they were also included in the curriculum as basic subjects of data science and artificial intelligence.

In 2017, we developed a smartphone application, termed Data Structure Viewer(DSV), to promote the understanding of data structures and algorithms. This application has been used in our lectures. In this article, we describe the evaluation of the DSVs used in lectures from 2017 to 2021 through the use of questionnaires. The students' evaluation and confirmation of the lecture content in class, revealed that the DSV contributed to the improvement of students' comprehension. We also discuss important points to be considered when using the DSV in lectures.

**Keywords:** Data structure · Smartphone application · HyFlex-type lecture

## 1 Introduction

Data structures and algorithms are taught in numerous universities as the foundation of computer science. For example, Introduction to Algorithms at the Massachusetts Institute of Technology (Cambridge, MA, USA) [1] has been developed as an online courseware, and lecture videos have been uploaded on YouTube. From 2013 to 2022, the first lecture had been viewed more than 4 million times. This course can also be used as basic knowledge for data science and artificial intelligence technology, and the need for this knowledge is increasing.

Algorithms and data structures include arrays, pointers, list structures (including stacks and queues), traversal of trees, binary trees, hashing, and sorting. In general, students learn the concepts of these data structures, understand their functions, and implement them in practice.

In recent years, students from various social and academic backgrounds have been enrolling in universities in Japan. If they only wish to learn the concepts of data structures, it is possible to attend lectures without considerable knowledge as a prerequisite. However, for those wishing to understand the mechanism of data structures, mathematical knowledge (e.g. number sequences and probabilities) will be required in certain situations. For example, to understand that
the number of comparisons in a binary search is proportional to $log n$ ($n$ is the number of data), knowledge of logarithms and number sequences are necessary. In previous lectures, mathematical knowledge was supplemented within the lectures, but some students rejected them simply because of the presence of mathematical formulas. In addition, the implementation of data structures is a difficult task for students who are not proficient in programming.

Two research questions in this study were as follows: (Q1) Does the use of a smartphone applications, termed Data Structure Viewer (DSV) improve students' comprehension when learning data structures? (Q2) What factors are necessary to improve comprehension when using the DSV?

The framework of this study is the same as [2]. That is, Budiman et al. [2] examined whether smartphone applications improve the comprehension of lectures on data structures. They used a control group to assess differences in comprehension. Since it was not possible to provide a control group for this study, a questionnaire to students who have been taking “Data Structures and Algorithms” for four years was used to evaluate differences in comprehension. The results showed that the smartphone application (DSV) contributed to lecture satisfaction and comprehension.

In this article, we describe and evaluate the lectures given in 2017, 2019, 2020, and 2021. Section 2 describes a system for displaying data structures and the use of smartphone application in lectures as related works. Section 3 describes DSVs and their evolution. Section 5 describes the details of the lectures, questionnaires, and their results. Section 6 discusses the results of the questionnaires.

## 2 Related Work

When teaching algorithms and data structures, demonstrating the operation of a program that implements the algorithms and data structures has been used as a method to promote understanding. Around 2000, Java applets were used for this purpose; however, these applets are no longer available.

Recently, a smartphone application for viewing the data structure has becomes available. “Algorithms: Explained and Animated”<fnref n="1" /> is an application that allows the viewing of data structures, such as lists and hashes, sorting, list search, graph search, and clustering. A book title “Algorithm Picture Book: 26 Algorithms in Picture” [3], describing the data structures included in the Algorithms: Explained and Animated application, is also available. This book and the smartphone application are a good combination for learning the concept of data structures.

In Python tutor [4] <fnref n="2" />, when you enter source code, Python tutor visualizes the execution of that code and allows you to see the values of variables. Python Tutor supports the Python language, Java, C/C++, JavaScript, and Ruby. By running the source code of a data structure in Python Tutor, it is possible to observe the function of the data structure. However, the development of some languages has

<footnote n="1">¹ Algorithms: http://algorithm.wiki/.</footnote>

<footnote n="2">² Python Tutor: https://pythontutor.com/.</footnote>
been discontinued. There are some restrictions, such as the inability to run long source codes.

Budiman et al. [5] discussed the contents of an application for learning data structures and the networks required for the application. Budiman et al. [2] conducted an evaluation experiment using the application for learning data structures and a traditional teaching method. The results showed statistically that learning outcomes were higher in the group that used the application. Irmayana et al. [6] also developed a similar application and conducted an evaluation experiment with students. As a result, 75% of the students responded that the learning of data structures and algorithms were optimized. The applications used in these studies differ from ours in that they visualize data structures and do not help students understand the principles of data structures. Note that the applications used in [2, 5, 6] are all Android OS only.

A review conducted by Kardy and Ghazal [7], surveyed 37 articles on mobile applications for science learning published in the Web of Science and Scopus databases between 2007 and 2014. The study concluded that future research should take advantage of new technologies, individual testing of the usefulness of application features, and develop additional strategies for using mobile applications for collaboration. In recent years, research has also been conducted using mobile applications. Zydnet and Warner [8] developed an application for learning basic mathematics, which was used in lectures. The application has online quizzes and allows students to monitor their understanding. By comparing classes that used the application with those that did not, they concluded that there was a significant difference in the retention of lecture content (75% and 55%, respectively).

## 3 DSV for Smartphones

The DSV is an application designed to assist students in understanding the principles and the function of data structures. Three versions of DSV (DSV17, DSV19, and DSV20) were developed in 2017, 2019, and 2020, respectively.

Firstly, we will explain the development policies common to all DSVs.

1. DSVs are constructed based on data structures implemented in the language used in the lectures.

2. The data structures used in DSVs are implemented in a simple manner, without using the classes included in the language. The goal of this approach is to present the code of the data structures in the lectures, as well as to promote understanding of the principles and the behavior of data structures.

3. The user can view the inside of the data structure. In the data structure using reference values, the reference values are displayed so that the user can intuitively understand the principles of the data structure.

[DSV17]. DSV17 was the first version of this application written in the Java language for Android smartphones. The data structures supported by this application are stack, queue, lists (array type), list (reference type), ordered list,
binary search tree, heap, and hash. DSV17 is available on Google Play; however there is no application for iOS devices.

[DSV19]. DSV19 is an application developed in collaboration with Asial Corporation (Tokyo, Japan). DSV19 was developed on the Monaca platform for smartphones.<fnref n="3" /> On this platform, it is possible to develop applications using JavaScript. We implemented the data structure in JavaScript and embedded the data structure to create DSV19. In this version, it is possible to use bubble sort, selection sort, insertion sort, quick sort, and merge sort in addition to the nine data structures available in DSV17. DSV19 was running on a Monaca debugger, so the convenience was not satisfactory.

[DSV20]. DSV20 is a smartphone application of DSV19 and is available on Google Play for Android smartphones<fnref n="4" /> and App Store for iOS devices. Therefore, students can download DSV20 to their own smartphones. The data structures available in DSV20 are the same as those in DSV19.

# 4 Educational Practice

## 4.1 Details of Educational Practice

This section describes the lecture in which the educational practice was conducted. This lecture takes place for 14 weeks in the fall semester and addressed to second-year students at the Department of Mathematical Sciences.

<table><caption>**Table 1.** Details of the lectures on educational practice.</caption><tr><td>Course title</td><td colspan="4">Data Structure and Algorithm I</td></tr><tr><td>Eligibility</td><td colspan="4">Second-year students, Fall semester</td></tr><tr><td rowspan="5">Course students</td><td>Year</td><td>Second-year students</td><td>Students in higher grade</td><td>Total number</td></tr><tr><td>2017</td><td>22 (68%)</td><td>10 (31%)</td><td>32</td></tr><tr><td>2019</td><td>64 (91%)</td><td>6 (9%)</td><td>70</td></tr><tr><td>2020</td><td>71 (89%)</td><td>9 (11%)</td><td>80</td></tr><tr><td>2021</td><td>74 (91%)</td><td>7 (9%)</td><td>81</td></tr></table>

The distribution of the number of students in each year is shown in Table 1. In all years, the majority of students were in their second year of study and few were in their third year or above. In particular, after 2019, ≥ 85% of the students were in the second year. Compared with 2017, the number of students enrolled in the course has significantly increased by more than double since 2019.

<footnote n="3">³ Monaca, https://monaca.io/.</footnote>

<footnote n="4">⁴ https://play.google.com/store/apps/details?id=jp.co.asial-algorithm&hl=ja&gl=US.</footnote>
By the time they attend this class, students would have attended 2h of lectures on Java language in the spring semester of their first year, 1h of lecture on Java language and 1h of lecture on HTML, and JavaScript in the fall semester of their first year, and 2h of lecture on Processing in the spring semester of their second year. Simultaneously to the aforementioned classes of data structure, they are also attending 2h graphical user interface programming classes.

The syllabus for this lecture includes the following three objectives:

- students will be able to read and execute various algorithms;

- students will be able to determine which algorithm is appropriate for a given problem;

- students will be able to express the computational complexity of algorithms in mathematical expressions and compare algorithms.

To achieve this goal, the lecture covered general data structures and algorithms such as lists (including stack and queue), binary tree, binary search tree, heap, and hash. Sorting was covered in another lecture due to time constraints.

Each lecture takes place in approximately the following order:

(1) review of the previous lecture;

(2) observation of data structures using DSV (2017);

(2') experience of data structures using DSV (2019, 2020, and 2021);

(3) explaining concepts with pseudo-code;

(4) explaining the source code by the lecturer (2017, 2019);

(4') running and explaining the source code in Python Tutor [4] (2020, 2021);

(5) exercises and Q&A;

(6) confirmation of lecture content (2020, 2021);

Although the content of these lectures has not changed, the applications and the fashion in which they are used in lectures have been altered over the years. Below, we explain the changes which occurred each year.

**2017:** The lecture was given using DSV17, and the language used was Java. In the lecture, we displayed the screen of the smartphone on the projector. However, since most of the students were using iPhones, they did not use the application themselves.

**2019:** Students used DSV19 on the Monaca debugger application. However, some students found this application difficult to use. The language used was JavaScript, and the application was produced based on the source code. We have also written a textbook [9] that assumes the use of the DSV.

**2020, 2021:** Due to the coronavirus disease 2019 (COVID-19) pandemic, the lectures in 2020 and 2021 took *HyFlex-type* lectures, which is a lecture given face-to-face and taken simultaneously by students at a distance. During this
period, the lecture content was organized so that only one data structure was
covered in each lecture.

The following six learning supports were incorporated: (1) Distribution of lecture materials; (2) Confirmation of lecture contents by Google Forms; (3) Individual questions using the Zoom breakout room; (4) Presentation of URLs for questions during lectures; (5) Demonstration of the execution process of the source code; and (6) Use of DSV20.

Support (2) is the confirmation of the lecture content covered in each lecture. This confirmation is performed via Google Form, and is set up so that students can check their score (0–4 points) as soon as they provide an answer. Students can deepen their understanding of the lectures by repeatedly answering the questions, as if they were practicing calculations. Support (5) was performed using Python Tutor.

Support (6) is an application for smartphones, which is the subject of this
article. Since DSV20 is available on Google Play and App Store, students can use
them independently at any time. Exercises using the application were conducted
in all lectures. The language used was JavaScript.

## 4.2 Evaluation of the Practice

**Evaluation Outline:** In this study, we used questionnaires to evaluate the applications and lectures. Three types of questionnaires were used. The first is a questionnaire regarding the DSV and lecture, which was administered after the last lecture. The content of the questionnaire is explained in the next section. The second questionnaire is a confirmation of the lecture contents (support (2)); it measures the level of understanding of the students by confirming the lecture contents from the second lecture to the fourteenth lecture in 2020 and 2021. The last questionnaire is a class evaluation questionnaire conducted by the university, and administered in all courses. Owing to the many items in the questionnaire, we used items which we thought were relevant to the DSVs used.

**Questionnaire Concerning the Lecture and DSV:** We explained to the students that the questionnaire would be used for the future development of applications and the publication of the article. We instructed the students not to answer if they were not satisfied with the purpose of usage of the questionnaire. As a result, we received responses from 18(2017), 61(2019), 44(2020), and 38(2021) students. Items of the open-ended questionnaire are shown in Table 2; items of the 4-point scale questionnaire and the mean values of the ratings are shown in Table 3. A 4-point scale was used for scoring (4 denoted high satisfaction and 1 denoted low satisfaction).

Table 4 shows the analysis <fnref n="5" /> of text mining on the free text of the questionnaires from 2019 to 2021. The three most frequent results of the annual analysis

<footnote n="5">⁵ Analysis by user local text mining tool (https://textmining.userlocal.jp/).</footnote>
<table><caption>**Table 2.** Evaluation items for the lecture and Data Structure Viewer (DSV) (opened questionnaire).</caption><thead><tr><td>Item number</td><td>Item</td></tr></thead><tbody><tr><td>1</td><td>Please indicate what you think is good about the DSV</td></tr><tr><td>2</td><td>Please indicate what you think is bad about the DSV</td></tr></tbody></table>

<table><caption>**Table 3.** Average values of the evaluation of the Data Structure Viewer (DSV) (4-point scale), N = 18(2017), 61(2019), 44(2020), 38(2021).</caption><thead><tr><td>Item number</td><td>Item</td><td>2017</td><td>2019</td><td>2020</td><td>2021</td></tr></thead><tbody><tr><td>1</td><td>Did the application help you learn?</td><td>3.06</td><td>2.44</td><td>3.30</td><td>3.39</td></tr><tr><td>2</td><td>Did you want to use the application for yourself?</td><td>2.61</td><td>2.18</td><td>2.68</td><td>2.68</td></tr><tr><td>3</td><td>Did the application help you understand stack behavior better?</td><td>3.17</td><td>2.48</td><td>3.43</td><td>3.42</td></tr><tr><td>4</td><td>" queue behavior better?</td><td>3.22</td><td>2.49</td><td>3.30</td><td>3.39</td></tr><tr><td>5</td><td>" list (array) behavior better?</td><td>3.17</td><td>2.57</td><td>3.32</td><td>3.26</td></tr><tr><td>6</td><td>" list (reference type) behavior better?</td><td>3.22</td><td>2.42</td><td>3.11</td><td>3.32</td></tr><tr><td>7</td><td>" sorted list behavior better?</td><td>3.11</td><td>2.51</td><td>3.09</td><td>3.13</td></tr><tr><td>8</td><td>" binary search tree behavior better?</td><td>3.28</td><td>2.62</td><td>3.18</td><td>3.18</td></tr><tr><td>9</td><td>" heap behavior better?</td><td>3.22</td><td>2.41</td><td>2.98</td><td>3.11</td></tr><tr><td>10</td><td>" hash behavior better?</td><td>3.11</td><td>2.39</td><td>2.89</td><td>3.03</td></tr></tbody></table>

regarding the good features (Table 2, item 1) and bad features (item 2) of the application are displayed. The discussion on this data is omitted for the sake of page numbers.

<table><caption>**Table 4.** Analysis of the free-text questionnaire concerning Data Structure Viewer (DSV).</caption><thead><tr><td>Category</td><td>Year</td><td>Word (frequency, term frequency-inverse document frequency[TF-IDF] score)</td></tr></thead><tbody><tr><td rowspan="3">Good</td><td>2019</td><td>easy to understand (6, 3.59), understand (6, 0.07), can do (6, 0.05)</td></tr><tr><td>2020</td><td>can do (7, 0.06), understand (4, 0.06), visual (3, 3.62)</td></tr><tr><td>2021</td><td>understand (12, 2.2), can do (12, 0.18), lecture (7, 1.13)</td></tr><tr><td rowspan="3">Bad</td><td>2019</td><td>update (4, 3), download (3, 0.66), application (3, 0.18)</td></tr><tr><td>2020</td><td>understand (3, 0.14), can do (3, 0.01), think (3, 0.01)</td></tr><tr><td>2021</td><td>arrays (3, 6.83), number of items (2, 1.61), reset (2,0.49)</td></tr></tbody></table>
In the questionnaire administered to the students, we asked which learning supports should remain in the lecture and which should be removed. The results are shown in Table 5.

<table><caption>**Table 5.** Percentage of students who would like learning supports in lectures to remain or be removed, N = 44 (2020, even numbered row), 38 (2021, odd numbered row).</caption><thead><tr><th>Learning support</th><th></th><th>(1)</th><th>(2)</th><th>(3)</th><th>(4)</th><th>(5)</th><th>(6)</th></tr></thead><tbody><tr><td rowspan="2">Continue</td><td>2020</td><td>84.1</td><td>52.3</td><td>22.7</td><td>40.9</td><td>47.7</td><td>65.9</td></tr><tr><td>2021</td><td>89.5</td><td>39.5</td><td>36.8</td><td>50.0</td><td>65.8</td><td>71.1</td></tr><tr><td rowspan="2">Neither</td><td>2020</td><td>13.6</td><td>38.6</td><td>75.0</td><td>54.5</td><td>43.2</td><td>29.5</td></tr><tr><td>2021</td><td>10.5</td><td>47.4</td><td>57.8</td><td>50.0</td><td>31.6</td><td>26.3</td></tr><tr><td rowspan="2">Discontinue</td><td>2020</td><td>2.3</td><td>9.1</td><td>2.3</td><td>4.5</td><td>9.1</td><td>4.5</td></tr><tr><td>2021</td><td>0.0</td><td>13.2</td><td>5.3</td><td>0.0</td><td>2.6</td><td>2.6</td></tr></tbody></table>

**Confirmation of Lecture Content:** This section explains the methodology used to check the lecture content. The confirmation of lecture content was performed by name to check the level of understanding for each student in 2019, 2020, and 2021. The submitted information was considered attendance to the lecture, but did not affect the student's grade. The items to be checked are shown below; all items were required:

- student's identification number and name;

- ability to use the data;

- student's level of understanding (scale 1 to 4; 4 denotes very well understood and 1 denotes not understood at all);

- what did you understand from the lecture (free description);

- what did you not understand from the lecture (free description);

- questions to confirm the content (4 choices or multiple elections). Each question is worth one point. The maximum score is 4 points and the minimum score is 0 points (only 2020, 2021).

Since students could check the content of the lecture as many times as they wished, they repeatedly answered the same questions to improve their scores.
<table><caption>**Table 6.** Comprehension of each lecture.</caption><thead><tr><th>Number of lectures</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th><th>11</th><th>12</th><th>13</th><th>14</th></tr></thead><tbody><tr><td rowspan="3">Self-assessment of comprehension</td><td>2019</td><td>-</td><td>2.76</td><td>2.56</td><td>2.56</td><td>2.66</td><td>2.86</td><td>2.75</td><td>1.87</td><td>2.47</td><td>2.74</td><td>2.78</td><td>2.49</td><td>-</td></tr><tr><td>2020</td><td>3.20</td><td>3.04</td><td>3.06</td><td>2.33</td><td>2.60</td><td>3.01</td><td>2.86</td><td>2.87</td><td>2.58</td><td>2.89</td><td>2.78</td><td>2.66</td><td>2.60</td></tr><tr><td>2021</td><td>3.22</td><td>3.26</td><td>3.27</td><td>2.91</td><td>3.08</td><td>3.06</td><td>3.25</td><td>3.25</td><td>2.85</td><td>2.97</td><td>3.05</td><td>2.88</td><td>3.00</td></tr><tr><td rowspan="2">Difference from 2019</td><td>2020</td><td>-</td><td>0.28</td><td>0.30</td><td>-0.23</td><td>0.04</td><td>0.35</td><td>0.00</td><td>0.12</td><td>0.71</td><td>0.42</td><td>0.00</td><td>0.17</td><td>-</td></tr><tr><td>2021</td><td>-</td><td>0.50</td><td>0.51</td><td>0.35</td><td>0.52</td><td>0.40</td><td>0.39</td><td>0.50</td><td>0.98</td><td>0.50</td><td>0.27</td><td>0.39</td><td>-</td></tr><tr><td rowspan="3">Average of confirmation of lecture content</td><td>2020</td><td>2.95</td><td>3.55</td><td>2.27</td><td>1.37</td><td>2.47</td><td>2.50</td><td>2.49</td><td>2.81</td><td>2.06</td><td>2.82</td><td>2.08</td><td>2.37</td><td>2.07</td></tr><tr><td>2021</td><td>3.04</td><td>3.79</td><td>2.71</td><td>3.09</td><td>3.18</td><td>3.17</td><td>3.08</td><td>3.39</td><td>3.31</td><td>3.25</td><td>3.05</td><td>3.01</td><td>2.55</td></tr><tr><td></td><td>3.22</td><td>3.64</td><td>2.49</td><td>1.53</td><td>2.67</td><td>2.51</td><td>2.71</td><td>2.91</td><td>1.87</td><td>2.67</td><td>2.63</td><td>2.51</td><td>1.68</td></tr><tr><td></td><td></td><td>3.43</td><td>3.75</td><td>3.46</td><td>3.46</td><td>3.48</td><td>3.37</td><td>3.33</td><td>3.58</td><td>3.44</td><td>3.62</td><td>3.65</td><td>3.49</td><td>3.24</td></tr></tbody></table>

Table 6 shows aggregate data of students who agreed to participate in the study. In rows 2–4 of Table 6, the mean levels of understanding of students from 2019 to 2021 are displayed. The data for 2020 and 2021 (lines 3 and 4) are the average of the comprehension levels indicated in the last response. Lines 5, 6 and 7, 8 show the difference between the data in 2020 and 2021 and those obtained in 2019, respectively. Because the order of the lectures was slightly changed, the differences are between lectures that covered the same content. For the third, and eleventh through thirteenth lectures, the differences were calculated for the same number of lectures. For the fourth through eleventh lectures in 2020 and 2021, the differences between the third through tenth lectures in 2019 were calculated.

Lines 9 and 11 in Table 6 are the average of the scores of the first confirmation of the lecture content in 2020 and 2021. Lines 10 and 12 are the average of the score of the last confirmation of the lecture contents in 2020 and 2021.

**Class Evaluation Conducted by the University:** There are approximately 20 questions in this class evaluation questionnaire, and each question is rated using a 5-point scale (5 denotes the best rating and 1 denotes the worst rating). The evaluations of three items related to the DSVs are shown in Table 7.

<table><caption>**Table 7.** Evaluation items and evaluation of the lecture (5 levels), N = 25(2017), 64(2019), 24(2020), 22(2021).</caption><thead><tr><th>Item number</th><th>Item</th><th>2017</th><th>2019</th><th>2020</th><th>2021</th></tr></thead><tbody><tr><td>1</td><td>The explanation was easy to understand.</td><td>3.60</td><td>2.68</td><td>2.92</td><td>3.09</td></tr><tr><td>2</td><td>The writing on the board, audio-visual materials, and handouts were appropriate and easy to understand.</td><td>3.80</td><td>2.87</td><td>3.63</td><td>3.41</td></tr><tr><td>3</td><td>Efforts were made to encourage students' willingness to participate and attend the class.</td><td>3.88</td><td>2.84</td><td>3.71</td><td>3.73</td></tr></tbody></table>
# 5 Discussion

## 5.1 Questionnaire Concerning the Lecture and DSV

**Evaluation of DSV:** Firstly, we describe the questionnaire on the DSV (Table 3). Item 1 refers to the usefulness of the DSVs: in 2019 (DV19), the evaluation value of DSVs temporarily and significantly decreased; however, in other years, it was increased an annual basis. This indicates that students consider the DSV to be useful. Item 2 is a question regarding willingness of students to use the DSV themselves. This item showed the same trend as item 1; nevertheless, the mean value was lower than that of item 1.

Items 3 through 10 are questions concerning each data structure. Similar to items 1 and 2, for all data structures, the worst rating was noted in 2019, and the ratings for 2017 (DSV17), 2020, and 2021 (DSV20) were good. The ratings noted in 2020 were better than those recorded in 2019 for all items. In addition, the ratings obtained in 2021 were better than those reported in 2020 for all items, except for item 3 (stack). However, the difference in item 3 was only 0.01. Comparison of the 2021 rating with the 2019 rating, reveled an increase of ≥ 0.6 (20%). The largest increase was observed in item 3 (stack), (i.e., 0.94, 32% up). When compared with 2017, five items showed an increase in rating. Item 8 (binary search tree), item 9 (heap), and item 10 (hash) showed a decrease in ratings.

These results suggest that the students who attended the course in 2021 and 2020 thought that the DSV20 helped them to better understand numerous data structures.

**Rating for Learning Supports:** This section describes the learning supports that students would like to continue using (Table 5). The item related to the DSV in Table 5 is learning support (6), which asks whether or not students wished to continue using the DSV. The percentages of students who wished to continue using the DSV in 2020 (row 2) and 2021 (row 3) are 65.9% and 71.1%, respectively. In contrast, very few students (4.5% and 2.6%, respectively) answered that they did not wish to continue using the DSV. The continue using the DSV was also the second highest among all learning supports, both in 2021 and 2022. This indicate that the students are willing to continue using DSV in the future.

The item for which students expressed the greatest desire to continue using was (1) open access to lecture materials. The percentage of students who did not wish to continue the study support was small for all items, indicating that we would like to continue using learning supports (1) - (6) in the future.

## 5.2 Confirmation of Lecture Content

In this section, we will discuss the students' own level of understanding and confirmation of the lecture content (Table 6) obtained in each lecture.
Comparison of data from 2020 and 2019 showed that the level of understanding increased for nine lectures, remained unchanged for two lectures, and decreased for one lecture. Comparison of data from 2021 and 2019 revealed that the level of understanding had increased for all lectures. Comparison of 2021 and 2020 data indicated that the level of understanding had also improved.

The average score of the lecture content check is an indicator of the level of understanding of the lecture. Since we proceeded with multiple responses, the final average score revealed improvements for all lectures (2020, 2021). The final average score did not reach 3.0 for only two lectures in 2020. In 2021, all final mean scores were $\ge$3.2. Repeated responses indicate that students' comprehension had improved.

For 2020, the correlation coefficient between the students' own understanding (row 3 of Table 6) and the mean score of the first confirmation of the lecture content (row 9) was 0.754, denoting a strong correlation ($p$-value: $0.514 \times 10^{-2}$). For 2021, the correlation coefficient was 0.741, also indicating a strong correlation ($p$-value: $0.423 \times 10^{-1}$). In other words, by using the students' own level of understanding, we can estimate the level of comprehension of each lecture.

The students' own level of understanding was higher in 2020 versus 2019. Furthermore, the level of comprehension was higher in 2021. Therefore, we can conclude that students' level of comprehension of lectures increased on an annual basis.

## 5.3 Class Evaluation Conducted by the University

We describe the class evaluation conducted by the university (Table 7). We focused on three items: (1) clarity of the explanations; (2) appropriateness and clarity of the lecture materials; and (3) efforts to encourage students' participation in the lectures. The DSV is a teaching material that supplements the clarity of explanations (items 1 and 2). In addition, the DSV displays data structures on the blackboard and lecture materials, and allows students to experience them at hand (item 3).

The year with the best ratings for all items was 2017, followed by 2021, 2020 and 2019. The number of students who participated was 32(2017), 70(2019), 80(2020), and 81(2021). In general, a small number of students allows us to conduct the lecture while observing the students; this helps the students understand the course better. In 2019, the number of students was very large and the DSV19 had problems with updates and stability, resulting in low ratings in all categories. In 2020 and 2021, the number of students was similar to that of 2019, but the lecture took place as a HyFlex lecture and the DSV20 was used.

As a result, the ratings for items 2 and 3 were the same as those recorded in 2017. Based on the evaluation of the DSV (Table 3) and the students' desire to continue using this tool (Table 5), we can assume that the DSV has a positive impact on comprehension.
## 5.4 Notes on Using the DSV in Lecture

Important factor to consider when using the application are discussed below. The change from DSV19 to DSV20, significantly improved the students' comprehension. The change in the DSV at that point was due to the release of DSV in the application store. DSV19 and DSV20 have the exact same data structures that DSV can handle. In the 2019 student questionnaire, the responses were related to the stability and download of the application. Since DSV19 runs on the Monaca debugger, it occasionally malfunctions. Moreover, updating DSV19 was cumbersome. These problems were addressed in DSV20. Thus, it is likely that the students were wary of applications that lacked stability, as they are used to stable smartphone applications, and thus rated them poorly.

In 2019, student stated that they would like to have operating instructions within the DSV. Hence, starting in 2020, we allocated time to explain the use of DSV in the first lecture. Firstly, a smartphone screen was displayed as part of the delivery screen to introduce the actual usage of the DSV. Next, as an exercise, we gave students a task using the DSV and asked them to provide the results. In other words, we explained the operability and convenience of the DSV in a step-by-step manner. Additionally, we explained the operation of the first data structure, stack, as carefully as possible. Consequently, in the survey responses after 2020, the opinions on the operation on the DSV were decreased considerably (Table 4, bad futures). Therefore, to create a usable application for students, the application needs to be improved. It is important to carefully explain how to present the application and its results, as well as how to use it.

# 6 Conclusion

We created an smartphone application, DSV, and its usefulness is presented in this article. In the process, we found out that simply creating an application is not enough to get students to use it. We found that it is necessary to carefully communicate the stability and usefulness of the application.

Unfortunately, the evaluation experiments in this paper have not shown that DSV alone has improved student comprehension. In the future, we intend to verify the effectiveness of DSV by comparing classes in which DSV was introduced with those in which it was not.

**Acknowledgments.** We express our sincere gratitude to Mr. Yuki Okamoto and Mr. Ryoichi Tsukada of Asial Corporation for their assistance in developing the application and publishing the book.

# References

1. Introduction to Algorithms. https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-006-introduction-to-algorithms-fall-2011/. Accessed 7 Jan 2022
2. Budiman, E., Pusnitasari, N., Wati, M., Widians, J.A., Tejawati, A., et al.: Mobile learning media for computer science course. In: 2018 International Electronics Symposium on Knowledge Creation and Intelligent Computing (IES-KCIC), pp. 262–267. IEEE (2018)

3. Ishida, M., Miyazaki, S.: Algorithm Picture Book: 26 Algorithms in Pictures. Shoeisha (2017). (in Japanese)

4. Guo, P.J.: Online python tutor: embeddable web-based program visualization for cs education. In: Proceeding of the 44th ACM Technical Symposium on Computer Science Education, pp. 579–584 (2013)

5. Budiman, E., Hairah, U., Haeruddin, H., Saudek, A.: Mobile networks for mobile learning tools. J. Telecommun. Electr. Comput. Eng. (JTEC) 10(1–4), 47–52 (2018)

6. Irmayana, A., et al.: The implementation of e-learning into mobile-based interactive data structure subject. In: 2017 5th International Conference on Cyber and IT Service Management (CITSM), pp. 1–5. IEEE (2017)

7. Kadry, S., Ghazal, B.: Design and assessment of using smartphone application in the classroom to improve students’ learning (2019)

8. Zydney, J.M., Warner, Z.: Mobile apps for science learning: review of research. Comput. Educ. 94, 1–17 (2016)

9. Onishi, K.: Learning data structures and algorithms with smartphone applications. Asial Corp. (2019). (in Japanese)