# Trends of Checklist Survey of Computer Operational Skills for First-Year Students: Over the Past Four Years

Daisuke Kaneko, Yukiya Ishida, Masaki Omata, Masanobu Yoshikawa, Takaaki Koga

► To cite this version:

Daisuke Kaneko, Yukiya Ishida, Masaki Omata, Masanobu Yoshikawa, Takaaki Koga. Trends of Checklist Survey of Computer Operational Skills for First-Year Students: Over the Past Four Years. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.423-428, 10.1007/978-3-031-43393-1_39. hal-05533295

HAL Id: hal-05533295

https://inria.hal.science/hal-05533295v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Trends of Checklist Survey of Computer Operational Skills for First-Year Students: Over the Past Four Years

Daisuke Kaneko¹✉, Yukiya Ishida², Masaki Omata³✉, Masanobu Yoshikawa³, and Takaaki Koga⁴

¹ Hokusei Gakuen University, Sapporo, Hokkaido, Japan
kaneko@hokusei.ac.jp

² Chitose Institute of Science and Technology, Chitose, Hokkaido, Japan
y-kawani@photon.chitose.ac.jp

³ University of Yamanashi, Yamanashi, Japan
omata@hci.media.yamanashi.ac.jp, yoshi@yamanashi.ac.jp

⁴ Saga University, Saga, Japan
d4062@cc.saga-u.ac.jp

**Abstract.** Information literacy has become extremely important not only in elementary and secondary education but in higher education as well. Thus, the quality of first-year information literacy education must be improved. To do this, it is necessary to visualize students' information literacy and improve the curriculum accordingly. For this study, the authors surveyed basic information and communication technology (ICT) knowledge and developed a self-assessment checklist to assess computer operational skills. The checklist (designed as placement and achievement tests) contained a total of 40 items across four categories: (a) computer operations, (b) internet/e-mail, (c) Word, and (d) Excel/PowerPoint. Students selected an item if they could perform the operation described in that item. The authors conducted the study over four years with participation from 17,086 students at six universities in Japan. The results indicate that the placement test score in 2018 was lower than in other years, and the achievement test scores were markedly higher than the placement test scores in all years. By category, (d) Excel/PowerPoint had lower placement test scores than the other categories. However, even in category (d), achievement test scores were higher than the placement test scores. The results suggest that the first-year information literacy education programs were effective in these universities.

**Keywords:** Information Literacy · Computer Operational Skills · First-Year Students · Self-Assessment Checklist

## 1 Introduction

The new courses of study (the fundamental standards for curriculums by the Ministry of Education) were announced in Japan in 2017 (2018 for high schools). These courses aimed to develop language skills, information literacy (including information morals),
and problem-solving skills as the qualifications and abilities that form the foundation of
learning. It is interesting to note that information literacy was stated for the first time as
part of the new courses of study. Over time, the importance of information literacy has
been established across primary, secondary, and higher education stages.

Subsequently, improving students’ ability to use information and the corresponding
development of information literacy education has been a major discussion in higher
education and across universities. In particular, information literacy education for first-
year students has been considered very important.

To improve the quality of information literacy education for first-year students, it
is necessary to measure students’ information literacy abilities. Previous literature pro-
vides insight into studies that focused on assessing these abilities. For example, Nishino
et al. [1] attempted to provide standardized placement test questions for knowledge of
information, and Fukada et al. [2] developed survey items on information ethics that can
be used at multiple universities.

However, these measurements have to be conducted as needed. The current study
proposes that by visualizing their abilities continuously, students can learn what they
need to learn. Additionally, the quality of the educational program can be improved by
analyzing the results.

We distinguished between measuring students’ basic information and communi-
cation technology (ICT) knowledge and their level of operational skills. We focused
on measuring students’ level of acquisition of basic ICT knowledge through a survey
consisting of 40 multiple-choice questions covering 14 learning areas [3].

We continued to investigate techniques to measure computer operational skills
required in information literacy education for first-year students. For example, it would
be possible to survey students’ computer operational skills using certificate examina-
tions such as Microsoft Office Specialist (MOS). However, having all students take
such exams consistently is extremely difficult. Considering the examination fee and the
number of available computers, the feasibility is low. Additionally, each university has
different requirements for operational skills that students should learn. Therefore, we
developed a self-assessment checklist for assessing computer operational skills [4].

We conducted this survey for four years at several universities. Specifically, we
examined the results of skill checks conducted on incoming first-year students from
2018 to 2021.

## 2 Computer Operational Skills Checklist

The self-assessment checklist (developed by the authors) allows students to evaluate
whether they can perform the operations described in the list. The checklist contains 40
items divided into the following four categories: (a) computer operations, (b) internet/e-
mail, (c) Word, and (d) Excel/PowerPoint.

Computer operations (a) consist of 10 items related to the operation of the com-
puter itself, such as mouse operations and file operations. Internet/e-mail (b) consists of
six items related to Internet browsing; for instance, using a browser, such as entering
URLs, and e-mail operations, such as sending attached files. Word (c) consists of 14
items related to Microsoft Office Word operations, such as changing the font and table
insertion. Excel/PowerPoint (d) consists of 10 items related to Microsoft Office Excel and PowerPoint operations, such as entering rudimentary functions and creating graphs. The students were also required to checkmark all items they could perform. Further, another option stating “I can do nothing” was set for each category. The purpose is to have students check this if they cannot check any other items in the category. It is possible to distinguish whether a student is not checking because of a lack of applicable items or because of a mistake.

Apart from these, we also included an item asking students about the types of computers and mobile devices they use.

# 3 Results from a Four-Year Survey

## 3.1 Participant Data

The results of checklist surveys from 2018 to 2021 were analyzed in this study. A placement test was administered to new students at the time of their enrollment, and the same survey was administered six months or one year later as an achievement test. The survey was administered to 17,086 students across six universities (national and private universities) in Japan.

Table 1 illustrates the number of students and universities for each year. Students who have taken the tests more than once are counted separately. The number of students for the achievement test in 2021 is lower than in other years since it reflects the data accumulated before writing the research paper.

<table><caption>Table 1. Number of students and universities for each year</caption><thead><tr><th></th><th>2018</th><th>2019</th><th>2020</th><th>2021</th><th>Total</th></tr></thead><tbody><tr><td>Placement Test (PT)</td><td>1,322 (3)</td><td>3,070 (5)</td><td>3,325 (5)</td><td>2,387 (5)</td><td>10,104</td></tr><tr><td>Achievement Test (AT)</td><td>2,284 (5)</td><td>1,870 (3)</td><td>2,078 (4)</td><td>750 (2)</td><td>6,982</td></tr></tbody></table>

Note: Number of students (universities)

## 3.2 Total Score

Table 2 illustrates the mean and standard deviation for each test conducted from 2018–2021. Firstly, we consider the trend of the total number of checked items (total score). Figure 1 illustrates a box-and-whisker plot of total scores. Scores on the placement test were low in 2018 and have since increased, with a slight drop in 2021. Scores on achievement tests show slight differences every year.

The results indicate that compared to the placement test, the achievement test scores have increased in all years. Furthermore, 77.9% of the students scored 36 points or higher on the achievement test. High achievement test scores must indicate that all participating universities had introduced information literacy education for first-year students as part of their regular curriculum.
<table><caption>**Table 2.** Mean and standard deviation for the test conducted each year</caption><thead><tr><th></th><th>18PT</th><th>18AT</th><th>19PT</th><th>19AT</th><th>20PT</th><th>20AT</th><th>21PT</th><th>21AT</th></tr></thead><tbody><tr><td>N</td><td>1,322</td><td>2,284</td><td>3,070</td><td>1,870</td><td>3,325</td><td>2,078</td><td>2,387</td><td>750</td></tr><tr><td>Total</td><td>23.14</td><td>36.28</td><td>27.24</td><td>36.97</td><td>28.38</td><td>36.98</td><td>26.69</td><td>37.78</td></tr><tr><td>PS = 40</td><td>10.04</td><td>6.05</td><td>9.58</td><td>4.97</td><td>9.12</td><td>5.09</td><td>9.43</td><td>4.09</td></tr><tr><td>(a) Computer</td><td>7.52</td><td>9.55</td><td>8.35</td><td>9.75</td><td>8.62</td><td>9.61</td><td>8.25</td><td>9.69</td></tr><tr><td>PS = 10</td><td>2.90</td><td>1.38</td><td>2.42</td><td>1.01</td><td>2.17</td><td>1.16</td><td>2.37</td><td>0.90</td></tr><tr><td>(b) Internet</td><td>4.28</td><td>5.54</td><td>4.64</td><td>5.56</td><td>4.76</td><td>5.63</td><td>4.46</td><td>5.64</td></tr><tr><td>PS = 6</td><td>1.69</td><td>1.03</td><td>1.47</td><td>0.90</td><td>1.39</td><td>0.82</td><td>1.45</td><td>0.75</td></tr><tr><td>(c) Word</td><td>7.99</td><td>12.90</td><td>9.61</td><td>13.09</td><td>10.02</td><td>13.11</td><td>9.43</td><td>13.27</td></tr><tr><td>PS = 14</td><td>4.28</td><td>2.28</td><td>4.06</td><td>1.93</td><td>3.86</td><td>1.95</td><td>4.04</td><td>1.62</td></tr><tr><td>(d) Excel/Ppt</td><td>3.35</td><td>8.29</td><td>4.63</td><td>8.58</td><td>4.99</td><td>8.63</td><td>4.55</td><td>9.19</td></tr><tr><td>PS = 10</td><td>2.99</td><td>2.46</td><td>3.30</td><td>2.13</td><td>3.24</td><td>2.11</td><td>3.25</td><td>1.55</td></tr></tbody></table>

Note: Above: mean, below: standard deviation, PS: perfect score

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** A box-and-whisker plot of total scores</figcaption></figure>

## 3.3 Scores by Category

Table 2 indicates the mean and standard deviations for each category of the self-assessment checklist. Figure 2 illustrates a box-and-whisker plot of the scores for each category. Similar to the trend observed in the total scores, placement test scores in 2018 in each category were also lower than in the following years. In addition, achievement test scores were higher than placement test scores in all categories.

In categories (a) and (b), many students achieved high scores on the placement test. In category (a), 66.5% of the students scored 9 or 10 points, and in category (b), 60.0% of the students scored 5 or 6 points. The percentage of students scoring high on the achievement test increased by 93.7% in category (a) and 91.5% in category (b). These
<figure><img src="image_5.png" /><figcaption>Fig. 2. A box-and-whisker plot of the scores for each category</figcaption></figure>

results suggest that most students had already used a computer in some form or another by the time they entered university and were familiar with computer operations and internet/e-mail.

Scores on the placement test slightly varied in category (c). However, 88.3% of students scored 12 or higher on the achievement test. In category (d), placement test scores are considerably lower than in the other categories. The achievement test scores are also lower than the other categories (except 2021) but demonstrate a large increase in performance compared to the placement test.

Table 3 illustrates the items in category (d) for which the percentage of students who checked the item was particularly low. The percentages in the table are calculated by summing the responses over the four years. All items are related to the operations of Excel. However, even for these low percentages, the percentages of these items in the achievement test were very high.

<table><caption>Table 3. Items in category (d) with particularly low percentages in the placement test</caption><thead><tr><td>Item</td><td>PT</td><td>AT</td></tr></thead><tbody><tr><td>I can copy formulas using absolute cell reference so that the cell address does not move relatively when using AutoFill</td><td>21%</td><td>70%</td></tr><tr><td>I can change the number of decimal places to the specified number of digits (round off to the second decimal point, and so on)</td><td>29%</td><td>80%</td></tr><tr><td>I can paste a graph created in Excel onto a Word document</td><td>33%</td><td>87%</td></tr></tbody></table>
# 4 Conclusions

To improve the quality of information literacy education for first-year students, it is necessary to visualize their information literacy abilities and capture the changes after imparting the necessary education. Subsequently, we developed a basic ICT knowledge survey and a self-assessment checklist to receive quantitative feedback. The checklist measured the level of computer operation skills of students. A survey using the developed checklist was administered to approximately 17,000 students from six universities as a placement and achievement test.

The total scores on the placement test and the achievement test, as well as the scores for each category, were compared over four years. The results demonstrate that the placement test scores in 2018 were lower than in other years. The reason the results of the placement test after 2019 are better than those of 2018 is not clear. However, computer literacy education in high schools has improved against the background of the revision of the courses of study. In particular, considering the large increase in (c) Word and (d) Excel/PowerPoint scores after 2019, it is possible that more exploratory classes using this software took place in upper secondary schools.

In addition, the achievement test scores were markedly higher than the placement test in all years. By category, (d) Excel/PowerPoint had lower placement test scores than the other categories since some items had a response rate of between 20%–30%. However, even in category (d), scores on the achievement test were considerably higher. It can be said that students had already been familiar with computer operations by the time they entered university, and the first-year information literacy education programs were effective at the universities where the survey was conducted.

As mentioned in the introduction, we are simultaneously surveying basic ICT knowledge in addition to the survey using the computer operational skills checklist. In the future, it is necessary to analyze the data in conjunction with the basic ICT knowledge survey and the skills checklist survey. In addition, it is also necessary to refine the survey items to better reflect students’ information literacy skills.

# References

1. Nishino, K., Kayama, M., Fuse, I., Takahashi, S.: Investigations and considerations of university freshmen's knowledge of the subject "information". In: IEICE Technical Report, ET2006-41, vol. 106, no. 249, pp. 29–34. IEICE, Tokyo (2006)

2. Fukada, S., Nakamura, A., Okabe, S., et al.: Analysis on judgmental and behavioral aspects of information ethics among university students. Jpn. J. Educ. Technol. 37(2), 97–105 (2013)

3. Kaneko, D.: Improving the fundamental ICT knowledge survey for first-year university students and the results of five years in a Japanese University. In: Proceedings of E-Learn, pp. 1117–1122. AACE, San Diego (2017)

4. Kaneko, D., Ishida, Y., Omata, M., Yoshikawa, M., Koga, T.: Development of a self-evaluation checklist of computer operational skills for first-year university students. In: Proceedings of E-Learn, pp. 530–534. AACE, San Diego (2018)