483

ORIGINAL
Does student evaluation of on-demand video lectures vary withstudent characteristics? A questionnaire survey of medical
students at Tokushima University
Harutaka Yamaguchi1, 2, Seito Kagoshima3, Yoshihiro Okura4, Yoshinori Nakanishi4, Keisuke Kawahito1, and Kenji Tani4
Department of General Medicine, Tokushima University Graduate School of Biomedical Sciences, Tokushima, Japan, 2Department of General
Medicine, Taoka Hospital, Tokushima, Japan, 3Medical student, Tokushima University, Tokushima, Japan, 4Department of General Medicine
and Primary Care, Tokushima University Hospital, Tokushima, Japan
1

Abstract : The popularity of online classes in university education has grown rapidly since the COVID-19 outbreak. This study aims to examine the association between the detailed characteristics / various contexts of students and the evaluation of online classes. An online questionnaire survey administered to third- and fourthgrade medical students of Tokushima University in September 2021—with 101 (42.8%) valid responses—showed
the respondents’ general impressions, comprehension, convenience, and desire for online classes even after the
pandemic ends were significantly higher for the on-demand lectures than for simultaneous online classes. Notably, students who did not do other things while watching video lectures (10.9%) rated on-demand lectures as
facilitating significantly higher impression, comprehension, and convenience than those who did other things(89.1%). A multivariate analysis revealed that students who did not do anything else while watching, whose waking time changed slightly, who commuted to school for shorter periods, who were not good at morning lectures,and who watched at high speed rated the video lectures highly. Thus, video lectures are appreciated by students
who are committed to learning, which is a positive result for the future of on-demand education. J. Med. Invest.
70 : 483-493, August, 2023
Keywords : online medical education, on-demand video lecture, medical student, questionnaire survey

INTRODUCTION
By the end of 2019, coronavirus disease (COVID-19) had
spread globally and was still rampant with the continual emergence of new mutations. Notably, some institutions were providing online medical education even before COVID-19 (1-3).
However, when medical students were restricted from attending
colleges and universities in person owing to the pandemic, thepopularity of online medical education increased rapidly andsignificantly (4-7).
The primary features of online medical education include
online lectures, webcasting, virtual group discussions, videoconferences, virtual simulators, and telemedicine (8). Online
medical education has several advantages over traditional
medical education, including easy accessibility, a safe and comfortable environment, repeated access to lectures at convenience,time flexibility, time-saving, and increased independence andself-learning capacities of students (8, 9). Conversely, its drawbacks include isolation from and minimal interaction with peers,
dependence on good internet access and poor technical skills of
educators, lack of hands-on exposure, lack of direct assessment,
inability to acquire medical skills required of a physician, confidentiality issues, and family-related distractions (8, 10, 11). Numerous students consider classroom lectures suitable for online
medical education (12-14), but not bedside teaching and practicaltraining (12, 13), The two types of online lectures are asynchronous lectures and synchronous lectures (15). Asynchronous
Received for publication December 12, 2022 ; accepted August 17, 2023.
Address correspondence and reprint requests to Harutaka Yamaguchi,
MD, PhD, Department of General Medicine, Tokushima University
Graduate School of Biomedical Sciences, 3-18-15 Kuramoto-cho,
Tokushima 770-8503, Japan and Fax : +81-88-633-9687. Email :yharutaka@tokushima-u.ac.jp

lectures are pre-recorded video lectures that students can watch
repeatedly and at any time ; these lectures allow for high-speed
watching, skipping, and watching videos while doing otherthings. With asynchronous lectures, some students can improvetheir learning efficiency by repeatedly watching parts that are
difficult to understand (16), while others—with low motivation—might watch the video lectures while doing something else
or appreciate them only for convenience. What kind of students
appreciate online lectures? Li et al. analyzed factors associatedwith groups that were satisfied and unsatisfied with online
education and reported that teachers’ professional titles andstudents’ years of study, country of origin (an Asian or Africancountry), and location of current residence (inside or outside
China) significantly influenced students’ satisfaction (17). Songet al. reported that male students had higher learning interest,
attention, and efficacy than female students (18). However, no
studies have examined the impact of more detailed student attributes (e.g., commute time, the nature of enjoying conversation,etc.) on students’ perceptions of online classes.
Studies have reported that students’ rhythm of life changed
during COVID-19 (16, 19, 20). However, to the best of our knowledge, no study has considered assessing how the evaluationof online classes differs between students whose life rhythm
changed after COVID-19 and those whose life rhythm did notchange. If only students who stay up late and oversleep highly
evaluate online classes, there is a concern that the number of
online classes will increase in the future.
At Tokushima University, almost all classes were conducted
face-to-face until February 2020, but from March 2020, commuting to school was prohibited owing to COVID-19 ; thus, online
classes were introduced. As of September 2021, first and second
graders had only had an online learning experience, and all fifthand sixth graders had already finished attending their classroom lectures and were training in clinical practices. In contrast,

The Journal of Medical Investigation Vol. 70 2023

484

H. Yamaguchi, et al. Evaluating on-demand video lectures

third- and fourth-grade medical students had experienced bothface-to-face and online classes.
Therefore, we aim to examine whether there are differencesin online class evaluation based on student characteristics andstudents whose life rhythm changed after COVID-19 and those
whose did not by administering an online questionnaire to thirdand fourth-grade medical students at Tokushima University.

MATERIALS AND METHODS

Impact of COVID-19 on Tokushima University
The School of Medicine at Tokushima University has a six-year
course, with each year beginning in April and ending in March.
The program’s structure is as follows : lectures on general liberalarts and some specialized subjects in the first grade ; lectureson and practical experimentation in anatomy, physiology, biochemistry, and other subjects in the second grade ; lectures onand experimentation in pharmacology, microbiology, pathology,
and others in the third grade ; and lectures on clinical subjects
are held from January in the third grade to December in the
fourth grade. Since April 2020, most lectures have been delivered as on-demand videos, while only some were delivered in
the synchronous online class format. During the COVID-19
pandemic, online classes were introduced with a short preparation period ; therefore, most of the online classes other than thePBL (problem-based learning) tutorial were one-way by teacherswith audio recorded on PowerPoint (Microsoft) slides. Moreover,
all experiments and hands-on practice have been provided in an
on-demand format or as simultaneous online classes.

ParticipantsThis study was conducted with the approval of the Tokushima
University Hospital Ethics Committee. An online questionnaire
was administered to third-grade (112 ; 69 male and 43 female)and fourth-grade students (124 ; 70 male and 54 female). With
the approval of the Academic Affairs Division and the School of
Medicine of Tokushima University, we sent emails to 236 participants three times, on September 2, 8, and 15, 2021. Responses
from the participants were collected until September 21, 2021.
The questionnaire survey was conducted before the clinicalclerkship because it begins in January for fourth-grade students.
We added questionnaire briefing documents, including information on voluntary participation ; no personal information wascollected. Students were assured that they would not face any
disadvantage for not attempting the questionnaire ; they were
not rewarded for answering it either.

Questionnaire
Table 1 presents the English translation of the original Japanese questionnaire. Response time was assumed to be approximately seven minutes. The questionnaire included categories
such as gender, grade, living environment, commute time, andgood home internet environment and IT skills. The questionnaire also collected information on students’ personality attributes, such as having a sociable personality, engaging in
active conversation during breaks, and whether they are goodat morning classes. Questions for on-demand lectures included increasing the playback speed, skipping the lecture video,
watching the video repeatedly, and doing other things while
watching the video. We considered the last question slightly
sensitive for medical students ; therefore, they were free toprovide a conservative answer. Therefore, in this study, we
provided the choice of “seldom” next to “none” to detect whether
students were doing other things while watching video lectures. Furthermore, we used a multi-choice question to ask

regarding “other things” : email / SNS, watching TV / videos,playing games, reading books, and others. Additionally, thequestionnaire included questions regarding lifestyle changes
before and after the COVID-19 pandemic, such as wake-up time,
bedtime, physical condition, duration of using a smartphone,duration of reading books, duration of playing games, durationof watching TV / videos, daily time to exercise, frequency ofpart-time jobs, and frequency of participating in club activities.
For each of the on-demand video lectures / simultaneous online
classes / on-demand video experimentation / simultaneous online
experimentation, participants evaluated “general impression,”“comprehension,” and “convenience” as very bad, bad, good, orvery good, as well as answered “whether they hope for the sameformat” with strongly disagree, disagree, agree, or strongly
agree. We also inquired about “asynchronous online lectures”
and “synchronous online lectures.” Asynchronous online lectures are on-demand lectures that students can access at any
time, while synchronous online lectures are live online classes,
including PBL (problem-based learning) tutorials.

Statistical analysis
All statistical analyses were performed using IBM SPSS
Statistics ver. 27. Results were expressed as medians (interquartile range). The Wilcoxon signed-rank test was used tocompare on-demand and synchronized online classes in termsof general impression, comprehension, convenience, and whether they hoped for the same format. Additionally, the Wilcoxon
signed-rank test was used to compare changes in sleep andlifestyle before and after the COVID-19 pandemic. In the Wilcoxon analyses, we replaced the 4-point scale and 7-point scalewith the following numbers : “very bad 1, bad 2, good 3, very
good 4,” “strongly disagree 1, disagree 2, agree 3, strongly agree4,” and “none 1, 1 / month 2, 2 / month 3, 1 / week 4, 2 / week 5,
3 / week 6, more 7.” Regarding various situations of students(e.g., grade, gender, living environment, commute time, home
network environment, IT skills, sociable personality, active
conversation during breaks, and good at morning classes), theresponses to each item were divided into two groups (for mostof the items : agree or disagree ; how they watch on-demand lectures : none or others). For analyses, commute time was divided
into two groups in two ways : ≤ 5 minutes vs. > 5 minutes and ≤ 10
minutes vs. > 10 minutes. As for sleep, the time to wake up and
go to bed was divided into two groups : unchanged or earlier andone or more hours later. In each of the two groups, the Mann–
Whitney test was used to compare the general impressions,comprehension, convenience, and whether they hope to receive
the on-demand and the live online lectures in the same format.
A multiple regression analysis was performed using items withp < 0.2. A p-value of less than 0.05 was considered significant.
The sample size was examined using G * power (21). A totalof 94 responses were required when the effect size was 0.3 (moderate), the power was 0.8, and the significance level was lessthan 0.05. For 236 third- and fourth-grade medical students, thetarget number of responses would be met if the response rate
was 40% ; therefore, all third- and fourth-grade students weretargeted.

RESULTSA total of 105 medical students (52 male and 53 female) provided their responses. Three students did not respond to thequestion regarding the evaluation of on-demand video lectures,
whereas one student did not respond to the question regarding
the living environment. After excluding these four responses,
a total of 101 responses (a valid response rate of 42.8%) were

The Journal of Medical Investigation Vol. 70 August 2023

485

Table 1. Questionnaire
No.

Questions and choices

1

Please tell us your gender. - Male, Female, No answer

2

What is your grade? - third grade, fourth grade

3

What is your current living environment? - Parent’s house, Living alone, Other

4

Please tell us how long you spend commuting to school.
- ≤ 5, 5 < ≤ 10, 10 < ≤ 20, 20 < ≤ 30, 30 < (minutes)

5

Do you have a good network connection at home?
- strongly agree, agree, disagree, strongly disagree

6

Do you have good IT skills?
- strongly agree, agree, disagree, strongly disagree

7

Do you think you are sociable?
- strongly disagree, disagree, agree, strongly agree

8

Do you have active conversations during breaks?
- strongly disagree, disagree, agree, strongly agree

9

Are you good at morning classes?
- strongly disagree, disagree, agree, strongly agree

10

How often do you increase the playback speed?
- none, a quarter, half, almost all11

How often do you skip the lecture videos?
- none, a quarter, half, or almost all

12

Do you watch the videos repeatedly if they are difficult to understand?
- strongly disagree, disagree, agree, strongly agree

13

How often do you do other things while watching video lectures?
- none, seldom, sometimes, or often

14

Questions for those who answered “often or sometimes” in the previous question : What do you do during class? - use e-mail andSNS, read books, play games, watch TV or videos
We would like to ask you about the changes in your lifestyle before the COVID-19 pandemic and around May 2021, when you werebanned from posting at the university.

15

What is your wake-up time?
Before the COVID-19 pandemic. - 4, 5, 6, 7, 8, 9, 10, 11, 12 (o’clock)- 4, 5, 6, 7, 8, 9, 10, 11, 12 (o’clock)Around May 2021.

16

What time do you go to bed?
Before the COVID-19 pandemic. - 21, 22, 23, 24, 1, 2, 3, 4, 5 (o’clock)- 21, 22, 23, 24, 1, 2, 3, 4, 5 (o’clock)Around May 2021.

17

On average, over the course of a week, how much time do you spend using your smartphone per day?
Before the COVID-19 pandemic.
- (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)
Around May 2021. - (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)

18

On average, over the course of a week, how much time do you spend reading books per day?
Before the COVID-19 pandemic.
- (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)
Around May 2021. - (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)

19

On average, over the course of a week, how much time do you spend playing game (including time spent playing on smartphones)
per day?
Before the COVID-19 pandemic.
- (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)
Around May 2021. - (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)

20

On average, over the course of a week, how much time do you spend watching TV and videos (including on smartphones) per day?
Before the COVID-19 pandemic.
- (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)
Around May 2021. - (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)

21

On average, over the course of a week, how much time do you spend exercising per day?
Before the COVID-19 pandemic.
- (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)
Around May 2021. - (1) < 1, (2) 1 ≤ < 2, (3) 2 ≤ < 3, (4) 3 ≤ < 4, (5) 4 ≤ < 5, (6) 5 ≤ (hours)

486

H. Yamaguchi, et al. Evaluating on-demand video lectures

22

How often do you work part-time?
- (1) none, (2) 1 / month, (3) 2 / month, (4) 1 / week, (5) 2 / week, (6) 3 / week, (7) more

23

How often do you participate in club activities, including club activities outside the university?
- (1) none, (2) 1 / month, (3) 2 / month, (4) 1 / week, (5) 2 / week, (6) 3 / week, (7) more

24

Tell us about your physical condition.
Before the COVID-19 pandemic. - (1) very bad, (2) bad, (3) good, (4) very good
- (1) very bad, (2) bad, (3) good, (4) very goodAround May 2021
Let us ask you about online classes.

25

What is your general impression of on-demand video lectures compared to face-to-face classes?
- very bad, bad, good, very good26

How is your comprehension of on-demand video lectures compared to face-to-face classes?
- very bad, bad, good, very good27

How do you find the convenience of on-demand video lectures compared to face-to-face classes?
- very bad, bad, good, very good28

Do you hope for on-demand video lectures even if after the COVID-19 pandemic ends?
- strongly disagree, disagree, agree, strongly agree

29-40

The four questions above were asked in the same way, for simultaneous online classes, on-demand video experimentation, andsimultaneous online experimentation, using the same choices for each category.

analyzed.
In the final sample, there were 50 men and 51 women (theresponse rate was 37.4% for men and 54.6% for women) (Table2). Thus, the response rate of women was significantly higher(p = 0.011). There were 53 third graders and 48 fourth graders(the response rate was 47.3% for third graders and 38.7% for
fourth graders) ; thus, there was no grade-based bias (p = 0.59).
As for the living environment, approximately 69.3% of the students lived alone, of whom 30.7% lived with their parents. No
student chose the option “others.” The most frequent commutetime was 5 minutes or less, and 70% of the respondents chose
10 minutes or less. All students who commuted to school for 5
minutes or less lived alone, and 66 out of 72 students (91.7%) who
commuted to school for 10 minutes or less lived alone ; therefore,
there was a significant association between living environmentand commute time (both ≤ 5 vs. 5 < and ≤ 10 vs. 10 < ; p < 0.001).
Most students had a good internet connection at home, but only
about half of them were confident about their IT skills. Of the
41 students who evaluated themselves as having a sociable
personality, 40 responded that they had active break-time conversations, indicating a significant association (p < 0.001). Approximately 80% of students said they were not good at morningclasses.
Table 3 describes how students watched on-demand video
lectures. Most students (84.2%) watched the lectures at a higher playback speed, but a small proportion of students (31.7%)
skipped while watching the video. Most students (95.0%) responded that they watched hard-to-understand points repeatedly. No significant association was found between students
increasing playback speed and skipping or watching repeatedly(p = 0.97, 0.53, respectively) ; however, most students (89.1%)
did other things while watching video lectures. Multiple choicequestions regarding the breakdown of “other things” revealed
that 38 students (37.6%) used email / SNS, five students (5.0%)
watched TV / videos, five students (5.0%) played games, five
students chose others, and none of the students read books.
Furthermore, among the five students who watched TV / videos
and the five who played games, only one did both ; the remaining
eight responses were from eight different students. Similarly,
five students who chose “others” were different from those whochose watching TV / videos and playing games.

Table 4 presents the evaluation results of on-demand video lectures and simultaneous online classes. All the items, including
general impression, comprehension, convenience, and whether tohope for the same format in the future, were significantly highly
evaluated for on-demand video lectures. For on-demand video
lectures, more than 95% of the respondents rated the general impression, comprehension, and convenience as good or very good.
In contrast, 82.2% responded to the item “hope for the sameformat in the future” with “agree” or “strongly agree,” which
was lower than the previous three items. For on-demand video
experimentation and simultaneous online experimentation, thenumbers of respondents were 58 or 59 and 51 or 52, respectively.
These were about half of the respondents in the on-demand video
lectures and simultaneous online classes, resulting in a very low
response rate. Therefore, they were excluded from subsequent
analyses.
Table 5 presents the relationship between student evaluationof on-demand video lectures and various student attributes(grade, gender, living environment, commute time, IT skills, andhome network environment), students’ characteristics (sociable
personality, active conversation during breaks, and good atmorning classes), and how they watched on-demand video lectures. The evaluation of the general impression and convenienceof on-demand lectures by students living alone was significantly
higher compared to that of students living with their parents(p = 0.049 and 0.002, respectively), and they strongly hoped
for the same format in the future (p = 0.006). The evaluationof students who commuted to school for 10 minutes or lesswas significantly higher than that of students who commuted
to school for longer than 10 minutes (p = 0.007 and 0.023, respectively), and they hoped for the same format in the future(p = 0.023). The evaluation of students with 5 minutes or lesscommute time was significantly higher for all items comparedto that of students with more than 5 minutes of commute time(p < 0.001). Students who evaluated themselves as having a sociable personality significantly appreciated their comprehension
and convenience and hoped for the same format in the future(p = 0.002, 0.010, and 0.008, respectively). However, there wasno difference in general impressions between students who evaluated themselves as sociable and those who did not (p = 0.198).
Students who evaluated themselves as active conversationalists

487

The Journal of Medical Investigation Vol. 70 August 2023
Table 2. Students’ characteristics
Sex (n = 101)

Male

Female

50

51

n
(%)
Grade (n = 101)

(49.5)(50.5)

Grade 3

Grade 4

n

53

48

(%)

(52.5)(47.5)

Dwelling environment (n = 101)

Parents’ house

Single

Other

n

31

70

0

(%)

(30.7)(69.3)(0.0)≤5

5 < ≤ 10

10 < ≤ 20

Commute time (n = 101)

20 < ≤ 30

30 minutes <

n

39

33

13

10

6

(%)

(38.6)(32.7)(12.9)

(9.9)

(5.9)

Strongly Disagree

Disagree

Agree

Strongly Agree

Good home network environment (n = 99)
n

0

4

42

53

(%)

(0.0)

(4.0)

(42.4)(53.5)

Good at IT skills (n = 101)
n

5

45

36

15

(%)

(5.0)

(44.6)(35.6)(14.9)

n

11

47

26

15

(%)

(11.1)(47.5)(26.3)(15.2)

Sociable personality (n = 99)

Active conversation during breaks (n = 101)
n

8

27

42

24

(%)

(7.9)

(26.7)(41.6)(23.8)

n

8

13

17

63

(%)

(7.9)

(12.9)(16.8)(62.4)

Not good at morning lectures (n = 101)

Table 3. How to watch on-demand video lectures (n = 101)
Watch at high speed

None

A quarter

Half

Almost all

n

16

6

9

70

(%)

(15.8)

(5.9)

(8.9)

(69.3)None

A quarter

Half

Almost all

69

13

12

7

Skip
n
(%)
Watch repeatedly(68.3)(12.9)(11.9)(6.9)

Strongly disagree

Disagree

Agree

Strongly agree

n

2

3

33

63

(%)

(2.0)

(3.0)

(32.7)(62.4)Do other things while watching

None

SeldomSometimes

Often

n

11

42

37

11

(%)

(10.9)(41.6)(36.6)(10.9)

While 84% of students watched videos at high speed, only 31% of the students skipped video lectures ; 89% of students answeredthat they did something else while watching video lectures.

H. Yamaguchi, et al. Evaluating on-demand video lectures

488
Table 4.

Comparison of on-demand video lectures and simultaneous online classes

Generalimpression

Comprehension

Convenience

Hope for thesame format

Very bad

Bad

Good

Very goodOD(n = 101)

1
(1.0)

4
(4.0)

50
(49.5)

46(45.5)

ST(n = 101)

4
(4.0)

30
(29.7)

58(57.4)

9(8.9)

OD(n = 101)

1
(1.0)

4
(4.0)

50
(49.5)

46(45.5)

ST(n = 100)

3
(3.0)

31
(31.0)

59(58.4)

7(7.0)

OD(n = 101)

2
(2.0)

1
(1.0)

38
(37.6)

60(59.4)

ST(n = 101)

8
(7.9)

29
(28.7)

53(52.5)

11(10.9)

Strongly disagree

Disagree

Agree

Strongly agree

OD(n = 101)

3
(3.0)

15
(14.9)

35(34.7)

48(47.5)

ST(n = 100)18(18.0)

30(30.0)

42(42.0)

10(10.0)

p-value< 0.001< 0.001< 0.001< 0.001

The on-demand video lectures were highly evaluated by respondents in all four items (Wilcoxon signed-rank test).
OD : on-demand video lectures, ST : simultaneous online classes

Table 5.

Relationship between the evaluation of on-demand video lectures and various situations and students’ individualities
General impression

Comprehension

Convenience

Hope for the sameformat

Grade

0.420

0.213

0.773

0.805

Sex

0.972

0.488

0.231

0.974

0.049

0.262

Living environment
Parents’ house

3.0 (3.0–4.0)

Single3.5 (3.0–4.0)

Commute time A

0.007

0.227

0.002

0.006

3.0 (3.0–4.0)

3.0 (2.5–3.5)

4.0 (3.0–4.0)

4.0 (3.0–4.0)

0.023

0.023

≤ 10 minutes

4.0 (3.0–4.0)

4.0 (3.0–4.0)

4.0 (3.0–4.0)> 10 minutes

3.0 (3.0–3.5)

3.0 (3.0–4.0)

3.0 (2.0–4.0)< 0.001< 0.001< 0.001< 0.001≤ 5 minutes

4.0 (3.0–4.0)

4.0 (3.0–4.0)

4.0 (4.0–4.0)

4.0 (3.0–4.0)> 5 minutes

3.0 (3.0–4.0)

Commute time B

3.0 (3.0–4.0)

3.0 (3.0–4.0)

3.0 (3.0–4.0)

Home network environment

0.791

0.650

0.565

1.000

IT skills

0.976

0.369

0.459

0.979

Morning lectures

0.053

0.032

0.054

0.081

good at morning lectures

3.0 (3.0–4.0)

3.0 (3.0–4.0)

3.0 (3.0–4.0)

3.0 (2.0–4.0)

not good at morning lectures

3.0 (3.0–4.0)

4.0 (3.0–4.0)

4.0 (3.0–4.0)

3.5 (3.0–4.0)

0.154

0.013

0.367

Active conversation during breaks
scarce3.0 (3.0–4.0)

often
Sociable personality

4.0 (3.0–4.0)
0.198

no
yes
Watch at a high speed

0.046
no

3.0 (3.0–3.5)

yes

3.0 (3.0–4.0)

0.016
3.0 (2.5–4.0)
4.0 (3.0–4.0)

0.002

0.010

0.008

3.0 (3.0–4.0)

3.0 (3.0–4.0)

3.0 (3.0–4.0)

4.0 (3.0–4.0)

4.0 (3.0–4.0)

4.0 (3.0–4.0)

0.186

0.221

0.132

Watch by skipping

0.760

0.833

0.563

0.646

Watch repeatedly

0.602

0.666

0.513

0.602

Do other things while watching

0.011

0.013

0.025

0.093

none

4.0 (4.0–4.0)

4.0 (4.0–4.0)

4.0 (4.0–4.0)

4.0 (3.5–4.0)

yes (seldom ~ often)

3.0 (3.0–4.0)

3.0 (3.0–4.0)

4.0 (3.0–4.0)

3.0 (3.0–4.0)

The p-value of each item and the median value (interquartile range) for items less than p < 0.1 are shown (Mann–Whitney’s U test).
For analyses, we replaced the 4-point scale with the following numbers : “very bad 1, bad 2, good 3, very good 4” and “strongly disagree 1,
disagree 2, agree 3, strongly agree 4.”

489

The Journal of Medical Investigation Vol. 70 August 2023
during breaks significantly appreciated their comprehension
and hoped for the same format in the future (p = 0.013 and0.016, respectively). Students who were not good at morning
lectures evaluated their comprehension, general impression,and convenience of on-demand lectures higher than those who
were good (p = 0.032, 0.053, and 0.054, respectively) and tended
to hope for the same format in the future (p = 0.081). However,there was no significant relationship between students who were
not good at morning lectures and how they watched on-demand
lectures. The evaluation of the general impression of the lecturesby students who increased the playback speed was significantly
higher than that of those who did not (p = 0.046). Students who
did not do other things while watching on-demand lectures had
significantly higher evaluations of their general impressions,comprehension, and convenience compared to students who did
other things while watching the lectures (p = 0.011, 0.013, and0.025, respectively). Conversely, students who did not do otherthings while watching on-demand lectures tended to have ahigher evaluation of their hope for the same format in the futurethan those who did other things, but not significantly (p = 0.093).
Grade, gender, IT skills, and whether to skip or watch the lecture
video repeatedly did not have a significant effect on students’evaluations of the items.
Table 6 presents changes in lifestyles before and after the
COVID-19 pandemic. Wake-up time was 7.0 (7.0–8.0) hours(before) and 9.0 (8.0–0.0) hours (after), bedtime was 24.0 (24.0–
25.0) hours (before) and 26.0 (24.0–26.0) hours (after), respectively, with a median delay of 2 hours (both p < 0.001). Table 6
also includes changes in lifestyles before and after the COVID19 pandemic. Regarding other lifestyle habits, the durationof using smartphones, duration of reading books, duration ofplaying video games, duration of watching TV / videos, and thefrequency of part-time jobs increased significantly, while the frequency of club activities decreased significantly. Furthermore,
the daily time spent exercising tended to decrease. There was nochange in the physical condition by self-evaluation.
Regarding the relationship between changes in wake-up timeand bedtime before and after the COVID-19 pandemic and theevaluation of on-demand lectures, 16 students (16.0%) reported
no change or an earlier wake-up time, and 84 students (84.0%)
reported waking up one hour later. There were 36 students

Table 6.

(35.6%) who went to bed at the same time or earlier, and 65
students (64.4%) who went to bed one hour or more later. General impressions of on-demand lectures tended to be higher for
those who went to bed at the same time or earlier than those
who went to bed one hour later (p = 0.087). Post hoc analyses
compared students whose wake-up time and bedtime were delayed by 2 hours or more (n = 55 and 37, respectively) with those
whose wake-up time and bedtime did not change or become
earlier (n = 16 and 36, respectively). The analyses showed thatstudents whose wake-up time and bedtime were the same or earlier tended to give higher scores (p = 0.074, 0.086, respectively).
Regarding the degree of comprehension, convenience, and hopefor the same format using on-demand lectures, no relationshipwas found with changes in wake-up time and bedtime by 1 houror more.
There were a few significant differences in the evaluationof simultaneous online lectures (not shown in the table). The
students with a commute time of 10 minutes or less evaluatedtheir hope for the same format in the future highly (p = 0.044)
and tended to evaluate general impressions higher (p = 0.084),
and students who were not good at morning lectures tended toappreciate convenience (p = 0.071). Students who went to bed one
or more hours later tended to feel that online lectures were moreconvenient than those who did not (p = 0.068). All other items
had no significant effect at p > 0.1.
The results of the multiple regression analyses are shownin Table 7. Regarding general impressions, students who were
not good at morning lectures, students who watched videos athigh speed, students who had shorter commute time, students
who did not do other things while watching, and students whose
wake-up time changed slightly rated the lectures highly (Table
7A). Students who were more active during breaks, who had
shorter commute time, and who did not do other things while
watching rated comprehension higher (Table 7B). Furthermore,
students who evaluated themselves as sociable, who had shorter
commute time, and who did not do other things rated convenience higher (Table 7C). Regarding whether they hope for thesame format in the future, students who were less comfortablewith morning lectures and those who had shorter commutetimes rated it higher (Table 7D).

Changes in waking time, bedtime, and other lifestyle habits before and after the COVID-19 pandemic
Median (interquartile range)
pre

post

p value

Wake-up hour

7.0 (7.0 - 8.0)

9.0 (8.0 - 10.0)< 0.001

Bedtime hour

24.0 (24.0 - 25.0)

26.0 (24.0 - 26.0)< 0.001

Physical condition

3.0 (3.0 - 4.0)

3.0 (3.0 - 4.0)

0.869

Duration of using smartphones (hours / day)

4.0 (3.0 - 5.0)

5.0 (4.0 - 6.0)

< 0.001

Duration of watching TV/video (hours / day)

2.0 (2.0 - 3.0)

3.0 (3.0 - 4.0)

< 0.001

Duration of playing video games (hours / day)

1.0 (1.0 - 2.0)

1.0 (1.0 - 3.0)

0.002

Duration of reading books (hours / day)

1.0 (1.0 - 1.0)

1.0 (1.0 - 2.0)

< 0.001

Frequency of part-time jobs

4.0 (1.0 - 6.0)

5.0 (3.0 - 6.0)

< 0.001

Frequency of club activities

5.0 (1.0 - 6.0)

1.0 (1.0 - 3.0)

< 0.001

Daily time to exercise (hours / day)

2.0 (1.0 - 2.0)

1.0 (1.0 - 2.0)0.073

Wilcoxon signed-rank test.
For analyses, we replaced the 4-point scale of “Physical condition” with the following numbers : “very bad 1, bad 2, good3, very good 4” and “Frequency of part-time jobs/club activities” “none 1, 1 / month 2, 2 / month 3, 1 / week 4, 2 / week 5,
3 / week 6, more 7.”

H. Yamaguchi, et al. Evaluating on-demand video lectures

490
Table 7.

Multiple regression analysis

A) General Impression
95% confidence interval

Partial regression
coefficient

lower

upper

Commute time

-0.176-0.265-0.087< 0.001

Morning lectures

0.198

0.085

0.310

0.001

p value

Watch videos at high speed

0.142

0.003

0.049

0.234

Difference in wake-up time

-0.097-0.160-0.0330.003

Do other things while watching videos-0.152-0.281-0.023

0.021

Variables : living environment, commute time, morning lecture, active conversation, watch videos at high speed, do otherthings, sociable personality, difference in wake-up time, difference in bedtimeB) Comprehension

Active conversation

95% confidence interval

Partial regression
coefficient

lower

upper

0.195

0.056

0.334

p value
0.006

Commute time

-0.184-0.332-0.0360.015

Do other things while watching videos-0.116-0.218-0.0130.027

Variables : commute time, morning lecture, active conversation, sociable personality, watch videos at high speed, do otherthingsC) Convenience
95% confidence interval

Partial regression
coefficient

lower

upper

Commute time

-0.140-0.240-0.0410.006

Sociable personality

0.179

0.045

0.313

0.009

Do other things while watching videos-0.160-0.302-0.0180.028

p value

Variables living environment, commute time, morning lecture, sociable personality, do other thingsD) Hope for the same format
95% confidence interval

Partial regression
coefficient

lower

upper

Commute time

-0.193-0.322-0.0640.004

Morning lecture

0.183

0.021

0.345

0.028

p value

Variables : living environment, commute time, morning lecture, active conversation, sociable personality, watch videos at highspeed, do other things

DISCUSSION
We conducted an online questionnaire survey targeting thirdand fourth-grade medical students at Tokushima University
who have experienced both face-to-face traditional educationand online education. Generally, pre-recorded lectures wererated higher than live online classes. A notable finding in themultivariate analysis was that medical students who did not doother things while watching video lectures evaluated their general impression, comprehension, and convenience significantly
higher than those who did other things while watching. Furthermore, those with shorter commute times, those who were notgood at morning lectures, those who increased playback speed,
those who had smaller wake-up time differences, those who wereactive conversationalists during breaks, and those with sociable
personalities rated pre-recorded video lectures highly.
On-demand video lectures were significantly highly evaluated, and the number of students who hoped for the same format
in the future was as high as 82.2% (Table 4). Conversely, 52.0%of the respondents hoped for synchronous online classes in the
future. Stoehr et al. reported that almost no student wanted
online classes before COVID-19 (mean of 14%), but after the
COVID-19 pandemic, the number of students who hoped foronline learning increased (mean of 42%) (22). Furthermore,
Tayem et al. conducted a questionnaire survey with medical
students who were exposed to COVID-19 and found that only
28.6% wanted completely online classes after the pandemic,
whereas 73.3% of the students wanted partial online classes (23).

The Journal of Medical Investigation Vol. 70 August 2023
Al-Balas et al. (9) reported that 5.2% of students desired completee-learning, 75.6% desired a mixed format, and 19.4% desired
traditional learning, indicating most students wanted a hybrid
format. Kim et al. (24) reported that 62.2% of students preferred
online classes, whereas 29.5% preferred offline classes ; theyalso reported that, of the students who preferred online classes,
75.5% preferred recorded video lectures and 11.3% preferred live
online classes, indicating that an overwhelming majority preferred recorded video lectures. Additionally, they reported that
38.8% of students hoped for online classes in the future, evenafter the COVID-19 pandemic is over ; 45.5% hoped for blended
classes ; and 15.7% hoped for offline classes (24). In their report,
84.3% of students hoped for full or partially online classes in the
future. Our survey also showed that 82.2% of students hoped forpre-recorded video lectures in the future ; therefore, this result
agrees with that of Kim et al.
Students who did not do other things while watching video lectures rated on-demand lectures significantly higher than those
who did other things while watching the lectures (Table 5, Tables 7A-C). Many students wanted on-demand education ; however, students who evaluated on-demand education highly were
not those who were less engaged but those who could concentrateon learning. High appreciation of on-demand lectures, from students who do not do other things while watching, supports thecontinuation and development of future online classes. However,
31% of students skipped watching online classes, and 89% of students did other things while watching online classes. Furthermore, many students who do other things while watching online
classes have a relatively lower evaluation of online classes ; thus,
this result means that the quality of education in the current
online classes is low. Video learning is superior to face-to-facelectures in certain aspects, such as easy accessibility, a safe andcomfortable environment, time flexibility, and repeated access,but learning through video lectures has a drawback : students
are prone to concentration loss (8). Dong et al. claimed that themost important aspect of online lectures is attracting students(15). Therefore, educators will be required to devise more attractive lectures so that students do not do anything else while
watching video lectures or to encourage students to watch online
classes through interactive communication between teachersand students and formative evaluation of students’ level of understanding and attainment.
Active conversationalists during breaks and sociable students
highly rated comprehension and hope for the same format after
the COVID-19 pandemic ends (Table 5). In the multivariate
analyses, active conversationalists during breaks rated comprehension (Table 7B), whereas sociable students rated convenience(Table 7C). We assumed that students who enjoyed break-time
conversations would underestimate video lectures because
video lectures would reduce their chances of meeting friends
in the classroom, but the results were the opposite. Stoehr et al.
reported that medical students find it difficult to build relationships with classmates because online classes can lead to social
isolation (22). Thus, students who are not active conversationalists tend to be socially isolated and have a great disadvantage
when taking online classes. In contrast, students who are active
conversationalists could avoid “isolation” by actively meeting
their peers outside the classroom and contacting them throughvarious means, such as social media, thereby reducing the disadvantages of online classes. This may be the reason students whoare social and active conversationalists can appreciate online
classes. Nonetheless, extra care should be taken in online classes
to prevent isolation, especially for less sociable students.
Students with shorter commute times rated on-demand classes significantly higher on all four items of on-demand classes in
the multivariate analysis (Tables 7A-D). Regarding the living

491

environment, students living alone evaluated their general impression and convenience highly and significantly hoped for thesame format in the future (Table 5). Ibrahim et al. (25) asserted
that 59.2% of students agreed that online classes would savetime, while AlQhtani et al. (26) reported that 60.4% of students
found it convenient. Tayem et al. (23) also reported that 78.2%of students agreed that online lectures offer flexibility andconvenience. Students who spend a lot of time commuting toschool were expected to appreciate online classes, but the results
indicated the opposite. A possible reason is that all students who
commute to school for less than 5 minutes live alone—91.7%
of students with a commute time of fewer than 10 minutes live
alone. Concentrating on online lessons while living with family is challenging (11, 15) and as a result, students who spend
considerable time commuting to school, that is, students wholive with their families, may have a lower evaluation of online
classes. Another possible reason is that if students need to attend
lectures or experiments at the university, those living far awayfrom the university might not be able to return home in time andmight attend them in the university library. In that case, the advantages of online learning are not being utilized at all ; rather,its disadvantages increase. Therefore, to avoid this when using
both online and face-to-face classes, separating the days of online
and face-to-face classes is crucial.
Students who were not good at morning classes valued on-demand lectures highly, which is understandable. Conversely,
students who were not good at morning classes did not skip or
do other things. This was a statistically significant result ; thistime flexibility can be considered an advantage of video lectures,as students can access them at any time.
Many students had later wake-up times, consistent with previous studies (Table 6) (16, 19, 20). Interestingly, when comparing students whose wake-up time did not change or conversely
became earlier with those whose rhythms changed (i.e., woke uplater or went to bed later), the former rated on-demand classes
more highly than the latter. To the best of our knowledge, nostudies have investigated the relationship between changes indaily life rhythm and class evaluations, and the present study’s
findings shed a positive light on on-demand classes.
Compared to on-demand lectures, there were only a few differences in synchronous classes. Of the 60 items in total, only 4
had a difference of p < 0.1. Additionally, only one item showed asignificant difference at p < 0.05 (future prevalence of students
whose commute time was 10 minutes or less). If it were a true
difference, that would mean that people living nearby appreciate
online live classes as well as on-demand video classes. However,unlike the evaluation of on-demand classes, there was no difference between students with a commute time of 5 minutes orless and more than 5 minutes, and no difference was found forthe living environment either. As the p-value of the future prevalence of students whose commute time was 10 minutes or less
was also high at p = 0.044, the possibility of an accidental result
cannot be denied.

Limitations
There are several limitations to this study. First, the valid
response rate was only 42.8%. Second, there was a bias between
the male and female respondents. Both of these limitations canlead to self-selection bias ; therefore, future studies with highresponse rates are needed to confirm our findings. Third, thissurvey did not investigate the learning achievement of students
through online classes, so the educational effects of online
classes could not be verified. Fourth, this survey was performed
during the COVID-19 pandemic ; therefore, the student responses may be strongly influenced by a desire to ensure safety
against infectious diseases. Additionally, because the university

492

H. Yamaguchi, et al. Evaluating on-demand video lectures

decided on whether to provide learning as video lectures or ontime lectures, a lecture with an originally low evaluation may
have been given as a live lecture, and it may not be the result ofa simple comparison between video lectures and online live lectures. Nevertheless, the fact that students who did not do otherthings while video learning evaluated video lectures highly is
the most meaningful finding of this study.

Conclusion
We found that medical students generally highly rate online
education. It showed that video lectures were rated higher thansynchronous classes, and students who did not do other things
while watching video lectures evaluated these lectures highly
compared to those who did other things. Although this implies
that students who are committed to learning tend to appreciate
video lectures more than those who are less motivated, currently,
many students do other things during recorded video classes,
and some kind of improvement measures are required to conduct effective recorded video classes. To confirm these findings,
future studies need to conduct surveys with (1) higher response
rates, (2) RCTs that compare recorded video lectures and face-toface lectures with the same content, and (3) RCTs that compare
recorded video lectures and online live lectures with the samecontent.

CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.

ACKNOWLEDGMENTS
We wish to thank all the participants and the staff membersof the Academic Affairs Division of the Department of Medicineof Tokushima University for their contributions, as well as Ms.
Yayoi Tagawa for her valuable secretarial support.

REFERENCES
1.

2.

3.

4.

5.

6.

Tang B, Coret A, Qureshi A, Barron H, Ayala AP, Law
M : Online lectures in undergraduate medical education : scoping review. JMIR Med Educ 10 : e11, 2018.
doi : 10.2196/mededu.9091
Pei L, Wu H : Does online learning work better than offline
learning in undergraduate medical education? A systematic
review and meta-analysis. Med Educ Online 24 : 1666538,
2019. doi : 10.1080/10872981.2019.1666538
Petit RK, Kinney M, McCoy L : A descriptive, cross-sectional study of medical student preferences for vodcast design,format and pedagogical approach. BMC Med Educ 17 : 89,
2017. doi : 10.1186/s12909-017-0926-z
Abbasi MS, Ahmed N, Sajjad B, Alshahrani A, Saeed S,
Sarfaraz S, Abduljabbar T : E-learning perception and satisfaction among health sciences students amid the COVID19 pandemic. Work 67 : 549-556, 2020. doi : 10.3233/
WOR-203308
Garg M, Eniasivam A, Satterfield J, Norton B, Austin E,
Dohan D : Rapid transition of a preclinical health systems
science and social justice course to remote learning in thetime of coronavirus. Med Educ Online 25 : 1812225, 2020.
doi : 10.1080/10872981.2020.1812225
Nimavat N, Singh S, Fichadiya N, Sharma P, Patel N,
Kumar M, Chauhan G, Pandit N : Online medical education in India - different challenges and probable solutionsin the age of COVID-19. Adv Med Educ Pract 12 : 237-243,
2021. doi : 10.2147/AMEP.S295728
7. Salari F, Sepahi V : Challenges of virtual medical sciences
education during the COVID-19 pandemic : a systematic
review. Educ Res Med Sci 10 : e117948, 2021. doi : 10.5812/
erms.117948
8. Althwanay A, Ahsan F, Oliveri F, Goud HK, Mehkari Z,
Mohammed L, Javed M, Rutkofsky IH : Medical education, pre- and post-pandemic era : a review article. Cereus
12 : e10775, 2020. doi : 10.7759/cureus.10775
9. Al-Balas M, Al-Balas HI, Jaber HM, Obeidat K, Al-Balas H,
Aborajooh EA, Al-Taher R, Al-Balas B : Distance learning inclinical medical education amid COVID-19 pandemic in Jordan : current situation, challenges, and perspectives. BMC
Med Educ 20 : 341, 2020. doi : 10.1186/s12909-020-02257-4
10. Wilcha RJ : Effectiveness of virtual teaching during the
COVID-19 crisis : systematic review. JMIR Med Educ
6 : e20963, 2020. doi : 10.2196/20963
11. Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair
L : Perceptions of medical students towards online teaching
during the COVID-19 pandemic- a national cross-sectional survey of 2721 UK medical students. BMJ Open
10 : e042378, 2020. doi : 10.1136/bmjopen-2020-042378
12. Tashkandi E : E-learning for undergraduate medical students. Adv Med Educ Pract 12 : 665-674, 2021. doi : 10.2147/
AMEP.S314509
13. Ikonne U, Campbell AM, Whelihan KE, Bay RC, Lewis
JH : Exodus from the classroom : student perceptions, lecture capture technology, and the inception of on-demandpreclinical medical education. J Am Osteopath Assoc
118 : 813-823, 2018. doi : 10.7556/jaoa.2018.174
14. Yagi S, Fukuda D, Ise T, Yamaguchi K, Kusunose K,
Kadota M, Kawabata Y, Matsuura T, Soga T, Yamada
H, Soeki T : Clinical clerkship students’ preferences andsatisfaction regarding online lectures during the COVID19 pandemic. BMC Med Educ 22 : 43, 2022. doi : 10.1186/
s12909-021-03096-7
15. Dong C, Lee DW, Aw DC : Tips for medical educators on
how to conduct effective online teaching in times of social
distancing. Proceedings of Singapore Healthcare 30 : 59-63,2021. doi.org/10.1177/2010105820943907
16. Nishimura Y, Ochi K, Tokumasu K, Obika M, Hagiya
H, Kataoka H, Otsuka F : Impact of the COVID-19 pandemic on the psychological distress of medical students inJapan : cross-sectional survey study. J Med Internet Res
23 : e25232, 2021. doi : 10.2196/25232
17. Li W, Gillies R, He M, Wu C, Liu S, Gong Z, Sun H : Barriers and facilitators to online medical and nursing education
during the COVID-19 pandemic : Perspectives from international students from low- and middle-income countries
and their teaching staff. Hum Resour Health 19 : 64, 2021.
doi : 10.1186/s12960-021-00609-9
18. Song Y, Wang S, Liu Y, Liu X, Peng A : Online education at
the medical School of Tongji University during the COVID19 pandemic- a cross-sectional study. BMC Med Educ
21 : 512, 2021. doi:10.1186/s12909-021-02951-x
19. Romero-Blanco C, Rodríguez-Almagro J, Onieva-Zafra
M D, Parra-Fernández M L , Prado-Laguna M DC,
Hernández-Martínez A : Sleep pattern changes in nursing
students during the COVID-19 lockdown. Int J Environ Res
Public Health 17 : 5222, 2020. doi : 10.3390/ijerph17145222
20. Ji Y, Shi Y, Zhou J, Li X, Qin R, Zhu Q : Analysis on thechange of college students’ life pattern and its impact during
the COVID-19 outbreak in China. Am J Health Behav
46 : 218-230, 2022. doi : 10.5993/AJHB.46.3.2
21. Faul F, Erdfelder E, Lang AG, Buchner A : G*Power 3 : A

The Journal of Medical Investigation Vol. 70 August 2023
flexible statistical power analysis program for the social,
behavioral, and biomedical sciences. Behav Res Methods
39 : 175-191, 2007. doi:10.3758/bf03193146
22. Stoehr F, Müller L, Brady A, Trilla A, Mähringer-Kunz
A, Hahn F, Düber C, Becker N, Wörns MA, Chapiro J,
Hinrichs JB : How COVID-19 kick-started online learning in medical education—the DigiMed study. PLoS One
16 : e0257394, 2021. doi : 10.1371/journal.pone.0257394
23. Tayem YI, Almarabheh AJ, Abo Hamza E, Deifalla A : Perceptions of medical students on distance learning during the
COVID-19 pandemic- a cross-sectional study from Bahrain.
Adv Med Educ Pract 13 : 345-354, 2022. doi : 10.2147/
AMEP.S357335
24. Kim JW, Myung SJ, Yoon HB, Moon SH, Ryu H, Yim
JJ : How medical education survives and evolves during

493

COVID-19 : our experience and future direction. PLoS One
15 : e0243958, 2020. doi : 10.1371/journal.pone.0243958
25. Ibrahim NK, Al Raddadi R, AlDarmasi M, Al Ghamdi
A, Gaddoury M, AlBar HM, Ramadan IK : Medical students’ acceptance and perceptions of e-learning during theCovid-19 closure time in King Abdulaziz University, Jeddah. J Infect Public Health 14 : 17-23, 2021. doi : 10.1016/j.
jiph.2020.11.007
26. AlQhtani A, AlSwedan N, Almulhim A, Aladwan R, Alessa
Y, AlQhtani K, Albogami M, Altwairqi K, Alotaibi F,
AlHadlaq A, Aldhafian O : Online versus classroom teaching for medical students during COVID-19 : measuring effectiveness and satisfaction. BMC Med Educ 21 : 452, 2021.
doi : 10.1186/s12909-021-02888-1