Session 4: Conversation Design

HAI ’19, October 6–10, 2019, Kyoto, JapanThe Design Method of the Virtual Teacher
Tetsuya Matsui

Seiji Yamada

Seikei university
t-matsui@st.seikei.ac.jp

National Institute of Informatics and SOKENDAI
seiji@nii.ac.jp

ABSTRACTeffective than those that gave lessons randomly [12]. Hastie et al.
showed that virtual tutors that remembered the last lesson increased
the students’ ability to learn; however, these tutors were perceived
as less likeable by students [6]. Scassellati et al. suggested a multiagent language education system that included both a real robot
and a virtual agent for deaf children [17].
However, robot teachers have some limitations. Kennedy et al.
showed that social robot tutoring positively affected students’ learning; however, applying social and adaptive behaviour to a robot hada negative effect on students’ learning [7]. Edwards et al. showedthat students felt a “teacher as robot" (telepresence robot operated
by a human teacher) to be more credible than a “robot as teacher"(autonomous robot teacher)[2]. These pieces of research showed
the difficulty of designing credible and effective robot teachers.
In this paper, one thing we focused on was the subject on which
VTs give a lesson. It was shown that robot teachers that taught science [19][15] and programming [13] were effective. These results
seemed to be due to humans expecting that robots have a copious
amount of knowledge on science, engineering and computer science. Also, robots that teach language positively affected students[8][11]. In the real world, many robot-like characters were usedin the education TV program. Figure 1 showed these charactersused in NHK(Japan Broadcasting Corporation)1 education program.
Left is "Garapiko", in "Okaasan-to-issyo"2 . Right is "Ra-bit-kun", in"Bitworld"3 .

Using robots and virtual agents as teachers in education is one of
the most important fields in HAI. Many pieces of work have been
published; however, little has been reported on the relationshipbetween the subject on which a virtual teacher (VT) gives a lesson
and the appearance of the VT. For example, are robot-like agentsusually effective regardless of the subject being taught? In this paper,
we hypothesized that the subject and the appearance of the VT
affect students’ level of understanding through the interaction of thetwo. To verify this, we conducted an experiment with two factors:
subject and VT appearance. Under all conditions, the participants
watched movies in which VTs gave a lesson, and they took a shorttest on the lesson and provided a subjective evaluation on the VTs.
As a result, the subject and appearance affected the short-test scores
through the interaction of the two. This result suggests a novel
design method that can be used to construct a VT.

CCS CONCEPTS• Computer systems organization → Embedded and cyberphysical systems; • Robotics → External interfaces for robotics.

KEYWORDS
virtual teacher, robot teacher, educational robotics, virtual agent,educational support system, appearance
ACM Reference Format:
Tetsuya Matsui and Seiji Yamada. 2019. The Design Method of the Virtual
Teacher. In Proceedings of the 7th International Conference on Human-AgentInteraction (HAI ’19), October 6–10, 2019, Kyoto, Japan. ACM, New York, NY,
USA, 5 pages. https://doi.org/10.1145/3349537.3351906

1

INTRODUCTION

In recent years, robots and virtual agents have played a social rolein the real world. “Teacher" is one of these roles. To overcome a
global teacher shortage, artificial intelligence in education (AIED)and robot teachers have been researched [3]. Edwards et al. showedthat students could accept robots as teachers, and they consideredrobots to be culturally neutral [4].
Methods of designing robot teachers and virtual teachers (VTs)have been widely researched. Leyzberg et al. showed that robot
teachers that gave personalized lessons to each student were more

Figure 1: The robot-like characters in NHK’s education program. Citation sources are shown in footnote.
However, little has been reported on which subject was suitable
for robot teachers. For example, is humanities (history, psychology
and philosophy) a suitable subject for robot teachers? Thus, we
propose two hypotheses:• H1: The subject and the VT appearance affect students’ levelof understanding through the interaction of the two.
• H2: A robot-like agent that teaches robot history is more
effective than a robot-like agent that teaches humanities.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
HAI ’19, October 6–10, 2019, Kyoto, Japan© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6922-0/19/10. . . $15.00
https://doi.org/10.1145/3349537.3351906

1 https://www.nhk.or.jp/
2 cited from https://www.nhk-sc.or.jp/studiopark/shop/goods/?program=207id=003
3 cited from https://www.nhk.or.jp/bitworld/about/

97

Session 4: Conversation Design

HAI ’19, October 6–10, 2019, Kyoto, JapanTo verify these hypotheses, we conducted an experiment with
two kinds of VTs. One VT had a human-like appearance and theother a robot-like one. It was reported that the appearance of virtual agents and robots affected their effect on people and the impressions they gave. Terada et al. showed that differences in appearance affected how effective product recommendation agentsare[18]. Many pieces of prior work have shown that the gender ofvirtual agents as shown by their appearance affected users’ decisionmaking [14][5][9]. Also, the skin colour of agents affected users’
impressions of them [16]. Komatsu et al. showed that the size andposition of robots’ facial parts affected whether users judged a robot
to be good at some social task (including education) or not [10]. In
this research, we also focused on the difference in VT appearance
and aimed to reveal the relationship between a subject being taught
and the appearance of a VT.

These questions were to measure the participants’ subjective
evaluation of the lessons. The participants answered all questions
on a seven-point Likert scale. We used the average of the scoresof Q1 and Q2 as a scale of “understandability perceived". We used
the average of the scores of Q3 and Q4 as a scale of “increased
motivation to learn". Also, we used the score of Q5 as a scale of“perceived credibility of VTs".
We conducted experiments on the web. All participants wererecruited from Yahoo! Crowd Sourcing4 and paid 80 yen (about
73 cents) as a reward. We obtained informed consent from all participants. Crump et al. showed the validity of the crowd sourcing
service for academic research[1].

2Before the main experiment, we conducted a preliminary experiment on the web. In the experiment, the participants read thelesson text instead of watching the lesson movie. In condition 1, theparticipants read the text on human history (see also Table 1). In
condition 2, they read the text on robot history. The lesson text was
the same as the text that the VTs spoke in the main experiment. The
questionnaires and other procedures were the same as the main
experiment. We conducted this preliminary experiment to confirm
that there was no difference between the short-test scores and otherscores for each text. For condition 1, we recruited 95 participants;
there were 63 males and 32 females ranging in age from 24 to 63
years for an average of 42.5 (SD = 8.0). For condition 2, we recruited
92 participants; there were 60 males and 32 females ranging in age
from 24 to 73 years for an average of 43.9 (SD = 10.1).
We conducted a two-way ANOVA on each score. As a result,
there were no significant differences for any score. Thus, we concluded that the two pieces of text did not have any difference interms of these scores. This showed that the significant differencesthat were observed in the main experiment were not derived froma difference in speech text.

2.1

EXPERIMENTAL DESIGN

We introduced two factors, and each factor had two levels. One
factor was the subject that the virtual teachers (VTs) taught: humanhistory (Ottoman Empire) and robot history. Table 1 shows someof the sentences that the VTs spoke for each level. The other factor
was the appearance of the VTs: human-like and robot-like. We used
our original human agent with a female-likeness and robot agent
as VTs. Figure 2 shows the VTs that we used in the experiment.

Figure 2: VTs that we used in experiment

2.2

Preliminary Experiment (Manipulation
Check)

Main Experiment

In the main experiment, for condition 1, we recruited 74 participants; there were 53 males and 21 females ranging in age from15 to 65 years for an average of 41.7 (SD = 9.9). For condition 2,
we recruited 74 participants; there were 56 males and 18 females
ranging in age from 25 to 72 years for an average of 44.7 (SD = 10.7).
For condition 3, we recruited 77 participants; there were 46 males
and 31 females ranging in age from 20 to 69 years for an average of
43.1 (SD = 9.8). For condition 4, we recruited 83 participants; there
were 54 males and 29 females ranging in age from 17 to 74 years
for an average of 43.4 (SD = 12.6).

Table 2 shows the conditions used in our experiment. Under
all conditions, the participants watched movies in which the VTs
conducted a lesson. All movies were about 240 seconds long.
Under all conditions, all participants took a short test and answered questionnaires after watching the movies. The short test
included six questions that asked questions regarding the lessons
that the VTs gave. All questions were four-choice questions. We
used the score of this short test as a scale to measure the effect of
the lessons. Table 3 shows examples of the short test.
Other questions were as follows.

3• Q1: How easy did you feel that the lesson was to understand?
• Q2: How much did you feel that you understood the lesson?
• Q3: How much interest did you have in the topic of the lesson
after watching the lesson?
• Q4: How much did you want to know more about the topic
in the lesson?
• Q5: How credible did you feel that the teacher in the lesson
was?

RESULT

Figure 3 shows the averages of the scores for the short test foreach condition. The error bars mean standard error. We conducted
a two-way ANOVA on the scores for the short test. The results
were subject: F (1, 304) = 3.442, p = 0.0645, VT: F (1, 304) = 4.890,
p = 0.0278 and subject × VT: F (1, 304) = 4.513, p = 0.0344. There
4 https://crowdsourcing.yahoo.co.jp/

98Session 4: Conversation Design

HAI ’19, October 6–10, 2019, Kyoto, Japan

Table 1: Examples of speech text for each subject level
Human history

Robot history

The Ottoman Empire was an empire that ruled over a very large territory, and its capital was in Turkey.
Osman I, who was the founder of this empire, was a legendary and mysterious historical figure.
In 1453, the Ottoman Empire defeated the Eastern Roman Empire.
The Ottoman Emperor at that time was Mehmet II.
The word “robot" was coined by Karel Capek, a Czech novelist.
The world’s first humanoid robot was Eric, and he appeared in 1928.
After WW2, industrial robots were introduced in factories all over the world.
The world’s first company to introduce industrial robots was the General Motors Company.

Table 2: Conditions in experimentcondition
condition 1
condition 2
condition 3
condition 4

subject
human history
human history
robot history
robot history

history, VT: robot-like) was significantly lower than the other conditions. The average of the short-test scores for condition 1 (subject:human history, VT: human-like) was higher than condition 2. Inthese two conditions, the subject was human history, and the VTs
were different. Also, the average of the short-test scores for condition 3 (subject: robot history, VT: human-like) was almost the same
as condition 4 (subject: robot history, VT: robot-like). In these two
conditions, the subject was robot history, and the VTs were different. The average for condition 4 was higher than condition 2. Inthese two conditions, the VT was robot-like, and the subjects weredifferent. Also, the average for condition 1 was almost the same
as condition 3. In these two conditions, the VT was human-like,and the subjects were different. These results shows that the bothof our hypotheses, H1 and H2, were supported by the experiment.
The subject and the VTs’ appearance affect the level of understandthrough the interaction of the two, and a robot-like VT that teaches
robot history is more effective than a robot-like VT that teaches
human history.
From these results, we concluded:

virtual teacher (VT)human-like VT
robot-like VT
human-like VT
robot-like VT

were significant differences in interaction; thus, we conducted asimple main effect test. The results were subject when VT washuman-like agent: F (1, 304) = 0.036, p = 0.8492, subject when
VT was robot-like agent: F (1, 304) = 7.920, p = 0.0052, VT whensubject was human history: F (1, 304) = 9.399, p = 0.0024 and VT
when subject was robot history: F (1, 304) = 0.004, p = 0.9511.

• The subject and the VTs’ appearance affected the short-testscores through the interaction of the two.
• The human-like VT had an advantage over the robot-like
VT when the subject was humanities.
• The robot-like agent’s effect was lower than the humanlike agent when the subject was humanities; however, thisdisadvantage was not observed when the subject was robot
history.

Figure 3: Average of short-test scores for each condition

These conclusions suggest methods for designing VTs and for
education in which virtual agents or robots are used as teachers.
We showed that the effect of the robot-like agent depends on thesubject. If VTs give a lesson on humanities (for example, historicalscience), a human-like VT will be more desirable than a robot-likeagent. If the aim is to use robot-like VTs, it is desirable that the subject be computer science (for example, robot science). Edwards etal. reported that students considered robots to be culturally neutral[4]; however, our result suggested that the robots were not completely neutral among academic fields. This result also suggests thatusing robots or robot-like agents regardless of subject is probably
unreasonable.
Table 4 showed that there were no significant differences for“understandability perceived", “increased motivation to learn" and“perceived credibility of VTs". These results showed that subject
and VT did not affect these scales in this experiment. There wasTable 4 shows the averages and SDs of the scores for “understandability perceived", “increased motivation to learn" and “perceived
credibility of VTs". We conducted a two-way ANOVA on thesescores. As a result, there were no significant differences for eachscore.

4

DISCUSSIONIn Figure 3, there was a significant main effect for VT (p < 0.05).
Also, there was a marginal difference for the main effect for subject(p < 0.07). However, these main effects were superseded by subject
× VT interaction.
From the sub-effect tests, there were significant simple main
effects for subject when the VT was the robot-like agent (p < 0.01)
and for VT when the subject was human history (p = 0.01). The
average of the short-test scores for condition 2 (subject: human99Session 4: Conversation Design

HAI ’19, October 6–10, 2019, Kyoto, Japan

Table 3: Examples from short test
human history
robot history

Who was the Ottoman Emperor during the peak period of the Ottoman Empire?
A: Ottoman I
B: Mehmet II
C: Serim I
D: Sulayman I
Where was the world’s first completely upright bipedal walking humanoid robot developed?
A: the USA
B: the UK
C: the Czech Republic
D: Japan

Table 4: Averages and SDs of each scale

understandability
perceived

increased motivation
to learn

perceived credibilityof VTscondition
condition 1
condition 2
condition 3
condition 4
condition 1
condition 2
condition 3
condition 4
condition 1
condition 2
condition 3
condition 4

average
5.01
5.01
5.19
5.10
4.78
4.54
4.77
4.84
4.68
4.53
4.70
4.61

appearance affected the short-test score through the interaction of
the two, and the robot-like VT effect when giving the humanities
lesson was significantly lower than the other conditions. These
results showed that we should choose the appearance of a VT inconsideration of the subject that the VT will teach. Also, the robotlike VT had a limitation in terms of effect.

SD
1.01
1.24
1.09
1.29
1.23
1.29
1.09
1.29
1.10
0.96
1.03
1.06

ACKNOWLEDGMENTS
This research was partially supported by JSPS KAKENHI “Cognitive
Interaction Design" (No. 32626118005).

REFERENCES[1] Matthew JC Crump, John V McDonnell, and Todd M Gureckis. 2013. Evaluating
Amazon’s Mechanical Turk as a tool for experimental behavioral research. PloS
one 8, 3 (2013), e57410.
[2] Autumn Edwards, Chad Edwards, Patric R Spence, Christina Harris, and Andrew
Gambino. 2016. Robots in the classroom: Differences in students’ perceptionsof credibility and learning between “teacher as robot” and “robot as teacher”.
Computers in Human Behavior 65 (2016), 627–634.
[3] Bosede I Edwards and Adrian D Cheok. 2018. Why not robot teachers: Artificial
intelligence for addressing teacher shortage. Applied Artificial Intelligence 32, 4(2018), 345–360.
[4] Bosede I Edwards, Idris O Muniru, Nosiba Khougali, Adrian D Cheok, and Rui
Prada. 2018. A Physically Embodied Robot Teacher (PERT) as a Facilitator forPeer Learning. In 2018 IEEE Frontiers in Education Conference (FIE). 1–9.
[5] Rosanna E Guadagno, Jim Blascovich, Jeremy N Bailenson, and Cade Mccall. 2007.
Virtual humans and persuasion: The effects of agency and behavioral realism.
Media Psychology 10, 1 (2007), 1–22.
[6] Helen Hastie, Mei Yii Lim, Srini Janarthanam, Amol Deshmukh, Ruth Aylett,Mary Ellen Foster, and Lynne Hall. 2016. I remember you!: Interaction with memory for an empathic virtual robotic tutor. In Proceedings of the 2016 international
conference on autonomous agents & multiagent systems. 931–939.
[7] James Kennedy, Paul Baxter, and Tony Belpaeme. 2015. The robot who tried too
hard: Social behaviour of a robot tutor can negatively affect child learning. In
2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).
67–74.
[8] James Kennedy, Paul Baxter, Emmanuel Senft, and Tony Belpaeme. 2016. Social
robot tutoring for child second language learning. In The Eleventh ACM/IEEEInternational Conference on Human Robot Interaction. 231–238.
[9] Yanghee Kim, Amy L Baylor, and E Shen. 2007. Pedagogical agents as learningcompanions: the impact of agent emotion and gender. Journal of Computer
Assisted Learning 23, 3 (2007), 220–234.
[10] Takanori Komatsu and Masahiro Kamide. 2017. Designing robot faces suited
to specific tasks that these robots are good at. In 2017 26th IEEE International
Symposium on Robot and Human Interactive Communication (RO-MAN). 1–5.
[11] Sungjin Lee, Hyungjong Noh, Jonghoon Lee, Kyusong Lee, Gary Geunbae Lee,
Seongdae Sagong, and Munsang Kim. 2011. On the effectiveness of robot-assisted
language learning. ReCALL 23, 1 (2011), 25–58.
[12] Daniel Leyzberg, Aditi Ramachandran, and Brian Scassellati. 2018. The Effect of
Personalization in Longer-Term Robot Tutoring. ACM Transactions on HumanRobot Interaction (THRI) 7, 3 (2018), 19.
[13] Sarah Müller, Bianca Bergande, and Philipp Brune. 2018. Robot Tutoring: On the
Feasibility of Using Cognitive Systems as Tutors in Introductory Programming
Education: A Teaching Experiment. In Proceedings of the 3rd European Conferenceof Software Engineering Education. 45–49.
[14] Jeunese Payne, Andrea Szymkowiak, Paul Robertson, and Graham Johnson. 2013.
Gendering the machine: Preferred virtual assistant gender and realism in selfservice. In International Workshop on Intelligent Virtual Agents. 106–115.
[15] Alex Polishuk and Igor Verner. 2017. An elementary science class with a robot
teacher. In International Conference on Robotics and Education RiE 2017. 263–273.
[16] Brent Rossen, Kyle Johnsen, Adeline Deladisma, Scott Lind, and Benjamin Lok.
2008. Virtual humans elicit skin-tone bias consistent with real-world skin-tone

no relationship between the short-test scores and these scores inTable 4.
Last, this research has some limitations. First, we used only two
subjects and two kinds of VTs. Other subjects and VTs would be
important, for example, the effect of the gender of a human-like VT.
Second, we used only virtual agents as the VTs, not a real robot.
Third, in this research, we did not observe the effects of the subject
and the VTs’ appearance on the participants’ subjective evaluation
scores (as shown in Table 4). These scores were not relevant tothe short-test score; however, increased motivation to learn andthe credibility of VTs seemed to be important in gaining students’interest and making them continue to study. Thus, a design method
for increasing these scores seems to be required. Last, we did notconsider the age and gender of the participants. These are our future
work.

5

CONCLUSION

In this paper, we aimed to suggest a design method for VTs. We
hypothesized that there was an interaction between the subject onwhich VTs give a lesson and the appearance of the VTs. Also, wehypothesized that robot-like VTs that teach robot history would be
more effective than robot-like VTs that teach humanities. To verify
these hypotheses, we conducted an experiment with two factors,
and each factor had two levels. One factor was the subject beingtaught, and the levels were human history and robot history. The
other factor was the VTs’ appearance, and the levels were humanlike and robot-like. In the experiment, the VTs gave a lesson to theparticipants, and the participants took a short test regarding thelesson and answered a questionnaire after watching the lesson. We
conducted a two-way ANOVA on the short-test score and otherscores. As a result, it was revealed that the subject and the VTs’100

Session 4: Conversation Design

HAI ’19, October 6–10, 2019, Kyoto, Japan[18] Kazunori Terada, Liang Jing, and Seiji Yamada. 2015. Effects of agent appearance
on customer buying motivations on online shopping sites. In Proceedings of the
33rd annual acm conference extended abstracts on human factors in computing
systems. 929–934.
[19] Igor M Verner, Alex Polishuk, and Niv Krayner. 2016. Science class with RoboThespian: using a robot teacher to make science fun and engage students. IEEE
Robotics & Automation Magazine 23, 2 (2016), 74–80.

biases. In Intelligent virtual agents. 237–244.
[17] Brian Scassellati, Jake Brawer, Katherine Tsui, Setareh Nasihati Gilani, Melissa
Malzkuhn, Barbara Manini, Adam Stone, Geo Kartheiser, Arcangelo Merla, Ari
Shapiro, et al. 2018. Teaching language to deaf infants with a robot and a virtualhuman. In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems. 553.

101