51

論文

Technology-Enhanced Collaborative Foreign Language Learning
Using Communication Robots.
KATORI, Mari
(Aomori Public University/Hakodate Future University)

1. Introduction“Communication Robots” (CR) or robots that converse with people, are now
relatively commonplace. We can now find several such robots in Japan, used forvarious purposes or in various environments such as in commercial applications or
school classes. Lately, some research has also been done regarding effective
communications between a user and a robot, or systems that use virtual agents(Sugano & Shibuya, 1997; Sumi & Nagata, 2010).
This study examines whether CR can be effective ‘tandem partners’ in foreign
language learning for Japanese university students. The concept of tandem learning
here is borrowed from Brammerts (2005). ‘Tandem” means something like “riding
together”. Brammerts notes five key features of tandem learning: communication foran educational purposes, natural communication, pair-work, collaboration, andautonomy.
Five university students participated in this research as a case study. They
used CR for three months and took a standardized speaking test as a pre- and post-test.
During their study period, transcriptions of their speech were graded for readability.
Finally, participants were interviewed with a view to discovering any mental effects oflearning with CR. The observed positive mental effects are explained with reference tothe concepts of “Media Equation” (Reeves & Nass, 1996) and “Captology” (computersas persuasive technology; Fogg, 2003).
2. Background of the research
2.1 Communication Robots
In recent years, we can find various types of robots in our lives, from talking
microwave ovens to learning AI-type humanoid robots, and some of these are being
adopted for educational purposes. Figure 1 shows an actual CR in Japan called“Pepper”. It is used for a variety of purposes in various environments, includingcommercial applications and classes. It has friendly body characteristics, and contains
a speech recognition system. It is capable of some non-verbal communication (NVC)

KATORI, Mari

52

including hand behaviors, nodding, and body movements. According to Sugano and
Shibuya (1997), NVC, especially hand behaviors, are important for communication
between humans and robots. Because of those features, users can feel as if the robot
and they are genuinely communicating. Pepper can also show letters or pictures on atablet on its chest, which can be a useful tool in assisting communication.
Unfortunately, its face is fixed and immobile, so Pepper is not able to show any facial
expressions. While this robot looks very useful, it is not designed for language learning.
Pepper can exhibit only pre-scripted behaviors. Users have to input intricate programs
in advance – essentially fully pre-scripted conversations with the lines Pepper should
speak, annotated with NVC motions, together with short expected answers or
keywords from the human interlocutor that Pepper can detect and use to locate where
it is in the script.

Figure 1. Pepper.

Figure 2. Musio,

Figure 2 shows another communication robot called “Musio”. This small CR
has been developed specifically for language learning. Its body is simple, meaning thatit does not show any body movement, including the hand behaviors that Sugano andShibata value, though its arms can be posed. However, it has many facial expressions(See figure 3). According to Sumi and Nagata (2010), these are particularly importantin communications between a human and a system. Indeed, observation ofhuman-Musio interaction showed people quickly and easily understanding the CR’s
status or responses and engaging emotionally themselves. Musio has some English
learning modes such as chat mode, drill mode, Edu mode (pair learning mode),
dictionary mode, and game mode. It uses English and some Japanese, and it is notnecessary for users to input intricate programs in advance. In my research, I adopted
this CR as a tandem partner for students for English learning because it is of alearning AI-type, as capable of learning through natural language interaction duringTechnology-Enhanced Collaborative Foreign Language Learning Using Communication Robots.

53

conversations with users. Moreover,since Musio creates a transcript ofstudents’ utterances, students canreview their speaking practices later,
and also the transcripts can be
graded for readability. As mentioned
above,

accordingtoBrammerts(2005), good “tandem” work involves
pair-work, communication for an
educational purposes, collaboration,
and autonomy. In this research, I
considered this CR as a “tandem
partner” of students and used casestudies to examine how students

Figure 3. Musio’s expressions.

developed their English speaking
skills.

2.2 The importance of non-verbal communication (NVC)
Figure 4 illustrates the importanceof non-verbal communication (NVC) for first
impressions.

AccordingtoWiener&Mehrabian (1968), a person’s first impression
will be mainly (93%) decided by the person’s
NVC such as facial expression, posture, or
tone. This idea is known as the rule of
Mehrabian. Therefore, in order to be alikable and friendly tandem partner, showing
effective NVC is an important factor for

Figure 4. The rule of Mehrabian.

robots. This is one of the reasons I chose
Musio for my research.
3. Method
In this research, a CR which can show many facial expressions as NVC was
adopted as a conversation practice partner for subjects. Five Japanese university
students participated in this research as a case study. They were students of a public
university in the northern part of Japan, majoring in economics or management. All ofthem were senior students. Students took the TOEIC Speaking IP (Institutional
Program) test as a pre- & a post-test and later those results were examined statistically.

KATORI, Mari

54

They had communication practicefor about 60 minutes per session,usually once a week, for 12 weeks.
The sessions were held in my officewith the CR, Musio (see figure 5).
This research was conducted in
2018.
The

CR

has

severallearning modes such as academy
mode, which is divided into twomode, “free chat mode” and “drill Figure 5. Talking with Musio.
mode”, then “Edu mode” (pairwork mode), “dictionary mode” (both English-Japanese and Japanese-English
dictionary), “game mode” (role playing game mode) and “radio mode” (BBC). Anotherfeature is the ability to create transcriptions of free chat with the students. Thisfeature is particularly useful for students because, generally, it is very difficult and notpractical to create transcriptions of conversations of actual English conversation
practices between an instructor and a student. After every practice session with the CR,
students’ transcriptions were printed out and given to learners so they could review
and think back over their conversations. The transcriptions were also analyzed
quantitively observing such things as number of sentences, number of words, andreadability (using the “Flesch reading ease” scale). Finally, after three months ofpractice sessions, students had an interview session with an author to examine theirstudy experiences with the CR.
During practice sessions, students mainly used the drill mode and chat mode.
The drill mode has two patterns, one a general conversation practice mode and theother a business conversation practice mode. In the general conversation practice mode,
they would talk about general topics. While in the business conversation practice mode,business topics are used. Each practice has three steps. In the first step, the CR plays
the conversation drill showing a transcription on its display (the small screen which
doubles as its face). Students can listen to the conversation and read the displayedtranscription. In the next step, subjects do role-playing practice, taking turns with the
CR and while reading the transcription. In the final step, students have the drill
conversation without seeing the English transcription, but they can see Japanesetranslations of each reply as prompts. In the second and third steps, students’ replies
are evaluated by the CR. When a subject’s reply is evaluated as good, the CR returns
short comments such as “good” or “well-done”, and then goes on with the conversation.
If a subject’s replies are not evaluated as good, the CR instead returns “bad”, “again” orTechnology-Enhanced Collaborative Foreign Language Learning Using Communication Robots.

55“sorry,” and students have to repeat the sentence until the CR accepts it. The following
examples are samples of Musio’s practice mode drill conversations.
Example 1: general conversation practice mode
Bill: Wendy, where do you live?
Wendy: I actually live in the apartment building near the park.
Bill: Oh, really? You are so lucky to be living there!
Wendy : Yeah, it takes only about five minutes for me to get to work.
Bill: I heard there’s also a nice bridge nearby that you can go to weekends!
Wendy : Yeah, but it’s too cold. So, I don’t go there much.
Bill: Oh! You should definitely go there often. It’s important that you exercise.
Wendy: Yeah, I’ll keep that in mind.
Example 2: business conversation practice mode
Daniel: What’s wrong?
Sally: I made a mistake in the meeting today.
Daniel: Oh, what kind of mistake?
Sally: I printed out the wrong materials for the meeting.
Daniel: Oh…we all make mistakes! It’s only been a month since you started
work, so they’ll understand.
Sally: I made the same mistake last week. I didn't mean to, but it justhappened again!
Daniel: In that case, I think you should concentrate more when you work. Yeah,
if you do so, I’m sure that you won’t make the same mistake again.
Sally: Thank you for your advice. See you later.
In the practice sessions, every time after practicing using drill mode, students used
free chat mode for 10 to 15 minutes.
4. Results
4.1 Chat’s Transcriptions
The following Transcription 1 shows the first free chat practice of a student,
as detected and transcribed by the CR. It is apparent that sometimes the speech
recognition system didn’t work well – for example, it is unlikely that the student
claimed to be from ‘album’; this was presumably a Japanese placename such as‘Aomori’. Likewise, the student probably said “hobbies”, but it was replaced “phobias”.
And, occasionally the conversations didn’t mesh even when the CR did correctly
recognize what the student said.

KATORI, Mari

56

Transcription 1. (the first free chat practice)Student R: where are you from?
Robot: I am originally from AIPIA. Now I live in Tokyo. Where are you?
Student R: I am from album. [sic]Robot: She also loved the album that came out in 1969, Rhymes and Reasons.
Student R: Moltres. [sic]Robot: I’m sorry?
Student R: Do you have any phobias? [sic]Robot: I have a cat.
Student R: me too
Robot: Oh, it’s really good.
After three months, the overall impression was that both the students andthe CR had improved. The students’ utterances became more natural andconversations seemed to go better (refer transcription 2). After 12 practices, although
Musio’s replies were sometimes irrelevant and unnatural so the student had to give up
and start a different topic, every student felt that not only had they themselves
improved but also the CR seemed to have learned somehow and its speech recognition
system worked better.
Transcription 2. (three months later)
Student R: What language can you speak?
Robot: Mandarin is an interesting language too.
Student R: How many language [sic]can you speak?
Robot: Oh, you'll definitely need subtitles. There's a bit of English, but also
Portuguese and Chinese.
Student R: What field do you want to go?
Robot: I want to become a teacher, different field from my parents.
Student R: Teachers?
Robot: Math teacher.
4.2 Quantitative results
Table 1 shows the quantitative results of transcriptions comparing the firstweek and the last week of their practice period. After three months practicing with
the CR, as table 1 illustrates, I examined the change between the first week and thefinal (twelfth) week in number of words, number of characters, number of sentences,
average number of words per sentence, and average number of characters per word instudents’ chat. All these figures increased: students came to use more words andTechnology-Enhanced Collaborative Foreign Language Learning Using Communication Robots.

57

sentences, and longer words and sentences, after their 3 months of practice. However,readability levels remained stable at the “very easy” level. Table 2 describes the Flesch
Reading Ease Score and levels for each score.
Table 1. Quantitative Results
Student R

Student A

Student K

Student D

Student M

Averages

1st time 12th week 1st time 12th week 1st time 12th week 1st time 12th week 1st time 12th week 1st time 12th week

Counts
Words

17

144

28

122

61

170

21

108

48

132

35

135.2

Characters

76

574

106

504

235

646

78

400

176

542

134.2

533.2

Sentences

6

33

7

26

14

42

7

23

9

24

8.6

29.6

Words per sentence

2.8

4.2

4

4.6

4.3

3.9

3

4.6

5.3

5.5

3.88

4.56

Characters per word

4.1

3.7

3.3

3.8

3.5

3.5

3.3

3.4

3.4

3.9

3.52

3.66

84.5

98.5

100

89.0

97.0

97.9

100.0

97.8

100.0

87.8

96.3

94.2

Averages

Readability
Flesch Reading Ease
Scorespre

TOEIC Speaking Score
CEFR

post

pre

post

pre

post

pre

post

pre

post

pre

post

120

120

80

90

90

120

100

110

130

130

104

114

5

5

4

4

4

5

5

5

6

6

4.8

5.0

(CEFR; Common European Framework of Reference for Languages)

Table 2. Flesch Reading Ease Score
Score

Level (approximate grade)

0-30

very difficult

30-50

difficult (university level)

50-60

fairly difficult (high school level)

60-70

standard (8th-9th grade level)

70-80

easy (7th grade level)

80-90

easy (6th grade level)

90-100very easy (5th grade level)

In my estimation, the main reason for this was the CR’s speech recognition
system, not the students themselves. Because the speech recognition system waslimited, even having subjectively improved somewhat from week 1, students madeshort simple sentences. To increase the probability of the CR understanding them andtherefore of the conversation being successful, they typically communicated insingle-clause sentences.
The bottom part of Table 1 shows the score of TOEIC speaking test. The
average results are 104 and 114 for pre-test and post-test respectively, and there wasno significant differences by two-tailed t-test (t=1.83, p=1.42; the result is not
significant at p<.05). Likewise, when I look at the CEFR (Common European
Framework of Reference for Languages) grade, the average results are very close (4.8

KATORI, Mari

58

and 5.0), a difference which a two-tailed t-test shows to be similarly not significant at

p<.05 (t=1.00, p=0.37). Accordingly, these quantitative results show that while 12
weeks practice encouraged students to use more words and sentences, but it was notvery effective to improve their English competence statistically.
4.3 Qualitative results
However, after participating in this research, students were also interviewed
about the CR. Although a few issues were mentioned, overall, the students weresatisfied with their tandem partner. Students felt the CR was more patient thanhumans. Therefore, they felt little anxiety when talking with the CR. Additionally, forexample, they were not afraid of repeating the same mistakes. They noted that
because of the CR’s facial expression, the CR looked very friendly. In addition,transcripts were very useful to review their conversations, they said. The following is asummary of subjects’ oral answers in the post-interview.

-Students were not afraid of mistakes.
-Felt little anxiety when talking with a CR.
-Felt like a real conversation because of CR’s facial expression.
-The CR can deal with a huge amount of data.
-Created transcripts were very useful.
-The CR was more patient than a human.
-The CR did not get tired.
-Can communicate (practice) anytime, anywhere with Wi-Fi.
-CR was not perfect so that they felt affinities with CR.
-Felt both (students and CR) were developing.
-Cannot chat naturally because of time lags.
-Sound recognition system is not perfect.

5. Discussion
5.1 Media Equation and Captology
Finally, I would like to mention two concepts that may be useful when
thinking about new technology in this context. They are called the Media Equation
and “captology.” According to Reeves & Nass (1996), people treat computers and new
media like real people. For instance, they found, when people use a certain computeror media repeatedly, people start to feel friendly feelings (camaraderie) toward thedevice, as if they were dealing with a person and developing some kind of friendship or
relationship. Likewise, in my research, students gradually came to view the CR as a

Technology-Enhanced Collaborative Foreign Language Learning Using Communication Robots.

59

kind of friend. We conjecture that this helped their motivation to come to the officeand practice. During the practice sessions, at least, they looked like they enjoyedconversing with the CR, and enjoyment would seem to be motivating and an
important factor in improving a skill.
The other concept that may be useful in discussing this topic is “captology”.
This term was coined by Fogg (2003), and means “computers as persuasive technology.”
The CR is perhaps not obviously persuading students, but it can be seen as persuasivein the sense that it kept students coming back to finish more practice mode sessions
and have extended free chat sessions with it. Students did not receive any university or
course credit for participating in the research, so if the sessions were in any way anegative experience the students would have been unlikely to continue or toparticipate so fully. Indeed, students seemed to enjoy the sessions, and were happy toreceive positive feedback (visually as happy expressions, and verbally as “good” and soon) from the CR.
In the very near future, many CRs will appear, and many schools oruniversities will adopt those robots for their classes. Figure 6 shows my hypothesis
which illustrates a hierarchy of speech recognition systems. I applied Maslow’s
hierarchy of needs theory for this model. Maslow’s hierarchy of needs theory is often
applied to marketing, personnel management, communication studies and other areas,
and I think it can be gainfully applied to the communication between humans and CRs

Figure 6. Hierarchy of speech recognition systems (hypothesis) - Application of
Maslow’s hierarchy of needs theory (Williams,1992)

KATORI, Mari

60

too. In my opinion, the CR used in this research is in the 3rd layer now. I think thatCRs are not likely to be language teachers even if their AI systems are much moredeveloped than they are now. However, if AI systems develop enough for CRs to get
into the 2nd layer, where talking with the CR is very enjoyable, they may become goodtandem partners in the near future.
6. Conclusion
In this article, I examined whether a CR can be an effective partner for pairwork in foreign language learning. I focused on a CR developed by data scientists andnatural linguists for educational purposes, which is of an AI-type and designed to learn
during conversation, and creates a transcript of the student’s utterances so students
can review it later. According to Brammerts (2005), good “tandem” work has fivefeatures: communication for an educational purpose, natural communication,
pair-work, collaboration, and autonomy. In this research, I considered the CR as a“tandem partner” of students and used case studies to examine how students
developed their English speaking skills. Furthermore, I tried to explain theeffectiveness of CRs in this role using two concepts, Media Equation and Captology. In
conclusion, the CR used in this research supplied some features as a good tandem
partner such as: communication for an educational purpose, pair-work, collaboration,
and autonomy (four of Brammerts’ five features, missing only ‘natural communication’).
Actually, I found from the interview results that students feel that the CR is already agood partner despite some issues with the technology. Therefore I would say, if CR’s AI
systems are developed significant above where they are now, CRs could be much better
tandem partners in the near-term future.
Acknowledgements
Part of this research was founded by the Aomori Academic and Culture
Promotion Foundation (公益財団法人 青森学術文化振興財団).

References
Brammerts, H. (2005). Autonomes Sprachenlernen im Tandem: Entwicklung eines
Konzepts. In H. Brammerts, & K. Kleppin (Eds.), Selbstgesteuertes

sprachenlernen im tandem (Ein handbuch 2. Auflage), 9-16. Tübingen,Germany: Stauffenburg. (Original work published 2001)
Fogg, B. J. (2003). Persuasive Technology Using Computer to Change What We Thinkand Do. San Francisco, CA: Morgan Kaufmann Publishers.
Reeves, B. & Nass, C. (1996). The media Equation: How people treat computers,

Technology-Enhanced Collaborative Foreign Language Learning Using Communication Robots.

61

television, and New Media Like Real People and Places, New York, NY:
Cambridge University Press & CSLI Publications.
Sugano,S.

&

Shibuya,K.

(1997).

Anthropomorphic

robotsfor

nonverbal

communication. JRSJ, 15(7), 975-978.
Sumi, K., & Nagata, M. (2010). A virtual agent as persuasive technology. Psychology of

Persuasion. Chapter 6, 1-19. Nova Science Publishers, Inc., 29(1), 80-89.
Wiener, M., & Mehrabian, A. (1968). Language within Language: Immediacy, achannel in verbal communication. New York, NY: Appleton-Century-Croft.
Williams, F. (1992). The new communications. Belmont, CA: Wadsworth.
RobotsPepper by Softbank Robotics, Tokyo.
MUSIO by AKA, Santa Monica, CA.