EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Everyone is Talking: Artificial Intelligence + Chatbot
for Improving Oral Proficiency in EFL Classrooms
Makoto Shishido
Tokyo Denki University
Tokyo, Japanshishido@mail.dendai.ac.jp
Abstract: This study introduces an e-learning system aimed at improving oral English
proficiency using artificial intelligence (AI) and multiple-choice chatbots. It enables students topractice oral drills such as pronunciation, shadowing, and conversations. With the developmentof speech recognition and pronunciation evaluation, the same type of responses as those by native
speaker instructors can be provided using this system. In addition, the advancement of chatbots
has enabled to practice several patterns of conversation that are not limited to certain fixed phrases.
This paper also reports the effectiveness of the system in improving oral proficiency in EFL
classrooms in Japan. The pre- and post-test results of the TOEIC and OPIc were compared andsignificant improvements were observed. The results of the students’ questionnaires showed
positive feedback such as lower resistance to speaking English and repeated practice at anytime
and anywhere.

Introduction
In Japan, English education has been centered on a grammar-translation method of teaching, and oral
language activities involving speech are not being actively conducted (Saito, 2007). Another problem is that speech
practice is often conducted by native English-speaking instructors, and Japanese English as a Foreign Language (EFL)
teachers tend to avoid handling it.
To address this problem, the author has implemented several experimental teaching methods. One was to use
Skype to practice conversations with near-native English-speaking instructors in the Philippines (Sakamoto, et al,2017). In this method, one instructor was assigned to teach a group of four students, and this could only be conducted
once per semester. Because the use of a human instructor created problems in terms of cost and securing humanresources, the author and his team decided that with the development of ICT, teaching materials using AI would beappropriate and aimed to develop a new system.
The second attempt the author made was to develop an e-learning system for smartphones, called Shikibu,with NTT Learning Systems (Shishido, 2018). Shikibu was an AI-based speech practice system that used speech
recognition by Apple’s Siri and IBM Watson’s AI for automatic responses. In the demonstration experiment, the results
showed a certain level of effectiveness, but the problem was that the content of the teaching materials was specialized
for the TOEIC Speaking exam and thus not suitable for university students. However, the survey results showed that
61% of the participants found the Shikibu system enjoyable, 69% felt a sense of accomplishment after practicing
English using the system, and 80% considered the system to be effective.
In the third experiment, SpeakBuddy, a smartphone application, was used to demonstrate improvements inspeech skills (Shishido, 2019). SpeakBuddy was a system that used Google speech recognition to practice business
English conversations. The effectiveness measurement showed -0.14 of development of oral proficiency in the OPIc
results. Although business-focused content was unfamiliar to university students, making it challenging learningmaterial for the classroom, positive results were obtained in the answers to questionnaires about attitudes toward thesystem.
Based on these experiences, the author decided to conduct a new demonstration experiment by creatingteaching materials whose content was tailored to the interests of university students and whose length could be handled
in 90–100 min, which is the typical length of a university lecture in Japan. The material created was the ELSTIntroducing Japan in English (Shishido, 2021). In creating this material, with Signwave Inc., the author created e-Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

learning materials and textbooks equipped with pronunciation evaluation and speech recognition by iFLYTEK. The
results indicated that speaking practice with e-learning materials using speech recognition and AI yielded positive
results. The students enjoyed studying with AI and viewed it as a new experience. They believed that their speaking
skills had improved, and their competence and OPIc scores had increased. A 0.27 increase in the effectiveness was
obtained in the OPIc results (Shishido, 2021).
These three experiments with an AI-based learning system proved that e-learning materials utilizing
pronunciation assessment and speech recognition were beneficial. Therefore, in this study, the author developed a newlearning system to further evolve ELST. New features added to the ELST system included shadowing in the reading
section and the introduction of a multiple-choice chatbot in the speaking section.
Shadowing has been reported to be effective in improving reading proficiency, particularly in oral reading
practice. A strong relationship has been found between shadowing and learners’ fluency in L2 production (Zakeri,2014). It has also been reported that shadowing exercises can significantly improve oral proficiency and can be appliedin university English teaching (Wang, 2017).
The next step was to introduce multiple-choice chatbots in oral practice. Chatbots have mainly been used forshort messages and website Q&As; however, recently, conversational chatbots such as Andy in the Google Play Store
have been developed. Huang et al. (2016) also reported the effectiveness of chatbots. Based on these studies, wedeveloped a new ELST using AI and multiple-choice chatbots.

Study
We now introduce the newly developed ELST-Virtual Travel around the World e-learning system and its
textbook that incorporates shadowing and multiple-choice chatbots. To develop this system, we used iFLYTEK’s AI
for speech recognition and pronunciation evaluation, with Signwave Inc. and Seibido Co., Ltd. cooperating in thedevelopment of paper-based textbooks. Virtual travel was chosen for the content of the teaching materials because we
wanted learners to broaden their views of the world, emphasizing practicality by using English conversation for travel,
and we could simulate world travel during restrictions on overseas travel during the COVID-19 pandemic. This was
designed for use in Japanese university classrooms for 90–100 min of lecture time, with 24 chapters for annual use.
The e-learning learning system consists of the following components:
1. Reading
1.1. Vocabulary Study (Pronunciation)
1.2. Vocabulary Study (Definition Quiz)1.3. Reading (Text)
1.4. Shadowing Practice
1.5. Comprehension Questions
1.6. Grammar Practice
2. Listening
2.1. Dictation Practice
2.2. Model Dialog
2.3. Role-Play Practice
2.4. Vocabulary Practice
2.5. Comprehension Check
3. Speaking
3.1. Speaking Practice with Chatbot
In the Reading section of the courseware, students are first asked to pronounce new words in the Vocabulary
Study (Pronunciation) and to check their understanding of the meanings in the Vocabulary Study (Definition Quiz).
In reading practice, that is Reading (Text), students listen to a sample read by a native speaker. Shadowing Practice
involves requesting students to engage in shadowing the reading text aloud (Figure 1). As speech perception andrecording skills improve through repeated shadowing, they can be mentally articulated with more accurate speech,
thus increasing the amount of information retained in the phonological loop and leading to an increase in the listening
ability (Tamai, 2005). Subsequently, comprehension of the English text is checked by answering Comprehension
Questions, and students are encouraged to check their understanding of the grammar presented in the text during

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Grammar Practice. The English text introduces each country and region of the world and describes the sightseeing
spots to visit, culture to experience, food, and other activities to engage in when traveling.

Figure 1: Example of shadowing practice
The Listening section includes a dictation exercise in which students listen to the audio and fill in the blanks.
Subsequently, students listen to the entire sample dialogues, practice pronunciation, and receive pronunciation
evaluation for each utterance in a role-play exercise (Figure 2), as well as conversational practice of each role’sutterances against a computerized voice. This is followed by a vocabulary study and questions to confirm
understanding of the content. The goal here is for students to master travel English conversation by practicing certain
fixed conversations that simulate situations they may encounter during their travel.

Figure 2: Example of role-play practiceThe Speaking section is a speech exercise using AI, a multiple-choice chatbot with speech recognition. The
computer asks the first question. The learner chooses one answer from the three options displayed and speaks. The
computer identifies the utterance using speech recognition and then asks the learner the next question based on his/heranswer. By repeating this process three times, 27 different conversation patterns can be practiced, starting with a single
question (Figure 3). A multiple-choice chatbot is used because chatbots are currently capable of presenting preprepared answers based on keywords (Figure 4). The learner selects an answer from the three options presented andthe corresponding next question is asked; thus, the technology is thought to be at present. As chatbot technology
evolves, more patterns and open-ended conversational exercises will become possible.
Students’ learning histories for each lesson and section can be managed and viewed on the ELST’s dedicatedmanagement system for instructors. Students can also view their study time, scores based on correct answers topractice questions, speaking time, and pronunciation assessment scores, as well as points based on study time and thenumber of correct answers. Students are notified weekly of their rankings based on their points, motivating them tocompete with each other and learn more.

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Figure 3: Structure of a multiple-choice chatbot

Figure 4: Example of conversation practice with a multiple-choice chatbot
The textbook is linked to the content of the e-learning system and consists of the following components:
1. Reading
1.1. Vocabulary Study
1.2. Reading
1.3. Shadowing Practice
1.4. Map Exercise
1.5. Comprehension Questions
1.6. Grammar Practice
2. Listening
2.1. Dictation
2.2. Dialog
2.3. Role-Play Practice
2.4. Vocabulary Practice
2.5. Comprehension Check
3. Speaking
3.1. Example
3.2. Dialog Practice with Multiple Choice Chat Bot
3.3. Destinations and Things to Do
4. Writing
4.1. Internet search and plan an itinerary

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

The accompanying textbook is designed to serve as a supplement to the e-learning system for students tokeep notes and records of study at hand and to be accessible at any time (Figure 5). The main exercises are the sameas those in the e-learning system, but the Map Exercise (Figure 6) in the Reading section is designed to encourage
students to use Google Maps or Google Earth to search for and view actual photographs and scenes of the countries
introduced in the English text, as well as to show illustrations of places of interest in the text. The Listening section
includes a dictation exercise in which students listen to the audio and fill in the blanks. They can write what they hearin the textbook to keep a record (Figure 7). The final writing exercise is not accompanied by the E-L system but is an
assignment or homework at the end of the lecture, in which students are asked to write an English essay about theirtravel itinerary and the places they like to visit (Figure 8). The assignment is not submitted through the ELST, but
through the LMS provided by the university.

Figure 5: Textbook example 1

Figure 6: Textbook example 2

Figure 7: Textbook example 3

Figure 8: Textbook example 4

Findings
An experimental measurement of the learning effectiveness of the newly developed ELST was conducted
between April 2022 and December 2022 with 27 second-year university EFL students in Japan. Although the students
had studied English for 9 years, including 8 years from the 5th grade of elementary school to the 3rd grade of highschool and their first year of university, their English proficiency at the start of the study in April was 254 on averageon the TOEIC score, and the results of the questionnaire survey showed that almost 97% of the students answered“No” when asked if they were good at or fluent in English; this would be classified as false beginners (Figure 9).

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Figure 9: Question about student proficiency level
The TOEIC and OPIc tests were conducted in April at the beginning of the study and in December at the endof the study, and the results of the pre- and post-tests were compared to demonstrate the effectiveness of the study.
TOEIC scores were obtained at the beginning and end of the study to verify the effectiveness of the use of learningmaterials, a method that has been used in studies such as that of Ishikawa et al. (2011). In addition to the TOEIC, theOPIc conducted by the ACTFL, which specializes in measuring the English speaking ability, was also included in theverification.
First, we discuss a comparison of TOEIC scores. At the beginning of the study, in April, the highest score
was 300, the lowest 65, the average 157.0, and the standard deviation 46.9 for the Listening section. By the end of thestudy in December, the highest score was 310, the lowest was 105, the average was 197.8, and the standard deviation
was 54.0 (Figure 10). The average score increased by approximately 40 points. For the Reading section, the highest
score was 220, the lowest was 55, the average was 97.0, and the standard deviation was 33.5 at the beginning of thestudy. At the end of the course, the highest score was 210, the lowest score was 50, the average score was 107.5, andthe standard deviation was 34.7 (Figure 11). The average score had improved by 10 points. In terms of total scores, atthe start of April, the highest score was 520, the lowest was 160, the average was 254.1, and the standard deviation
was 65.0. At the end of December, the highest score was 520, the lowest was 175, the average 304.4, and the standard
deviation was 71.2 (Figure 12). The average score increased by approximately 50 points. Twenty-one students
increased their total scores, one remained unchanged, and the scores of five students decreased. The score development
rate, which calculates the amount of increase in the score relative to the full TOEIC score of 990 based on the following
formula, showed an average increase of 7.8% in this experiment; 22.8% at the highest and -7.5% at the lowest (Figure
13).
SDR = (Post-Test Score - Pre-Test Score) / (Full Mark - Pre-Test Score) × 100

Figure 10: TOEIC listening score comparison

Figure 11: TOEIC reading score comparison

These results inferred that the increase in TOEIC scores, particularly listening scores, was significant and that speech
practice may have improved listening skills. The t-test results also showed significant differences in improvements inlistening and total scores: p = 0.005 for listening, p = 0.313 for reading, and p = 0.010 for the total score. Comparing
these results with those of my previous study (Shishido, 2021), I found a significant improvement in the TOEIC scores.

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Figure 12: TOEIC total score comparison

Figure 13: Score development rate

As for the OPIc results, at the beginning of the study in April, the highest was level 6, the lowest was level
1, the average was 2.07, and the standard deviation was 0.81. At the end of the study in December, the highest waslevel 6, the lowest level 1, the average was 2.56, and the standard deviation was 1.31, indicating an average
improvement of 0.49. According to Cohen (1988), the effectiveness of an independent two-group test can be measured
using the following formula:(Here, M2 and M1 denote the means of the pre-test and post-test samples, respectively, and SD1and SD2 denote the standard deviations of the pre-test and post-test samples, respectively.)
Table 1: Effectiveness Scale
Effectiveness
d

Small
0.20

Medium
0.50

Large
0.80

The effectiveness of d = 0.44 obtained in this experiment was almost double that of d = 0.27 observed in the previous
experiment by Shishido (2021). Based on the effectiveness of d = 0.44, a certain degree of effectiveness was achieved(Table 1). However, when the t-test was conducted, p = 0.118; no significant difference was observed (Figure 14).
Comparing the OPIc level differences between the pre-test and the post-test, levels of 12 students increased, those of
6 students decreased, and those of 9 students did not change (Figure 15). The up-and-down rate increased by 48 %.
up/down=48%

p=0.118

Figure 14: OPIc score comparison

Figure 15: OPIc level up and down

Second, we discuss the results of the questionnaire survey. The survey asked about their impressions of ELST
on a five-point scale, with five being strongly agree and one being strongly disagree. The first question received a very
high score of more than 3.4. Question 4, that is, whether they were enthusiastic about the program, scored 2.93 points.
The fifth question, that is, whether the ELST was effective, received a score of 3.57. The last question, that is, whether
the participants were satisfied with the ELST, received a score of 3.30. The score for whether they were enthusiastic
about the program was below three and did not receive a very high rating, but the score for whether they thought it
was effective was very high at 3.57, indicating that the students who used the program felt it was effective (Table 2
and Figure 16).

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

Table 2: Impressions of ELST
Questions
Were you interested in ELST?
Did you enjoy studying with ELST?
Did you feel achieved?
Did you feel enthusiastic?
Do you think ELST is effective?
Were you satisfied with ELST?

Agree⇔Disagree
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1
5 4 3 2 1

Figure 16: Impressions about ELST

Next, respondents were asked their opinion on the limitations of learning English using AI. The lowest score
was 2.1 for feeling uncomfortable talking to an AI partner. The score was 2.9 for AI not understanding the meaning of
the participant’s words 3.1 for feeling uncomfortable with an artificial voice, and 2.8 for only uniform conversation
being possible. The score for conversation being unnatural and not established was 2.37 points. Except for thediscrepancy in the voice, the scores were less than 3 points, proving that the negative evaluation of the learning systemusing AI was not high (Table 3 and Figure 17).
Table 3: Drawbacks of learning with AI
Statement
Agree⇔Disagree
I felt discomfort in speaking with AI. 5 4 3 2 1
AI did not understand what I said.
5 4 3 2 1
I felt an artificial voice strange.
5 4 3 2 1
I could practice only set phrases.
5 4 3 2 1
I felt a conversation with AI was 5 4 3 2 1
unnatural.

Figure 17: Drawbacks of learning with AI

When the participants were asked about the advantage of AI-based English language learning, the following
responses were obtained: less resistance to speaking English, 3.77; not embarrassed by mistakes, 3.93; AI responded
according to the participant’s words, 3.50; can repeat the same conversation, 3.93; ability to converse at a convenienttime, 3.90 points (Table 4 and Figure 18). The responses that received higher marks for the use of AI than for practicing
with a human partner included the following: less resistance to speaking English, not embarrassed by mistakes, being
able to repeat the same conversation, and being able to practice at a convenient time. These are assumed to be
extremely useful learning materials for false beginner learners who are not good at practicing speaking with a humanpartner.
Table 4: Advantage of learning with AI
Statement
I felt less resistance to speaking English.
It was not embarrassing to make a
mistake.
AI responded naturally.
I could practice repeatedly.
I could study anytime.

Agree⇔Disagree
5 4 3 2 1
5 4 3 2 1
5
5
5

4
4
4

3
3
3

2
2
2

1
1
1
Figure 18: Advantage of learning with AI

The last question asked was about the learning styles using AI. The result was a score of 3.57 for whether thestyle of learning English with AI was suitable. The score for whether the respondent liked to use AI-based English
learning in his/her private life in the future was 3.50 and 3.73 for whether the respondent liked to use AI-based English

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

learning in his/her university classes in the future. Overall, the scores were extremely high, indicating that the proposedmaterials fit the learning styles of the students, and that they would like to use them in the future (Table 5 and Figure
19).
matches with your
Table 5: Learning style using AI
learning style
3.8
Questions
Agree⇔Disagree
3.7
3.57
Do you think speaking practice with 5 4 3 2 1
3.6
3.5
AI matches with your learning style?
3.4
Would you like to use this type of app 5 4 3 2 1
3.3
privately?
Would you like to use this
3.50
Would you like to use thistype of system in your
Would you like to use this type of 5 4 3 2 1
3.73
type of app privately?
class?
system in your class?
Figure 19: Learning style using AI
The following notable positive comments were mentioned in the comments section of the survey.
・ My pronunciation improved.
・ I felt that my listening ability improved.
・ When I hear or see English, a scene comes to mind more often than before.
・ I felt my interest in English increased.
・ I learned a lot of vocabulary.
・ I personally think it is a good system because it has improved my English.
・ I feel more comfortable speaking English.
・ My TOEIC score went up.
・ I got used to listening and can hear a little better.
・ I could remember more English words by talking to AI than by writing them.
The aforementioned comparison of TOEIC and OPIc pre- and post-test scores in the experiment and resultsof the questionnaire survey indicated that the speech practice with e-learning materials using AI and a multiple-choicechatbot produced positive results. The students enjoyed studying with AI and viewed it as a new experience. They
also believed that their speaking proficiency had improved, and their TOEIC and OPIc scores had increased.
Respondents gave higher ratings for the use of AI than for practicing with a human partner, expressing that they felt
less resistance to speaking English, were not embarrassed even if they made mistakes, could practice the same
conversations repeatedly, and could practice at their convenient time. These proved to be extremely useful learningmaterials for false beginner learners who are not very good at practicing speaking with humans.

Conclusions
The results discussed thus far have proven that learning using AI and multiple-choice chatbots is beneficial.
If ICT technology evolves further, the further evolution of chatbots and the development of the metaverse may leadto the development of a system in which learners’ avatars can practice English conversations through a simulated
experience in a 3D space.
As for the use of chatbots, this time, the conversation proceeded using a multiple-choice system in which thelearner’s choice from three options of answers assumed in advance was confirmed by voice recognition. However, as
chatbot technology advances, natural and free conversations may become possible. In such a case, the practice will
not be limited to fixed phrases but will be similar to actual social situations between humans.
In addition, some students may feel that the e-learning system, which was developed on a computer, tablet,or smartphone, is far removed from realistic conversations in the real world. The solution to this problem is to placelearners in a virtual avatar space where they can practice conversing with the people they encounter, giving them asense of reality. The evolution of the metaverse will enable to place learners’ avatars in 3D space and have

Preview version of this paper. Content and pagination may change prior to final publication.

EdMedia + Innovate Learning 2023 Vienna - Vienna, Austria, July 10-14, 2023

conversations with other avatars they meet, facilitating to learn English and practice speaking through simulated
experiences that are similar to those in the real world.
In the next version of the ELST, we want to develop a learning system that enables the evolution of chatbots
and conversation practice using avatars in 3D space in the metaverse world.

Acknowledgment
This study was supported by JSPS KAKENHI (Grant Number 21K00688).

References
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Huang, X., & Jia, X. (2016). Corrective feedback on pronunciation: Students’ and teachers’ perceptions. International
Journal of English Linguistics, 6(6), pp. 245–254.
Ishikawa, Y. et al. (2011). TOEIC® Test Learning with ATR CALL BRIX: A Case Study of a University and a
Company. University ICT Promotion Association 2011 Annual Conference Proceedings, pp. 296–300.
Saito, C. (2007). Japanese and English: Another Centennial History of the English Language. Kenkyusha.
Sakamoto, M., Handa, J., Shishido, M., Sakai, K., & Aratame, N. (2017). Expectations and challenges regarding online
one-to-one instruction by Filipino external instructors. Language Learning and Educational Linguistics: 2016 Edition,
pp. 17–24. Edited by the Joint Editorial Board of the Japanese Association for English Education and the Japanese
Association for Educational Linguistics, Published by the Institute for Information Education, Waseda University.
March 31, 2017.
Shishido, M. (2018). Developing e-learning system for English conversation practice using speech recognition andartificial intelligence. In T. Bastiaens, J. Van Braak, M. Brown, L. Cantoni, M. Castro, R. Christensen, G. DavidsonShivers, K. DePryck, M. Ebner, M. Fominykh, C. Fulford, S. Hatzipanagos, G. Knezek, K. Kreijns, G. Marks, E.
Sointu, E. Korsgaard Sorensen, J. Viteli, J. Voogt, P. Weber, E. Weippl & O. Zawacki-Richter (Eds.), Proceedings of
EdMedia: World Conference on Educational Media and Technology (pp. 226–231). Amsterdam, Netherlands:
Association for the Advancement of Computing in Education (AACE).
Shishido, M. (2019). Evaluating e-learning system for English conversation practice with speech recognition andfuture development using AI. In J. Theo Bastiaens (Ed.), Proceedings of EdMedia + Innovate Learning (pp. 213–218).
Amsterdam, Netherlands: Association for the Advancement of Computing in Education (AACE).
Shishido, M. (2021). Developing and Evaluating an E-learning Material for Speaking Practice with the Latest AI
Technology. The IAFOR International Conference on Education – Hawaii 2021. Official Conference Proceedings. pp.
55–68. ISSN: 2189-1036.
Tamai, K. (2005), A Study on the Effectiveness of Shadowing as a Listening Instructional Method. Kazama Shobo.
Wang, X. (2017). The Study of Shadowing Exercise on Improving Oral English Ability for Non-English Major College
Students. World Conference on Management Science and Human Social Development (MSHSD 2017). Advances in
Social Science, Education, and Humanities Research, volume 120, pp. 195–200.
Zakeri, E. (2014). The Effect of Shadowing on EFL Learners’ Oral Performance in terms of Fluency. International
Journal of English Language Teaching, Vol. 2, No. 1, pp. 21–26.

Preview version of this paper. Content and pagination may change prior to final publication.