Everyone is Talking: Artificial Intelligence+ Chatbot for Improving Oral Proficiencyin EFL Classrooms
Makoto Shishido
Tokyo Denki UniversityTokyo, Japan

Contents

PreviousStudies

LiteratureReviewDevelopment

Examples ofChatbot

around theof ELST with

Virtual Travel
World

Evaluation ofEffectiveness
Pre-Test vsPost-Test

Comparison
Questionnaire

Discussion

Future WorkPrevious Studies

Oral Practice in EFL Classroom in Japan• Native Speakers• Filipino Teachers through Skype

AI for EFL•Shikibu•SpeakBuddy

ELST – English Listening and Speaking Test

• Previous Version

Literature Review

Artificial Intelligence• Development in E-Learning• ICT Technological Advancement• Shishido (2018, 2019)

Shadowing and Oral Practice• A strong relationship between shadowing
and learners' fluency in L2 production(Zakeri, 2014).
• Shadowing exercises can significantly
improve oral proficiency (Wang, 2017).

Chatbot for EFL• Andy in the Google Play Store• The effectiveness of using chatbots has
been reported by Huang and others (2016).

Development

New Features• Shadowing• Multiple Choice Chatbot

Multiple Choice Chatbot (27 Patterns)
AIStudentAIStudentAI

Student
I'd like to go to the Maritime Museum of theAtlantic in Halifax.

CanadaI'd like to go to Nova Scotia.

What would you like to do in Nova Scotia?

I'd like to go to the Halifax Town Clock.
I'd like to visit Prince Edward Island.
I'd like to go to the Notre Dame Basilica.

I'd like to visit the eastern part of Canada.

What part would you like to go in the easternpart?

I'd like to go to Montreal.

What would you like to do in Montreal?

I'd like to go to the Old Port of Montreal.
I'd like to go to the Montreal Biodome zoo.
I'd like to stroll around Old Quebec.

I'd like to go to Quebec City.

What would you like to do in Quebec?

I'd like to visit Château Frontenac.
I'd like to take a stroll along Dufferin Terrace fora magnificent view of the St. Lawrence River.
I'd like to visit Toronto Tower.

I'd like to go to Toronto.

What would you like to do in Toronto?

I'd like to visit Niagara Falls.
I'd like to watch a Blue Jays' baseball game.

What part of Canada would you like to visit?

I'd like to visit the central part of Canada.

What part would you like to go in the central
part?

I'd like to go to the Canadian Museum for
Human Rights.
I'd like to go to Winnipeg, Manitoba.

What would you like to do in Winnipeg?

I'd like to go to the Assiniboine Park and Zoo.
I'd like to go to the Forks National Historic Site.
I'd like to go to the Columbia Icefield.

I'd like to go to Calgary.

What would you like to do in Calgary?

I'd like to go to Lake Louise.
I'd like to go to the Canadian Rockies.
I'd like to visit Staley Park.

I'd like to go to Vancouver.

I'd like to visit the western part of Canada.

What part would you like to go in the western
part?

What would you like to do in Vancouver?

I'd like to visit the Capilano Suspension Bridge.
I'd like to go to Canada Place.
I'd like to go to the British Columbia Parliament
Building.

I'd like to go to Victoria.

What would you like to do in Victoria?

I'd like to go to Hatley Castle.
I'd like to go to Butchart Gardens.

I'd like to go to Northwest Territories.

What would you like to do in Northwest
Territories?

I'd like to go to Yellowknife and see northernlights.
I'd like to go to Inuvik and try snowmobiling.

Examples

ELST System

Vocabulary Study

Reading - Shadowing

Listening - Dictation

Speaking - Role Play PracticeSpeaking – Multiple Choice Chatbot

Textbook

TextbookEvaluation ofEffectivenessExperiment
70.0

64.5

Subject: False Beginner
Are you good at English?

60.0
50.0

32.3

40.0
30.0
20.0
10.0
0.0

12

3.2

0.0

0.0

345

Pre-Test vs Post-Test Comparison•Pre-Test (April)•TOEIC and OPIc
•Post-Test (December)•TOEIC and OPIc

TOEIC
Test of English for International Communication

ETS (Educational Testing Service)

Listening 495 points + Reading 495 points = Total 990 points

P=0.005p=0.313

p=0.010

TOEIC Listening
350
300

Post-Test

250
200
150
100
500050100

150200Pre-Test

250

300

350

TOEIC Reading
350
300

Post-Test

250
200
150
100
500050100

Pre-Test

150

200

250

TOEIC Total Score
600

Post-Test

500
400
300
200
10000100

200

300

Pre-Test

400

500

600

TOEIC Score Improvement
2521

Number of Student

20
15
105510Score Increased

No Change

Score Decreased

Score Development Rate (SDR)
SDR = (Post-Test Score – Pre-Test Score) / (Full Mark – Pre-Test Score) x 100

Example 1: TOEIC score increase 490 to 590(590-490) / (990-490) x 100 = 20%
Example 2: TOEIC score increase 890 to 990(990-890) / (990-890) x 100 = 100%

TOEIC Score Development Rate Improvement25%

22.80%

20%
15%
10%

7.80%

5%
0%
-5%
-10%

HighestAverage

Lowest-7.50%

OPIc Oral Proficiency Interview Computer
Internet-based Oral Proficiency Test

ACTFL (American Council on the Teaching of Foreign Languages)

Novice Low (1) to Advanced Low (9)OPIc Scale and Score
Oral Proficiency Interview - Computer

ACTFL Oral Proficiency Interview (OPI)p=0.118

OPIc Level Up and Down1412

# of student

10up/down = 48%

129866420Up

No ChangeDownEffectiveness

d=0.44
EffectivenessdSmallMedium

Large

0.20

0.50

0.80

(Here, M2 and M1 denote the means of the pre-test and post-test samples,respectively, and SD1 and SD2 denote the standard deviations of the pretest and post-test samples, respectively.)Questionnaire

Interest in ELST
4.00

3.43

3.00

Satisfied 3.30

2.00
1.00

Did you enjoy

3.47

0.00

Effective

3.63

Felt achievement

3.57
2.93

Enthusiastic

Felt disconfort in
speaking with AI
4.0
3.0

Unnatural
conversation 2.37

2.10

AI did not
2.90 understand what I
said

2.0
1.0
0.0

2.80

Only set phrases

3.10

Felt artificial voices
strangeless resistance to
speaking English
4.00

3.77

3.75

able to study anytime

3.90

not embarrassing to
make a mistake

3.50

3.93

3.25

3.50

able to practicerepeatedly

3.93

Responded naturallymatches with your learning
style
3.8

3.7

3.57

3.6

3.5

3.4

3.3

Would you like to use thistype of system in your3.73
class?

3.50
Would you like to use thistype of app privately?

Comments• My pronunciation improved.
• I felt that my listening ability improved.
• When I hear or see English, a scene comes to mind more often thanbefore.
• I felt my interest in English increased.
• I learned a lot of vocabulary.
• I personally think it is a good system because it has improved my English.
• I feel more comfortable speaking English.
• My TOEIC score went up.
• I got used to listening and can hear a little better.
• I could remember more English words by talking to AI than by writing it.

Discussion

Discussion• Respondents gave higher ratings to the use of artificial
intelligence than to practice with a human partner• Less resistance to speaking English• Not embarrassed even if they made mistakes• Practice the same conversations repeatedly• Practice whenever it was convenient for them
• Very useful learning materials for False Beginners who are
not very good at practicing speaking against humans.

Future Work

Future Work

• Free Conversation with
ChatbotーBard, ChatGPT• Character / Avatar• AR/VR• Metaverseこの写真 の作成者 不明な作成者 は CC BY-NC-ND のライセンスを許諾されていますResearch Grant Funding

This study was supported by JSPS KAKENHI Grant
Number 21K00688.