Developing and Evaluating
an E-Learning Material
for Speaking Practicewith the Latest AI Technology
Makoto Shishido
Tokyo Denki UniversityJapan

Contents• Introduction• Problems in Teaching English Speaking in Japan• Several Previous Trials• The First Trial with AI• Shikibu• The Second Trial with AI• Speakbuddy• The Third Trial with AI– Current study• ELST- Introducing japan in English• Future Possibilities

Problems in Teaching Speaking in Japan• Japanese EFL teachers’ pronunciation• High dependency on native speakers• Large class size (30 – 40 students / class)• Lack of teaching strategies• Lack of effective teaching materials

Expectations for improving speaking skills• University Education Reform – Emphasis onfour skills－especially speaking skills• Globalization• Inbound, Increasing Visitors from Overseas• 2020 Tokyo Olympics and Paralympics

E-Learning Materials for Speaking Practice• English Central• Callan Method• Shikibu• SpeakBuddy• ELST

English Central (2017年)

Voice Recognition + Artificial Intelligence• Advancement in Voice Recognition• Siri• Google Assistant• AI Speakers• Advancement in Artificial Intelligence• Watson (IBM)• Self-Driving• One to One practice• Anytime / Many times

The First Trial ー Shikibu (NTT Learning Systems)• Developed by NTT Learning Systems in Japan• Siri + Watson

Study• 26 freshmen of a university in Japan• Computer science majors• July to September, 2017
• For three months• During summer break• Self-study

Main Menu

Exercise Menu

Questionnaire
1. English learning － English ability
2. Shikibu’s functions－ effectiveness3. Contents
4. Benefits of using the app
5. Future use

Were you interested in an app with AI?
8%

23%

31%

38%

Strongly yes
Somewhat yesa littleNo

Did you enjoy?
48

28 108Satisfied?

-12

611-32

Felt achievement?

-52-10Effective?

17Enthusiastic?

Did you find AI conversation natural?
8%

50%

8%
34%

Yes
somewhat yes
rather noNo

Good features of practicing with AI
less resistance to speaking English

50

not embarrassing to make a mistake

54

35

15

Responded naturally

15 03850

8 0 Yes278

somewhat yes
rather no38

able to practice repeatedly

54
50

able to study anytime
0%

10%

20%

8 0 No

3830%

40%

50%

60%

70%

12 080%90%100%

practice with AI matched your learning style?
4%

19%

15%

62%

Yes
somewhat yes
rather noNo

The Second Trial ー SpeakBuddy (appArray)• Developed by appArray in Japan• Voice Recognition (Google) and Artificial IntelligenceStudy• 209 freshmen of a university in Japan• Information and Computer science majors• April to December 2018
• For nine months• In class study• Questionnaires - last week in May.

SpeakBuddy

Questionnaire
1. English learning － English ability
2. SpeakBuddy’s functions－ effectiveness3. Lesson contents
4. Benefits of the English conversation app using AI
5. Future useDid you enjoy?
80
60
40

27

200-20Satisfied?
7-40

Felt achievement?

-60
-80

Effective? 64

55-76

Enthusiastic?

GOOD FEATURES
less resistance tospeaking English
19%

able to study
anytime28%

not embarrassing tomake a mistake
26%

able to practice

Responded naturally

repeatedly

7%

20%

Summary• Practice English utterance• on a one to one basis• low cost
• anytime and anywhere• Less resistance to speaking English• Not feeling of embarrassment• making mistakes• able to repeat many times• Need further development• enhancing the performance of speech recognition

Third Trial － ELST (sinewave)• Introducing Japan in English (ELST)• Developed by Sinewave in Japan• Voice Recognition and Artificial Intelligence (iFLYTEK)Introducing Japan in English (ELST)• Sinewave and Seibido• Voice Recognition• Artificial Intelligence• Evaluation and Scoring• Textbook + E-Learning (with AI)• Reading• Listening• Speaking• WritingELST ー Introducing Japan in English (Menu)

Vocabulary PracticeVocabulary Test

Grammar Exercise

Reading - ShadowingDialog PracticeSpeaking PracticeComparisons of Pre-Test andPost-Test Results ofOPIc and TOEICOPIc Pre and Post Test Comparison
12

Number of Student

1010910886

6Pre-Test

44

Post-Test

220

12

3OPIc Level

14

TOEIC Pre and Post Test Comparison
7

Number of student

65
4Pre-test

3

Post-Tst

2
10Score range

Questionnaire

24%

1 Interest in ELST

0%

53

23%

23%

450%

Did you enjoy?
5.6
Improvement?

65

5.4

4

3.2

Felt achievement?

32
10

Satisfied? 4.4

2Effective?

3.6

Enthusiastic?

less resistance tospeaking English

able to studyanytime 7.8

able to practice 5.8
repeatedly

8765432
107

not embarrassing
8

to make a
mistake

5 Responded

naturally

Felt disconfort inspeaking with AI
1.5

1 0.6

Unnatural
1.2
conversation

0.5

AI did not

00.2

-0.5
-1

understand what I
said

-1
0.2
Only set phrases

Felt artificial voices
strangeSpeaking practice with AI matches with your learning style
20%

1

510%

23%

330%

437%

Would you like to use this kind of app privately?
23%

110%

517%

333%

437%

Questionnaires-Comment 1
新鮮学習

面白い

Subgraph:
勉強

01

03

02
Frequency:
2

AI

3

English-fun-study

楽しい

英語授業

45

思う6

画期的

AI-interesting-studyfreshQuestionnaire – Comment 2
Pronunciation
Improvement
ContentGood
Score
ChangeGoodGo up

Summary• The results seem positive and effective.
• The students had fun, liked, and enjoyedstudying with AI.
• It is a fresh experience for them.
• They felt improvement.
• Their abilities and OPIc scores went up.

Future Development
Free Conversation with AI
Using Multiple Choice Chat Bot
6 turns × 3 multiple choice = 27 patterns

An Example of Multiple-Choice Chatbot
AIStudentAIStudent
4.1.1. I live alone.

2.1. I’m going home.

3.1. Who is waiting?

4.1.2. My mom.
4.1.3. My dog.
4.2.1.I study science.

1. Where are you going? 2.2. I’m going to school.

3.2. What do you study? 4.2.2.I study economics.
4.2.3.I study math.
4.3.1. I want a jacket.

2.3. I’m going shopping.

3.3. What will you buy?

4.3.2.I will get a shirt.
4.3.3.I am not sure yet.