Japanese-English Code-switching
Speech Data Construction〇Sahoko Nakayama, Takatomo Kano, Quoc
Truong Do, Sakriani Sakti, Satoshi Nakamura
Nara Institute of Science and TechnologyJapanCopyright © Nara Institute of Science Technology All rights Reserved

1/20

Code-switchingCode-switching (CS):A speaker switches languages within a conversation
 [Inter-sentential code-switching]:
ああ、そうだってね。 On the honeymoon, they bought this. [Nakamura+2005](Oh,year, your’re right. On their honeymoon, they bought this) [Intra-sentential word-level code-switching]:
確率を与えたので、その確率を使ってrecountingします。[From NAIST lecture ](Since we gave a probability, we recalculate using that probability.) [Intra-sentential phrase-level code-switching]:
わたしね、that’s not the pointとか言われたもんね。 [Fujimura+2013](I was told that that’s not the point.)Loanword insertion:
中間言語を使った時のメリットに何があるか？ [From Naist lecture](What is the merit of using an interlingua?)
Copyright © Nara Institute of Science Technology All rights Reserved

2/20

Bilinguals and Code-switching
Code-switching plays a vital role in bilingualism. [McSwan+2000]
Bilingual speakers have increased in Japan.
Childrenwith a foreign parentChildren
who have lived abroad(people)25000(people)14000

21,180(people)70000

12,527

12000

60000

10000

50000

8000

40000

20000
15000

13,686

6000

10000
5000

Students
who study abroad

5,900

30000

4000

20000

2000

10000

00
1990

2016

(years)Data : MHLW

60,64323,63301977

2015(years)Data : MEXT

Copyright © Nara Institute of Science Technology All rights Reserved

2006

2016

(years)Data : JASSO

3/20

Reports of Code-switchingin Japanese-English Case

1. A Japanese child who lives in the US
59 code-switching in half an hour
[Nakamura+2005]
By aturukus

2. Bilingual children living in Japan, who have foreign parents
153 code-switching in 4 hours[Fujimura+2013]

Copyright © Nara Institute of Science Technology All rights Reserved

4 /20

Speech Recognition
Requirement

Realize to recognize every conversation<Usage scenes >1. Lecture, Meeting
翻訳された文章をtarget sentenceといいます。
[From NAIST lecture ](The translated sentence is called
target sentence)

2. Dialogue with a robot
えっと、日本語で”coincidence”って何ていうんだっけ？
(Um, what do you say “coincidence”
in Japanese?)

Copyright © Nara Institute of Science Technology All rights Reserved

5/20

Previous Research in ASR
Tried to train speech recognition to recognize code-switching
Chinese

code-switching (GMM-HMM) [Vu+2012]EnglishFrisiancode-switching (DNN-HMM) [Yilmaz+2016]
Dutch→Japanesecase has received scant research
EnglishWe focus on Japanese-English code-switching.
Copyright © Nara Institute of Science Technology All rights Reserved

6/20

Data Construction
Japanese-English code-switching ASR must be developed, but:
Problems
Previous collected
No large-scale data exists
code-switching speech data
①for training model Mainly from conversation②Time-consuming andexpensive for collecting

So we constructedJapanese-English code-switching
speech database by utilizing TTS

1. Mandarin-English [Lyu+2015]
2. Frisian-Dutch [Heuvel+2016]
3. Spanish-English [Solorio+2008]
4. Turkish-German[Herkenrath+2012]
5. Mandarin-Taiwanese[Lyu+2008]

etc…
Copyright © Nara Institute of Science Technology All rights Reserved

7/20

The Overview of Code-switching
Data Construction
1Japanesetext

2
Selectword/phrase

Text

3
Translate into
English4
Code-switchingtext

Speech5Synthesize &get time information

6
Find time to switch& trimming

7
Synthesize
English part8
Code-switchingspeech

Copyright © Nara Institute of Science Technology All rights Reserved

8/20

Data Resourcesin Text Data Construction
Available data resources:
Monolingual BTEC text data The ATR Basic Travel Expression Corpus(BTEC) Basic conversations in travel domains
We used Japanese/English BTEC text data called BTEC1,2,3, and 4.

Text
1
BTECJapanese text2
Selectword/phrase

3
Translate into
English4
Code-switchingtext
Copyright © Nara Institute of Science Technology All rights Reserved

9/20

Data Resourcesin Speech Data Construction
Available data resources:
Monolingual Japanese and English TTS system Use HMM-based Speech Synthesis System(HTS) Generated based on a bilingual BTEC speech data
Bilingual BTEC speech data
 Originally constructed to emphasize a speech translation study Use the data from 1 bilingual speakerSpeech
BTECJapanesetext
5Synthesize &get time information

Code-switchingtext

6
Find time to switch& trimming

7
Synthesize
English part

Join

Copyright © Nara Institute of Science Technology All rights Reserved

8Code-switching
speech10/20

Word-levelin Text Data Construction
 Intra-sentential word-level code-switching①

観光 バス の パンフレット は あり ます か(do you have any brochures for the sightseeing bus?)※Commonly select katakana characters

Copyright © Nara Institute of Science Technology All rights Reserved

11/20

Word-levelin Text Data Construction
 Intra-sentential word-level code-switching①

観光 バス の パンフレット は あり ます か(do you have any brochures for the sightseeing bus?)※Commonly select katakana characters②

観光 バス のは あり ます か(do you have any brochures for the sightseeing bus?)
パンフレット⇒ pamphlet
by Google TranslationAPI

Copyright © Nara Institute of Science Technology All rights Reserved

12/20

Word-levelin Text Data Construction
 Intra-sentential word-level code-switching①

観光 バス の パンフレット は あり ます か(do you have any brochures for the sightseeing bus?)※Commonly select katakana characters②

観光 バス のは あり ます か
(do you have any brochures for the sightseeing bus?)
パンフレット⇒ pamphlet
by Google TranslationAPI③

観光 バス の pamphlet は あり ます か(do you have any brochures for the sightseeing bus?)
Copyright © Nara Institute of Science Technology All rights Reserved

13/20

Phrase-levelin Text Data Construction
 Intra-sentential phrase-level code-switching
これ、ウルトラマンティガ か, what do you think this is?
(Yes, this (is) Ultraman-Tiga or what do you think this is?)① Reference それ は、 that’s not his arm.
[Nakamura+2005](Speaking of that, that’s not his arm)
ガス 入り の 炭酸 水 を 二 本 氷 と 一緒 に 持っ て き て ください
(Bring me two bottles of carbonic minerals and some ice please)
Trigger is Japanese particles:
wa, ga, wo, ni, he(e), to, ka, kara, yori, ba, temo, keredo,noni, node, kara, nari, nagara, tari, tsutsu

Copyright © Nara Institute of Science Technology All rights Reserved

14/20

Phrase-levelin Text Data Construction
 Intra-sentential phrase-level code-switching①②ガス 入り の 炭酸 水 を 二 本 氷 と 一緒 に 持っ て き て ください
(Bring me two bottles of carbonic minerals and some ice please)
ガス 入り の 炭酸 水 を(Bring me two bottles of carbonic minerals and some ice please)
二 本 氷 と 一緒 に 持っ て き て ください⇩by Google Translation API

bring me with two ice

Copyright © Nara Institute of Science Technology All rights Reserved

15/20

Phrase-levelin Text Data Construction
 Intra-sentential phrase-level code-switching①②

ガス 入り の 炭酸 水 を 二 本 氷 と 一緒 に 持っ て き て ください
(Bring me two bottles of carbonic minerals and some ice please)
ガス 入り の 炭酸 水 を(Bring me two bottles of carbonic minerals and some ice please)
二 本 氷 と 一緒 に 持っ て き て ください⇩by Google Translation API

bring me with two ice③

ガス 入り の 炭酸 水 を bring me with two ice.
(Bring me two bottles of carbonic minerals and some ice please)
Copyright © Nara Institute of Science Technology All rights Reserved

16/20

Speech Data Construction
Englishword/phrase

Japanese sentence(Does that include breakfast and dinner?)
昼食と夕食つきですか0

1.31

1.76

dinner
3.23

Trimming

Copyright © Nara Institute of Science Technology All rights Reserved

17/20Data Analysis
Statistics of Japanese-English
code-switching speech utterances
100%

161K
80%

749K
60%
40%

2295K

8873K

1147K

632K

20%
0%
Monolingual
Japanese

Word-level code- Phrase-level codeswitchingswitching

MonolingualEnglishJapanese words
English words※The number in the bar is the number of words.
Copyright © Nara Institute of Science Technology All rights Reserved

18/20

Conclusion
Problem
For Japanese-English
code-switching ASR① No large-scale JapaneseEnglish code-switching dataexists

We constructed
a Japanese-English
code-switching corpus

and② Time-consuming
expensive for collecting

Utilizing
Japanese and Englishbilingual TTS

Word level

Phrase level

code-switchingcode-switching

146k146k(12% English)(54% English)

Copyright © Nara Institute of Science Technology All rights Reserved

19/20

Future works
Investigate the quality byincreasing speakers andcomparing with naturalcode-switching speechUtilize it to enhance ourspeech recognition system
for the bilingual community

Copyright © Nara Institute of Science Technology All rights Reserved

20/20

The End

Copyright © Nara Institute of Science Technology All rights Reserved

21/20

Appendix

Copyright © Nara Institute of Science Technology All rights Reserved

22/20

Reference
M.Nakamura, “Developing codeswitching patterns of aJapanese/English bilingual child,” in Proceedings ofthe 4th International Symposium on Bilingualism, 2005,
pp.1679-1689
K.Fujimura, “Inevitable Language Outcome: The Use ofCode- switching and Code-mixing by Japanese People
Living in London, England,” The bulletin of Yasuda
women university, 2013, 41, 23-32
Jeff McSwan, “The architecture of the bilingual language
faculty: Evidence from intrasentential codeswitching,”
Bilingualism: Language and Cognition, vol.3, no.
1, pp.37-54, 2000.

Copyright © Nara Institute of Science Technology All rights Reserved

23/20

ReferenceJapanese Ministry of Health, Labour and Welfare,“Overviewof the population statistics in 2016 [in Japanese],”http://www.mhlw.go.jp/, 2016.
Japanese Ministry of Education, Culture, Sports, Science,and Technology, “School basic survey in 2015 [inJapanese],” http://www.mext.go.jp/, 2015.
Japan Student Service Organization, “Survey on japanese
student abroad situation in 2016 [inJapanese],”http://www.jasso.go.jp/, 2016.

Copyright © Nara Institute of Science Technology All rights Reserved

24/20

Reference
N. T. Vu, D. C. Lyu, J. Weiner, D. Telaar, T. Schlippe, F.
Blaicher, E. S. Chng, T. Schultz, and H. Li, “A firstspeech recognition system for mandarin-english
code-switch conversational speech,” in 2012 IEEE
International Conference on Acoustics, Speechand Signal Processing (ICASSP), March 2012, pp.
4889–4892.
Emre Ylmaz, Henk van den Heuvel, and David vanLeeuwen, “Investigating bilingual deep neural networks for automatic recognition of code-switching
Frisian speech,” Procedia Computer Science, vol.
81, pp. 159 –Copyright © Nara Institute of Science Technology All rights Reserved

25/20

Corpus sample
Text

Wordlevel

Phraselevel

ThisstudyGoogleTTS

私 の baggage が 見つかり ません。
(I can’t find my baggage)
これ は declaration して ません ね？
(You haven’t declared this have you?)
私 の 荷物 を please look for it.
(Please look for my luggage)
申告 する もの が Do you have?
(Do you have anything to declare?)Copyright © Nara Institute of Science Technology All rights Reserved

26/20

Available Data Resources
1. Monolingual BTEC text data
The ATR Basic Travel Expression Corpus(BTEC)
Basic conversations in travel domains
Collected by bilingual travel experts from Ja/En
sentence pairs in travel domain phrasebooks
We used Japanese-English BTEC text data called
BTEC1,2,3, and 4:Copyright © Nara Institute of Science Technology All rights Reserved

27/20

Available Data Resources
2. Bilingual BTEC speech dataConstructed to emphasize a speech translation
study
1015 Japanese/English sentence pairs wereselected from 16,000 BTEC sentences
Recording was done with 3 bilingual speakers
WAV audio was recorded with a frequency of16KHz,16 bits and a single channel
We only use the data from 1 bilingual speaker.

Copyright © Nara Institute of Science Technology All rights Reserved

28/20

Available Data Resources
3. English and Japanese HMM-Based speech synthesis

EngineTextSpeaker
Speech Training datasynthesis Sampling rate(TTS)
Frame size/shift

HMM-based Speech Synthesis
System(HTS)Japanese-English BTEC
One bilingual speaker
960 utterances
16kHz
25ms / 5ms

Feature vector

40-dimensional MGC, log F0Vocoder

WORLD vocoder

Copyright © Nara Institute of Science Technology All rights Reserved

29/20

SEAME(Mandarin-English)
corpus• South-East Asia Mandarin-English corpus• 156 speakers(36.8% are Malaysian while the rest
are Singaporean)• 192 hours of wav audio records• Speaking style: conversation and interview• In total 110,145 segments• 28,655(English)• 24,438(Mandarin)• 57,052(Code-switch)
Copyright © Nara Institute of Science Technology All rights Reserved

30/20

FAME!: Frisian radio broadcast
database• The regional public broadcaster of the province Frisian• The total duration of the manually annotated radio
broadcasts sums up to 18 hours extracted from radio
programs almost 50 years• Topic is about culture, history, literature, sports, nature,agriculture, politics, society and languages.
• 309 identified speakers and 233 unidentified speakers• Total number of code-switching case is 3837

Copyright © Nara Institute of Science Technology All rights Reserved

31/20