Recognition and Translation of
Code-switching Speech Utterances
Sahoko Nakayama1,2, Takatomo Kano1, Andros Tjandra1,
Sakriani Sakti1,2, Satoshi Nakamura1,2
1Nara Institute of Science and Technology, Japan
2RIKEN, Center for Advanced Intelligence Project AIP, Japan{nakayama.sahoko.nq1, kano.takatomo.km0, andros.tjandra.ai6,ssakti, s-nakamura}@is.naist.jp
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

1 /30

Background

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

2/30

Bilingualism

Bilingual speakers have increased in Japan
Children who have lived abroad

Data : MHLW

25000

Population

Population

15000
10000
500001990

2016
Year

Data : MEXT

14000

20000

Students who studied abroad

12000

60000

10000

50000

8000
6000
4000

40000
30000
20000

2000

10000

00

1977

Data : JASSO

70000

Population

Children with a foreign parent

2015
Year2006

2016
Year

Code-switching (CS) plays an important role in bilingualism [McSwan, 2000].

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

3/30

CS Definition
Speakers switch languages within a conversation.
 Word-level CS:
国会が the Equal Employment Opportunity Law に罰則を設けなかったので、空文だという意見があります。
(As the Diet did not put any teeth into the Equal Employment Opportunity
Law, some are of the opinion that it is a mere scrap of paper.) Phrase-level CS:
If I could make a suggestion, この議題についての討議を昼食までに終えて頂ければと思いますが。
(If I make a suggestion, would you finish discussing this subject by lunchtime?)

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

4/30

CS Coverage of This Study
The definition of what constitutes CS is controversial.
 Are loanwords word-level CS?

中間言語を使った時のメリットに何があるか？
(What is the merit of using an interlingua?) Are quotations phrase-level CS?
What do you think of the Japanese saying, "うそつきは泥棒の始まり"?
(What do you think of the Japanese saying, "Show me a liar and I'll show you athief"?)

Theoretically, they may not be CS.
But in this study we aim to properly handle as many cases as possible.

Therefore, we will try to handle these cases as well.
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

5/30

How Code Switching Occurs

Proficiency-driven CS• A speaker is competent in both languages• Easily able to switch from one language to another.
Deficiency-driven CS• A speaker is lack of competency of one language• Go back to another language.

As the data of deficiency-driven CS has not been obtained yet, we only
handle the proficiency-driven case in this work.

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

6/30

CS Challenges For ASR
 Standard Automatic Speech Recognition (ASR) is monolingual
JapaneseASR“こんにちは”EnglishASR“Hello”(Hello) Challenge with CS: need to handle multilingual input“これはstill waterですか？”(Is this still water?)JapaneseEnglishJapanese

Output text

ASR?

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

7/30

Previous Works

Several works have constructed CS ASR.
 Mandarin-English CS with phone merging and language
identification [Vu et al., 2012]
 Frisian-Dutch CS with bilingual deep neural networks [Yilmaz et al.,2012]⇒ Common aim: merely for transcribing CS-speech into CS-text
CS-speech

CS-text

ASR“父の手術には there is ahigh degree of risk.”(there is a high degree of risk inmy father's operation)

Copyright © Nara Institute of Science and Technology 2019. All Rights ReservedMonolingual
speakers

8/30

Proposed Approaches

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

9/30

Goal

Support monolingual speakers trying to understand CS speakers

CS-speech

CS-text
ASR“父の手術には there isa high degree of risk.”(there is a high degree of risk inmy father's operation)

CS-speech

Monolingual
speakers

Monolingual-text
ASR“there is a high degree ofrisk in my father's operation.”

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

Monolingual
speakers
10/30

Approaches

1.

Cascade approaches
1-1. CS2CS ASR + Mono-recovery BERT
1-2. CS2CS ASR + CS2Mono NMT

CS Ja-En
ASR

2.

CS Ja-En
Text

Post-processing

MonoEnText

Direct approaches
2-1. CS2Mono ASR with single-task learning
2-2. CS2Mono ASR with multi-task learning

CS Ja-En

ASR

MonoEnText

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

11/30

1. Cascade Approaches
1-1.

Cascade CS2CS ASR + Mono-recovery BERT

Post-processing

CS Ja-En
ASR

1-2.

CS Ja-En
Text

MaskedEn Text

Recovery(BERT)

MonoEnText

Cascade CS2CS ASR + CS2Mono NMT*
Post-processing

CS Ja-EnASR

CS Ja-En
Text

Translation (NMT)

MonoEn
Text*Neural Machine Translation (NMT)

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

12/30

2. Direct Approaches

2-1.

Direct CS2Mono ASR with single-task learning

CS Ja-En
ASR

2-2.

MonoEnText

Direct CS2Mono ASR with multi-task learning
CS Ja-En
CS Ja-En
Text

ASR
MonoEnText⇒

I will describe these proposed models one by one

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

13/30

1. Cascade Approaches
1-1.

Cascade CS2CS ASR + Mono-recovery BERT

Post-processing

CS Ja-En
ASR

1-2.

CS Ja-En
Text

MaskedEn Text

Recovery(BERT)

MonoEnText

Cascade CS2CS ASR + CS2Mono NMT*
Post-processing

CS Ja-EnASR

CS Ja-En
Text

Translation (NMT)

MonoEn
Text*Neural Machine Translation (NMT)

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

14/30

1-1. Mono-Recovery BERTBERT: bidirectional language model [Devlin et al., 2019]

Mono-recovery BERT Masks the 2nd language
 Recovers complete sentence of the 1st language
Source あのね，Charles may be a bad husband, but He’s 心の温かい人なのよ.
(you know, Charles may be a bad husband, but He’s a warm-heatedperson.)

Mask[MASK] [MASK] [MASK], Charles may be a bad husband, butHe’s [MASK] [MASK] [MASK].

Label

you know [PAD], Charles may be a bad husband, but He’s awarm-heated person.

Targetyou know, Charles may be a bad husband, but He’s a warmheated person.
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

15/30

1-1. Mono-Recovery BERT Architecture Based on the BERTBase model• Use multi-layer bidirectional Transformer [Vaswani et al., 2017]
pleasetakea

picture

forme

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Trm

Position
Embeddings

E0

E1

E2

E3

E4

E5

Token
Embeddings

EpleaseEtake

Ea

E[MASK]

Efor

Eme

Input

pleasetakea[MASK]

forme

OutputBidirectional
Transformer

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

16/30

Overview of Cascade BERT

OutputText

pleasetakea

picture

forme

BERT
Input
Text

pleasetakea[MASK]

forme

OutputText

pleasetakea

写真forme

写真formeASR
Input

pleasetakea

Given CS speech, we performs an ASR and produces CS text.
Then, we utilizes BERT to recover the monolingual text.
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

17/30

1. Cascade Approaches
1-1.

Cascade CS2CS ASR + Mono-recovery BERT

Post-processing

CS Ja-En
ASR

1-2.

CS Ja-En
Text

MaskedEn Text

Recovery(BERT)

MonoEnText

Cascade CS2CS ASR + CS2Mono NMT*
Post-processing

CS Ja-EnASR

CS Ja-En
Text

Translation (NMT)

MonoEn
Text*Neural Machine Translation (NMT)

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

18/30

1-2. NMT Model Sequence-to-sequence NMT with attention [Bahdanau et al., 2015]
Outputpleasetakeapicture

for

me

Decoder

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

E

E

E

E

E

E

Token
Embeddings

Attention
BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

Token
Embeddings

Eplease

Etake

Ea

E写真

Efor

Eme

Input

please

take

写真for

me

EncoderaCopyright © Nara Institute of Science and Technology 2019. All Rights Reserved

19/30

Overview of Cascade NMT

pleasetakeapicture

for

me

NMTpleasetakea

写真formepleasetakea

写真forme

写真formeASR

pleasetakea

Given CS speech, we performs an ASR and produces CS text.
Then, we utilizes NMT to translate from CS to monolingual text.

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

20/30

2. Direct Approaches

2-1.

Direct CS2Mono ASR with single-task learning

CS Ja-En
ASR

2-2.

MonoEnText

Direct CS2Mono ASR with multi-task learning
CS Ja-En
CS Ja-En
Text

ASR
MonoEnText
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

21/30

2-1. Direct Single-task Learning Sequence-to-sequence ASR with attention [Chan et al., 2016] [Tjandra et al., 2017]

Outputpleasetakeapicture

for

me

Decoder

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM

TokenEmbeddings

EE

E

E

E

E

Attention

Encoder

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTMplease

takea

Input

写真for

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

me

22/30

2. Direct Approaches

2-1.

Direct CS2Mono ASR with single-task learning

CS Ja-En
ASR

2-2.

MonoEnText

Direct CS2Mono ASR with multi-task learning
CS Ja-En
CS Ja-En
Text

ASR
MonoEnText
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

23/30

2-2. Direct Multi-task Learning Multi-task learning improves learning efficiency with a shared encoder.
Code-switching Output

aplease

take

LSTM

LSTM

E

E

Monolingual Output

写真formeplease

take

⋯

LSTM

LSTM

LSTM

⋯LSTM

⋯

E

E

E

⋯

E

apicture

formeAttention
SharedEncoder

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTMInputSpeechpleasetakea

写真forCopyright © Nara Institute of Science and Technology 2019. All Rights Reservedme24/30

Experiments

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

25/30

Datasets

Artificial speech

CS text created by translating part of BTEC
Google TTS speech
CS text created by a CS user
Bilingual reading speech

Natural speech
JaBTECCS User
CreateEnBTECCS
text

Anotherbilingual

Read1KCSspeech All artificial CS : Train 50K, Test 500
 Mix natural CS : Train 50K, Test 500
All texts are segmented with BERT wordpiece tokenizer.
Wordpiece: subword units proposed to effectively deal with rare words
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

26/30

Cascade Text-to-Text Results
Pre-experiment: check the difference between with ASR error and without
Wordpiece error rate* (%) ofphrase-level CS on BERT model
Artificial CS Mix Natural
Test
CS TestAll Artificial CS
9.75
14.96
Mix Natural CS
10.37
13.21
*𝑊𝑜𝑟𝑑𝑝𝑖𝑒𝑐𝑒 𝐸𝑟𝑟𝑜𝑟 𝑅𝑎𝑡𝑒[%] =Wordpiece error rate (%) of phrase-level
CS on NMT model
Artificial CS Mix NaturalTest
CS TestAll Artificial CS
7.56
16.34
Mix Natural CS6.34
19.16

𝑇𝑜𝑡𝑎𝑙 𝑊𝑜𝑟𝑑𝑝𝑖𝑒𝑐𝑒 𝐸𝑟𝑟𝑜𝑟𝑠
× 100
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑊𝑜𝑟𝑑𝑝𝑖𝑒𝑐𝑒𝑠 𝑖𝑛 𝑅𝑒𝑓𝑒𝑟𝑒𝑛𝑐𝑒

 NMT model translated all artificial CS test utterances well, but theBERT model translated the mixed natural CS test utterances better.
 Natural CS has more complex Japanese phrases, so the NMT task
became more difficult.

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

27/30

Cascade Speech-to-Text Results

Wordpiece error rate (%) of phrase-level
CS on cascade of ASR+BERT
Artificial CS Mix NaturalTest
CS TestAll Artificial CS
17.80
47.98
Mix Natural CS
18.62
29.22Wordpiece error rate (%) of phrase-level
CS on cascade of ASR+NMT
Artificial CS Mix NaturalTest
CS TestAll Artificial CS
11.07
43.52
Mix Natural CS
11.32
25.29

 However, in speech-to-text, NMT outperformed BERT.
 It seems a more difficult task for BERT because the ASR error increased
the number of [MASK] tokens.
 Therefore, we compare the cascade of ASR+NMT with other models.

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

28/30

Cascade and Direct Results

Wordpiece Error Rate[%]Cascade of ASR+NMT

Direct ASR

Multi-task Learning

45.00
40.00
35.00
30.00

25.00
20.00
15.00
10.00
5.00
0.00

All artificial CS test

Mix natural CS test

All Artificial CS Model

All artificial CS test

Mix natural CS test

Mix Natural CS Model Direct approaches seem to be better than the cascade model.
• They can learn speech information directly.
 Multi-task learning tends to have the best performance.
• It can improve learning efficiency and accuracy.
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

29/30

Conclusion
 We compared four ways to translate CS speech:
1.
2.
3.
4.

Cascade CS2CS ASR + mono-recovery BERT
Cascade CS2CS ASR + CS2Mono NMT
Direct CS2Mono ASR with single-task learning
Direct CS2Mono ASR with multi-task learning The results reveal that
• ASR error makes the task difficult for mono-recovery BERT.
• Direct approaches seem to be better than the cascade model.
• Multi-task learning tends to have the best performance.
 In the future, we will further investigate how natural the translation
results are by conducting an evaluation by bilingual people.
 As this paper is targeted only to proficiency-driven CS, we will handle
deficiency-driven CS in the future.
Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

30/30

References•Jeff McSwan, “The architecture of the bilingual language faculty: Evidence from intrasentential code-switching,” Bilingualism:
Language and Cognition, vol. 3, no. 1, pp. 37–54, 2000.

•Maria Lourdes S Bautista, “Tagalog-english code switching as a mode of discourse,” in Proc. of Asia Pacific Education Review,
2004, vol. 5, pp. 226–233.

• Ngoc Thang Vu, Dau-Cheng Lyu, Jochen Weiner, Dominic Telaar, Tim Schlippe, Fabian Blaicher, Eng-Siong Chng, Tanja Schultz, andHaizhou Li, “A first speech recognition system for Mandarin-English code-switch conversational speech,” in Proc. of ICASSP, Kyoto,Japan, 2012, pp.4889–4892.
• Emre Yilmaz, Henkvan den Heuvel, and David van Leeuwen, “Investigating bilingual deep neural networks for automatic
recognition of codeswitching Frisian speech,” in Proc. of SLTU, Yogyakarta, Indonesia, 2016, vol. 81, pp. 159 – 166.
• Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: pre-training of deep bidirectional transformers forlanguage understanding,” in Proc. of NAACL-HLT, 2019, pp. 4171–4186.
• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,“Attention is all you need,” in Proc. of NIPS, 2017, pp. 5998–6008.
• Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In
Proceedings of ICLR, 2015.

• William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, “Listen, attend and spell: A neural network for large vocabulary
conversational speech recognition,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2016, pp. 4960–4964.
• Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura, “Listening while speaking: Speech chain by deep learning,” CoRR, vol.
abs/1707.04879, 2017. [Online]. Available: http://arxiv.org/abs/1707.04879• Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, QinGao, Klaus Macherey, et al. “Google’s neural machine translation system: Bridging the gap between human and machinetranslation,” CoRR, vol. abs/1609.08144, 2016.

Copyright © Nara Institute of Science and Technology 2019. All Rights Reserved

31/30