ZERO-SHOT CODE-SWITCHING ASR AND TTS WITH
MULTILINGUAL MACHINE SPEECH CHAIN
1,211,21,2Sahoko Nakayama , Andros Tjandra , Sakriani Sakti , Satoshi Nakamura
1Nara Institute of Science and Technology, Japan
2RIKEN, Center for Advanced Intelligence Project AIP, Japan{nakayama.sahoko.nq1, andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp
2. Proposed Method

1. Introduction

Code-switching(CS):switching languages withina conversation
CS challenges for ASR & TTS:
need to handle themultilingual input
Existing approaches• Just developed on ASR orTTS• Only focused on a single
language pair
• Trained in a supervisedfashionGoal• CS tasks of ASR and TTS on
multiple language pairs• Trained semi-supervisedlearning• Zero-shot CS:
performing the unknown
CS w/o directly learning it

Proposed Machine Speech Chain [Tjandra et al., 2017,2018,2019]:
a closed-loop architecture that enables ASR & TTS to assist each otherMethod: embedding Language Identification Discrimination (LID) to machine speech chain
Multilingual Speech Chain
𝑥=𝑦ො = "𝑐ℎ𝑟"ASR𝑥ො =𝑥ො =ASR𝑥ො =𝑦ො = "𝑐ℎ𝑟"
TTS

𝑥=TTS
𝑦 = "𝑐ℎ𝑎𝑟" 𝑥ො =ASR+ 𝑦ො = ["𝑐ℎ𝑟“, "𝑙𝑛𝑔“]LID𝑥ො =𝑦ො = ["𝑐ℎ𝑟“, "𝑙𝑛𝑔“] SPKREC𝑧
TTS
𝑦 = ["𝑐ℎ𝑟“, "𝑙𝑛𝑔“]𝑥ො =(c) Proposed multilingual speech chain𝑦ො = "𝑐ℎ𝑟"𝑥=

SPKREC𝑦ො = "𝑐ℎ𝑟"𝑧𝑦 = "𝑐ℎ𝑟"(b) Multi-speaker speech chain
with speaker recognition(SPKREC)(a) Basic speech chain

Training Process for Multilingual Speech Chain𝑥=𝑥=𝑦 = ["𝑐ℎ𝑟“,"𝑙𝑛𝑔“]ASR+ 𝑦ො = ["𝑐ℎ𝑟“,"𝑙𝑛𝑔“]
LID
SPKREC 𝑧𝑥ො =ASR+LID

TTSTTS𝑥ො =En Ja
En JaZhZhtext
speechMonolingual train & test set(a) Supervised learning on
monolingual data

Zh-En
Zero-shot
CS text𝑦ො = ["𝑐ℎ𝑟“,"𝑙𝑛𝑔“]

SPKREC𝑧𝑥=

ASR+LID𝑦ො = ["𝑐ℎ𝑟“, "𝑙𝑛𝑔“] SPKREC𝑧TTS𝐿𝑜𝑠𝑠𝐿𝑜𝑠𝑠𝑦 = ["𝑐ℎ𝑟“, "𝑙𝑛𝑔“]𝑥ො =En-Ja Ja-Zhtext only
CS train & test set

En-Ja Ja-Zh
speech only
CS train & test set

Zh-En
Zero-shot
CS speechb Unsupervised learning on code-switching data

3. ASR Evaluation
I. Proposed model on zero-shot CS (Known language)
Monolingual (CER)
Code-switching (CER)
ModelJaEn
Zh EnJaCS JaZhCS ZhEnCS
Baseline: Supervised training on monolingual data only
Ja25k+En25k+Zh25k (paired)
8.85% 8.48% 5.11% 14.06% 16.91% 16.04%
Proposed Machine Speech chain: Semi-supervised training on two CS data
+ EnJaCS10k+JaZhCS10k (unpaired) 9.18% 12.71% 5.93% 11.56% 8.31% 10.52%+EnJaCS10k+ZhEnCS10k (unpaired) 8.93% 12.34% 5.67% 11.18% 9.21% 9.71%+ZhEnCS10k+JaZhCS10k (unpaired) 8.91% 14.45% 6.08% 11.85% 10.40% 11.29%
Topline: Supervised training on CS data
+EnJaCS10k+JaZhCS10k(paired)10.18% 12.32% 7.93% 8.94% 6.70% 8.09%+EnJaCS10k+ZhEnCS10k(paired)11.04% 10.91% 7.48% 10.81% 7.26% 8.07%+ZhEnCS10k+JaZhCS10k (paired)
10.98% 11.57% 7.22% 10.34% 7.72% 7.98%+EnJaCS10k+JaZhCS10k+ZhEnCS10k 10.48% 10.43% 6.88% 8.68% 6.98% 8.05%
Language used as pair
Language used as unpair
Zero-shot CS• Improved ASR in the multilingual CS test
• Performed well on an unknown CS test

II. Proposed model on zero-shot CS(Unknown language)

(CER)
40%

Baseline[Ja25k+En25k+Zh25k (paired)]30%

Proposed[+EnJa10k+JaZh10k+EnFr5k(unpaired)]

20%Topline[+EnJa10k+JaZh10k+EnFr5k(paired)]10%
0%

Zero-shot CS

FrZh test• Also improved on zero-shot CS withunknown Fr language

Model:
Encoder-decoder attention ASR
Data: Monolingual BTEC
III. Proposed model on zero-shot CS (CS natural speech)
CS created from BTEC
• Improved even in the case of using CS natural speech (see paper for more details)
4. TTS Evaluation
AB preference subjective evaluation

5. Conclusion
Setup
Model: Tacotron TTS,
DeepSpeaker SPKREC

EnJaCS
JaZhCS•ZhEnCS
0%

20%

40%

Without LngEmb

Same

60%

80%

100%

Proposed (With LngEmb)•Maintained TTS quality
betterEspecially on the switch
position between two
languages• Introduced a zero-shot CS ASR &TTS
• Proposed multilingual machine
speech chain with LID• Improved the performance of themultilingual CS• Also performed well on the unknown
CS without directly learning it