From Speech Chain to Multimodal Chain:
Leveraging Cross-modal Data Augmentation for Semi-supervised Learning
Johanes Effendi1,2, Andros Tjandra1, Sakriani Sakti1,2, Satoshi Nakamura1
1 Nara Institute of Science and Technology
2 RIKEN Center for Advanced Intelligence Project (AIP){johanes.effendi.ix4, andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Part of this work is supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237 as
well as NII CRIS Contract Research 2019 and Google AI Focused Research Awards Program

Copyright © NAIST 2020
All Rights Reserved

Introduction• Most machine learning system are mostly trained on parallel dataset– However, this is not always applicable– Need ability to learn from unpaired data

• Feedback links to use unpaired dataset– Dual NMT (He+, 2016); CycleGAN (Zhu+, 2017)src_sent

src to trg

Reconstruction loss

hyp_trg_sent
hyp_src_sent

trg to src[1] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. In Advances in Neural Information Processing Systems, pages 820–828, 2016.
[2] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International
Conference on, 2017.
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan2Speech Chain• Model of human communication• Providing a technology with the ability to listen and speak“Goodafternoon”“How are you?”Sensory
nervesAuditory feedback

Speaking

Speechrecognition
Listening

Motornerves

Sensory
nerves“Good afternoon”
Recognized words

Speaking

SpeechSynthesis
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan3

Previously: Machine Speech Chain [Tjandra+, 2017]• Develop a closed loop speech chain model based on deep learning• The first deep learning model that integrates human speech perception and production
behaviours“Goodafternoon”
Sensory
nervesAuditory feedback“How are you?”

Speaking
Motornerves

Auditory feedback
Speaking[1] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. CoRR, abs/1707.04879, 2017.

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan4

Machine Speech Chain
Speaking

Feedback
Listening

A closed-loop architecture:→ In training stage:
 Allow to train with labeled and unlabeled data (semi-supervised learning) Allow ASR and TTS to teach each other using unlabeled data and generate useful feedback→ In Inference stage: Possible to use ASR & TTS module independently
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan5

Speech Chain Limitation• Speech chain is still limited to speech and textual modalities• What if all the text and speech data has been used? Can we use other data?
“a cake”Sensorynerves

Visual feedback
Auditory feedback

Motornerves

Speaking• Natural communication is auditory + visual
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

6

Proposed: Multimodal Machine Chain• Visual chain = image captioning (IC) + image retrieval (IR)• Multimodal chain = speech chain + visual chain

Speech Chain[Tjandra+ , 2017]
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Multimodal Machine Chain(Proposed Method)7Model Details• ASR: Listen, Attend, and Spell [Chan+, 2016]– LSTM encoder-decoder ASR model• TTS: Tacotron [Wang+, 2017]– encoder-decoder speech synthesis model• IC: Show, Attend, and Tell [Xu+, 2015]– Using ResNet encoder• IR: Joint Multimodal Embedding [Vilalta+, 2015]– Modified from full network embedding to normal encoder embedding

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

88 / 36

Framework Training Mechanism

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

9

Possibility of Data Condition
Assume we index all of our data row into ii for speech, jj for text, and kk for image• Type 1: Pairedii == jj == kk(i.e.: every speech has transcription, every image has caption)• Type 2: Unpairedii != jj != kk(i.e.: speech, text, image are available but no correlation)• Type 3: Single modality
ii xor jj xor kk(only speech or text or image is available)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan10

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

ASR

ICTTS

IR

Type 1(ASR, TTS, IC, IR)supervised learning

11

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Type 2a and 2c.
(ASR → TTS, TTS → ASR)
speech chain12

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Type 2b and 2c.
(IC → IR, IR → IC)visual chain13

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Type 3b. (ASR → IR → IC)speech chain → visual chain

Improving IC even without
image and text data14

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Type 3c. (IC → TTS → ASR)
visual chain → speech chain

Improving ASR even without
speech and text data15

Experiment

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan16

Dataset• Experiment using Flickr30k dataset(train 29k, dev 1k, test 1k)• Photos of everyday activities andscenes• One image has five caption sentences• Single speaker synthesized speechusing Google TTS

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, JapanDataSpeech

Text

Image

# Data

(D1)

○○○

2000

(D2)

∆∆

∆

7000

(D3x)

∆x

x

10000

(D3z)

xx

∆

10000

○ = available paired; ∆ = available unpaired; x = not available

17

Training Mechanism

Type 1: Paired speech-text-image data exists(Supervised learning)• Separately train ASR, TTS, IC, and IR
Type 2: Speech, text, and image data exists, but unpaired(Unsupervised learning)a. speech data: speech chain ASR → TTS
b. image data: visual chain IC → IRc. text data: speech chain TTS → ASR, visual chain IR → IC
Type 3: Single modality data (either speech, text, or image exist)(Unsupervised learning)a. text data : (TTS → ASR) || (IR → IC)b. speech data : (ASR → TTS) || (ASR → IR → IC)c. image data : (IC → IR) || (IC → TTS → ASR)

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

Partition of the Flickr30k dataset
DataSpeechTextImage# Data(D1)○○○

2000(D2)∆∆∆

7000(D3x)

∆xx

10000

(D3z)

xx

∆

10000

○ = available paired; ∆ = available unpaired; x = not available

18Evaluation Metric• ASR: word error rate
– comparing edit distance of hypothesis transcription and ground truth transcription• TTS: L2-norm^2– Comparing generated mel-spectrogram with ground truth mel-spectrogram• IC: BLEU
– Bilingual Evaluation Understudy [Papineni+, 2002]– commonly used in NMT to compare using matching n-grams• IR: Recall @ k– Proportion of relevant items in the top-k retrieved result• IR: med r– Highest ranked ground truth result

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan19

Experiment Settings• Using the train, dev, and test set of Flickr30k• Experiment purpose:– Not to show that the multimodal chain can outperform a baseline that was only trained with a small
dataset– But to identify how much we can improve performance when the required data are no longer
available• Trained a topline to show that our model works as good as other model whensupervised training is possible– ASR and TTS using Wall Street Journal (WSJ-SI284) speech dataset– IC and IR using Flickr30k dataset

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan20

Topline Results (when a large amount of paired data exists)
IC and IR performance on Flickr30k

ASR and TTS performance on WSJ-SI284
Data

ASR
TTS
CER(%) ↓ L2-norm2↓

Data

IC
BLEU1↑

IRR@10 ↑Med r ↓

Kim et al. (2017)

11.08-Xu et al. (2015)

67.00

Tjandra et al. (2018)

6.60

0.682

Vilalta et al (2017)

-59.8

6Ours

6.87

0.653Ours

66.27

62.4

5• Train each model independently using supervised training• In fully supervised condition, our model works as well as previously published studies.

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan21

Experiment Results (when only a small amount of paired data exists)
ASR and TTS results on Flickr30k
Data

ASR WER (%) ↓TTS L2-norm2↓
Initial model: ASR and TTS
Supervised learning – Type 1

D1 2k
81.31
0.874
Speech+text
Proposed: speech chain ASR→TTS and TTS→ASR
Semi-supervised learning – Type 2a and 2c+D2 7k
10.60
0.714
Speech/text
Proposed: visual chain → speech chain
Semi-supervised learning – Type 3c+D3z 10k
7.970.645
Image only
Topline: ASR and TTS separately
Supervised learning – Full data
All 29k
2.370.398

IC and IR results on Flickr30k
Data

IC BLEU1↑ IR R@10↑ IR med r↓
Initial model: IC and IR
Supervised learning – Type 1

D1 2k
33.91
26.88
34
Image+text
Proposed: visual chain IC→IR and IR→IC
Semi-supervised learning – Type 2b and 2c+D2 7k
42.11
28.14
31
Image/text
Proposed: speech chain → visual chain
Semi-supervised learning – Type 3b+D3x 10k
43.08
28.44
30Speech only
Topline: IC and IR separately
Supervised learning – Full data
All 29k66.27
62.42
5ASR improvement even without speech and text data available
Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan

22

Conclusion and Future Work• Empower speech chain into multimodal chain• Visual chain by jointly training IC and IR model• ASR improvement even without speech and text data available• Jointly train both speech and visual chain together• Investigate result on natural multispeaker speech dataset

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan23

• End of slide

Johanes Effendi @ AHC Lab., NAIST/RIKEN AIP, Japan24