言語処理学会 第26回年次大会 発表論文集 (2020年3月)

From Speech Chain to Multimodal Chain:
Leveraging Cross-modal Data Augmentation forSemi-supervised Learning
Johanes Eﬀendi1,21Andros Tjandra1

Sakriani Sakti1,2Nara Institute of Science and Technology2

Satoshi Nakamura1,2

RIKEN AIP

{johanes.effendi.ix4, andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

12Introduction

Recently, approaches that utilize learning fromfeedback links which enable model training with unpaired datasets have gained attention. He et al. [1]and Cheng et al. [2] recently published work that
proposed a mechanism called dual learning in neuralmachine translation (NMT). It can leverage monolingual data to improve NMT. Other similar systems
in image modality such as CycleGAN [3] was also
proposed. However, most only work with the same
domain between the source and the target.
In speech processing, the speech chain framework[4, 5] was proposed to integrate human speech perception and production behaviors that utilize bothautomatic speech recognition (ASR) model, andtext-to-speech synthesis (TTS) inside a chain mechanism. This approach enables semi-supervised training for both ASR and TTS, even without paired
speech-text data, which is often unavailable. Perhaps this is the first framework that was constructed
on a diﬀerent domain (speech vs. text). However,this study is limited to speech and textual modalities.
In this study, we proposed a multimodal machine
chain, which is an improvement from speech chain.
We proposed a collaborative image captioningretrieval model collaboration in visual chain, whichworks closely with the speech chain through text
modality. Our new framework mimics the mechanism of the entire human communication systemwith auditory and visual sensors. The overall system and its comparison with speech chain can beseen in Fig. 1.

Training Mechanism

The sequence-to-sequence model in closed loop architecture allows us to train our entire model in asemi-supervised fashion by concatenating both thelabeled and unlabeled data. To further clarify thelearning process, we describe the mechanism basedon the availability condition of the training data:
1. Paired speech-text-image data exist:
Given complete multimodal dataset D1, all themodels in the chain are trained separately supervisedly.
2. Unpaired speech, text, images data exist:
In this case, speech, text, and image data are
available in D2, but they are unpaired. The
reconstructed data are used to calculate reconstruction loss for each respective model. For
example, with the text-only dataset D2y , TTS
generates speech utterance x̂ for the ASR. On
the other hand, image captions y retrieve imagesẑ, which are reconstructed into text ŷ using theIC model.
3. Single modality data exist:
In this case, only a single data (speech or image)is available, and the others are empty. For example, if only image data are available in dataset
D3z , first we perform unrolled process IC→IRin the visual chain (See 2(b)). The generated
image caption ŷ is then used to perform unrolled process TTS→ASR in the speech chain(See 2(c)).
We are trying to learn whether in the situation
where only image data exist (3) we can still improve the ASR performance through a learning pro-― 1012 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

x=ASRx̂ =ŷ = "text"

x=ASRŷ = (caption)IRẑ=ŷ = "text"

TTSx̂ =y = "text"

TTSx̂ =

ŷ = (caption)

IC

z=(b)

(a)

Figure 1: Architecture of (a) speech chain framework [4], and (b) our proposed multimodal chain mechanism.
Table 3: IC and IR performancecess from a visual chain to a speech chain by leverIC
IRBLEU1
R@10↑
med r↓
Baseline: IC & IR
(Supervised learning - Type 1)
D1 2k*
33.91
26.88
34
Data

aging cross-modal data augmentation.

3

Experiment Set-up andResultProposed: visual chain IC→IR and IR→IC(Semi-supervised learning - Type 2)+D2 7k
42.11
28.14
31

Table 1: Training data partition for Flickr30k with

Proposed: speech chain → visual chain(Semi-supervised learning - Type 3)+D3x 10k
43.08
28.44
30

three conditions: (1) available paired data are denoted as ⃝, (2) available data but unpaired are denoted as ▲, and unavailable data are denoted as ×.
SetSpeechx

TextyImagez#TrainingType

D1
D2
D3x
D3z

⃝
▲
▲×⃝
▲××⃝
▲×▲

2000
7000
10000
10000

1 (paired)
2 (unpaired)
3 (unpaired)
3 (unpaired)

Topline: IC & IR separately(Supervised learning - Full data)
Dall 29k66.27
62.42
5Then, we demonstrate how much we can improve
performance when the required data are no longer
available, using our proposed chain mechanism. We
used the initial model to train the speech chain us-We ran our experiment with the Flickr30k dataset

ing D2 7k data and achieved 10.60% WER and 0.714[6] that has 31,014 photos of everyday activities andL2-norm2 as shown in Table 2. Finally, using the ICscenes. Similar to other image captioning datasets,
each image has five captions with a vocabulary ofmodel that was trained semi-supervisedly through18k words. To maintain balance between source anddataset which enables it to be used in speech chain.

target, we selectively used one or five images depend-

By this way, we achieved about 2.6% WER improve-ing on the task target. We generated speech fromment over the original speech chain [4] that was only

the Flickr30k captions using single speaker Google

trained using the speech and text datasets.

TTS. We used sequence-to-sequence encoder-decoder

proposed strategies makes improvement of ASR andmodel for ASR [7], Tacotron [8], show-attend-tell

TTS possible, even without any speech or text data,

model for IC [9], and Neural IR Embedding [10] forwith the help of a visual chain.

IR.

In the third block of Table 3 we show that the visual chain can also be improved using speech data,

Table 2: ASR and TTS performanceData

D1 2k*ASRTTS
WER(%)
L2-norm2
Baseline: ASR & TTS(Supervised learning - Type 1)
81.31
0.874

Type 2(a)&2(c), we decoded the image-only D3z

Our

by the help of speech chain. There was about 1 point
improvement in terms of BLEU for IC (high is good)and median r for IR (low is good). This result also

Proposed: speech chain ASR→TTS and TTS→ASR(Semi-supervised learning - Type 2)+D2 7k
10.60
0.714

implies that using our proposed learning strategy,

Proposed: visual chain → speech chain(Semi-supervised learning - Type 3)+D3z 10k
7.97
0.645

out image and text datasets available. Therefore, weTopline: ASR & TTS separately(Supervised learning - Full Data)
Dall 29k
2.37
0.398

speech modality, but also reversely.

the IC and IR model can be improved even withshowed that it also works not only from image-to-― 1013 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.

4Conclusions[5] Andros Tjandra, Sakriani Sakti, and Satoshi

We described a novel approach for cross-modal

Nakamura. Machine speech chain with one-shotdata augmentation that upgrades a speech chain intoa multimodal chain. We improved the speech chain

speaker adaptation. In Interspeech 2018, 19th

using an image-only dataset, bridged by our visual

Communication Association, Hyderabad, India,chain, and vice-versa. Therefore, we conclude that it

2-6 September 2018., pages 887–891, 2018.

is still possible to improve ASR, even without speech[6] B. A. Plummer, L. Wang, C. M. Cervantes,

and text data available, with our proposed multi-J. C. Caicedo, J. Hockenmaier, and S. Lazeb-modal chain. In the future, we will jointly train bothnik.

the speech and visual chain so that both can also beto-phrase correspondences for richer image-to-updated together.

sentence models. In 2015 IEEE International
Conference on Computer Vision (ICCV), pages

5

Annual Conference of the International Speech

Acknowledgement

Flickr30k entities:

Collecting region-2641–2649, Dec 2015.

Part of this work is supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237
as well as NII CRIS Contract Research 2019 and[7] William Chan, Navdeep Jaitly, Quoc Le, andOriol Vinyals. Listen, attend and spell: A
neural network for large vocabulary conversa-Google AI Focused Research Awards Program.

tional speech recognition. In 2016 IEEE Inter-Referencesnational Conference on Acoustics, Speech andSignal Processing (ICASSP), pages 4960–4964.

[1] Di He, Yingce Xia, Tao Qin, Liwei Wang, Neng-IEEE, 2016.

hai Yu, Tie-Yan Liu, and Wei-Ying Ma. Duallearning for machine translation. In Advances in[8] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stan-

Neural Information Processing Systems, pageston, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly,820–828, 2016.

Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy
Bengio, Quoc Le, Yannis Agiomyrgiannakis,[2] Yong Cheng, Zhaopeng Tu, Fandong Meng,Junjie Zhai, and Yang Liu.

Rob Clark, and Rif A. Saurous. Tacotron: To-Towards robustwards end-to-end speech synthesis. 2017.

neural machine translation. In Proceedings ofthe 56th Annual Meeting of the Association for[9] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun

Computational Linguistics (Volume 1: Long Pa-Cho, Aaron Courville, Ruslan Salakhudinov,pers), pages 1756–1766, Melbourne, Australia,

Rich Zemel, and Yoshua Bengio. Show, attend

July 2018. Association for Computational Lin-and tell: Neural image caption generation withguistics.

visual attention. In International conference onmachine learning, pages 2048–2057, 2015.

[3] Jun-Yan Zhu, Taesung Park, Phillip Isola, andAlexei A Efros. Unpaired image-to-image trans-[10] Armand Vilalta, Dario Garcia-Gasulla, Ferranlation using cycle-consistent adversarial net-Parés, Eduard Ayguadé, Jesus Labarta, E Ulises

works. In Computer Vision (ICCV), 2017 IEEE

Moya-Sánchez, and Ulises Cortés. Studying theInternational Conference on, 2017.

impact of the full-network embedding on multi-[4] Andros Tjandra, Sakriani Sakti, and Satoshi
Nakamura. Listening while speaking: Speechmodal pipelines. Semantic Web, (Preprint):1–15.

chain by deep learning. CoRR, abs/1707.04879,
2017.

― 1014 ―Copyright(C) 2020 The Association for Natural Language Processing.
All Rights Reserved.