Improving ASR with Multimodal Machine Chain∗◎ Johanes Eﬀendi1,2 , Andros Tjandra1,† , Sakriani Sakti1,2 , Satoshi Nakamura1,2(1 NAIST, 2 RIKEN AIP)

1

Introduction
Gathering a substantial amount of parallel data

vestigate the possibility of applying the chain mechanism in a simple multisource multimodal model.

is one of the major problems in building a multimodal model. Previously, a machine speech chain

3

Chain Components[3] was proposed to enable the training of ASR andFor ASR and TTS models, we use a similar model

TTS to assist each other in semi-supervised learning.

as Tjandra et al. [3], which is based on a sequence-

It successfully avoids the need for a large amountof paired speech and text data by using a speech

to-sequence architecture. Then, for IC, we removed
the last two layers of ResNet model, and use the hid-or text only unpaired data in a closed-loop mecha-

den representation to be attended by an attentional

nism. However, this work has not yet addressed vi-LSTM-based text decoder. We use the AttnGAN

sual modality, although visual modality is also one

model for IG, which generates the image iterativelyof the important senses in human communication.

until the 128x128 pixels resolution. For ImgSp2Txt,

In this study, we formulate a multimodal model

we averaged the output layer of ASR and IC, tocollaboration between automatic speech recognition

enables the combination of these two models in an(ASR), text-to-speech synthesis (TTS), image cap-ensembling fashion.

tioning (IC), and image generation (IG), in which
they support each other under a weakly-supervised

4

Experiment Settings

chain training strategy. The results reveal that withWe used Flickr8k dataset [2] with Flickr8k Audiothe help of IC and IG, ASR and TTS can still be[1] which has 8000 photos of everyday activities andimproved using an image-only dataset.

events. Each image has ﬁve captions with multispeaker speech. We make a data partition from the26000 images of the training set, into several subsets

Multimodal Machine Chain
We proposed a multimodal machine chain towith diﬀerent modality type. To simulate a small

mimic overall human communication and accommo-number of paired multimodal data, the ﬁrst par-date visual modality. This framework emphasizes

tition consists of 800 images, which is the smallest

that human communication is not only auditory butpartition. The second partition is the unpaired mul-also visual. In addition to the previously publishedtimodal data which has 1500 images. Finally, thespeech chain, we added a visual chain, which is com-last two partitions are the speech-only and image-

posed as a collaboration between an image caption-

only subset with 1850 images each. We trained theing model and an image generation model. We callmodel step-by-step following the order of the parti-

this MMC1 for the multimodal chain with dual-looption (represented by every row in Table 1).

architecture that uses text modality as a bridge (SeeFig. 1).

automatic metric for each task. We used character

Then, inspired by how human brain process multi-or word error rate (CER/WER) to evaluate ASR

ple sensories in a uniﬁed manner, we also see the pos-and bilingual evaluation understudy (BLEU) withsibility to introduce sharing between ASR and IC,

4-grams (B4) for the IC. For TTS, we used L2-norm2

given that both of them has the same target modal-

metric to calculate the error between the generated

ity, which is text. As an alternative, we also pro-mel-spectrogram and its reference. Finally, we used

posed a single-loop multimodal chain, which we callthe inception score (IS) to measure how realistic theMMC2. With this alternative chain, we want to in-IG output was. Each model is evaluated with the∗We evaluated each model with the commonly used

Multimodal Machine Chain による音声認識の改善、◎ジョハネスエフェンディ1,2 、アンドロスチャンドラ 1,† , サクリア二サクチイ 1,2 、中村 哲 1,2（1 NAIST, 2 RIKEN AIP)†This work was done when he was at NAIST, he is currently at Facebook AI, USA.

Fig. 1

Structure of: (a) speech chain [3] (b) proposed single-loop multimodal chain (MMC1), (c) proposeddual-loop multimodal chain (MMC2).
Table 1

Performance of proposed MMC1 and MMC2 compared with label propagation method in Flickr8k

multispeaker natural speech dataset. The last line is the topline system when the 6k images with thecorresponding ﬁve captions and ﬁve speech utterances for each image are available.
Training

Partition Type#ImageLabel Propagation(Semi-Supervised)

Multimodal (P)+ Multimodal (U)+ Sp only (U)+ Img only (U)Multimodal (P)+ Multimodal (U)+ Sp only (U)+ Img only (U)Multimodal (P)

800
1500
1850
1850
800
1500
1850
1850
6000

ProposedMultimodal Chain(Semi-Supervised)
Topline (Supervised)

test set of Flickr8k dataset.

ASR
CER↓
36.35
39.57
46.04
36.35
15.10
12.37
12.06
5.76

MMC1ICTTS
B4↑L22 ↓
12.75
0.77
12.53
0.77
0.63
11.41
12.75
0.77
13.22
0.59
13.28
0.56
13.29
0.56
19.91
0.50

IGIS↑
5.90
7.04
7.20
5.90
8.29
9.12
9.11
9.66

MMC2
ImgSp2Txt
TTS
CER↓
B4↑
L22 ↓
26.67
32.23
0.77
27.45
33.59
0.77
28.87
35.75
0.63
30.31
35.38
26.67
32.23
0.77
14.88
55.15
0.65
13.81
58.03
0.62
12.32
59.66
0.61
5.16
79.88
0.50

IGIS↑
5.90
7.04
7.20
5.90
10.12
10.65
9.95
9.66

these results show that the cross-modal augmentation enabled by our proposed multimodal chain has

5

Result and Conclusion
We used label propagation to compare our ex-periment results.

Label propagation is a simple

semi-supervised method that continues the trainingof a model with generated pseudo-labels from nolabelled data. In ASR case, for example, we canuse an ASR model to transcript a speech only data,so that the transcription hypothesis can be used asa pseudo-label to continue the ASR training semi-

shown to be eﬀective to further improve the ASR
model, even with an image only dataset.

6Acknowledgements
Part of this work is supported by JSPS KAKENHI

Grant Number JP17H06101 and the Google AI Focused Research Awards Program.

References

supervised. However, the result in Table 1 shows[1] D. Harwath and J. Glass. Deep multimodal se-that this method was not able to improve the ASR

mantic embeddings for speech and images. Inand ImgSp2Txt. We suspect that this is because

Proc. of IEEE ASRU, pp. 237–244, 2015.

this method needs more data to initialize the modeltraining in a supervised manner.
The next part of Table 1 shows the result of our
proposed method.

We can see that with multi-modal chain, the CER can be improved from 15.10%
to 12.06% by using the speech only and image
only data. The improvement can also be seen in[2] C. Rashtchian, P. Young, M. Hodosh, andJ. Hockenmaier. Collecting image annotations
using amazon’s mechanical turk.

In Proc. ofthe NAACL HLT 2010 Workshop on Creating
Speech and Language Data with Amazon’s Mechanical Turk, pp. 139–147, 2010.

ImgSp2Txt CER and B4 score, which performs bet-

[3] A. Tjandra, S. Sakti, and S. Nakamura. Listen-ter than MMC1 in low-data condition. We also ob-ing while speaking: Speech chain by deep learn-

served that the performance of other tasks such as

ing. In Proc. of the IEEE ASRU, pp. 301–308,

IC, TTS, and IG can be maintained. In conclusion,

Dec 2017.