言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Unifying Speech Recognition and Generation withMachine Speech Chain
Andros Tjandra1,2
1

Sakriani Sakti1,2Satoshi Nakamura1,2

Nara Institute of Science and Technology, Japan
2
RIKEN AIP, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@.is.naist.jp

1

Introduction

Human speech chain was first introduced by Denes
et al. [2], describes the basic mechanism involved inspeech communication when a spoken message travels from the speaker’s mind to the listener’s mind
(Fig. 1). It consists of a speech production mechanism in which the speaker produces words and generates speech sound waves, transmits the speech waveform through a medium (i.e., air), and creates aspeech perception process in a listener’s auditory
system to perceive what was said. Over the pastfew decades, tremendous research eﬀort has struggled to understand the principles underlying natural speech communication. Many attempts have also
been made to replicate human speech perception andproduction by machines to support natural modality in human-machine interactions. In this paper,
we proposed a closed-loop between ASR and TTS
model to construct a machine speech chain mechanism. By using similar sequence-to-sequence architecture and interchangeable source-target domain, it
allows us to train both labeled and unlabeled data ina single loop. The ASR model transcribes the unlabeled speech utterances, then TTS reconstructs thespeech waveform based on the predicted text fromASR. In the opposite case, the TTS reconstructs
the speech features, then ASR reconstructs the text
based on predicted speech from TTS. Based on ourknowledge, this is the first deep learning model that
integrates human speech perception and production
behaviours.

Figure 1: Speech chain [2] and related spoken language technologies.

2Machine Speech Chain

In Fig. 2(a), we illustarted an overview of machine
speech chain mechanism. It consists of a sequence-tosequence ASR [1], a sequence-to-sequence TTS [3],and a loop connection from ASR to TTS and fromTTS to ASR. The key idea is to jointly train both
the ASR and TTS models. As mentioned above, thesequence-to-sequence model in closed-loop architecture allows us to train our model on the concatenation of both the labeled and unlabeled data. Forsupervised training with labeled data (speech-textpair data), both models can be trained independently
by minimizing the loss between their predicted target sequence and the ground truth sequence. However, for unsupervised training with unlabeled data(speech only or text only), both models need to support each other through a connection.
To further clarify the learning process during unsupervised training, we unrolled the architecture as
follows:― 183 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Figure 2: (a) Overview of machine speech chain architecture. Examples of unrolled process: (b) from ASR toTTS and (c) from TTS to ASR.
• Unrolled process from ASR to TTS

paired them with corresponding text. For our test

scribes the unlabeled input speech, while TTS

set, we used all 510 utterances from the BTEC default test set. For the unsupervised learning step,

reconstructs the original speech waveform based

we chose 40,000 speech utterances just from BTEC1

on the output text from ASR. Fig. 2(b) illustrates the mechanism. We may also treat it asand 40,000 text utterances from BTEC1.

Given the unlabeled speech features, ASR tran-

an autoencoder model, where the speech-to-text
ASR serves as an encoder and the text-to-speech

3.1

TTS as a decoder.

For the speech features, we used a log magnitude spectrogram extracted by short-time Fourier• Unrolled process from TTS to ASR
Given only the text input, TTS generates speech
waveform, while ASR also reconstructs the original text transcription given the synthesizedspeech.

Features Extraction

Fig. 2(c) illustrates the mechanism.

Here, we may also treat it as another autoencoder model, where the text-to-speech TTS
serves as an encoder and the speech-to-text ASR

transform (STFT). We extracted the spectrogramwith STFT (50-ms frame length, 12.5-ms frame shift,2048-point FFT). After getting the spectrogram, weused the squared magnitude and a Mel-scale filterbank with 40 filters to extract the Mel-scale spectrogram.

After getting the Mel-spectrogram, wesquared the magnitude spectrogram features.We normalized each feature into 0 mean and unit vari-as a decoder.

ances. Our final set is comprised of 40 dims logMel-spectrogram features and a 1025 dims log mag-3

nitude spectrogram. For the text, we converted all ofExperiments

the sentences into lowercase and tokenize them into

To gather a large single speaker speech dataset, wea character sequence.

utilized Google TTS to generate a large set of speech
waveform based on basic travel expression corpus(BTEC) English sentences. For training and devel-3.2

Model Details

opment we used part of the BTEC1 dataset, and for

Our ASR model is a encoder-decoder with an at-testing we used the default BTEC test set. For su-tention mechanism. On the encoder side, we used apervised training on both the ASR and TTS models,log-Mel spectrogram as the input features, which are

we chose 10,000 speech utterances that were paired

projected by a fully connected layer, processed bywith their corresponding text. For our development

three stacked BiLSTM layers with 256 hidden units.

set, we selected another 3000 speech utterances andOn the decoder side, we use LSTM with 512 hidden― 184 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

units, followed by an MLP attention and a softmax

4function.

Acknowledgements
Part of this work was supported by JSPS

Our TTS model hyperparameters are generally theKAKENHI

same as the original Tacotron, except that we used

JP17K00237.

Grant

Numbers

JP17H06101

andLeakyReLU instead of ReLU for most of the parts.
On the encoder sides, the CBHG used K = 8 different filter banks instead of 16 to reduce our GPU
memory consumption. Our TTS predicted four consecutive frames in one time step to reduce the num-References[1] Dzmitry Bahdanau, Jan Chorowski, Dmitriy
Serdyuk, Philemon Brakel, and Yoshua Ben-

ber of time steps in the decoding process.

gio. End-to-end attention-based large vocabulary
speech recognition. In Acoustics, Speech and Sig-

3.3

nal Processing (ICASSP), 2016 IEEE Interna-Experiment Result

tional Conference on, pp. 4945–4949. IEEE, 2016.
Table 1 shows our result on the single-speaker ASR
and TTS experiments. For the ASR experiment, we[2] P.B. Denes and E. Pinson. The Speech Chain.
Anchor books. Worth Publishers, 1993.

generated best hypothesis with beam search (size=5). We used a character error rate (CER) for evaluating the ASR model. For the TTS experiment, we[3] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stan-reported the MSE between the predicted log Mel andthe log magnitude spectrogram to the ground truth.
We also report the accuracy of our model that predicted the last speech frame. We used diﬀerent values for α and text decoding strategy for ASR (in theton, Yonghui Wu, Ron J Weiss, Navdeep
Jaitly, Zongheng Yang, Ying Xiao, Zhifeng
Chen, Samy Bengio, et al. Tacotron: Towards
end-to-end speech synthesis.

arXiv preprint

arXiv:1703.10135, 2017.

unsupervised learning stage) with a greedy search ora beam search.
Table 1: Experiment result for single-speaker test
set.
Data

Hyperparameters
gen.
αβ
mode

Paired(10k)---+
Unpaired(40k)

0.25
0.5
0.25
0.5

1
1
1
1

greedygreedybeam 5beam 5

ASR
CER
(%)

TTS
Mel

Raw

10.06

7.068

9.376

5.83
5.75
5.44
5.77

6.212
6.247
6.243
6.201

8.485
8.418
8.441
8.435

The result show that after ASR and TTS models
have been trained with a small paired dataset, theystart to teach each other using unpaired data andgenerate useful feedback. Here we improved both
ASR and TTS performance. Our ASR model reduced CER by 4.6% compared to the system thatwas only trained with labeled data. In addition toASR, our TTS also decreased the MSE and the endof speech prediction accuracy.

― 185 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.