Unifying Speech Recognition and Generation withMachine Speech Chain
Andros Tjandra1,2
1

Satoshi Nakamura1,2

Sakriani Sakti1,2Nara Institute of Science and Technology, Japan2
RIKEN AIP, Japan{andros.tjandra.ai6, ssakti, s-nakamura}@.is.naist.jp

1

Introduction
Human speech chain was first introduced by Denes

et al. [2], describes the basic mechanism involved inspeech communication when a spoken message travels from the speaker’s mind to the listener’s mind
(Fig. 1). It consists of a speech production mechanism in which the speaker produces words and generates speech sound waves, transmits the speech waveform through a medium (i.e., air), and creates aspeech perception process in a listener’s auditory
system to perceive what was said. Over the pastfew decades, tremendous research eﬀort has struggled to understand the principles underlying natu-Figure 1: Speech chain [2] and related spoken language technologies.

2Machine Speech Chain

ral speech communication. Many attempts have also

In Fig. 2(a), we illustarted an overview of machine

been made to replicate human speech perception andspeech chain mechanism. It consists of a sequence-to-production by machines to support natural modal-

sequence ASR [1], a sequence-to-sequence TTS [3],ity in human-machine interactions. In this paper,
we proposed a closed-loop between ASR and TTS

and a loop connection from ASR to TTS and fromTTS to ASR. The key idea is to jointly train bothmodel to construct a machine speech chain mecha-

the ASR and TTS models. As mentioned above, thenism. By using similar sequence-to-sequence archi-sequence-to-sequence model in closed-loop architec-tecture and interchangeable source-target domain, it

ture allows us to train our model on the concate-allows us to train both labeled and unlabeled data innation of both the labeled and unlabeled data. Fora single loop. The ASR model transcribes the unla-supervised training with labeled data (speech-text

beled speech utterances, then TTS reconstructs thespeech waveform based on the predicted text frompair data), both models can be trained independentlyASR. In the opposite case, the TTS reconstructs

get sequence and the ground truth sequence. How-the speech features, then ASR reconstructs the text

ever, for unsupervised training with unlabeled data

based on predicted speech from TTS. Based on our(speech only or text only), both models need to sup-knowledge, this is the first deep learning model thatport each other through a connection.

integrates human speech perception and production
behaviours.

by minimizing the loss between their predicted tar-To further clarify the learning process during unsupervised training, we unrolled the architecture as
follows:

Figure 2: (a) Overview of machine speech chain architecture. Examples of unrolled process: (b) from ASR toTTS and (c) from TTS to ASR.
• Unrolled process from ASR to TTS

paired them with corresponding text. For our test

scribes the unlabeled input speech, while TTS

set, we used all 510 utterances from the BTEC default test set. For the unsupervised learning step,

reconstructs the original speech waveform based

we chose 40,000 speech utterances just from BTEC1

on the output text from ASR. Fig. 2(b) illustrates the mechanism. We may also treat it asand 40,000 text utterances from BTEC1.

Given the unlabeled speech features, ASR tran-an autoencoder model, where the speech-to-text
ASR serves as an encoder and the text-to-speech
TTS as a decoder.
• Unrolled process from TTS to ASR
Given only the text input, TTS generates speech
waveform, while ASR also reconstructs the original text transcription given the synthesizedspeech.

Fig. 2(c) illustrates the mechanism.

Here, we may also treat it as another autoencoder model, where the text-to-speech TTS
serves as an encoder and the speech-to-text ASR
as a decoder.

3.1

Features Extraction

For the speech features, we used a log magnitude spectrogram extracted by short-time Fouriertransform (STFT). We extracted the spectrogramwith STFT (50-ms frame length, 12.5-ms frame shift,2048-point FFT). After getting the spectrogram, weused the squared magnitude and a Mel-scale filterbank with 40 filters to extract the Mel-scale spectrogram.

After getting the Mel-spectrogram, wesquared the magnitude spectrogram features.We normalized each feature into 0 mean and unit variances. Our final set is comprised of 40 dims log
Mel-spectrogram features and a 1025 dims log mag-3Experiments
To gather a large single speaker speech dataset, wenitude spectrogram. For the text, we converted all of
the sentences into lowercase and tokenize them intoa character sequence.

utilized Google TTS to generate a large set of speech
waveform based on basic travel expression corpus(BTEC) English sentences. For training and devel-

3.2

Model Details

opment we used part of the BTEC1 dataset, and for

Our ASR model is a encoder-decoder with an at-testing we used the default BTEC test set. For su-tention mechanism. On the encoder side, we used apervised training on both the ASR and TTS models,log-Mel spectrogram as the input features, which are

we chose 10,000 speech utterances that were paired

projected by a fully connected layer, processed bywith their corresponding text. For our development

three stacked BiLSTM layers with 256 hidden units.

set, we selected another 3000 speech utterances andOn the decoder side, we use LSTM with 512 hidden

units, followed by an MLP attention and a softmax
function.

4Acknowledgements
Part of this work was supported by JSPS

Our TTS model hyperparameters are generally theKAKENHI

same as the original Tacotron, except that we used

JP17K00237.

GrantNumbers

JP17H06101

andLeakyReLU instead of ReLU for most of the parts.
On the encoder sides, the CBHG used K = 8 different filter banks instead of 16 to reduce our GPU
memory consumption. Our TTS predicted four consecutive frames in one time step to reduce the number of time steps in the decoding process.

References[1] Dzmitry Bahdanau, Jan Chorowski, Dmitriy
Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary
speech recognition. In Acoustics, Speech and Sig-3.3

nal Processing (ICASSP), 2016 IEEE Interna-Experiment Result

tional Conference on, pp. 4945–4949. IEEE, 2016.
Table 1 shows our result on the single-speaker ASR
and TTS experiments. For the ASR experiment, we
generated best hypothesis with beam search (size=5). We used a character error rate (CER) for evaluating the ASR model. For the TTS experiment, wereported the MSE between the predicted log Mel andthe log magnitude spectrogram to the ground truth.
We also report the accuracy of our model that predicted the last speech frame. We used diﬀerent values for α and text decoding strategy for ASR (in theunsupervised learning stage) with a greedy search ora beam search.
Table 1: Experiment result for single-speaker testset.
Data

Hyperparametersgen.
αβmodePaired(10k)---+Unpaired(40k)

0.25
0.5
0.25
0.5

11
1
1

greedygreedybeam 5beam 5

ASR
CER
(%)

TTSMel

Raw

10.06

7.068

9.376

5.83
5.75
5.44
5.77

6.212
6.247
6.243
6.201

8.485
8.4188.441
8.435

The result show that after ASR and TTS models
have been trained with a small paired dataset, theystart to teach each other using unpaired data andgenerate useful feedback. Here we improved both
ASR and TTS performance. Our ASR model reduced CER by 4.6% compared to the system thatwas only trained with labeled data. In addition toASR, our TTS also decreased the MSE and the endof speech prediction accuracy.

[2] P.B. Denes and E. Pinson. The Speech Chain.
Anchor books. Worth Publishers, 1993.
[3] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep
Jaitly, Zongheng Yang, Ying Xiao, Zhifeng
Chen, Samy Bengio, et al. Tacotron: Towards
end-to-end speech synthesis.
arXiv:1703.10135, 2017.

arXiv preprint