Machine Speech Chain with Deep Learning∗
Andros Tjandra1,2 , Sakriani Sakti1,2 , Satoshi Nakamura1,2 (NAIST1 , RIKEN AIP 2 )1

INTRODUCTION

illustrates the mechanism. We may also treat
it as an autoencoder model, where the speech-

In this paper, we develop a closed-loop speech

to-text ASR serves as an encoder and the text-chain model based on deep learning and construct ato-speech TTS as a decoder.

sequence-to-sequence model for both ASR and TTS• Unrolled process from TTS to ASRtasks as well as a loop connection between thesetwo processes. The sequence-to-sequence model in

Given only the text input, TTS generates

closed-loop architecture allows us to train our model

speech waveform, while ASR also reconstructs

on the concatenation of both labeled and unlabeled

the original text transcription given the syn-data. While ASR transcribes the unlabeled speechfeatures, TTS attempts to reconstruct the original

thesized speech. Fig. 1(c) illustrates the mechanism. Here, we may also treat it as anotherspeech waveform based on text from ASR. In theautoencoder model, where the text-to-speech

opposite direction, ASR also reconstructs the origi-TTS serves as an encoder and the speech-to-nal text transcription given the synthesized speech.

text ASR as a decoder.

To the best of our knowledge, this is the first deeplearning model that integrates human speech per-3

ExperimentTask

ception and production behaviors.

2

Machine Speech Chain
An overview of our proposed machine speechchain architecture is illustrated in Fig. 1(a). It consists of a sequence-to-sequence ASR [1], a sequenceto-sequence TTS [2], and a loop connection fromASR to TTS and from TTS to ASR. The key idea
is to jointly train both the ASR and TTS models. As mentioned above, the sequence-to-sequencemodel in closed-loop architecture allows us to train
our model on the concatenation of both the labeled
and unlabeled data. For supervised training withlabeled data (speech-text pair data), both models
can be trained independently by minimizing theloss between their predicted target sequence andthe ground truth sequence. However, for unsupervised training with unlabeled data (speech only ortext only), both models need to support each otherthrough a connection.

onSingle-SpeakerTo gather a large single speaker speech dataset,
we utilized Google TTS to generate a large set ofspeech waveform based on basic travel expression
corpus (BTEC) English sentences. For training anddevelopment we used part of the BTEC1 dataset,and for testing we used the default BTEC test set.
For supervised training on both the ASR and TTS
models, we chose 10,000 speech utterances that werepaired with their corresponding text. For our development set, we selected another 3000 speech utterances and paired them with corresponding text.
For our test set, we used all 510 utterances from theBTEC default test set. For the unsupervised learning step, we chose 40,000 speech utterances just fromBTEC1 and 40,000 text utterances from BTEC1.
3.1

Features Extraction

For the speech features, we used a log magnitude
spectrogram extracted by short-time Fourier trans-To further clarify the learning process during un-form (STFT). We extracted the spectrogram withsupervised training, we unrolled the architecture asSTFT (50-ms frame length, 12.5-ms frame shift,follows:

2048-point FFT). After getting the spectrogram,• Unrolled process from ASR to TTSGiven the unlabeled speech features, ASR
transcribes the unlabeled input speech, while
TTS reconstructs the original speech waveform
based on the output text from ASR. Fig. 1(b)∗深層学習を用いたスピーチチェーン

we used the squared magnitude and a Mel-scalefilterbank with 40 filters to extract the Mel-scale
spectrogram. After getting the Mel-spectrogram,
we squared the magnitude spectrogram features.We
normalized each feature into 0 mean and unit vari-Fig. 1 (a) Overview of machine speech chain architecture. Examples of unrolled process: (b) from ASR toTTS and (c) from TTS to ASR.
ances. Our final set is comprised of 40 dims log
Mel-spectrogram features and a 1025 dims log magnitude spectrogram. For the text, we converted all

Table 1set.
Dataof the sentences into lowercase and tokenize theminto a character sequence.
3.2

Model Details

Our ASR model is a encoder-decoder with an attention mechanism. On the encoder side, we usedExperiment result for single-speaker test
Hyperparametersgen.
αβmodeASRCER(%)

TTSMelRaw

Paired
(10k)

---

10.06

7.068

9.376

+Unpaired(40k)

0.25
0.5
0.25
0.5

11
1
1greedy
greedybeam 5beam 5

5.83
5.75
5.44
5.77

6.212
6.247
6.243
6.201

8.485
8.418
8.441
8.435

a log-Mel spectrogram as the input features, which
are projected by a fully connected layer, processed

have been trained with a small paired dataset, theyby three stacked BiLSTM layers with 256 hidden

start to teach each other using unpaired data andunits. On the decoder side, we use LSTM with 512

generate useful feedback. Here we improved bothhidden units, followed by an MLP attention and aASR and TTS performance. Our ASR model re-softmax function.

duced CER by 4.6% compared to the system that

Our TTS model hyperparameters are generally

was only trained with labeled data. In addition tothe same as the original Tacotron, except that weused LeakyReLU instead of ReLU for most of theASR, our TTS also decreased the MSE and the endparts. On the encoder sides, the CBHG used K = 84diﬀerent filter banks instead of 16 to reduce ourGPU memory consumption.

Our TTS predicted

four consecutive frames in one time step to reduceof speech prediction accuracy.

ACKNOWLEDGEMENTS
Part of this work was supported by JSPS

KAKENHI

Grant

Numbers

JP17H06101

andJP17K00237.

the number of time steps in the decoding process.
3.3

Experiment Result

Table 1 shows our result on the single-speaker
ASR and TTS experiments. For the ASR experiment, we generated best hypothesis with beam
search (size= 5). We used a character error rate(CER) for evaluating the ASR model. For the TTS
experiment, we reported the MSE between the predicted log Mel and the log magnitude spectrogram

References[1] D. Bahdanau,

J. Chorowski,

D. Serdyuk,

P. Brakel, and Y. Bengio. End-to-end attentionbased large vocabulary speech recognition.
In Acoustics, Speech and Signal Processing(ICASSP), 2016 IEEE International Conferenceon, pp. 4945–4949. IEEE, 2016.

to the ground truth. We also report the accuracy of[2] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu,

our model that predicted the last speech frame. We

R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao,

used diﬀerent values for α and text decoding strat-

Z. Chen, S. Bengio, et al. Tacotron: Towards

egy for ASR (in the unsupervised learning stage)

end-to-end speech synthesis.

with a greedy search or a beam search.

arXiv:1703.10135, 2017.

The result show that after ASR and TTS models

arXiv preprint