IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

1

Machine Speech Chain
Andros Tjandra, Nonmember, IEEE, Sakriani Sakti, Member, IEEE, and Satoshi Nakamura, Fellow, IEEE

Abstract—Despite the close relationship between speech perception and production, research in automatic speech recognition(ASR) and text-to-speech synthesis (TTS) has progressed moreor less independently without exerting much mutual influence. In
human communication, on the other hand, a closed-loop speechchain mechanism with auditory feedback from the speaker’s
mouth to her ear is crucial. In this paper, we take a step further
and develop a closed-loop machine speech chain model based on
deep learning. The sequence-to-sequence model in closed-loop
architecture allows us to train our model on the concatenationof both labeled and unlabeled data. While ASR transcribes
the unlabeled speech features, TTS attempts to reconstruct theoriginal speech waveform based on the text from ASR. In the
opposite direction, ASR also attempts to reconstruct the originaltext transcription given the synthesized speech. To the best of
our knowledge, this is the first deep learning framework that
integrates human speech perception and production behaviors.
Our experimental results show that the proposed approachsignificantly improved performance over that from separate
systems that were only trained with labeled data.
Index Terms—speech chain, ASR, TTS, deep learning

I. Introduction

S

PEECH chain, a concept introduced by Denes et al.
[1], describes the basic mechanism involved in speechcommunication when a spoken message travels from the
speaker’s mind to the listener’s mind (Fig. 1). It consistsof a speech production mechanism in which the speakerproduces words and generates speech sound waves, transmits
the speech waveform through a medium (i.e., air), and createsa speech perception process in a listener’s auditory system toperceive what was said. Over the past few decades, researchers
have struggled to understand the principles underlying natural speech communication. Many attempts have also been
made to replicate human speech perception and production
with machines to support natural modality in human-machineinteractions.
To date, the development of advanced spoken language
technologies based on automatic speech recognition (ASR)
and text-to-speech (TTS) has enabled machines to process
and respond to basic human speech. Various ASR approaches
have relied on acoustic-phonetics knowledge [2] in earlier
works to template-based schemes with dynamic time warping(DTW) [3], [4] and data-driven approaches with rigorous statistical modeling of a hidden Markov model-Gaussian mixture
model (HMM-GMM) [5], [6]. In a similar direction, TTS
technology has gradually shifted from the foundation of a rulebased system using waveform coding and an analysis-synthesis
method [7]–[9] to waveform unit concatenation [10], [11] anda more flexible approach using the statistical modeling of a
The author are with the Nara Institute of Science and Technology, Japanand RIKEN, Center for Advanced Intelligence Project AIP, Japan (email:andros.tjandra.ai6@is.naist.jp, ssakti@is.naist.jp, s-nakamura@is.naist.jp).

Figure 1. Speech chain [1] and related spoken language technologies.

hidden semi-Markov model-GMM (HSMM-GMM) [12], [13].
Recently, after the resurgence of deep learning, interest has
also surfaced in the possibility of utilizing a neural approachfor ASR and TTS systems. Many state-of-the-art performances
in ASR [14]–[16] and TTS [17]–[19] tasks have been successfully constructed based on neural network frameworks.
However, despite the close relationship between speech perception and production, ASR and TTS research has progressedmore or less independently without exerting much influenceon each other. In human communication, on the other hand, aclosed-loop speech chain mechanism has a critical auditory
feedback mechanism from the speaker’s mouth to her ear(Fig. 1). In other words, the hearing process is critical, not only
for the listener, but also for the speaker. By simultaneously
listening and speaking, the speaker can monitor her volume,
articulation, and the general comprehensibility of her speech.
Processing the information further, the speaker’s brain can plan
what she will say next. Children who lose their hearing often
have difficulty producing clear speech due to their inability tomonitor their own speech [20].
Unfortunately, investigating the inherent links between thesetwo processes is very challenging. Difficulties arise because
methodologies and analyses are necessarily quite different
when they are extracting the underlying messages from speech
waveforms, as in speech perception, or generating an optimum
dynamic speaking style from the intended message, as inspeech production. Until recently, it was impossible in ajoint approach to reunite the problems shared by both modes.
However, due to deep learning’s representational power, many
complicated hand-engineered models have been simplified
by letting deep neural nets (DNNs) learn their way from
input to output spaces. With this newly emerging approach tosequence-to-sequence mapping tasks, a model with a common
architecture can directly learn the mapping between variablelength representations of different modalities: text-to-text se-

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

2

Figure 2. (a) Overview of machine speech chain architecture. Examples of unrolled process: (b) from ASR to TTS and (c) from TTS to ASR.

quences [21], [22], speech-to-text sequences [23], [24], textto-speech sequences [25], and image-to-text sequences [26],etc.
Therefore, in this paper, we take a step further and develop
a closed-loop speech chain model based on deep learning andconstruct a sequence-to-sequence model for both ASR andTTS tasks, as well as a loop connection between these twoprocesses. The sequence-to-sequence model in closed-loop
architecture allows us to train our model on the concatenationof both labeled and unlabeled data. While ASR transcribes
the unlabeled speech features, TTS attempts to reconstruct
the original speech waveform based on text from ASR. In
the opposite direction, ASR also reconstructs the original texttranscription given the synthesized speech. To the best of ourknowledge, this is the first deep learning model that integrates
human speech perception and production behaviors.
Our contributions in this paper include:1) Basic machine speech chain that integrates ASR and
TTS and performs on single-speaker task.
2) Multi-speaker speech chain with a speaker-embedding
network for handling speech with different voice characteristics.
3) Machine speech chain with a straight-through estimatorto allow end-to-end feedback loss through discrete units
or subwords.
II. Related Works
Approaches that utilize learning from source-to-target andvice-versa, as well as feedback links, remain scant. He etal. [27] quite recently published a work that addressed amechanism called dual learning in neural machine translation.
Their system has a dual task: source-to-target language translation (primal) versus target-to-source language translation(dual). The primal and dual tasks form a closed loop andgenerate informative feedback signals to train the translation
models, even without the involvement of a human labeler.
This approach was originally proposed to tackle training data
bottleneck problems. With a dual-learning mechanism, thesystem can leverage monolingual data (in both the source andtarget languages) more effectively. First, they construct one
model to translate from the source to the target language andanother to translate from the target to the source language.
After both the first and second models have been trained witha small parallel corpus, they start to teach each other using
monolingual data and generate useful feedback with language
model likelihood and reconstruction error to further improve
the performance.
Another similar work in neural machine translation was
introduced by Cheng et al. [28], [29]. This approach also
exploited monolingual corpora to improve neural machinetranslation. Their system utilizes a semi-supervised approachfor training neural machine translation (NMT) models on
the concatenation of labeled (parallel corpora) and unlabeled(monolingual corpora) data. The central idea is to reconstruct monolingual corpora using an autoencoder in which thesource-to-target and target-to-source translation models serveas the encoder and decoder, respectively.
In this manuscript, we addressed similar problems in spoken
language processing tasks. This paper presents a novel mechanism that integrates human speech perception and production
behaviors. With a concept that resembles dual learning in
NMT, we utilize the primal model (ASR) that transcribes
the text given the speech versus the dual model (TTS) thatsynthesizes the speech given the text. However, the maindifference between NMT is that the domain between thesource and the target here are different (speech versus text).
While ASR transcribes the unlabeled speech features, TTS
attempts to reconstruct the original speech waveform basedon the text from ASR. In the opposite direction, ASR also
attempts to reconstruct the original text transcription given
the synthesized speech. Nevertheless, our experimental results
show that the proposed approach also identified a successful
learning strategy and significantly improved performance overthat of separate systems that were only trained with labeled
data.
After our preliminary work, several works have discussed
our methods and built on top of them. Karita et al. [30] forma text and speech autoencoder and train unpaired data with

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

reconstruction loss. Ren et al., [31] replaced the LSTM-based
encoder-decoder with Transformer modules for both ASR and
TTS and achieved good performance with small paired speechtext in single speaker dataset. Rosenberg et al. [32] exploredthe effect of data augmentation by using TTS on the larger
experiment. Kurata et al. [33] improved ASR performanceby adding feature reconstruction loss during training. Ueno
et al. [34] use synthetic speeches from multi-speaker TTS
to improve their Acoustic2Word speech recognition system.
Baskar et al. [35] proposed an alternative to backpropagate
through discrete variables by using a policy-gradient method,compared to our proposal using a straight-through estimator.
Hori et al. [36] replaced TTS with text-to-encoder (TTE) to
avoid the need for modeling the speaking style during thereconstruction.

3

Algorithm 1 Speech Chain Algorithm
1: Input:Paired speech and text dataset DP , text-only dataset YU , speechonly dataset XU , supervised loss coefficient α, unsupervised loss coefficient β
2: repeat3:
A. Supervised training with speech-text data pairs
4:
Sample paired speech and text
(xP , yP ) = ([x1P , .., xSPP ], [y1P , .., yTPP ])from DP with speech length S P and text length T P .
5:
Generate a text probability vector by ASR using teacher forcing:P , xP ; θ
pyt = P(yt |y<t
AS R ), ∀t ∈ [1..T P ]6:
Generate best predicted speech by TTS using teacher forcing:P , yP ; θx̂Ps = argmax P(z|x<s
T T S ); ∀s ∈ [1..S P ]z

7:Calculate the loss for ASR and TTS

III. Basic Machine Speech Chain
A. Overview
We start by explaining an overall basic machine speechchain mechanism. For a better understanding, we illustrated
the speech chain loop in Fig. 2(a). Speech chain consists ofa sequence-to-sequence ASR, a sequence-to-sequence TTS,
and a loop connection from ASR to TTS and from TTS to
ASR. The key idea is to jointly train both the ASR and TTS
models. As mentioned above, the sequence-to-sequence modelin closed-loop architecture allows us to train our model on
the concatenation of both the labeled (paired) and unlabeled(unpaired) data.
To further clarify the learning process during supervised andunsupervised training, we unrolled the architecture as follows:
1) Paired speech-text training for ASR and TTS
Given the labeled data (speech-text paired data), bothmodels can be trained independently by minimizing theloss between their predicted target sequence and theground truth sequence via teacher forcing.
2) Unpaired speech data only (ASR → TTS)
Given the unlabeled speech features, ASR transcribes
the unlabeled input speech, while TTS reconstructs theoriginal speech waveform based on the output text from
ASR. Figure 2(b) illustrates the mechanism. We may
also treat it as an autoencoder model, where the speechto-text ASR serves as an encoder and the text-to-speech
TTS as a decoder.
3) Unpaired text data only (TTS → ASR)
Given only the text input, TTS generates speech waveform, while ASR also reconstructs the original texttranscription given the synthesized speech. Figure 2(c)illustrates the mechanism. Here, we may also treat it
as another autoencoder model, where the text-to-speech
TTS serves as an encoder and the speech-to-text ASR
as a decoder.
With such autoencoder models, ASR and TTS can teacheach other by adding a reconstruction term of the observed
unlabeled data to the training objective. Details of the algorithm can be found in Alg. 1.

. Eq. 13 & 14

PP`AS
R = LAS R (y , py ; θAS R )P
P P`T T S = LT T S (x , x̂ ; θT T S )

8:
9:
10:
11:
12:

13:

(1)
(2)

B. Unsupervised training with unpaired speech and text
# Unpaired text data (TTS → ASR):
UU
Sample text yU = [yU
1 , .., yT U ] from Y
Generate speech by TTS: x̂U ∼ PT T S (·|yU ; θT T S )Generate text probability vector by ASR from TTS’s predicted speechusing teacher forcing:U∀t ∈ [1..T U ]pyt = P(yt |yU<t , x̂ ; θAS R );Calculate the loss between original text yU and reconstruction probability vector py
`U
AS R

=

LAS R (yU , py ; θAS R )

(3)

14:
15:
16:
17:

# Unpaired speech data (ASR → TTS):
Sample speech xU = [x1U , .., xSU ] from XU
UGenerate text by ASR: ŷU ∼ PAS R (·|xU ; θAS R )Generate speech by TTS from ASR’s predicted text using teacher
forcing:UUx̂U∀s ∈ [1..S ]s = argmax PT T S (z|x<s , ŷ ; θT T S );18:

Calculate the loss between original speech xU and generated speechx̂Uz`TUT S
19:
20:=

LT T S (xU , x̂U ; θT T S )# Loss combination:
Combine all weighted loss into a single loss variable
P
U
U`ALL = α ∗ (`TPT S + `ASR ) + β ∗ (`T T S + `AS R )

21:

(5)

Calculate TTS and ASR parameters gradient withthe derivative of `ALL w.r.t θAS R , θT T S
G AS R
GT T S

22:

(4)

=
=

∇θAS R `∇θT T S `(6)
(7)

Update TTS and ASR parameters with gradient descentoptimization (SGD, Adam, etc)
θAS R ← Optim(θAS R , G AS R )θT T S ← Optim(θT T S , GT T S )23: until convergence of parameter θT T S , θAS R

(8)
(9)

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

B. Sequence-to-Sequence Model for ASR
A sequence-to-sequence model is a neural network that
directly models conditional probability P(y|x), where x =[x1 , ..., xS ] is the sequence of the (framed) speech featureswith length S and y = [y1 , ..., yT ] is the sequence of labelswith length T . Figure 3 shows the overall structure of theattention-based encoder-decoder model that consists of encoder, decoder, and attention modules.

4of hidden units for the decoder. Finally, the decoder is an
autoregressive model that predicts target sequence probability
pyt = P(yt |ct , hdt , y<t ; θAS R ) = P(yt |x, y<t ; θAS R ) at time t basedon previous output y<t , current context information ct , andhidden state hdt . The loss function for ASR can be formulated
as:
T
C
1 XX
1(yt = c) ∗ log pyt [c], (13)
`AS R = LAS R (y, py ) = −T t=1 c=1
where C is the number of output classes. Input x for speech
recognition tasks is a sequence of feature vectors like the log
Mel-scale spectrogram. Therefore, x ∈ RS ×D , where D is thenumber of feature dimensions and S is the total frame length
for an utterance. Output y, which is a speech transcription
sequence, can be either a phoneme or grapheme (character)sequence.
C. Sequence-to-Sequence Model for TTS

Figure 3. Sequence-to-sequence ASR architecture: the encoder consists of afully connected layer + stack bidirectional LSTM and the decoder consists ofa unidirectional LSTM with attention mechanism.

The encoder task processes input sequence x, projects
it through several layers (e.g., long short-term memory(LSTM) [37]/gated recurrent unit (GRU) [38], convolution,
fully connected), and outputs representative information he =[he1 , ..., heS ] for the decoder. The attention module is an extension scheme that helps the decoder find relevant informationon the encoder side based on the current decoder hidden states[21], [24]. Attention modules produce context information ct
at time t based on the encoder hidden states hes and decoder
hidden states hdt :
ct =

S
X

at (s) ∗ hes

(10)

s=1

at (s) = Align(hes , hdt )exp(Score(hes , hdt ))
= PS
.
e d
s=1 exp(Score(h s , ht ))(11)

There are several variations for score functions [39]:hhes , hdt i,
dot product



 e|
e d
d
Score(h s , ht ) = 
(12)
h
W
h
,
bilinear
s t
s




V s| tanh(W s [hes , hdt ]), MLP
where Score : (R M × RN ) → R, M is the number of hidden
units for the last layer of encoder and N is the number

Figure 4. Sequence-to-sequence TTS (Tacotron) architecture with frame
ending binary prediction. (FC = Fully Connected, CBHG = Convolution Bank
+ Highway + bi-GRU)

Parametric speech synthesis resembles a sequence-tosequence task where we generate speech given a sentence.
Using a sequence-to-sequence model, we model the conditional probability between P(x|y), where y = [y1 , ..., yT ] is thesequence of characters with length T and x = [x1 , ..., xS ] is thesequence of (framed) speech features with length S . From theIEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

sequence-to-sequence ASR model perspective, we now havean inverse model for reconstructing the original speech given
the text.
In this work, our core architecture is based on Tacotron [25]with several structural modifications. Figure 4 illustrates ourmodified Tacotron. On the encoder side, we project our input
characters with an embedding layer. The character vectors are
fed into several fully connected layers followed by a non-linear
activation function. We pass the result into the CBHG block(1-D Convolution Bank + Highway + bidirectional GRU) witheight filter banks (filter size ranging from 1 to 8). The CBHG
output is expected to produce representative information he =[he1 , ..., heT ] for the decoder.
Our modified decoder has one input layer and three output
layers (instead of two as in the original Tacotron). The first
output layer generates a sequence of log Mel-scale spectrogram frames x M = [x1M , ..., xSM ]. At the s-th step, the input
layer is fed by a previous step-log Mel-scale spectrogramM, and then several fully connected layers and a non-linear
x s−1
activation function are processed. Next, we use a stacked
LSTM with a multilayer perceptron (MLP) attention withalignment and context history [40] to extract the expected
context c s information based on the current decoder input andencoder states he . We project the context with a fully connected
layer to predict the Mel-scale spectrogram.
The second output layer reconstructs log-magnitude spectrogram xR = [x1R , ..., xSR ] given the first layer generated output
x M . After we get complete the sequences of the log Melscale spectrogram, we feed them into a CBHG block followed
by a fully connected layer to predict the log magnitude
spectrogram.
The third output layer generates binary prediction b s ∈ [0, 1](1 if the s-th frame is the end of speech, otherwise 0) basedon the current log-Mel spectrogram generated by the first
output layer and expected context c s from the decoder withthe attention layer. We add the binary prediction layer because
the output from the first and second decoder layers is a realvalue vector, and we cannot use an end-of-sentence (eos) token
to determine when to stop the generation process. Based on
our initial experiment, we found that our modification helped
Tacotron determine the end of speech more robustly than
forcing the decoder to generate frames with a 0 value at the endof the speech. We also enable our model to learn from multiple
speakers by concatenating the projected speaker embedding
into the input before the LSTM layer, first output regression
layer, and second output regression layer.
For training the TTS model, we used the following loss
function:S`T T S = LT T S (x, x̂) =1X M
kx − x̂ sM k22 +kxRs − x̂Rs k22
S s=1 s, (14)− (b s log(b̂ s ) + (1 − b s ) log(1 − b̂ s ))where x̂ = (x̂ M , x̂R , b̂) are the predicted Mel-scale spectrogram,the magnitude spectrogram, and the end-of-frame probability,
and x = (xM , xR , b) is the ground truth. In the decoding
process, we use the Griffin-Lim [41] algorithm to iteratively
estimate the phase spectrogram and reconstruct the signal

5with the inverse short-time Fourier transform (STFT) from thepredicted magnitude and phase spectrogram.
D. Experiment on Single-Speaker Task
To verify our proposed method, we experimented on a
corpus with a single speaker because, until recently, most
TTS systems by deep learning are trained on a single speakerdataset.
We utilized a natural speech single-speaker dataset named
LJSpeech [42] that contains about 13,100 utterances. Because
there is no official dev and test split from this dataset, weshuffled it and randomly took 94% (total 12,314 utts) for
training, 3% (total 393 utts) for dev, and 3% (total 393 utts)
for the test set. Later, we split the train-set again to smaller
ratio to compare the result between the paired only and paired+ unpaired data. In Figure 5, we illustrate how we split thetraining data for supervised (baseline) with small ratio of
paired data, supervised (upperbound) with full paired dataand semi-supervised with paired, unpaired text and unpaired
speech. For the unpaired speech and text, there are 70%
remaining unused data. To get the unpaired text, we take it byrandomly sample 35% (without replacement) from the unused
data. For the unpaired speech, we take it from the remaining
35% data. Therefore, there is no overlap between unpaired
speech and unpaired text dataset.

Figure 5. Illustration for training data split between three different scenarios.

1) Feature Extraction: For the speech features, we extracted two different sets of features: Mel spectrogram andmagnitude spectrogram. Both the Mel spectrogram and magnitude spectrogram are extracted based on STFT with the librosapackage [43]. All speech waveforms were sampled at 16 kHz.
Given the raw speech waveform, we applied pre-emphasis(coefficient 0.97) and extracted the spectrogram with STFT
(50-ms frame length, 12.5-ms frame shift, 2048-points FFT).
After getting the spectrogram, we applied absolute and log
operations to extract the log magnitude spectrogram features.
To generate the Mel spectrogram features, we extracted the 80dims Mel-scale coefficients from the magnitude spectrogram
followed by log operation. Our final set is comprised ofan 80-dimension log-Mel spectrogram and 1025-dimension
log magnitude spectrogram. The log magnitude spectrogramIEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

features are used by TTS and the log-Mel spectrogram features
are used by both TTS and ASR.
For the text, we converted all of the sentences into lowercase
and replaced some punctuation marks (for example, ” into’). In the end, we have 26 letters (a-z), six punctuation
marks (,:’?.-), and three special tags (
, 
, <spc>)
to denote the start, end of sentence, and spaces between words.
2) Model Details: Our ASR model is an encoder-decoderwith an attention mechanism. On the encoder side, we useda log-Mel spectrogram as the input features (in unsupervised process, the log-Mel spectrogram was generated by
TTS), which are projected by a fully connected layer and a
LeakyReLU (l = 1e−2) [44] activation function, and processedby three stacked bidirectional LSTM (BiLSTM) layers with
256 hidden units for each direction (512 hidden units). We
applied sequence subsampling [24], [45] to reduce the memory
usage and computation time on the each LSTM layer andreduced the length of the speech features eight times shorter.
On the decoder side, the input character is projected with a
128-dims embedding layer and fed into a one-layer LSTM
with 512 hidden units. We calculated the attention matrix
with an MLP scorer (Eq. 12) followed by a fully connected
layer and a softmax function. In the decoding phase, thetranscription was generated by beam-search decoding (size =5), and we normalized the log-likelihood score by dividing it
by its own length to prevent the decoder from favoring theshorter transcriptions. We did not use any language model orlexicon dictionary in this work.
Our TTS model hyperparameters are generally the same as
the original Tacotron, except that we used LeakyReLU insteadof ReLU for most of the parts. On the encoder side, the CBHG
used K = 8 different filter banks instead of 16 to reduce our
GPU memory consumption. On the decoder side, we used a
two-stacked LSTM instead of a GRU with 256 hidden units.
Our TTS predicted four consecutive frames in one time-stepto reduce the number of time-steps in the decoding process.
Both the ASR and TTS models are implemented with the
PyTorch library 1 .
3) Experiment Results: For the ASR, we compare thecharacter error rate (CER) between different scenarios inTable I. For the TTS experiment, we did both objective andsubjective evaluations. In the objective evaluation, we compare
the L2-norm squared between the predicted and ground truth
log Mel-spectrogram in Table II. We experimented with adifferent ratio between the paired and unpaired data fromthe LJSpeech dataset. In the subjective evaluation, based on
the quality of the synthesized speech using mean opinion
score (MOS) test based on five-point scale (5: very good 1: very poor). We compare three systems: 1) baseline withpaired speech-text 30%, 2) speech chain with paired speechtext 30%, unpaired text 35% and unpaired speech 35 (nooverlap)% and 3) upperbound paired speech-text 100%. To
generate the samples, we randomly picked 20 utterances fromthe test set. In total, we have 27 subjects and each subject
1 PyTorch https://github.com/pytorch/pytorch

6evaluates 60 utterances. We report the subjective evaluation
result in Figure 7.
Table IASR experiment result for LJSpeech single-speaker natural speech dataset.
Supervised (Baseline)
Unpaired
PairedCER (%)
Text
Speech
Enc-Dec Att
10%
31.7
Enc-Dec Att
20%
9.9
Enc-Dec Att
30%
6.8
Enc-Dec Att
40%
4.9
Enc-Dec Att
50%
4.1
Semi-supervised (Speech Chain)
Enc-Dec Att
10%
45%
45%
12.3
Enc-Dec Att
20%
40%
40%
5.6
Enc-Dec Att
30%
35%
35%
4.7
Enc-Dec Att
40%
30%
30%
3.8
Enc-Dec Att
50%
25%
25%
3.5
Supervised (Upperbound)
Enc-Dec Att
100%
3.1
Model

Figure 6. Side-by-side CER (%) comparison between baseline, speech chain
and upperbound with different percentage of paired speech-text data.

Table II
TTS experiment result for LJSpeech single-speaker natural speech dataset.
Supervised (Baseline)
Unpaired
Model
PairedL2-norm2
Text
Speech
Enc-Dec Att
10%
1.05
Enc-Dec Att
20%
0.91
Enc-Dec Att
30%
0.71
Enc-Dec Att
40%
0.69
Enc-Dec Att
50%
0.66
Semi-supervised (Speech Chain)
Enc-Dec Att
10%
45%
45%
0.87
Enc-Dec Att
20%
40%
40%
0.73
Enc-Dec Att
30%
35%
35%
0.66
Enc-Dec Att
40%
30%
30%
0.65
Enc-Dec Att
50%
25%
25%
0.64
Supervised (Upperbound)
Enc-Dec Att
100%
0.606

The results show that after the ASR and TTS models
are trained with a small paired dataset, they start to teacheach other using unpaired data and generate useful feedback.
Here, we improved both the ASR and TTS performance
significantly compared to only using a portion of the paired
dataset. We provided some samples from the single speakerIEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

Figure 7. MOS with 95% confidence interval between baseline, speech chain
and upperbound scenario.

speech chain TTS experiments on https://speech-chain-singlespk-demo.netlify.com/.

72) Unpaired speech data only (see Fig. 8(b)) Given
only the speech utterances x from unpaired dataset DU ,ASR generates the text transcription ŷ (with greedy
or beam-search decoding) and SPKEMB provides aspeaker-embedding vector z = SPKEMB(x). Given thegenerated text and the original speaker vector z, TTS
then reconstructs the speech waveform x̂ = T T S (ŷ, z)
via teacher forcing. We then calculate the loss `TUT S
between x and x̂.
3) Unpaired text data only (see Fig. 8(c)) Given only
the text transcription y from unpaired dataset DU ,we need to sample speech from the available datasetx̃ ∼ (DP ∪ DU ) and generate a random speaker vectorz̃ = SPKEMB(x̃) from SPKEMB. Then, TTS generates
the speech utterance x̂ with greedy decoding. Given
the generated speech x̂, ASR reconstructs the text ŷ =AS R(x̂) via teacher forcing. We then calculate the loss`U
AS R between y and ŷ.
We combine all losses together and update both the ASR andTTS model:

E. Discussion
In this section, we presented a basic speech chain mechanism and demonstrated the ability to train both ASR and TTS
modules with paired and unpaired speech and a text dataset.
However, there is a limitation in the unpaired training:
1) For training unpaired text, given an unpaired text, we
can only generate speech with a specific speaking style.
The speaking style is limited based on the speaker set
that we used in the supervised TTS training.
2) In the unpaired speech training, the ASR transcribes a
sentence. However, our TTS can only reconstruct the
speech if the speaker identity from the unpaired speech
is provided and the speaker embedding for that person
has been seen during the supervised training.
IV. Machine Speech Chain Frameworkwith Speaker Adaptation
A. Overview
Figure 8 illustrates the updated speech chain mechanism.
Similar to the earlier version, it consists of a sequence-tosequence ASR [24], [46], a sequence-to-sequence TTS [25],and a loop connection from ASR to TTS and from TTS to
ASR. The key idea is to jointly train the ASR and TTS models.
The difference is that, in this version, we integrate a speakerrecognition (SPKEMB) model inside the loop illustrated inFig. 8(a). As mentioned above, we can train our modelon the concatenation of both labeled (paired) and unlabeled(unpaired) data. We describe the learning process below.
1) Paired speech-text dataset (see Fig. 8(a)) Given the
speech utterances x and the corresponding text transcription y from dataset DP , both the ASR and TTS
models can be trained independently. Here, we canPtrain ASR by calculating the ASR loss `AS
R directlywith teacher forcing. For TTS training, we generate aspeaker-embedding vector z = SPKEMB(x), integrate z
information with TTS, and calculate the TTS loss `TPT S
via teacher forcing.

P
PUU` = α ∗ (`AS
R + `T T S ) + β ∗ (`AS R + `T T S )(15)

θAS R ← Optim(θAS R , ∇θAS R `)(16)

θT T S ← Optim(θT T S , ∇θT T S `),(17)

where α, β are hyperparameters to scale the loss between
the supervised (paired) and unsupervised (unpaired) loss, and∇θAS R `, ∇θT T S ` are the gradient of combined loss ` w.r.t. ASR
θAS R and TTS parameters θT T S .
Figure 8 illustrates the updated speech chain mechanism.
Similar to the earlier version, it consists of a sequence-tosequence ASR [24], [46], a sequence-to-sequence TTS [25],and a loop connection from ASR to TTS and from TTS to
ASR. The key idea is to jointly train the ASR and TTS models.
The difference is that, in this version, we integrate a speakerembedding (SPKEMB) model inside the loop illustrated inFig. 8(a). As mentioned above, we can train our modelon the concatenation of both labeled (paired) and unlabeled(unpaired) data. We describe the learning process below.
1) Paired speech-text dataset (see Fig. 8(a)) Given the
speech utterances x and the corresponding text transcription y from dataset DP , both the ASR and TTS
models can be trained independently. Here, we canPtrain ASR by calculating the ASR loss `AS
R directlywith teacher forcing. For TTS training, we generate aspeaker-embedding vector z = SPKEMB(x), integrate z
information with TTS, and calculate the TTS loss `TPT S
via teacher forcing.
2) Unpaired speech data only (see Fig. 8(b)) Given
only the speech utterances x from unpaired dataset DU ,ASR generates the text transcription ŷ (with greedy
or beam-search decoding) and SPKEMB provides aspeaker-embedding vector z = SPKEMB(x). Given thegenerated text and the original speaker vector z, TTS
then reconstructs the speech waveform x̂ = T T S (ŷ, z)
via teacher forcing. We then calculate the loss `TUT S
between x and x̂.

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

8

Figure 8. (a) Overview of proposed machine speech chain architecture with speaker recognition; (b) Unrolled process with only speech utterances and no texttranscription (speech → [ASR,SPKEMB] → [text + speaker vector] → TTS → speech); (c) Unrolled process with only text, but no corresponding speechutterance ([text + speaker vector by sampling SPKEMB] → TTS → speech → ASR → text). Note: grayed box is the original speech chain mechanism.

3) Unpaired text data only (see Fig. 8(c)) Given only
the text transcription y from unpaired dataset DU ,we need to sample speech from the available datasetx̃ ∼ (DP ∪ DU ) and generate a random speaker vectorz̃ = SPKEMB(x̃) from SPKEMB. Then, TTS generates
the speech utterance x̂ with greedy decoding. Given
the generated speech x̂, ASR reconstructs the text ŷ =AS R(x̂) via teacher forcing. We then calculate the loss`U
AS R between y and ŷ.
We combine all losses together and update both the ASR andTTS model:
PPUU` = α ∗ (`AS
R + `T T S ) + β ∗ (`AS R + `T T S )On the top of dz , we attach a linear projection and softmax activation function to calculate the probability along all possible
N speakers.
z = SPKEMB(x)(21)

py = Softmax(zWz )(22)

To optimize the speaker representation model, there areseveral loss functions such as negative log-likehood:`NLL = −NX

1(y = n) ∗ log py [n],

(23)

n=1

(18)

θAS R ← Optim(θAS R , ∇θAS R `)(19)

θT T S ← Optim(θT T S , ∇θT T S `),(20)

where α, β are hyperparameters to scale the loss between
the supervised (paired) and unsupervised (unpaired) loss, and∇θAS R `, ∇θT T S ` are the gradient of combined loss ` w.r.t. ASR
θAS R and TTS parameters θT T S .
B. Speaker Recognition and Embedding
Speaker recognition is a task to determine the identity of the
speaker based on a spoken utterances. Another related tasks to
speaker recognition is speaker identification, where the speaker
identification model needs to predict if a pair of speech arecome from same identity or not. By generating a embeddingthat correspond to the speaker identity, it can be used to predictboth tasks. There are several traditional methods for speakerrecognition such as i-vectors [47] and PLDA-based approach[48]. Since the deep learning approach become more popular,
several deep learning architectures (DeepSpeaker [49], [50])have been proposed to directly learn speaker representationfrom speech features. In Figure 9 we illustrate DeepSpeakerarchitecture in more details.
To generate a speaker representation for speaker recognitiontask, we assume our input is a speech feature x ∈ RS ×din . Then,
we construct a deep neural network by stacking convolution,
recurrent, pooling, etc and generate a fixed size vector z ∈ Rdz .

or distance-based such as triplet loss [51], [52]:X`T RI =
max(kza − z p k22 +kza − zn k22 , 0)

(24)

a,p,n
ya =y p 6=yn

where a, p, n are the anchor, positive and negative example
and za , z p , zn are their embedding respectively. In the training
stage, those losses could be combined together and improved
the final model performance [49].

Figure 9. Deep learning based speaker embedding (DeepSpeaker) architecture.

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

9

C. Sequence-to-Sequence TTS with One-shot Speaker Adaptation

A parametric TTS can be formulated as a sequence-tosequence model where the source sequence is a text utterancey = [y1 , .., yT ] with length T and the target sequence is a
speech feature x = [x1 , .., xS ] with length S . Our model objective is to maximize P(x|y; θT T S ) w.r.t. TTS parameter θT T S .
We build our model upon the basic structure of the “Tacotron”
TTS [25] and “Deep Speaker” models (Section IV-B).
The original Tacotron is a single-speaker TTS system basedon a sequence-to-sequence model. Given a text utterance,
Tacotron produces the Mel spectrogram and linear spectrogram
followed by the Griffin-Lim algorithm to recover the phase andreconstruct the speech signal. However, the original model is
not designed to incorporate speaker identity or to generatespeech from different speakers.
On the other hand, Deep Speaker is a deep neural speakerembedding system (here denoted as “SPKEMB”). Given asequence of speech features x = [x1 , .., xS ], Deep Speakergenerates an L2-normalized continuous vector embedding z.
If x1 and x2 are spoken by the same speaker, the trained Deep
Speaker model will produce the vector z1 = SPKEMB(x1 ) andthe vector z2 = SPKEMB(x2 ), which are close to each other.
Otherwise, the generated embeddings z1 and z2 will be far fromeach other. By combining Tacotron with Deep Speaker, we cando “one-shot” speaker adaptation by conditioning the Tacotron
with the generated fixed-size continuous vector z from Deep
Speaker with a single speech utterance from any speaker.
Here, we adopt both systems by modifying the original
Tacotron TTS model to integrate the Deep Speaker model.
Figure 10 illustrates our proposed model. From the encoder
module side, the architecture and building blocks are same as
the description from Section III-C.
On the decoder side, we have an autoregressive decoder. To
produce different speech based on the original speaker target,
we generate speaker-embedding vector z = S PKEMB(x M ).
This speaker embedding z is generated using only one utterance of the target speaker; thus it is called “one-shot” speaker
adaptation. After that, we integrate speaker vector z with alinear projection and sum it with the last output from the FC
layer. Then, we apply two LSTM layers to generate current
decoder state hds . To retrieve the relevant information between
the current decoder state and the entire encoder state, we calculate the attention probability a s (t) = Align(het , hds ); ∀t ∈ [1..T ]P
and the expected context vector c s = T1 a s (t) ∗ het . Then,
we concatenate the decoder state hds , context vector c s , andprojected speaker-embedding z together into a vector, followedby two fully connected layers to produce the current time-step
Mel spectrogram output x sM .
In the training stage, we optimized our proposed model byminimizing the following loss function:

Figure 10. Proposed model: sequence-to-sequence TTS (Tacotron) + speaker
information via neural speaker embedding (Deep Speaker).

`T T S==LT T S (x, x̂, z, ẑ) SX  M
 γ1 kx s − x̂ sM k22 +kxRs − x̂Rs k22
s=1−γ2 b s log(b̂ s ) + (1 − b s ) log(1 − b̂ s )!
hẑ, zi+γ3 1 −,kẑk2 kzk2

(25)

where γ1 , γ2 , γ3 are our sub-loss hyperparameters, andx M , xR , b, z are the ground-truth Mel spectrogram, linear spectrogram, and end-of-speech label and speaker-embedding vector from the real speech data, respectively. x̂ M , x̂R , b̂ represent
the predicted Mel spectrogram, linear spectrogram, and endof-speech label, respectively, and speaker-embedding vectorẑ = SPKEMB(x̂ M ) is the predicted speaker vector from the
Tacotron output. Here, `T T S consists of three different loss
formulations: Eq. 25 line 1 applies L2-norm squared error
between the ground truth and predicted speech as a regressiontask, Eq. 25 line 2 applies binary cross entropy for end-ofspeech prediction as a classification task, and Eq. 25 line
3 applies cosine distance between the ground-truth speakerembedding z and predicted speaker-embedding ẑ, which isthe common metric for measuring the similarity between
two vectors; furthermore, by minimizing this loss, we also
minimize the global loss of speaker style [53], [54].

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

Table III
Character error rate (CER (%)) comparison between results of supervised
learning and those of a semi-supervised learning method, evaluated on
test eval92 set (without any lexicon & language model on the decoding
step)

Table IV
L2-norm squared on log-Mel spectrogram to compare the supervised
learning and those of a semi-supervised learning method, evaluated on
test eval92 set. Note: We did not include standard Tacotron (without
SPKEMB) into the table since it cannot output various target speakers.

Model
CER (%)
Supervised training:
WSJ train si84 (paired) → Baseline
Att Enc-Dec [58]17.01
Att Enc-Dec [59]17.68
Att Enc-Dec (ours)
17.35
Supervised training:
WSJ train si284 (paired) → UpperboundAtt Enc-Dec [58]8.17
Att Enc-Dec [59]7.69
Att Enc-Dec (ours)
7.12
Semi-supervised training:
WSJ train si84 (paired) + train si200 (unpaired)Label propagation (greedy)
17.52
Label propagation (beam=5)
14.58
Proposed speech chain (Sec. IV)
9.86

Model
L2-norm2
Supervised training:
WSJ train si84 (paired) → Baseline
Proposed Tacotron (Sec. IV-C) (ours)
1.036
Supervised training:
WSJ train si284 (paired) → Upperbound
Proposed Tacotron (Sec. IV-C) (ours)
0.836
Semi-supervised training:
WSJ train si84 (paired) + train si200 (unpaired)
Proposed speech chain (Sec. IV + Sec. IV-C)
0.886

10

D. Experiment on Multi-speaker Task
1) Corpus Dataset: In this study, we ran our experimenton the Wall Street Journal (WSJ) CSR Corpus [55]. The
complete data are contained in an SI284 (SI84+SI200) dataset.
We followed the standard Kaldi [56] s5 recipe to split thetraining set, development set, and test set. To reformulate
the speech chain as a semi-supervised learning method, we
prepared SI84 and SI200 as paired and unpaired training sets,
respectively. SI84 consists of 7138 utterances (about 16 hoursof speech) spoken by 83 speakers, and SI200 consists of
30,180 utterances (about 66 hours) spoken by 200 speakers(without any overlap with speakers of SI84). We use “dev93”
to denote the development and “eval92” for the test set.
2) Feature and Text Representation on WSJ dataset: For
the feature extraction, we use the same configuration as
Section III-D1 The text utterances were tokenized as charactersand mapped into a 33-character set: 26 alphabetic letters (az), 3 punctuation marks (’.-), and 4 special tags hnoisei,
hspci,hsi, and h/si as noise, space, start-of-sequence, and endof-sequence tokens, respectively. Both the ASR input and TTS
output shared the same text representation.
3) Model Details: For the ASR and TTS encoder-decoder,we use a same setting as Section III-D2. We set the sub-loss
hyperparameter in Eq. 25 with γ1 = 1, γ2 = 1, γ3 = 0.25.
For the speaker recognition model, we used the Deep
Speaker model and followed the original hyperparameters in
the previous paper. However, our Deep Speaker is only trainedon the WSJ SI84 set with 83 unique speakers. Thus, themodel is expected to generalize effectively across all remaining
unseen speakers to assist the TTS and speech chain training.
We used Adam optimization with a learning rate of 5e − 4 for
the ASR and TTS models and 1e − 3 for the Deep Speakermodel. All of our models in this paper are implemented with
PyTorch [57].
4) Experiment Results: Table III shows the ASR resultsfrom multiple scenarios evaluated on eval92. In the first block,
we trained our baseline model by using paired samples fromthe SI84 set only, and we achieved 17.35% CER. In the

Figure 11. MOS with 95% confidence interval between baseline, speech chain
and upperbound scenario.

second block, we trained our model with paired data of thefull WSJ SI284 data, and we achieved 7.12% CER as ourupperbound performance. In the last block, we trained ourmodel with a semi-supervised learning approach using SI84 aspaired data and SI200 as unpaired data. For comparison withother models trained with semi-supervised learning, we carriedout label-propagation [60]. Label propagation is a simple way
to do semi-supervised learning. First, we train initial modelwith paired speech-text DP . The pre-trained model is used
to generate the hypothesis from the unpaired speech XU .
Later, we add the unpaired speech and their correspondent
hypothesis into training set and treat them as a paired dataset.
Our result showed that by using label-propagation with beamsize=5, we successfully reduced the CER to 14.58%. Nevertheless, our proposed speech-chain model could achieve asignificant improvement over all baselines (paired only andlabel-propagation) with 9.89% CER, close to the upperboundresults.
For the TTS experiment, we did both objective and subjective evaluations. In the objective evaluation, we calculated
the difference with L2-norm squared between ground truth
and the predicted log-Mel spectrogram and presented theresult on Table IV. We observed similar trends with the ASR
results, where the semi-supervised training with speech chain
method improved significantly over the baseline and close
to the upperbound result. In the subjective evaluation, based

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

on the quality of the synthesized speech using mean opinion
score (MOS) test based on five-point scale (5: very good 1: very poor). To generate the samples, we randomly picked
20 utterances from the test set. In total, we have 26 subjects and each subject evaluates 60 utterances. We report the
subjective evaluation result in Figure 11. We provided somesamples from multi-speaker speech chain TTS experiments on
https://speech-chain-multi-spk-demo.netlify.com/.
E. Discussion
In this section, we introduced an improved speech chain
mechanism by integrating a speaker recognition model inside
the loop. By using the new system, we eliminated the downsidefrom our basic speech chain, where we are unable to incorporate the data from unseen speakers. We also extended thecapability of TTS to generate speech from an unseen speakerby implementing one-shot speaker adaptation. Thus, the TTS
can generate speech with a similar voice characteristic onlywith a single utterance example. Inside the speech chain loop,
the ASR also gets new data from the combination between atext sentence and an arbitrary voice characteristic. Our results
show that after we deployed the speech-chain loop, the ASR
system achieved significant improvement compared to thebaseline (supervised training only) and other semi-supervised
technique (label propagation). Like the trends in ASR, the TTS
system also showed improvement compared to the baseline(supervised training only).
V. End-to-end Feedback Loss on Speech Chain
A. Overview
In the speech chain mechanism, given speech featuresx = [x1 , .., xS ] (e.g., Mel spectrogram) and text y = [y1 , .., yT ],we fed the speech to the ASR module and the ASR decoder
generated continuous vector hdt step by step. To calculate
probability vector py = [py1 , .., pyT ], we applied the softmax
function pyt = softmax(hdt ) to decoder output hdt . For eachclass probability mass in pyt , pyt [c] was defined as:exp(hdt [c]/τ),pyt [c] = PCdi=1 exp(ht [i]/τ)∀c ∈ [1..C].

11

Another generation process to decode ASR transcription uses its own prediction to generate probability
vector pyt . There are many different generation methods, such as greedy decoding (1-best beam-search)(ỹt = argmax pyt [c]), beam-search, or stochastic samc

pling (ỹt ∼ Cat(pyt )).
After the generation process, we obtained probability vector
py and applied discretization from continuous probability
vector pyt to ỹt either by taking the class with the highest
probability or sampling from a categorical random variable.
After getting a single class to represent the probability vector,
we encoded it into vector [0, 0, .., 1, .., 0] with one-hot encoding
representation and gave it to the TTS as the encoder input. The
TTS reconstructs Mel spectrogram x̂ with the teacher-forcing
approach. The reconstruction loss is calculated by:S`TrecT S = Lrec
T T S (x, x̂) =1X M
kx − x̂ sM k22 ,S s=1 s(27)

where x̂ sM is the predicted (or reconstructed) Mel spectrogramand x sM is the ground-truth spectrogram at s-th time-step.
We directly calculated the gradient from the reconstruction
loss w.r.t. the TTS parameters (∂`TrecT S / ∂θT T S ) because all
the operations inside the TTS module are continuous anddifferentiable. However, we could not calculate the gradient from the reconstruction loss w.r.t. the ASR parameters(∂`TrecT S /∂θAS R ) because we have a discretization operation frompyt → onehot(ỹt ). Therefore, we applied a straight-throughestimator to enable the loss from `TrecT S to pass through discretevariable ỹt .
B. End-to-End Feedback Loss
1) Straight-through Argmax: The straight-through estimator [62], [63] is a method for estimating or propagating gradients through stochastic discrete variables. Its main idea is to
backpropagate through discrete operations (e.g., argmax pyt [c]cor sampling ỹt ∼ Cat(pyt )) like an identity function. We
describe the forward process and the gradient calculation witha straight-through estimator in Fig. 12.

(26)

Here, C is the total number of classes, hdt ∈ RC are the logits
produced by the last decoder layer, and τ is the temperatureparameters. Setting temperature τ using a larger value (τ > 1)
produces a smoother probability mass over classes [61].
For the generation process, we generally have two different
methods:
1) Conditional generation given ground truth (teacher forcing):
If we have paired speech and text (x, y), we can generate
pyt from autoregressive ASR decoder DecAS R (yt−1 , he ),conditioned to ground-truth text yt−1 in the current timestep and encoded speech feature he = EncAS R (x). At theend, the length of probability vector py is fixed to T
time-steps.
2) Conditional generation given previous step model prediction:

Figure 12. Straight-through estimator on arg max function. Given input
x and model parameters θ, we calculate categorical probability mass P(x; θ)and apply discrete operation argmax. In the backward pass, the gradient fromstochastic node y to P(x; θ), ∂y/∂P(x; θ) ≈ 1 is approximated by identity.

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

In the implementation, we created a function with different
forward and backward operations. For the argmax one-hot
encoding function, we formulated the forward operation:z̃t = argmax pyt [c](28)ỹt = onehot(z̃t ).

(29)

c

Here, we describe ỹt as a one-hot encoding vector with thesame length as the pyt vector. When the loss is calculated andthe gradients are backpropagated from loss `TrecT S , we formulate
the backward operation:
∂ỹt
≈ 1.
∂pyt

(30)

Therefore, when we backpropagate the loss from Eq. 27
with the straight-through estimator approach, we calculate the
TTS reconstruction loss gradient w.r.t. θAS R :T∂`TrecT S X∂`TrecT S ∂ỹt
∂pyt
=
·
·
∂θAS R∂ỹt∂pyt ∂θAS R
t=1

≈

T
X
∂`rec

TTS

t=1

∂ỹt

·1·

∂pyt
.
∂θAS R(31)

122) Straight-through Gumbel-Softmax: Besides taking
argmax class from probability vector pyt , we also generated
a one-hot encoding by sampling with the Gumbel-Softmax
distribution [64], [65]. Gumbel-Softmax is a continuous
distribution that approximates categorical samples and thegradients can be calculated with a reparameterization trick.
For Gumbel-Softmax, we replaced the softmax formula for
calculating pyt (Eq. 26):
exp((hdt [c] + gc )/τ),pyt [c] = PCdi=1 exp((ht [i] + gi )/τ)∀c ∈ [1..C],(33)

where g1 , .., gC are i.i.d. samples drawn from Gumbel (0, 1)
and τ is the temperature. We sample gc by drawing samples
from the uniform distribution:uc ∼ Uniform(0, 1)
gc = − log(− log(uc )),∀c ∈ [1..C].

(34)
(35)

To generate a one-hot encoding, we define our forward
operation:z̃t = argmax pyt [c](36)ỹt = onehot(z̃t ).

(37)

c

(32)

At the backpropagation time, we use the same straightthrough estimator (Eq. 30) to allow the gradients to flow
through the discrete operation.
3) Combined Loss for ASR: Our final loss function for ASR
is a combination from negative likelihood (Eq. 13) and TTS
reconstruction loss (Eq. 27) by sum operation:
F
rec
`AS
R = `AS R + `T T S .

(38)

To summarize our explanation in this section, we provide
an illustration in Fig. 13 that explains how sub-losses `AS R
and `TrecT S are backpropagated to the rest of the ASR and TTS
modules.
C. Experiment on Multi-speaker Task in Supervised Settings

Figure 13. Given speech feature x, ASR generates a sequence of probability
py = [py1 , py2 , ..., pyT ]. If we have a ground-truth transcription, we cancalculate LAS R (Eq. 13). The TTS module generates speech features and we
calculate reconstruction loss Lrec
T T S (Eq. 27). After that, the gradients basedon LAS R are propagated through the ASR module, and the gradients based on
Lrec
T T S are propagated through the TTS and ASR modules by a straight-throughestimator.

1) Dataset: We evaluated the performance of our proposed
method on the WSJ dataset [55]. Our settings for the training,development, and test sets are the same as the Kaldi s5
recipe [56]. We trained our model with WSJ-SI284 data. Our
validation set was dev 93 and our test set was eval 92. For
the feature extraction and text tokenization, we use the samesetting as Section IV-D2.
2) Model Details: For the ASR model, we used a standard
sequence-to-sequence model with an attention module (Section III-B). We use the same encoder setting as Section III-D2.
On the decoder sides, we projected one-hot encoding fromthe previous character into a 256-dims continuous vector with
an embedding matrix, followed by one unidirectional LSTM
with 512 hidden units. For the attention module, we used thecontent-based attention + multiscale alignment (denoted as“Att MLP-MA”) [40] with a 1-history size. In the evaluation
stage, the transcription was generated by beam-search decoding (size = 5), and we normalized the log-likelihood scoreby dividing it by its own length to prevent the decoder fromfavoring shorter transcriptions. We did not use any language

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

Table VASR experiment result on WSJ dataset test eval92.
Baseline (`AS R )ModelAtt MLP [58]Att MLP + Location [58]Att MLP [59]Att MLP-MA (ours) [40]Proposed (`AS R + `TrecT S )Modelpyt generationST
Att MLP-MA
argmax
Teacher forcing
Att MLP-MA
gumbelAtt MLP-MA
argmaxGreedyAtt MLP-MA
gumbelCER (%)
11.08
8.17
7.12
6.43
CER (%)
5.75
5.7
5.84
5.88

model or lexicon dictionary in this work. In the training
stage, we tried ST-argmax (Section V-B1) and ST-Gumbel
softmax (Section V-B2). We also tried both teacher forcing
and greedy decoding to generate ASR probability vectors py
in the training stage. For each scenario, we treated temperatureτ = [0.25, 0.5, 1, 2] as our hyperparameter and searched for thebest temperature based on the CER on the development set.
For the TTS model, we used the modified Tacotron whichis explained in Section III-C and we use same settings as
Section III-D2.
3) Experiment Results: For our baseline, we trained an
encoder-decoder with MLP + multiscale alignment with a
1-history size [40]. We also added several published results
to our baseline. All of the baseline models were trained byminimizing negative log-likelihood `AS R (Eq. 13).
All the models in the proposed section were trained witha combination from two losses, `AS R + `TrecT S , and the ASRparameters were updated based on the gradient from the sumof the two losses. We have four different scenarios, mostof which provide significant improvement compared to thebaseline model that is only trained on LAS R loss. With teacher
forcing and sampling from Gumbel-softmax, we obtained 11%
relative improvement compared to our best baseline Att MLPMA.
VI. Conclusion
This paper demonstrated a novel machine speech chain
mechanism based on deep learning. The sequence-to-sequencemodel in closed-loop architecture allows us to train our modelon the concatenation of both labeled and unlabeled data. We
explored applications of the model in various tasks, including
single speaker synthetic speech and multi-speaker natural
speech. Our experimental results in both cases show thatthe proposed approach enabled ASR and TTS to improve
performance by teaching each other using only unpaired data.
We also introduced an extension to allow backpropagation
through discrete output from the ASR module with a straightthrough estimator. In the future, it is necessary to further
validate the effectiveness of our approach on various languages
and conditions (i.e., spontaneous, noisy, and emotion).
VII. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.

13

References[1] P. Denes and E. Pinson, The Speech Chain, ser. Anchor books. WorthPublishers, 1993. [Online]. Available: https://books.google.co.jp/books?
id=ZMTm3nlDfroC[2] K. H. Davis, R. Biddulph, and S. Balashek, “Automatic recognition of
spoken digits,” Acoustic Society of America, pp. 627–642, 1952.
[3] T. K. Vintsyuk, “Speech discrimination by dynamic programming,”
Kibernetika, pp. 81–88, 1968.
[4] H. Sakoe and S. Chiba, “Dynamic programming algorithm quantization
for spoken word recognition,” IEEE Transaction on Acoustics, Speechand Signal Processing, vol. ASSP-26, no. 1, pp. 43–49, 1978.
[5] F. Jelinek, “Continuous speech recognition by statistical methods,” IEEE,
vol. 64, pp. 532–536, 1976.
[6] J. G. Wilpon, L. R. Rabiner, C. H. Lee, and E. R. Goldman, “Automatic
recognition of keywords in unconstrained speech using hidden Markov
models,” IEEE Transaction on Acoustics, Speech and Signal Processing,vol. 38, no. 11, pp. 1870–1878, 1990.
[7] J. N. Holmes, I. G. Mattingly, and J. N. Shearme, “Speech synthesis byrule,” Language and speech, vol. 7, no. 3, pp. 127–143, 1964.
[8] J. P. Olive, “Rule synthesis of speech from dyadic units,” in Proceedingsof ICASSP, 1977, pp. 568–570.
[9] Y. Sagisaka, “Speech synthesis by rule using an optimal selection of
non-uniform synthesis units,” in Proceedings of ICASSP, 1988.
[10] Y. Sagisaka, N. Kaiki, N. Iwahashi, and K. Mimura, “Atr υ-talk speech,”in Proceedings of ICSLP, 1992, pp. 483–486.
[11] A. Hunt and A. Black, “Unit selection in a concatenative speechsynthesis system using a large speech database,” in Proceedings of
ICASSP, 1996, pp. 373–376.
[12] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,“Simultaneous modeling of spectrum, pitch and duration in HMM-based
speech synthesis,” in Proceedings of Eurospeech, 1999, pp. 2347—2350.
[13] K. Tokuda, T. Kobayashi, and S. Imai, “Speech parameter generationfrom HMM using dynamic features,” in Proceedings of ICASSP, 1995,
pp. 660—-663.
[14] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition withdeep recurrent neural networks,” in Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, 2013,
pp. 6645–6649.
[15] D. Palaz, M. M. Doss, and R. Collobert, “Convolutional neural networksbased continuous speech recognition using raw speech signal,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International
Conference on. IEEE, 2015, pp. 4295–4299.
[16] T. N. Sainath, R. J. Weiss, A. W. Senior, K. W. Wilson, and O. Vinyals,“Learning the speech front-end with raw waveform cldnns.” in Interspeech, vol. 2015, 2015.
[17] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speechsynthesis using deep neural networks,” in Proceedings of the IEEE
International Conference on Acoustics, Speech, and Signal Processing(ICASSP), 2013, pp. 7962–7966.
[18] A. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves,N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016.
[19] S. O. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky,
Y. Kang, X. Li, J. Miller, A. Ng, J. Raiman, S. Sengupta, and
M. Shoeybi, “Deep voice: Real-time neural text-to-speech,” arXiv
preprint arXiv:1702.07825, 2017.
[20] N. R. Council, Hearing Loss: Determining Eligibility forSocialSecurity
Benefits,
R.
A.
Dobie
and
S.
V.
Hemel, Eds.
Washington, DC: The National Academies
Press, 2004. [Online]. Available: https://www.nap.edu/catalog/11099/
hearing-loss-determining-eligibility-for-social-security-benefits[21] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by
jointly learning to align and translate,” arXiv preprint arXiv:1409.0473,
2014.
[22] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence-to-Sequence learningwith neural networks,” in Advances in neural information processing
systems, 2014, pp. 3104–3112.
[23] J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end
continuous speech recognition using attention-based recurrent NN: First
results,” arXiv preprint arXiv:1412.1602, 2014.
[24] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A
neural network for large vocabulary conversational speech recognition,”in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE
International Conference on. IEEE, 2016, pp. 4960–4964.

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY[25] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly,
Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al., “Tacotron: A fully end-toend text-to-speech synthesis model,” arXiv preprint arXiv:1703.10135,
2017.
[26] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov,
R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image
caption generation with visual attention,” in Proceedings of the 32nd
International Conference on Machine Learning, ICML 2015, Lille,
France, 6-11 July 2015, 2015, pp. 2048–2057.
[27] D. He, Y. Xia, T. Qin, L. Wang, N. Yu, T. Liu, and W.-Y. Ma, “Duallearning for machine translation,” in Advances in Neural Information
Processing Systems, 2016, pp. 820–828.
[28] Y. Cheng, W. Xu, Z. He, W. He, H. Wu, M. Sun, and Y. Liu, “Semisupervised learning for neural machine translation,” arXiv preprint
arXiv:1606.04596, 2016.
[29] R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), vol. 1, 2016, pp. 86–96.
[30] S. Karita, S. Watanabe, T. Iwata, A. Ogawa, and M. Delcroix, “Semisupervised end-to-end speech recognition,” Proc. Interspeech 2018, pp.
2–6, 2018.
[31] Y. Ren, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Almost
unsupervised text to speech and automatic speech recognition,” arXiv
preprint arXiv:1905.06791, 2019.
[32] A. Rosenberg, Y. Zhang, B. Ramabhadran, Y. Jia, P. Moreno, Y. Wu,
and Z. Wu, “Speech recognition with augmented synthesized speech,”arXiv preprint arXiv:1909.11699, 2019.
[33] G. Kurata and K. Audhkhasi, “Multi-task ctc training with auxiliary feature reconstruction for end-to-end speech recognition,” Proc. Interspeech
2019, pp. 1636–1640, 2019.
[34] S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Multi-speakersequence-to-sequence speech synthesis for data augmentation inacoustic-to-word speech recognition,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2019, pp. 6161–6165.
[35] M. K. Baskar, S. Watanabe, R. Astudillo, T. Hori, L. Burget, andJ. Černockỳ, “Self-supervised sequence-to-sequence asr using unpaired
speech and text,” arXiv preprint arXiv:1905.01152, 2019.
[36] T. Hori, R. Astudillo, T. Hayashi, Y. Zhang, S. Watanabe, andJ. Le Roux, “Cycle-consistency training for end-to-end speech recognition,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6271–
6275.
[37] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neuralcomputation, vol. 9, no. 8, pp. 1735–1780, 1997.
[38] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of
gated recurrent neural networks on sequence modeling,” arXiv preprint
arXiv:1412.3555, 2014.
[39] M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” arXiv preprint
arXiv:1508.04025, 2015.
[40] A. Tjandra, S. Sakti, and S. Nakamura, “Multi-scale alignment andcontextual history for attention mechanism in sequence-to-sequencemodel,” To appear in IEEE SLT 2018, 2018.
[41] D. Griffin and J. Lim, “Signal estimation from modified short-time
fourier transform,” IEEE Transactions on Acoustics, Speech, and Signal
Processing, vol. 32, no. 2, pp. 236–243, 1984.
[42] K. Ito, “The LJ speech dataset,” https://keithito.com/LJ-Speech-Dataset/,
2017.
[43] B. McFee, M. McVicar, O. Nieto, S. Balke, C. Thome, D. Liang,
E. Battenberg, J. Moore, R. Bittner, R. Yamamoto, and et al., “librosa
0.5.0,” Feb 2017.
[44] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified
activations in convolutional network,” arXiv preprint arXiv:1505.00853,
2015.
[45] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using
RNN encoder-decoder for statistical machine translation,” arXiv preprint
arXiv:1406.1078, 2014.
[46] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “Endto-end attention-based large vocabulary speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International
Conference on. IEEE, 2016, pp. 4945–4949.
[47] Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel scheme forspeaker recognition using a phonetically-aware deep neural network,” in142014 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP). IEEE, 2014, pp. 1695–1699.
[48] P. Matějka, O. Glembek, F. Castaldo, M. J. Alam, O. Plchot, P. Kenny,
L. Burget, and J. Černocky, “Full-covariance ubm and heavy-tailed plda
in i-vector speaker verification,” in 2011 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2011,
pp. 4828–4831.
[49] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan,
and Z. Zhu, “Deep speaker: an end-to-end neural speaker embedding
system,” arXiv preprint arXiv:1705.02304, 2017.
[50] Y. Zhu, T. Ko, D. Snyder, B. Mak, and D. Povey, “Self-attentive speakerembeddings for text-independent speaker verification,” Proc. Interspeech
2018, pp. 3573–3577, 2018.
[51] K. Q. Weinberger and L. K. Saul, “Distance metric learning for large
margin nearest neighbor classification,” Journal of Machine Learning
Research, vol. 10, no. Feb, pp. 207–244, 2009.
[52] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2015, pp. 815–823.
[53] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther,“Autoencoding beyond pixels using a learned similarity metric,” inInternational Conference on Machine Learning, 2016, pp. 1558–1566.
[54] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for realtime style transfer and super-resolution,” in European Conference on
Computer Vision. Springer, 2016, pp. 694–711.
[55] D. B. Paul and J. M. Baker, “The design for the wall street journal-based
csr corpus,” in Proceedings of the workshop on Speech and Natural
Language. Association for Computational Linguistics, 1992, pp. 357–
362.
[56] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel,
M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi speech recognition toolkit,” in IEEE
2011 Workshop on Automatic Speech Recognition and Understanding. IEEE Signal Processing Society, Dec. 2011, iEEE Catalog No.:
CFP11SRW-USB.
[57] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation inpytorch,” 2017.
[58] S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-toend speech recognition using multi-task learning,” in Acoustics, Speechand Signal Processing (ICASSP), 2017 IEEE International Conference
on. IEEE, 2017, pp. 4835–4839.
[59] A. Tjandra, S. Sakti, and S. Nakamura, “Attention-based wav2text
with feature transfer learning,” in 2017 IEEE Automatic Speech
Recognition and Understanding Workshop, ASRU 2017, Okinawa,
Japan, December 16-20, 2017, 2017, pp. 309–315. [Online]. Available:
https://doi.org/10.1109/ASRU.2017.8268951[60] X. Zhu and Z. Ghahramani, “Learning from labeled and unlabeled datawith label propagation,” Tech. Rep., 2002.
[61] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neuralnetwork,” arXiv preprint arXiv:1503.02531, 2015.
[62] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating
gradients through stochastic neurons for conditional computation,” arXiv
preprint arXiv:1308.3432, 2013.
[63] G. Hinton, “Neural networks for machine learning, Coursera video
lectures,” 2012. [Online]. Available: https://www.coursera.org/course/
neuralnets[64] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization withgumbel-softmax,” 2017. [Online]. Available: https://arxiv.org/abs/1611.
01144[65] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution:
A continuous relaxation of discrete random variables,” arXiv preprint
arXiv:1611.00712, 2016.

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. X, NO. X, MM YYYY

Andros Tjandra received his B.S degree in Computer Science (cum laude) from the Faculty of Computer Science, Universitas Indonesia, Indonesia, in2014. Later, he continued his studies and obtained
his M.S (cum laude) in 2015, also from the Facultyof Computer Science, Universitas Indonesia. He iscurrently taking a doctoral course at the Graduate
School of Information Science, Nara Institute ofTechnology, Japan. His research interests include
machine learning, speech recognition, speech synthesis, and natural language processing.

Sakriani Sakti received her B.E. degree in Informatics (cum laude) from Bandung Institute ofTechnology, Indonesia, in 1999. In 2000, she received a DAAD-Siemens Program Asia 21st Century Award to study Communication Technology
at the University of Ulm, Germany, and received
her MSc degree in 2002. During her thesis work,
she worked with the Daimler-Chrysler Research andTechnology Speech Understanding Department inUlm, Germany. Between 2003 and 2009, she worked
as a researcher at ATR SLC Labs, Japan, and during
2006 through 2011 she worked as an expert researcher at NICT SLC Groups,
Japan. While working with ATR-NICT, Japan, she continued her study (20052008) at the Dialog Systems Group University of Ulm, Germany, and received
her PhD degree in 2008. She was actively involved in collaboration activities
such as the Asia Pacific Telecommunity Project (2003-2007), A-STAR, and
U-STAR (2006-2011). In 2009-2011, she served as a visiting professor in
the Computer Science Department, University of Indonesia (UI), Indonesia.
From 2011, she has been an assistant professor at the Augmented Human
Communication Laboratory, NAIST, Japan. She served also as a visiting
scientific researcher at INRIA Paris-Rocquencourt, France, from 2015 to 2016
under the JSPS Strategic Young Researcher Overseas Visits Program for
Accelerating Brain Circulation. She is a member of JNS, SFN, ASJ, ISCA,
IEICE, and IEEE. Her research interests include statistical pattern recognition,
speech recognition, spoken language translation, cognitive communication,and graphical modeling framework.

15

Satoshi Nakamura is a Professor at the Graduate
School of Information Science, Nara Institute of
Science and Technology, Japan, Honorary Professorof the Karlsruhe Institute of Technology, Germany,
and an ATR Fellow. He received his B.S. from the
Kyoto Institute of Technology in 1981 and Ph.D.
from Kyoto University in 1992. He was Associate
Professor of the Graduate School of Information
Science, Nara Institute of Science and Technology,
from 1994 to 2000. He was Director of the ATR
Spoken Language Communication Research Laboratories from 2000 to 2008 and Vice President of ATR from 2007 to
2008. He was Director General of Keihanna Research Laboratories and the
Executive Director of the Knowledge Creating Communication ResearchCenter, National Institute of Information and Communications Technology,
Japan, from 2009 to 2010. He is currently Director of the Augmented
Human Communication Laboratory and a full professor of Nara Institute of
Science and Technology’s Graduate School of Information Science. He isinterested in modeling and systems of speech-to-speech translation and speech
recognition. He is one of the leaders in speech-to-speech translation researchand has been participating in various speech-to-speech translation research
projects throughout the world, including C-STAR, IWSLT, and A-STAR. He
received the Yamashita Research Award, Kiyasu Award from the Information
Processing Society of Japan, Telecom System Award, AAMT Nagao Award,
Docomo Mobile Science Award in 2007, and ASJ Award for DistinguishedAchievements in Acoustics. He received the Commendation for Science andTechnology from the Minister of Education, Science and Technology andthe Commendation for Science and Technology from the Minister of Internal
Affairs and Communications. He also received the LREC Antonio ZampolliAward 2012. He has been an Elected Board Member of the International
Speech Communication Association, ISCA, since June 2011, IEEE Signal
Processing Magazine Editorial Board Member since April 2012, IEEE SPS
Speech and Language Technical Committee Member since 2013, and IEEE
Fellow since 2016.