INTERSPEECH 2020
October 25–29, 2020, Shanghai, China

Incremental Machine Speech Chain
Towards Enabling Listening while Speaking in Real-time
Sashi Novitasari1 , Andros Tjandra1 , Tomoya Yanagita1 , Sakriani Sakti1,2 , Satoshi Nakamura1,2
1

2

Nara Institute of Science and Technology, JapanRIKEN, Center for Advanced Intelligence Project (AIP), Japan{sashi.novitasari.si3,andros.tjandra.ai6,yanagita.tomoya.yo8,ssakti,s-nakamura}@is.naist.jp

Abstract

starts recognition after receiving a complete speech utterance
from TTS, and TTS begins its synthesis after receiving a complete sentence from ASR. As a result, there is a significant delay
when encountering long utterances.
By contrast with machines, humans can listen to what they
speak in real-time. When perceiving their own speech, speakers
simultaneously evaluate their speech and generate their next utterances based on their evaluations of their perceived speech.
Several studies have investigated the importance of auditory
feedback in speech perception as well as in speech production[11, 12, 13]. It is done by constructing a delayed auditory feedback (DAF) device that extends the time between speech andauditory perception. A study by Badian et al. [12] found thatusing DAF with a 175-millisecond delay has been shown to induce mental stress, which was measured as changes in biochemical and cardiovascular variables. Another study also found thatthe effect of a few hundred milliseconds delay can disturb people, and this effect disappears immediately by stopping speaking [13]. Thus, if there is a delay in hearing, humans are unable
to continue their speech.
In this work, we attempt to mimic the human speech chain
mechanism closely and reduce the delay of feedback within themachine speech chain. The challenge is that the generation
or recognition, and feedback of the spoken utterances must be
done based on incomplete sequence information with minimal
delay. Therefore, we propose an incremental machine speechchain mechanism to improve the learning quality of end-to-end
incremental ASR (ISR) and incremental TTS (ITTS) through ashort-term closed loop. The proposed mechanism also aims toenable real-time feedback generation during inference. By enabling the real-time feedback generation, we can move a step
closer to achieve an ASR or TTS that is able to adapt simultaneously to the environment unsupervisedly, similar to human.

Inspired by a human speech chain mechanism, a machine
speech chain framework based on deep learning was recently
proposed for the semi-supervised development of automatic
speech recognition (ASR) and text-to-speech synthesis (TTS)
systems. However, the mechanism to listen while speaking canbe done only after receiving entire input sequences. Thus, there
is a significant delay when encountering long utterances. By
contrast, humans can listen to what they speak in real-time, andif there is a delay in hearing, they won’t be able to continue
speaking. In this work, we propose an incremental machine
speech chain towards enabling machine to listen while speaking
in real-time. Specifically, we construct incremental ASR (ISR)
and incremental TTS (ITTS) by letting both systems improve
together through a short-term loop. Our experimental results
reveal that our proposed framework is able to reduce delays due
to long utterances while keeping a comparable performance tothe non-incremental basic machine speech chain.
Index Terms: incremental speech chain, speech recognition,text-to-speech synthesis

1. Introduction
Automatic speech recognition (ASR) and text-to-speech synthesis (TTS) systems are crucial for human-machine interaction. ASR represents the speech perception system, while TTS
represents the speech production system. Researchers havebeen working on ASR and TTS technology for many decades.
Current state-of-the-art ASR [1, 2, 3] and TTS [4, 5] consistof attention-based sequence-to-sequence (seq2seq) neural networks for end-to-end processing.
Despite its remarkable performance, the development of
ASR and TTS has progressed more or less independently ofeach other. Motivated by the human speech chain mechanism [6], which has a feedback loop phenomenon between
speech production and a hearing system, a machine speechchain framework was recently proposed for the simultaneousdevelopment of ASR and TTS systems [7, 8, 9, 10]. In the machine speech chain framework, both ASR and TTS components
are pre-trained in supervised training using a limited amount oflabeled data. Then, by establishing a closed feedback loop between the listening component (ASR) and speaking component(TTS), both components can assist each other in unsupervised
learning. This loop enables a machine that can learn, not only
to listen or speak but also to listen while speaking.
However, standard attention-based ASR and TTS require
attention to whole input sequences while producing output. Also, current machine speech chain uses only standard
attention-based ASR and TTS components that do not work incrementally. Therefore, the mechanism to listen while speaking
can be done only after receiving entire input sequences. ASR

Copyright © 2020 ISCA

2. Related Works
Many researches on ISR and ITTS have been conducted in order to develop a simultaneous speech translation system. The
aim is to construct a speech translation system that can mimic
human interpreters and translate incoming speech from a source
language to the target language in real-time [14, 15, 16].
Several studies on the conventional ASR approach basedon the hidden Markov model (HMM) and hybrid systems[17, 18, 19, 20] have shown that the framework can recognize
speech in real-time. However, the HMM-based ASR cannot
perform end-to-end recognition, which is the current state-ofthe-art approach with deep learning. By contrast, research onneural ISR is still very limited. Jaitly et al. [21] might be thefirst group that has proposed a neural transducer framework thatcan recognize speech segment-by-segment with a fixed window.
Another study has investigated attention-transfer ISR (AT-ISR)

4372

http://dx.doi.org/10.21437/Interspeech.2020-2034

[22], which learns from attention-based non-incremental ASR
for end-to-end speech recognition with a low delay.
For TTS, developing ITTS is very challenging; the standard
framework commonly requires language-dependent contextual
linguistics of a full sentence to produce a natural-sounding
speech waveform. Existing studies of ITTS have mainly been
conducted on a model based on HMM [23, 24, 25, 26]. The
first study that attempted to synthesize speech in real-time using neural ITTS was proposed by Yanagita et al. [27]. Recently,
another ITTS was proposed based on a prefix-to-prefix framework [28].
Those previously published works were only concernedwith ISR and ITTS tasks individually. By contrast, this study
investigates the joint of incremental learning between ISR and
ITTS that attempts to mimic the process in the human speechchain. The idea here is to construct well performing low-delaysystems by evaluating the short-term output of a system using
another system and jointly improving both of them. The mechanism of the incremental machine speech chain is based on abasic machine speech chain [8] and the incremental steps arelearned through attention transfer [22]. In this work, we perform the inference process with separated ASR and TTS, andalso inference with connected ASR and TTS to do a feedback
generation.

ber of unlabeled training data. Unlabeled data is data that either
speech without transcribed text or text without corresponding
speech. In this phase, ASR and TTS support each other by doing feedback passing through a loop that connects them. The
loop between ASR and TTS consists of two unrolled processes:• ASR-to-TTS. ASR transcribes a speech utterance X,
with a length S, into a sentence text Ŷ, and then TTS
generates a speech utterance X̂ based on ASR output Ŷ.
A training loss is calculated based on the original speech
X and TTS speech X̂.
• TTS-to-ASR. Given a completed sentence text Y with alength T , TTS generates speech X̂ and ASR transcribes
the TTS speech X̂ into text Ŷ. A loss is calculated basedon the original text Y and ASR output text Ŷ.

4. Incremental Machine Speech Chain
An incremental machine speech chain follows the idea of thehuman speech chain and the basic machine speech chain by establishing a loop that connects ISR and ITTS. The main difference is that the data passing between ISR and ITTS is done
within a short time without waiting for a complete utterance.
4.1. Components
4.1.1. Seq2seq Incremental Speech Recognition System
Seq2seq ISR works with only a low delay through shortsegment-based recognition [21, 22]. ISR predicts sentence textŶ with length T from a full speech utterance X with length
S in N recognition steps. The recognition procedure for eachS
recognition step n = [1, ..., N ], where N = W, is below:

3. Basic Machine Speech Chain
Speaking^x =y= text

TTS

Feedback^y = text

^x =
ASR

x=

1. Encode Xn , a segment of W speech frames from X,
where W<S.

^y = text

Listening

2. Decode and predict Ŷn , a segment of Kn text tokens
from Ŷ, where 0 ≤ Kn < T , until an end-of-block token
predicted by attending encoder states from Xn .

(a)

^x =

LossTTS ( x , ^x )^y = text

LossASR ( y , ^y )3. Shift the input window W frames and keep the model
states.

ASR

TTS^y = text
ASR

^x =

We use attention transfer ISR (AT-ISR) [22] in our incremental machine speech chain to limit ISR construction complexity while maintaining reliable recognition performance. Attention transfer teaches AT-ISR, the student model, to mimic thealignment from the teacher model or non-incremental ASR thatprovides Xn -Yn pairs based on the attention alignment. ATISR learns Yn and an end-of-block token as the output targetof Xn . In the attention-based alignment, all Xn lengths are
uniform (W ), while the length of each text segment Yn can bedifferent.

TTS

y= text

x=
(b)

(c)

Figure 1: Overview of machine speech chain [8] (a). The feedback loop is unrolled into two processes: ASR to TTS (b) andTTS to ASR (c).
Fig. 1 illustrates an overview of the machine speech chain[7, 8, 9, 10]. Machine speech chain trains seq2seq ASR andTTS together by connecting them via a loop. Both ASR and
TTS frameworks are based on seq2seq neural networks thatconsist of encoder and decoder components with an attention
mechanism [29].
Machine speech chain was originally proposed for semisupervised learning for ASR and TTS that consists of twostages: supervised and unsupervised training. The supervised
training stage is done to train ASR and TTS independently witha small amount of labeled training data. The labeled training
data consist of pairs of speech and transcribed text. This stage
acts as a knowledge initialization phase for both components.
After the supervised training phase, ASR and TTS are
trained jointly by using the pre-trained models and a large num-4.1.2. Seq2seq Incremental Text-to-Speech Synthesis System
Seq2seq ITTS performs speech generation without waiting for
a complete sentence text input [27, 28]. We construct ITTS
using attention transfer from non-incremental ASR. The nonincremental ASR provides attention-based alignment between
fixed-size speech segments and variable-length text segments,
which is sufficient to create ISR. ITTS, with attention transferfrom non-incremental ASR, learns how to process a variablelength token sequence to produce at least W speech frames,
given the length of a speech segment in the attention-basedalignment is W . The ITTS output length here is at least W
frames because, during training, we combine subsequent speechsegments that did not align to any token (Kn =0) to the neighboring segment that aligns with an output token. We apply the4373

Step n = 2

Step n = 1

cesses the passed data in an incremental step. The closed-loop
between ISR and ITTS is unrolled into the following processes:• ISR-to-ITTS. In each recognition step n, ISR processes
a speech segment Xn and produces an output segmentŶn , while ITTS reconstructs the speech segment X̂n byencoding Ŷn , as illustrated in Fig. 2(a). Model parameters are updated using averaged ITTS losses from eachincremental step, as formulated in Eq. 1.
N
1 X
LossITTS =LossITTSn (Xn , X̂n )
(1)
N n=1

LossITTS n =1 (x n=1 , x^ n=1 ) LossITTS n =2 (x n =2 , ^x n=2 )^x 𝑛=2=^x 𝑛=1=
ITTS

ITTS

...

^y 𝑛=2= “ d e ”^y 𝑛=1= “ a b c ”
ISR

ISR

x 𝑛=2=

x 𝑛=1=• ITTS-to-ISR. For each step n, ITTS synthesizes aspeech segment X̂n by taking a text segment Yn . ISR
then transcribes the ITTS speech segment X̂n and produces Ŷn . An illustration of this process can be seen in
Fig. 2(b). Training loss is calculated based on each Yn
and Ŷn pair, which are averaged as can be seen in Eq. 2.

Full speech =(a)Step n = 1

Step n = 2

LossISR n=1 ( y n=1 , ^y n=1 ) LossISR n=2 ( y n=2 , ^y n=2)
^y 𝑛=1= “ a b c ”
ISR

LossISR =

ISR

...

^x 𝑛=1=ITTS

y 𝑛=1= “ a b c ”
Full text = a b c d e f(2)

4.3. Incremental Machine Speech Chain Learning Approach
Basic machine speech chain framework was originally proposed
for semi-supervised learning, in which the intermediate output
in an unrolled feedback loop process is generated through agreedy decoding mechanism. In this work, as our focus is not on
semi-supervised learning, we explore two approaches for intermediate output generation during joint training via closed-loop:
teacher-forcing approach and greedy approach.
In the joint training that synthesizes the intermediate output
with a teacher-forcing approach, the first system in an unrolledloop process generates an output sequence through a teacherforcing decoding mechanism. The joint training here is doneusing labeled data.
Joint training that generates the intermediate output with agreedy approach is done using unlabeled data. Here, the output
generation in the first system of an unrolled-loop process is done
through a greedy decoding mechanism.

^x 𝑛=2=ITTS

1 X
LossISRn (Yn , Ŷn )N n=1
N^y 𝑛=2= “ d e ”

y 𝑛=2= “ d e ”(b)Figure 2: Unrolled processes in incremental machine speechchain closed-loop: ISR-to-ITTS (a) and ITTS-to-ISR (b).
same alignment as ISR for ITTS to reduce components delay
incompatibility during joint training.
Speech generation with our ITTS follows the following procedure in each step n = [1, ..., N ]:1. Encode Yn , a segment of Kn tokens from token sequence Y, where 1 ≤ Kn < T .
2. Decode and predict X̂n , a segment of Wn speech frames
from speech utterance X̂, where W ≤ Wn < S, by attending Yn until a stop flag is predicted.

5. Experiment Setting
5.1. Dataset
We used Wall Street Journal (WSJ) [30] dataset for ASR andTTS construction with the following setting: SI-84, SI-200, andSI-284 as the training sets, dev93 as the development set, andeval92 as the test set. The SI-84 set consisted of 16 hours ofspeech by 83 speakers and the SI-200 set consisted of 66 hoursof speech by 200 speakers that did not overlap with SI84 set.
The SI-284 set was a combination of SI-84 and SI-200 sets. The
SI-84 and SI-284 were utilized to train the ASR and TTS during independent training, while the SI-200 set was utilized forsystems joint training with a closed-loop. All speech utterances
had a sampling rate of 16-kHz.
For both ASR input and TTS output, speech utterances wererepresented as 80-dimension log Mel spectrograms, where eachfeature frame had a length of 50 ms that shifted by 12.5 ms fromthe previous frame. In the ASR output and TTS input sides, thetext data was represented as a sequence of character units.

3. Shift the input window Kn tokens and keep the model
states.
4.2. ISR and ITTS Training Mechanism
Similar to the basic machine speech chain, the training of anincremental machine speech chain also consists of two stages:
ISR and ITTS independent training without a closed-loop and
ISR-ITTS joint training with a closed-loop.
4.2.1. Stage 1: ISR and ITTS Independent-Supervised Training
ISR and ITTS are trained independently by using paired speechtext data. Here attention transfer from non-incremental ASR isapplied to train ISR and ITTS, where both of the incremental
systems are trained using the same data as the teacher model.
4.2.2. Stage 2: ISR-ITTS Joint Training with a Short-term
Closed-Loop
ISR and ITTS support each other to jointly improve themselves
by establishing a short-term closed-loop. Here each time thefirst component finishes an incremental step, it passes the output to the second component. The second component then pro-

5.2. Model Configuration
5.2.1. ASR
Non-incremental ASR and ISR had identical seq2seq structures.
The encoder consisted of a feed-forward neural network layer

4374

Table 1: Performances of ASR (character error rate (CER)) and TTS (L2-norm2 between ground truth and predicted Mel spectrogram).
(nat-sp = natural speech as ASR input; nat-txt = natural text as TTS input; syn-sp = TTS output as ASR input; rec-txt = ASR output
as TTS input; indep-trn: independent training; chain-trn-greedy: joint training via speech chain with greedy intermediate output
generation; chain-trn-teachforce: joint training via speech chain with teacher-forcing intermediate output generation).

Data
ASR and TTS with independent training
indep-trn (SI-84)
indep-trn (SI-284)
ASR and TTS with machine speech chain
indep-trn (SI-84) + chain-trn-greedy (SI-200)
indep-trn (SI-84) + chain-trn-teachforce (SI-200)

ASR (CER(%))
Non-incremental
Incremental(delay: 7.88 sec)(delay: 0.84 sec)
nat-spsyn-spnat-sp syn-sp

TTS (L2-norm2 )Non-incremental
Incremental(delay: 103 characters) (delay: 30 characters)
nat-txt
rec-txt
nat-txt
rec-txt

17.33
7.16

27.03
9.60

17.81
7.97

44.54
19.99

0.99
0.75

1.02
0.77

1.04
0.84

3.62
1.31

11.21
7.27

11.52
6.30

14.23
9.43

32.43
12.78

0.80
0.77

0.82
0.80

0.86
0.79

1.35
1.26

performance to the non-incremental speech chain. Here we allowed the ISR and ITTS to take contextual inputs, which consisted of look-back and look-ahead blocks [22, 27, 28, 34] thatare the blocks before and after the main segment respectively,
to enrich the information in the main input segment. The ISR
input segment for an incremental step consisted of four main
speech blocks with two look-back and four look-ahead blocks,
which we decided based on optimum delay configuration in previous AT-ISR work [22]. The ISR delay here was equal to 0.84
sec. The ITTS input segment in an incremental step consistedof the main character block with two look-back and four lookahead character blocks. The ITTS main input size range was between one to four blocks, with an average of two blocks. Herea character block consisted of five characters, the average word
length in the training data. Our best proposed incremental systems were ISR and ITTS that were trained jointly by applying
teacher-forcing approach to generate the intermediate output.
Given a natural input, the best proposed ISR achieved 9.43%
CER with a delay of 0.84 sec, while the non-incremental ASR
that was trained with the basic machine speech chain achieveda CER of 7.27% but with a delay of 7.88 sec. The best ITTS,
given a natural input, was able to predict speech features with aloss of 0.79 to the ground truth by only waiting for 30 characters on average, while the non-incremental TTS with the same
training data achieved a synthesis loss of 0.77 but has to wait
for 103 characters on average to begin the synthesis.
Incremental machine speech chain framework successfully
improves ISR and ITTS performances. The improvement of
ISR and ITTS occurred on both natural input and synthetic input processing. It shows that the short-term feedback loop between the incremental systems are able to leverage their training
quality. Here we also demonstrated a real-time feedback generation. This is an important step to achieve a system that canlisten while speaking in real-time.

(512 units) that was followed by three bidirectional LSTM layers (256 units each). Each bidirectional LSTM layer applied hierarchical sub-sampling [1, 31]. As a result, an encoder state in
the encoder’s final layer represented eight speech frames. Here
we defined eight speech frames (0.14 sec) as a speech block.
The ASR decoder consisted of a character embedding layer(256-dims), an LSTM layer (512 units) with an attention mechanism, and a softmax layer. We applied an MLP-scoring function that used a previously proposed multi-scale alignment andcontextual history [32] in the attention component. The text
generation during inference was done by greedy-decoding toprevent an additional delay.
5.2.2. TTS
Our TTS followed the TTS structure in previous machine
speech chain work [8], which was a modification from TTS
Tacotron [4]. The model hyperparameters were generally sameas those in the original Tacotron. The modification was madeby replacing the rectified linear unit (ReLU) function with theleaky ReLU (LReLU) function. The CBHG (1-D convolutionbank + highway network + bidirectional GRU) module used
K = 8 filter banks. The decoder consisted of two LSTM (256units) layers. Our TTS generated 4 consecutive frames for each
decoding step, thus reducing the number of total decoding steps.
Our TTS implemented a speaker recognition component
similar to that in previous machine speech chain work [9]. Here
we used a DeepSpeaker model [33] with the same hyperparameters as in the previous machine speech chain research. The
speaker recognition component was trained using SI-84 set.

6. Experimental Results and Analysis
Our experiment result can be seen in Table 1. The baselines
are ISR and ITTS that were trained independently using SI-84
set. The toplines are the non-incremental systems that weretrained independently with SI-284 set. The machine speechchain mechanism for the non-incremental systems followed thebasic mechanism (see Sec. 3), while the incremental systems
followed the incremental mechanism (see Sec. 4). Here thesystem evaluation was done based on natural input and synthetic input. The synthetic input was generated by the target
system’s counterpart system, which was trained under the same
training condition, by processing a natural input. For the incremental system, the synthetic input of an incremental step was
the output from the processing of a short segment of natural input. We can consider the output of synthetic input processing as
the feedback for the system that produced the synthetic input.
The results show that our proposed framework could reduce the delay when encountering long utterances with a close

7. Conclusion
Our aim was to mimic the process of the human speech chain
by constructing an incremental machine chain and allowing ISR
and ITTS to improve together through a short-term loop. Our
experimental results reveal that our proposed framework can reduce the delay when encountering long utterances while keeping a close performance to a non-incremental speech chain andalso outperforming the baseline ISR and ITTS.

8. Acknowledgement
Part of this work is supported by JSPS KAKENHI Grant Numbers JP17H06101 and JP17K00237.

4375

9. References[18] H. Sak, M. Saraclar, and T. Gungor, “On-the-fly lattice rescoring for real-time automatic speech recognition,” in Proc. INTERSPEECH 2010, 2010, pp. 2450–2453.

[1] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in 2016 IEEE international conference on acoustics,
speech and signal processing (ICASSP). IEEE, 2016, pp. 4945–
4949.

[19] C. Lavania and J. Bilmes, “Reducing total latency in online realtime inference and decoding via combined context window andmodel smoothing latencies,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017,
pp. 2791–2795.

[2] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen,
Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina et al., “Stateof-the-art speech recognition with sequence-to-sequence models,”in 2018 IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP). IEEE, 2018, pp. 4774–4778.

[20] V. Peddinti, Y. Wang, D. Povey, and S. Khudanpur, “Low latency
acoustic modeling using temporal convolution and LSTMs,” IEEE
Signal Processing Letters, vol. 25, no. 3, pp. 373–377, 2018.
[21] N. Jaitly, Q. V. Le, O. Vinyals, I. Sutskever, D. Sussillo, and
S. Bengio, “An online sequence-to-sequence model using partial
conditioning,” in Advances in Neural Information Processing Systems, 2016, pp. 5067–5075.

[3] N.-Q. Pham, T.-S. Nguyen, J. Niehues, M. Mller, and A. Waibel,“Very deep self-attention networks for end-to-end speech
recognition,” in Proc. INTERSPEECH 2019, 2019, pp. 66–70.
[Online]. Available: http://dx.doi.org/10.21437/Interspeech.20192702[22] S. Novitasari, A. Tjandra, S. Sakti, and S. Nakamura, “Sequenceto-sequence learning via attention transfer for incremental speech
recognition,” in Proc. Interspeech 2019, 2019, pp. 3835–3839.
[Online]. Available: http://dx.doi.org/10.21437/Interspeech.20192985[4] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss,
N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio et al.,“Tacotron: Towards end-to-end speech synthesis,” in Proc.
INTERSPEECH 2017, 2017, pp. 4006–4010. [Online]. Available:
http://dx.doi.org/10.21437/Interspeech.2017-1452[23] T. Baumann and D. Schlangen, “Evaluating prosodic processing
for incremental speech synthesis,” in Proc. INTERSPEECH 2012,2012, pp. 438–441.

[5] M. Chen, M. Chen, S. Liang, J. Ma, L. Chen, S. Wang,and J. Xiao, “Cross-lingual, multi-speaker text-to-speech synthesis using neural speaker embedding,” in Proc. INTERSPEECH 2019, 2019, pp. 2105–2109. [Online]. Available:
http://dx.doi.org/10.21437/Interspeech.2019-1632[24] T. Baumann, “Decision tree usage for incremental parametricspeech synthesis,” in 2014 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), 2014, pp.
3819–3823.

[6] P. B. Denes and E. N. Pinson, The Speech Chain: The Physics
and Biology of Spoken Language, ser. Science/communication.
New York, N.Y: W.H. Freeman, 1993. [Online]. Available:
https://books.google.co.jp/books?id=UBtiAAAAMAAJ[25] M. Pouget, T. Hueber, G. Bailly, and T. Baumann, “HMM training strategy for incremental speech synthesis,” in Proc. INTERSPEECH 2015, 2015, p. 12011205.

[7] A. Tjandra, S. Sakti, and S. Nakamura, “Machine speech chain,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 976–989, 2020.

[26] T. Yanagita, S. Sakti, and S. Nakamura, “Incremental TTS forJapanese language,” in Proc. INTERSPEECH 2018, 2018, pp.
902–906.

[8] ——, “Listening while speaking: Speech chain by deep learning,”
in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2017, pp. 301–308.

[27] ——, “Neural iTTS: Toward synthesizing speech in real-timewith end-to-end neural text-to-speech framework,” in Proc. ISCA
Speech Synthesis Workshop (SSW), 2019, pp. 183–188.

[9] ——, “Machine speech chain with one-shot speaker adaptation,”in Proc. of INTERSPEECH 2018, 2018, pp. 887–891.

[28] M. Ma, B. Zheng, K. Liu, R. Zheng, H. Liu, K. Peng, K. Church,and L. Huang, “Incremental text-to-speech synthesis with prefixto-prefix framework,” 2019.

[10] ——, “End-to-end feedback loss in speech chain framework via
straight-through estimator,” in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019,
pp. 6281–6285.

[29] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol.
abs/1409.0473, 2015.

[11] J. S. Perkell, M. Matthies, H. Lane, F. H. Guenther, R. WilhelmsTricarico, J. Wozniak, and P. Guiod, “Speech motor control:
Acoustic goals, saturation effects, auditory feedback and internal
models,” Speech Communication, vol. 22, pp. 227–250, 1997.

[30] D. B. Paul and J. M. Baker, “The design for the Wall Street
Journal-based CSR corpus,” in Proceedings of the workshop on
Speech and Natural Language. Association for Computational
Linguistics, 1992, pp. 357–362.

[12] M. Badian, E. Appel, D. Palm, W. Rupp, W. Sittig, and K. Taeuber, “Standardized mental stress in healthy volunteers induced bydelayed auditory feedback (DAF),” European Journal of ClinicalPharmacology, vol. 16, pp. 171–176, 1979.

[31] A. Graves, “Supervised sequence labelling,” in Supervised Sequence Labelling with Recurrent Neural Networks, ser. Studies
in Computational Intelligence. Berlin: Springer, 2012, pp. 5–13.

[13] K. Kurihara and K. Tsukada, “SpeechJammer: a systemutilizing artificial speech disturbance with delayed auditory
feedback,” CoRR, vol. abs/1202.6106, 2012. [Online]. Available:
http://arxiv.org/abs/1202.6106[32] A. Tjandra, S. Sakti, and S. Nakamura, “Multi-scale alignmentand contextual history for attention mechanism in sequence-tosequence model,” in 2018 IEEE Spoken Language Technology
Workshop (SLT). IEEE, 2018, pp. 648–655.

[14] T. Mieno, G. Neubig, S. Sakti, T. Toda, and S. Nakamura, “Speedor accuracy? a study in evaluation of simultaneous speech translation,” in Proc. INTERSPEECH 2015, 2015, pp. 2267–2271.

[33] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y. Cao, A. Kannan, and Z. Zhu, “Deep Speaker: an end-to-end neural speakerembedding system,” arXiv preprint arXiv:1705.02304, 2017.

[15] S. Nakamura, K. Sudoh, and S. Sakti, “Towards machine speechto-speech translation,” Interpreting Technologies, no. 17, pp. 81–
87, 2019.

[34] T. N. Sainath, C.-C. Chiu, R. Prabhavalkar, A. Kannan, Y. Wu,
P. Nguyen, and Z. Chen, “Improving the performance of online neural transducer models,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2018, pp. 5864–5868.

[16] C. Fügen, A. H. Waibel, and M. Kolss, “Simultaneous translationof lectures and speeches,” Machine Translation, vol. 21, pp. 209–
252, 2007.
[17] M. Saraclar, M. Riley, E. Bocchieri, and V. Goffin, “Towards automatic closed captioning : Low latency real time broadcast news
transcription,” in International Conference on Spoken Language
Processing (ICSLP), 2002.

4376