Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier

Japanese Neural Incremental
Text-to-speech Synthesis Frameworkwith an Accent Phrase Input
TOMOYA YANAGITA1 , SAKRIANI SAKTI2 , (Member, IEEE) and SATOSHI NAKAMURA3 ,(Fellow, IEEE)
1
2
3

Nara Institute of Science and Technology, Ikoma, Nara 630-0101, Japan (e-mail: yanagita.tomoya.yo8@is.naist.jp)
Japan Advanced Institute of Science and Technology, Nomi, Ishikawa 923-1292, Japan (e-mail: ssakti@jaist.ac.jp)
Nara Institute of Science and Technology, Ikoma, Nara 630-0101, Japan (e-mail: s-nakamura@is.naist.jp)

Corresponding author: Sakriani Sakti (e-mail: ssakti@jaist.ac.jp)
Part of this work was supported by JSPS KAKENHI Grant Numbers JP21H05054 and JP21H03467.

ABSTRACT Work in the development of neural incremental text-to-speech (iTTS), which is attracting
increasing attention, has recently pursued low-latency processing by generating speech on the fly before
reading complete sentences. Most current state-of-the-art iTTS systems use a prefix-to-prefix neural iTTS
framework with look-ahead of 1-2 unit segments (i.e., phonemes or words). However, since the Japanese
language is based on accent phrase units that are longer than words, using a prefix-to-prefix neural iTTS
with a look-ahead approach increases latency. Here, we propose an alternative to the end-to-end neural iTTS
architecture that does not apply look-ahead input when synthesizing speech chunks. We further propose
a method to use information from the previous time step by connecting the synthesized vector and themodel’s internal state to the current time step. We experimentally investigated the latency of various iTTS
systems with different modeling and synthesis chunks. The experimental results show that, for Japanese, theproposed iTTS is able to synthesize better speech quality, with a similar latency range, than the conventional
baseline prefix-to-prefix neural iTTS with word units. Moreover, we found that our proposed approach
improved the prosodic naturalness among synthesized units in the Japanese language. Subjective evaluations
also revealed that the proposed approach with an incremental unit of two accent phrases achieved the bestscores in Japanese iTTS systems.
INDEX TERMS incremental speech synthesis, end-to-end, Japanese language, accent phrase unit

I. INTRODUCTION

Speech-to-speech translation (S2ST) is an innovative technology that translates speech signals from a source language
to another language, enabling people of different languages
to communicate in their native tongues. S2ST systems commonly consist of three components [1]: automatic speech
recognition (ASR), machine translation (MT), and text-tospeech (TTS) synthesis. In operation, such a system firstrecognizes the source language speech as a source language
text, automatically translates this into a target language text,
and finally synthesizes the target language speech by TTS.
In conventional systems, the process is done sentence bysentence. Conventional S2STs produce translated speechwith significant latency for the longer speech segments oflectures and meetings, thus creating difficulty for listeners
who are struggling to follow the speaker’s talk or an ongoing
VOLUME 4, 2016

conversation.
In contrast to conventional S2ST systems, human interpreters generally break sentences into smaller chunks
and incrementally translate them based on partial information with minimum latency [2]. Researchers have recently started to develop incremental speech-to-speech translation/interpretation systems toward a simultaneous interpretation process for humans. One critical difference fromstandard S2ST systems is that, in an incremental approach,each component (ASR, MT, TTS) needs to generate on-thefly output before it receives a complete sentence. In thispaper, we focus on incremental TTS (iTTS).
End-to-end TTS systems [3]–[7] have been proposed
based on sequence-to-sequence modeling. Unlike parametric
TTS systems [8]–[10], the neural end-to-end architecturesimplifies models so that the neural network directly maps
1Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

input features to speech outputs or acoustic feature outputs.
Developments of neural vocoders [11]–[16] to reconstructspeech from acoustic features or a noise sequence have also
made remarkable progress. Therefore, the speech quality ofthe end-to-end architecture has reached the level of humanspeech [4].
Hidden Markov models (HMMs) have previously been
applied to iTTS [17]–[20]. Taking into account the performance improvement by end-to-end TTS systems, sequenceto-sequence modeling has also been applied to iTTS recently [21]–[26]. In English, a prefix-to-prefix framework[22] was proposed that allows waits for look-ahead of 12 words in iTTS. Although this prefix-to-prefix iTTS usedphoneme sequences as input and produced good speech
quality, it could not automatically control the look-ahead
length. Another work [24] proposed a prefix-to-prefix iTTS
with reinforcement learning to control the tradeoff between
look-ahead words and speech quality. Other research [23]analyzed the look-ahead effects in a prefix-to-prefix iTTS andfound that the look-ahead word length significantly affected
quality. From this analysis, Stephenson et al. [25] proposed amethod that predicts look-ahead text using a language model.
A similar method [26] was then also proposed. These related
works [22]–[26] use phoneme sequences as input features
and word units for a synthesis chunk.
In a Japanese iTTS, based on the HMM framework [20],the input features are sequences of phonemes and the linguistic features of accent phrase units. Since accent phrases
are longer than a word, a word-based, prefix-to-prefix neuralnetwork cannot be simply applied to a Japanese iTTS. When
we apply the prefix-to-prefix neural network to Japanese, thelook-ahead length is 1-2 accent phrases. Using 1-2 accentphrases with the look-ahead approach does not produce anunacceptable latency. We previously presented a preliminary
result of a neural Japanese iTTS system [21] that uses accentphrases and phonemes without a look-ahead approach. This
paper is an extended version with newer modeling, a deeper
analysis of the synthesis unit, and comparisons of latency
and quality with those of the related works. Furthermore, we
propose an additional method, using various accent featuresin addition to accent phrases, and use Parallel WaveGan [15]and Tacotron2 [4] to improve speech quality. For the Japanesebaseline prefix-to-prefix iTTS, we use a morpheme unit as
the synthesis chunk to minimize latency and then compare it
to our proposed approaches. The latency and quality of the
proposed iTTS are analyzed and then compared to the resultsof prefix-to-prefix iTTS.
II. JAPANESE ITTS INPUTSJapanese is a pitch-accent aspect language. Its accent ofeach mora, which indicates the relative pitch change in theaccent phrase, plays an important role in prosody, which
resembles tone types in tonal languages [27]. One mora
is approximately equivalent to one hiragana character. The
Japanese word “ha shi" can mean either a bridge or a pair ofchopsticks. If the pitch changes from high to low, it means
2a bridge; if the pitch changes from low to high, it means apair of chopsticks. Such pitch information is represented notin words or phonemes but in accent phrase units, which haveone accent type that changes a pitch from high to low. An
accent phrase has only one accent type, and the type depends
on the context. Accent type is critical for representing themeaning in a given context, although it does not represent aphoneme sequence. A Japanese TTS needs another input for
the accent phrase.
Fig. 1 shows a converting process from a surface text to inputs: a phoneme and an accent type in the accent phrase. The
Japanese surface text has no word boundary, such as a spacecharacter in English, and a morpheme unit is useful to obtain
the phoneme and the accent type. Consequently, morpheme
analysis detects morphemes in the text, and each morpheme
includes pronunciation, a part-of-speech tag, and accent information for constructing the accent type. To obtain features
for an accent phrase, morpheme units are reconstructed bymora units. Pronunciations are converted to a sequence ofphonemes with a pronunciation dictionary. Furthermore, theaccent phrase boundary is detected by the part-of-speechtags, and the accent type of the accent phrase is obtained fromrules with part-of-speech and accent information of the mora
unit. Finally, phonemes and accent types are used as inputs
for the Japanese TTS, and using features in the accent phrase
for Japanese end-to-end TTS systems is known to improve
speech quality [28]–[30].
Features of the accent phrase are assigned to each phoneme
and defined on the mora unit. Therefore, the same feature of
the accent phrase will be assigned to each phoneme. Fig. 2
shows accent phrase features of "kyo o wa" ("Today is") on
mora units in the accent phrase. We use five features in theaccent phrases. A1 is the difference between the position of
the phoneme in mora units and the positions of the accenttype. For example, A1 of the phoneme ”o” in the second mora
unit is 1 because the difference between the position of thephoneme ”o” in the mora unit and the mora’s position of theaccent type is 2−1 = 1. We expect A1 to increase the numberof features related to the accent type and mora units. A2 andA3 are the forward and backward positions of the mora in theaccent phrase, that is, A2 of the phoneme ”o” in the second
mora unit is 2, and A3 of the phoneme ”o” in the second mora
unit is also 2. A4 is the number of moras in the accent phrase,and A5 is the accent type of the accent phrase.
III. METHODSThis paper deals with neural TTS, which consists of two
steps: a neural encoder-decoder model to infer acoustic features from input sequences and a neural vocoder to synthesize
speech from acoustic features.
The baseline prefix-to-prefix iTTS uses a word unit as thesynthesis unit [22]. As described in Section II, a morpheme
unit is useful in obtaining inputs for Japanese TTS systems.
In later experiments, the Japanese baseline prefix-to-prefix
iTTS used morpheme units as the synthesis chunk instead ofword units.
VOLUME 4, 2016

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

FIGURE 1. Flowchart of extracting Japanese TTS inputs

B. WORD-BASED ITTS

Unlike sentence-based TTS, iTTS uses a partial synthesis
chunk instead of the full sentence. We use a prefix-to-prefix
iTTS as a baseline, where the synthesis chunk is one word
(top table in Fig. 3). The prefix-to-prefix iTTS uses a lookahead approach to account for the following speech changes.
The look-ahead approach waits for k words before the encoder process. The look-ahead length is determined by thefollowing function:g(t) = min{t + k, |x1:N |},FIGURE 2. Japanese accent features in an accent phrase

A. SENTENCE-BASED TTS

A sentence-based TTS processes a text sentence by sentence,that is, a synthesis chunk is a full sentence. A full sentence
containing N words is represented with a sequence of words
x1:N = [x1 , ..., xN ], where the word xt = [x1t , ..., xst t ]includes an input sequence of phonemes with a length st .
The encoder transforms the input sequence into anotherfeature sequence as hidden states h1:N = Enc(x1:N ) =[h1 , ..., hN ] = [h11 , ..., hs11 , ..., h1N , ..., hsNN ], where Enc(·)
represents the encoder’s process.
After getting the encoder’s hidden states, the decoder infers acoustic features. The chunk of acoustic features for theword yt = [yt1 , yt2 , ...] is estimated by h1:N and y<t , wherey<t = [y1 , ..., yt−1 ] are sequences of acoustic features until
the previous words. More specifically, the i-th frame for any
word is as follows:(3)

where |x1:N | indicates the total number of words in thesentence.
Under the condition of the look-ahead approach, the sequence of hidden states for the word is represented byh1:g(t) = Enc(x1:g(t) ) = [h1 , ..., hg(t) ]. In other words, thesequence of hidden states is conditioned by the g(t) words.
Therefore, the i-th acoustic feature for the word and the
speech waveform of the word are as follows:
yti = Dec(h1:g(t) , y<t ◦ yt,<i ),(4)wt = ϕ(yt ).

(5)

C. PROPOSED ACCENT-PHRASE-BASED ITTS

where Dec(·) denotes the decoder’s process, yt,<i =[yt1 , ..., yti−1 ], and ◦ is the concatenation of two sequences. Finally, the sentence’s speech waveform w1:N =[w1 , ..., wN ] = [w11 , w12 , ..., ] is as follows:As described earlier, an accent phrase is important for representing Japanese intonation and meaning. Therefore, we
propose Japanese iTTS on the basis of the accent phrase
as the synthesis chunk. In contrast to subsection III-B, thefull sentence containing M accent phrases is represented
with a sequence of the accent phrase x′1:M = [x′1 , ..., x′M ],′rtwhere the accent phrase x′t = [x′1t , ..., xt ] includes aninput sequence of phonemes with length rt and a sequenceof accent features (bottom table in Fig. 3). We propose two
methods to estimate acoustic features for the accent phrase.
The first method is dec+in. The encoder’s hidden states of
the accent phrase are observed:

w1:N = ϕ(y1:N ),′rth′t = Enc(x′t ) = [h′1t , ..., ht ].

yti = Dec(h1:N , y<t ◦ yt,<i ),

(1)

(2)

where ϕ(·) represents the neural vocoder’s process, y1:N are
acoustic features of the full-sentence.
VOLUME 4, 2016

(6)

Acoustic features in the accent phrase yt′ = [yt′1 , yt′2 , ...] areestimated by h′t and the last acoustic feature for the previous
3

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

FIGURE 3. Synthesis chunks and their elements for a sentence-based TTS, a prefix-to-prefix iTTS, and the proposed accent-phrase-based iTTS′p
accent phrase yt−1
. The i-th acoustic feature for the accentphrase is the following:′p′yt′i = Dec(h′t , yt−1
◦ yt,<i),(7)′where yt,<i= [yt′1 , yt′2 , ..., yt′i−1 ]. The h′t does not use thehidden vectors in the previous accent phrase x′t−1 .
Fig. 4 (a) shows how dec+in functions in the Japanese
iTTS system. The first accent phrase starts from the beginning of the sentence, and other accent phrases start fromits middle. We set the initial decoder’s inputs as the Mel
spectrogram’s last frame from the previous accent phrase.
The second method, dec+in+hidden, connects not only
the last acoustic feature but also the previous states of themodel to the current states of the model (Fig. 4 (b)). Therefore, the encoder’s hidden states and the acoustic feature areas follows:h′t = Enc(x′1:t−1 ◦ x′t ),

(8)

′
′
◦ yt,<i),yt′i = Dec(h′t , y<t

(9)

′
′
where y<t
= [y1′ , ..., yt−1].
′p′
yt−1 in (7) and y<t in (9) are different due to encoding
processes in (6) and (8). In using the second method, we
expect it to learn not only the acoustic feature time-series but
also the model’s internal state change.
Finally, the speech waveform of the accent phrase is thefollowing:
wt = ϕ(yt′ ).
(10)

IV. EXPERIMENT
A. EXPERIMENTAL CONDITIONS
1) Dataset and models

We used the JSUT dataset (version 1.1), which has 7,696
sentences (10 hours of audio sampled at 48-kHz, which
we down-sampled to 22.05-kHz) spoken by a single native
female speaker [31]. The data were divided into 7,196 pairs
4(speech and input sequences) for training, 250 pairs for thedevelopment set, and 250 pairs for the test set. We used Open
Jtalk1 for extracting the phoneme and accent features fromthe text and files with speech duration2 . We used a Geforce
RTX TITAN with a memory of 24 gigabytes.
The acoustic features were extracted by Fourier transform,
and our final set was composed of 80 dimensions of log Mel
spectrogram features. The size of the Fourier transform was
2,048 points. The frameshift and frame lengths were 10 and
50 milliseconds, respectively. We used Tacotron2 [4] to estimate acoustic features from inputs, and a Parallel WaveGan[15] to reconstruct speech from the acoustic features. Unlike
the original Tacotron2, we used a uni-directional LSTM toconnect hidden states of the model and Forward Attention
with Transit Agent [32] to quickly converge the attention.
We used an Adam [33] optimizer with a 32-batch size. The
learning rate was 1e-3.
We used the prefix-to-prefix model as a baseline iTTS.
The input feature is a phoneme sequence (Pho). To accommodate such pitch information, we used two input typesof Japanese iTTS to improve speech quality. Pho+AccTypeuses phonemes and only accent types (A5) in accent phrases.
We used two embedding layers for the phonemes and theaccent types. Then we concatenated two embedding outputs
as single input. Pho+AccFeats uses phonemes and bothaccent types (A5) and many accent features (A1, A2, A3,
and A4) in the accent phrases. We used six embedding layers
for the phonemes and many accent features and concatenatedthe embedding outputs as one input. Table 1 shows the sizeof each embedding layer in our experiment. The first column
indicates input features, and the second indicates the sizeof each embedding layer. We replaced low-frequency input
features with an unknown symbol using a threshold to deal
with unknown inputs. In Japanese iTTS, the vocabulary size
1 Open Jtalk – http://open-jtalk.sourceforge.net/
2 https://github.com/r9y9/jsut-lab
VOLUME 4, 2016

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input(a) dec+in: connecting the last synthesis vector to the decoder’s initial input(b) dec+in+hidden: connecting not only the last synthesis vector but also the model’s internal states to theencoder and decoder
FIGURE 4. Proposed approaches to Japanese iTTS

TABLE 1. Input feature types and embedding dimensions

Input feature Input and embedding dimensions
types
Phoneme feature dimension: 44,
Phoembedding dimension: 512
Phoneme feature dimension: 44 (TTS) or 46 (iTTS),
Pho+AccType embedding dimension: 480
A5-feature dimension: 23, embedding dimension: 32
Phoneme feature dimension: 44 (TTS) or 46 (iTTS),
embedding dimension: 432
A1-feature dimension: 26, embedding dimension: 16
Pho+AccFeats A2-feature dimension: 20, embedding dimension: 16
A3-feature dimension: 20, embedding dimension: 16
A4-feature dimension: 23, embedding dimension: 16
A5-feature dimension: 23, embedding dimension: 16

is increased by two due to the special characters that indicate
the middle.
Baseline TTS and iTTS systems were trained in sentencebased units. The speech was synthesized using each synthesis chunk by adding location symbols to differentiate the
VOLUME 4, 2016

unit’s location: 
 is the sentence’s start and 
 isthe sentence’s end. The terminating process of the decoder
differs in each model. The decoding process in the TTS iscontrolled by the stop flag [4]. The iTTS uses the stop flagand the alignment distribution to stop the decoder in order tosynthesize Mel spectrogram frames [22].
On the other hand, our proposed method was trained on
the accent-phrase-based units, and the decoding process wascontrolled only by the stop flag. The speech was synthesized using the accent phrase by adding location symbols
to differentiate the unit’s location: 
, 
, <m> isthe middle sentence’s start, and </m> is the middle sentence’s end. When we used dec+in, we connected only the
Mel spectrogram to each synthesis chunk. When we used
dec+in+hidden, we connected not only the Mel spectrogram
but also the RNN hidden states on each synthesis chunk.

5

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

2) Evaluation indexes

We used natural speech as a reference in our objective evaluation of speech quality. Then, we synthesized the speechusing various iTTS systems. We calculated a perceptualbased measure in terms of the fundamental frequency (F0)between natural speech and synthesized speech as follows:
T

Cf0 =

|f tar (t)|1X,
1200log2 0src
T t=1
|f0 (t)|(11)

where f0tar is F0 of natural speech, f0src is F0 of synthesized
speech, and 1200 cents represents a difference of 1-octave[18]. We also calculated the accuracy of the estimated spectrum using Mel cepstrum distortion [34] in dB, defined asfollows:
v
T u
D
u X
X
1 10
t2
(ytd − ŷtd )2 ,(12)
M CD =T ln(10) t=1
d=1

where t and d are the number of frames and the Mel cepstrum
dimensions, respectively. y is a Mel cepstrum component of
natural speech, ŷ is a Mel cepstrum component of synthesized speech.
In our subjective evaluation of speech quality, we calculated a mean opinion score (MOS) test [35] for the naturalness of changing lengths of incremental units. Subjects
listened to each presented bit of speech audio and rated theoverall quality based on its naturalness. We used a 5-point
MOS scale, where 5 indicated excellent speech utterances(very clear and completely natural) and 1 indicated badspeech utterances (unclear and completely unnatural). We
conducted subjective evaluations in Japanese with 13 native
speakers. Synthesized speech samples were evaluated from10 speech utterances per model.
To analyze latency, we used a re-speaking system thatsynthesized the same speech after playing natural speechwith each chunk. We measured the latency time from starting
an input sequence to finishing each bit of synthesis speech.
Then we calculated the frequency of latency in each model.
B. OBJECTIVE EVALUATION OF METHODSAs described above, we used two types of input featuresin Fig. 2 and a proposed approach that uses the previous
decoder’s input and hidden states of a model. In this section, we evaluate the differences in the input types and theeffectiveness of the models. The iTTS’s speech quality isobjectively evaluated for the differences in input and method.
Table 2 shows the results of the objective evaluations
with methods, input features, models, and synthesis chunks.
We made four observations regarding the Japanese iTTS.
First, the objective evaluation in F0 and MCD demonstrated
that the proposed iTTS system approaches the sentencebased TTS (see (3-1) vs. (3-3) or (3-1) vs. (3-4)). Second,
the proposed Japanese iTTS systems with dec+in+hidden
are more efficient than word-based iTTS systems that we
6

FIGURE 5. Relationship between latency and its frequency. The top figure is(1-2): baseline prefix-to-prefix iTTS with one morphoneme+look-1 (black:average latency of 0.464 seconds) and (3-3): iTTS with one accent phrase as
a synthesis chunk (red: average latency of 0.655 seconds). The middle figureis (1-3): baseline prefix-to-prefix iTTS with one morphoneme+look-2 (gray:average latency of 0.572 seconds) and (3-4): iTTS with two accent phrases as
a synthesis chunk (yellow: average latency of 1.20 seconds). The bottom
figure is (1-4):prefix-to-prefix iTTS with one morphoneme+look-3 (green:
average latency of 0.667 seconds).

reimplemented ourselves. Third, regarding the different input types in the accent phrase, Pho+AccFeats is more efficient than Pho+AccType. Finally, our proposed method
dec+in+hidden is better than dec+in, that is, using a large
amount of previous information is efficient.
C. OBJECTIVE EVALUATION OF RELATIONSHIPBETWEEN SPEECH QUALITY AND LATENCYAn accent phrase is longer than a morpheme. Therefore,
we must compare the latency of the baseline iTTS and our
proposed iTTS before subjectively evaluating our proposed
method. We analyzed the latency of the iTTS models withfive methods: (1-2), (1-3), (1-4), (3-3), and (3-4).
Fig. 5 shows latencies and their frequencies. The latenciesof the proposed iTTS were 0.655 seconds with one accentphrase and 1.20 seconds with two accent phrases. While thelatencies of the baseline methods are 0.572 seconds withlook-ahead of two morphemes and 0.667 seconds with lookahead of three morphemes. The latency of the proposed
method with one accent phrase is slightly slower but has
lower F0 error and MCD.
D. SUBJECTIVE EVALUATION OF NATURALNESS

Next, we conducted a MOS test as a subjective evaluationof naturalness under six experimental conditions: a baseline
iTTS with one morpheme as a synthesis chunk and no lookahead approach (1-0), a baseline prefix-to-prefix iTTS with
one morpheme as a synthesis chunk and a look-ahead-1 (12), a baseline prefix-to-prefix iTTS with one morpheme as asynthesis chunk and a look-ahead-2 (1-3), a topline TTS (31), our proposed iTTS with one accent phrase as a synthesis
chunk (3-3), and our proposed iTTS with two accent phrases
VOLUME 4, 2016

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

TABLE 2. Objective evaluations of proposed methods. Note that full sentence in a column of the synthesis chunk means a sentence unit as the synthesis chunk, 1morpheme+look-k means one morpheme as the synthesis chunk with a look-ahead of k length, and 1 accent phrase and 2 accent phrases mean one accentphrase as the synthesis chunk and two accent phrases as the synthesis chunk.

Methods
Input type: Pho(1-1): Baseline TTS(1-2): Baseline prefix-to-prefix iTTS
(1-3): Baseline prefix-to-prefix iTTS
(1-4): Baseline prefix-to-prefix iTTS
(1-5): Baseline prefix-to-prefix iTTS
(1-6): Baseline prefix-to-prefix iTTS
(1-7): Baseline prefix-to-prefix iTTS
Input type: Pho+AccType(2-1): Baseline TTS(2-2): Japanese iTTS
(2-3): Japanese iTTS
(2-4): Japanese iTTS
Input type: Pho+AccFeats(3-1): Topline TTS(3-2): Japanese iTTS
(3-3): Japanese iTTS
(3-4): Japanese iTTS

ModelsSynthesis chunk

F0 error [cent]

MCD [dB]

Sentence-based TTS
Word-based iTTS
Word-based iTTS
Word-based iTTS
Word-based iTTS
Word-based iTTS
Word-based iTTS

Fulll sentence
1 morpheme+look-1
1 morpheme+look-2
1 morpheme+look-3
1 morpheme+look-4
1 morpheme+look-5
1 morpheme+look-10

276.98318.99317.38311.162303.001304.235
290.438

7.03
7.24
7.20
7.20
7.21
7.14
7.08

Sentence-based TTS
Accent phrase-based iTTS with dec+in
Accent phrase-based iTTS with dec+in+hidden
Accent phrase-based iTTS with dec+in+hidden

Full sentence
1 accent phrase
1 accent phrase
2 accent phrases

237.10309.57273.04278.34

6.87
7.32
7.27
7.12

Sentence-based TTS
Accent phrase-based iTTS with dec+in
Accent phrase-based iTTS with dec+in+hidden
Accent phrase-based iTTS with dec+in+hidden

Full sentence
1 accent phrase
1 accent phrase
2 accent phrases

212.81290.84246.16230.03

6.84
7.33
7.08
6.94

prefix iTTS with look-ahead-2 and each proposed Japanese
iTTS, and a significant difference was confirmed (p<0.001).
V. CONCLUSION

FIGURE 6. Relationship between MOS and its average latency. Note that
MOS score of (1-0) is 1.14, (1-2) is 2.78, (1-3) is 2.83, (3-1) is 4.47, (3-3) is
3.45, and (3-4) is 3.87.

as a synthesis chunk (3-4). Method (1-0) used one morpheme
as a short sentence with the baseline TTS model.
The subjective evaluation results and each average latency
are shown in Fig. 6. The baseline iTTS that utilized eachmorpheme as a short sentence showed a lower quality thanthe baseline prefix-to-prefix iTTS systems, since that model
did not use look-ahead inputs [22]. The best model was the
topline TTS with all phonemes and accent features in thesentence; the latency was 4.63 seconds and longer than theothers. Although the objective results showed that the proposed iTTS system approached the level of the topline TTS,
the subjective results of the proposed iTTS showed room for
improvement. The intonation between accent phrases might
attract strong attention from the evaluators, or the Japanese
iTTS system might need more comprehensive information.
iTTS systems with accent phrases showed better quality
than baseline iTTS systems. Furthermore, a statistical significance test was conducted between the baseline prefix-toVOLUME 4, 2016

This paper proposed a novel Japanese end-to-end neural
iTTS architecture using an accent phrase unit. We presented
a method to connect the initial input by considering theacoustic time-series well as a method to connect the model’sinternal state. Moreover, we used two types of input features
for the accent phrase unit. We experimentally investigated thelatency of various iTTS systems with different modeling andsynthesis chunks.
We objectively evaluated the speech quality regarding thedifferences in input and method. The objective evaluation in
F0 and MCD demonstrate that the proposed iTTS systemapproaches the sentence-based TTS, while the MOS scoreof the proposed iTTS is still lower. The proposed Japanese
iTTS systems using the previous initial input and the previous
model’s internal state are more efficient than word-based
iTTS systems. The speech quality is improved by using many
features in the accent phrase unit for inputs. Moreover, usinga large amount of previous information is also efficient.
Futhermore, we also subjectively evaluated speech quality.
Our results reveal that the proposed method with one accentphrase had better MOS scores, with a similar latency range
between the baseline with look-ahead of two morphemes andthe baseline with look-ahead of three morphemes. A methodwith two accent phrases improved speech quality, although
the latency is slightly longer than in a baseline with a twomorpheme look-ahead approach.
ACKNOWLEDGEMENTS

Part of this work was supported by JSPS KAKENHI Grant
Numbers JP21H05054 and JP21H03467.
7

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

REFERENCES[1] S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro,
J.-S. Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto, “The atr multilingual speech-to-speech translation system,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 14, no. 2, pp. 365–376, 2006.
[2] F. Goldman-Eisler, “Segmentation of input in simultaneous translation,”
Journal of psycholinguistic Research, vol. 1, no. 2, pp. 127–140, 1972.
[3] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly,
Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis,
R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speechsynthesis,” in Proc. Interspeech, 2017, pp. 4006–4010.
[4] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen,
Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis,
and Y. Wu, “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in Proc. ICASSP, 2018, pp. 4779–4783.
[5] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: Fast, robust and controllable text to speech,” in Proc. NeurIPS,
vol. 32, 2019.
[6] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech
2: Fast and high-quality end-to-end text to speech,” in Proc. ICLR, 2021.
[7] C. Miao, S. Liang, M. Chen, J. Ma, S. Wang, and J. Xiao, “Flow-tts: A nonautoregressive network for text to speech based on flow,” in Proc. ICASSP,
2020, pp. 7209–7213.
[8] K. Tokuda, H. Zen, and A. W. Black, “An hmm-based speech synthesis
system applied to english,” in IEEE speech synthesis workshop. IEEE
Santa Monica, 2002, pp. 227–230.
[9] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using deep neural networks,” in Proc. ICASSP, 2013, pp. 7962–7966.
[10] H. Zen and H. Sak, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis,”in Proc. ICASSP 2015, 2015, pp. 4470–4474.
[11] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals,
A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu,“Wavenet: A generative model for raw audio,” in Proc. Interspeech, 9th
Speech Synthesis Workshop, 2016, p. 125.
[12] A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu,
G. Driessche, E. Lockhart, L. Cobo, F. Stimberg et al., “Parallel wavenet:
Fast high-fidelity speech synthesis,” in International Conference on Machine Learning. PMLR, 2018, pp. 3918–3926.
[13] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A flow-based generative network for speech synthesis,” in Proc. ICASSP 2019, 2019, pp.
3617–3621.
[14] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande,
E. Lockhart, F. Stimberg, A. Oord, S. Dieleman, and K. Kavukcuoglu,“Efficient neural audio synthesis,” in International Conference on Machine
Learning. PMLR, 2018, pp. 2410–2419.
[15] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial networks withmulti-resolution spectrogram,” in Proc. ICASSP, 2020, pp. 6199–6203.
[16] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks
for efficient and high fidelity speech synthesis,” in Advances in Neural
Information Processing Systems, vol. 33. Curran Associates, Inc., 2020,
pp. 17 022–17 033.
[17] T. Baumann and D. Schlangen, “Evaluating prosodic processing for incremental speech synthesis,” in Proc. Interspeech, 2012, pp. 438–441.
[18] M. Pouget, T. Hueber, G. Bailly, and T. Baumann, “HMM training strategy
for incremental speech synthesis,” in Proc. Interspeech, 2015, pp. 1201–
1205.
[19] M. Pouget, O. Nahorna, T. Hueber, and G. Bailly, “Adaptive latency forpart-of-speech tagging in incremental text-to-speech synthesis,” in Proc.
Interspeech, 2016, pp. 2846–2850.
[20] T. Yanagita, S. Sakti, and S. Nakamura, “Incremental tts for japanese
language,” in Proc. Interspeech, 2018, pp. 902–906.
[21] ——, “Neural itts: toward synthesizing speech in real-time with end-toend neural text-to-speech framework,” in Proc. Interspeech, 10th ISCA
Speech Synthesis Workshop, 2019, pp. 183–188.
[22] M. Ma, B. Zheng, K. Liu, R. Zheng, H. Liu, K. Peng, K. Church,and L. Huang, “Incremental text-to-speech synthesis with prefix-to-prefix
framework,” in Proc. Findings of ENLNP, 2020, pp. 3886–3896.
[23] B. Stephenson, L. Besacier, L. Girin, and T. Hueber, “What the future
brings: investigating the impact of lookahead for incremental neural TTS,”in Proc. Interspeech, 2020, pp. 215–219.
8[24] D. S. R. Mohan, R. Lenain, L. Foglianti, T. H. Teh, M. Staib, A. Torresquintero, and J. Gao, “Incremental text to speech for neural sequenceto-sequence models using reinforcement learning,” in Proc. Interspeech,
2020, pp. 3186–3190.
[25] B. Stephenson, T. Hueber, L. Girin, and L. Besacier, “Alternate endings:
improving prosody for incremental neural TTS with predicted future text
input,” in Proc. Interspeech, 2021, pp. 3865–3869.
[26] T. Saeki, S. Takamichi, and H. Saruwatari, “Incremental text-to-speechsynthesis using pseudo lookahead with large pretrained language model,”IEEE Signal Processing Letters, vol. 28, pp. 857–861, 2021.
[27] S. Yokomizo, T. Nose, and T. Kobayashi, “Evaluation of prosodic contextual factors for HMM-based speech synthesis,” in Proc. Interspeech, 2010,
pp. 430–433.
[28] Y. Yasuda, X. Wang, S. Takaki, and J. Yamagishi, “Investigation ofenhanced tacotron text-to-speech synthesis systems with self-attention for
pitch accent language,” in Proc. ICASSP, 2019, pp. 6905–6909.
[29] T. Fujimoto, K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda,“Impacts of input linguistic feature representation on Japanese end-toend speech synthesis,” in Proc. Interspeech, 10th ISCA Speech Synthesis
Workshop, 2019, pp. 166–171.
[30] K. KURIHARA, N. SEIYAMA, and T. KUMANO, “Prosodic featurescontrol by symbols as input of sequence-to-sequence acoustic modelingfor neural tts,” IEICE Transactions on Information and Systems, vol.
E104.D, no. 2, pp. 302–311, 2021.
[31] R. Sonobe, S. Takamichi, and H. Saruwatari, “Jsut corpus: free large-scalejapanese speech corpus for end-to-end speech synthesis,” arXiv preprint
arXiv:1711.00354, 2017.
[32] J.-X. Zhang, Z. Ling, and L. Dai, “Forward attention in sequence- tosequence acoustic modeling for speech synthesis,” in Proc. ICASSP, 2018,
pp. 4789–4793.
[33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inProc. ICLR, Y. Bengio and Y. LeCun, Eds., 2015.
[34] R. Kubichek, “Mel-cepstral distance measure for objective speech quality
assessment,” in Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, 1993, pp. 125–128 vol.1.
[35] Recommendation ITU-T P.800, “Methods for subjective determination oftransmission quality,” 1996.

TOMOYA YANAGITA received his master’s degree from Nara Institute of Science and Technology (NAIST) in 2018. He graduated from a
doctoral program at NAIST in March 2022. He iscurrently a researcher at the Augmented Human
Communication Laboratory at NAIST. He is amember of ASJ and IPSJ. His research interests include text-to-speech synthesis, deep learning, andincremental speech processing for simultaneous
speech translation systems.
VOLUME 4, 2016

Yanagita et.al et al.: Japanese Neural incremental Text-to-speech Synthesis Framework with an Accent Phrase Input

SAKRIANI SAKTI received her B.E. degree inInformatics (cum laude) from Bandung Instituteof Technology, Indonesia, in 1999. In 2000, she
received the DAAD-Siemens Program Asia 21stCentury Award to study in Communication Technology, University of Ulm, Germany, and received
her MSc degree in 2002. During her thesis work,
she worked with the Speech Understanding Department, DaimlerChrysler Research Center, Ulm,
Germany. Between 2003 and 2009, she worked asa researcher at ATR SLC Labs, Japan, and during 2006–2011, she worked
as an expert researcher at NICT SLC Groups, Japan. While working with
ATR-NICT, Japan, she continued her studies (2005–2008) with Dialog
Systems Group, University of Ulm, Germany, and received her Ph.D. degreein 2008. She has been actively involved in collaboration activities such
as Asian Pacific Telecommunity Project (2003–2007), A-STAR, and USTAR (2006–2011). In 2009–2011, she served as a visiting professor in
the Computer Science Department, University of Indonesia (UI), Indonesia.
In 2011–2017, she was an assistant professor at the Augmented Human
Communication Laboratory, NAIST, Japan. She also served as a visiting
scientific researcher of INRIA Paris-Rocquencourt, France, in 2015–2016,
under the JSPS Strategic Young Researcher Overseas Visits Program for
Accelerating Brain Circulation. In 2018–2021, she was a research associate
professor at NAIST and a research scientist at RIKEN, Center for AdvancedIntelligent Project (AIP), Japan. Currently, she is an associate professor
at JAIST, adjunct associate professor at NAIST, visiting research scientist
at RIKEN AIP, and adjunct professor at the University of Indonesia. She
is a member of JNS, SFN, ASJ, ISCA, IEICE, and IEEE. She is also a
committee member of IEEE SLTC (2021-2023) and an associate editor of
the IEEE/ACM Transactions on Audio, Speech, and Language Processing(2020-2023). She is also the chair of ELRA/ISCA Special Interest Group on
Under-resourced Languages (SIGUL) and a Board Member of Spoken Language Technologies for Under-Resourced Languages (SLTU). Her researchinterests include statistical pattern recognition, graphical modeling framework, deep learning, multilingual speech recognition and synthesis, spoken
language translation, affective dialog system, and cognitive-communication.

VOLUME 4, 2016

SATOSHI NAKAMURA is a professor of the
Graduate School of Science and Technology, Nara
Institute of Science and Technology, Japan, andthe Honorarprofessor of Karlsruhe Institute ofTechnology, Germany. He received a B.S. from
Kyoto Institute of Technology in 1981 and a
Ph.D.from Kyoto University in 1992. He was an
associate professor of the Graduate School of Information Science at Nara Institute of Science andTechnology in 1994-2000 and the director of the
ATR Spoken Language Communication Research Laboratories in 20002008 and a vice president of ATR in 2007-2008. He was the director generalof the Keihanna Research Laboratories and the executive director of theKnowledge Creating Communication Research Center,National Institute ofInformation and Communications Technology, Japan in 2009-2010 and a
project leader of the Tourism Information Analytics Team at the Center for
Advanced Intelligence Project (AIP) of RIKEN Institute in 2017-2021. He iscurrently the director of the Augmented Human Communication laboratory
and a full professor of the Graduate School of Information Science at
Nara Institute of Science and Technology. He interests include modelingand systems of speech-to-speech translation and speech recognition. He isa leading researcher of speech-to-speech translation and has been participating in various worldwide speech-to-speech translation research projects,
including C-STAR, IWSLT, and A-STAR. He received the LREC Antonio
Zampolli Prize in 2012. He also received the Yamashita Research Award,
the Kiyasu Award from the Information Processing Society of Japan, theTelecom System Award, the AAMT Nagao Award, the Docomo Mobile
Science Award in 2007, the ASJ Award for Distinguished Achievements in
Acoustics, the Commendation for Science and Technology by the Ministerof Education, Culture, Sports, Science and Technology, and the Commendation for Science and Technology by the Minister of Internal Affairs andCommunications. He was an elected board member of the International
Speech Communication Association, ISCA in 2011-2018, an IEEE Signal
Processing Magazine Editorial Board Member in 2012-2014, and an IEEESPS Speech and Language Technical Committee Member in 2013-2015. He
is an IEEE Fellow, ISCA Fellow, IPSJ Fellow, and ATR Fellow.

9