USING LOCAL PHRASE DEPENDENCY STRUCTURE INFORMATION
IN NEURAL SEQUENCE-TO-SEQUENCE SPEECH SYNTHESIS

Nobuyoshi Kaiki†, Sakriani Sakti†‡
and Satoshi Nakamura†‡†Nara Institute of Science and Technology, Japan,‡RIKEN AIP, Japan

Background[Background]→ Neural networks have made it possible to produce speechsynthesis with very high quality→ Naturalness of the prosody is insufficient when using thespeech synthesis for reading novels or storybooks

Input: 警官は 走って 逃げる 泥棒を 追いかけた。
phoneme + accent information:k e ^ e k a N w a # h a ^ sh i ! cl t e # n i ^ g e ! r u #do^robooo#o^ikake!ta([Objective]
Generate more natural prosody that reflects the speaker's
intention using local dependency phrase structureThis sentence has two syntax tree
and different intonations

Table 1 Prosodic Symbols (Accent) [1]

Initial rising
Accent nucleus
Accent phrase boundary
Sentence end（Declarative）
Sentence end（Interrogative）
Pause

Prosodic symbols^！
＃（？
＿right-branching boundary

left-branching boundary

警官は 走って 逃げる 泥棒を 追いかけた
警官は 走って 逃げる 泥棒を 追いかけた
The policeman ran away the thief chased after The policeman ran away the thief chased afterSyntax tree１

Syntax tree２[1] K. Kurihara, N. Seiyama, T. Kumano, “Prosodic Features Control by Symbols as Input of Sequence-to-Sequence Acoustic Modeling for Neural TTS,”IEICE Trans. Inf. & Syst., Vol.E104–D, no2, Feb. 2021

2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

2/8

Proposed Approach[Proposed Approach] Using local phrase dependency→ Proposed 1: With prosodic symbols that represent
the depth at phrase boundaries→ Proposed 2: With prosodic symbols that reflects a folded model ofphrase & accent components based on prosodic
generation control mechanism
Proposed 1

Baseline
Initial rising
Accent nucleus
Accent phrase boundary
Sentence end (Declarative)
Sentence end (Interrogative)
Pause

Fujisaki Model[2]Prosodic symbols^！
＃（？
_Initial rising
Accent nucleus
Accent phrase boundary(syntactic dependency distance)

Punctuation mark

Proposed 2

Prosodic symbols^！
#1, #2, #3,#4, #5, #6,

警官は 走って 逃げる 泥棒を 追いかけた
k e e k a N w a h a sh i cl t eni ge r u do r o bo oo o ik ake t a
Baseline:k e ^ e k a N w a # h a ^ sh i ! cl t e # n i ^ g e ! r u # d o ^ r o b o o o # o ^ i k a k e ! t a (Proposed1 (accent + local phrase dependency structure):
Syntax tree 1:
k e ^ e k a N w a #4 h a ^ sh i ! cl t e #1 n i ^ g e ! r u #1 d o ^ r o b o o o #1 o ^ i k a k e ! t a
Syntax tree 2:
k e ^ e k a N w a #1 h a ^ sh i ! cl t e #3 n i ^ g e ! r u #1 d o ^ r o b o o o #1 o ^ i k a k e ! t a
Proposed2 (based on the processes of generating the F0 contour):
Syntax tree 1:
k e / e k a N w a ¥ #4 h a / sh i ¥ cl t e n i / g e ¥ r u d o / r o b o o o ¥ o / i k a k e ¥ t a
Syntax tree 2:
k e / e k a N w a ¥ h a / sh i ¥ cl t e #3 n i / g e ¥ r u d o / r o b o o o ¥ o / i k a k e ¥ t a
Input:
Phoneme:2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

Accent command (rising)
Accent command (falling)
Phrase command(syntactic dependency distance）

Prosodic symbols/¥#2, #3,#4, #5, #6,

Punctuation marksyntactic dependency distance:
#4

#1

#3

警官は 走って逃げる 泥棒を 追いかけた

警官は 走って 逃げる 泥棒を追いかけた

The policeman chased the thief who ran and ran away

The policeman ran and chased the thief who ran away

Syntax tree１

Syntax tree 2[2] H. Fujisaki, S. Nagashima, “A model forSynthesis of pitch contours of connected speech,”
Annual Report of Engineering Research Institute,
University of Tokyo, Vol.28, pp.53-60, 1969

3/8

Speech Database and Pre-processing[Speech Database]→ Arabian Nights (reading a story)→ Single speaker→ 11,615 sentences→ 26 hours 26 minutes[Text processing]
Text⇒ Open Jtalk[3] ⇒ Identification of phoneme,
Accent phrases、Morphological analysis results

Morphological Analysis Results⇒ Chabocha[4] ⇒ Syntax tree

[Speech processing]
Sentences ⇒ CTC Segmentation[5]Sentences ⇒ Montreal-Forced-Aligner[6]⇒ split sentence⇒ phoneme segmentation[3] "Open JTalk," http://open-jtalk.sourceforge.net/[4] “CaboCha/南瓜: Yet Another Japanese Dependency Structure Analyzer,“http://taku910.github.io/cabocha/[5] J. Nishitoba, "Introduction to CTC Segmentation," https://tech.retrieva.jp/entry/2020/10/02/143338 (in Japanese)[6] "Montreal Forced Aligner," https://montrealcorpustools. github.io/Montreal-Forced-Aligner/2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

4/8

Synthesized Speech: Pause Generation
Comparison of phonemes and prosodic symbols based on two syntax tree candidates
Baseline:

Baseline:

Proposed１:

Proposed 2:

Syntaxtree１

Syntaxtree１

Syntaxtree 2

Syntaxtree 2

Syntax tree１
警官はSyntax tree２

走って 逃げる 泥棒を 追いかけた[The policeman] [ran] [run away] [the thief] [chased after]→ The policeman chased the thief who ran and ran away

警官は走って 逃げる 泥棒を 追いかけた[The policeman] [ran] [run away] [the thief] [chased after]→ The policeman ran and chased the thief who ran away

2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

5/8

Synthesized Speech: F0 Resetting
Baseline:

Baseline:

Proposed１:
Proposed2:
left-branching boundary: lower F0 than the previous phrase
Syntaxtree１

Syntaxtree１

Syntaxtree 2

Syntaxtree 3right-branching boundary: higher F0 than the previous phrase

Syntaxtree 2left-branching boundary: lower F0 than the previous phrase

Syntaxtree 3

Syntax tree１

Syntax tree ２

Syntax tree ３

私は、 白い 屋根の大きい 家が 好きだ。

私は、 白い 屋根の大きい家が 好きだ。

私は、白い 屋根の大きい 家が 好きだ。

[I] [white] [roof] [big] [house] [like][I] [white] [roof] [big] [house] [like]→ I like white houses with big roofs.

[I] [white] [roof] [big] [house] [like]→ I like big houses with white roofs.

2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

6/8

Subjective Evaluation of Naturalness[Participants]→ 13 native speakers of Japanese→ Listened to the synthesized speech andjudged naturalness of prosody[Evaluation sentences]

Proposed 2

Proposed 1

Baseline

Baseline vs.
Proposed 1

Baseline vs. Proposed 1 vs.
Proposed 2
Proposed 2

100%→ Select 20 sentences from 250 sentences in the created DB→ AB Test: 60 pair speech utterances
Baseline vs Proposed 1:20 sentences * 2
Baseline vs Proposed 2:
20 sentences * 2
Proposed1 vs Proposed 2: 20 sentences * 2

75%50%
25%[Results]→ Proposed methods were judged to besignificantly more natural than the baseline
・Proposed1 ＞ Baseline
68% ＞ 32%
・Proposed2 ＞ Baseline
62% ＞ 38%
→ No significant difference between proposedmodel 1 and 2 (significant difference 5%)0%2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

7/8

Conclusion[Purpose]
To synthesize more natural prosody→ Incorporated new prosody symbol of syntactic dependency for neural end-to-end TTS[Method]
Proposed two models:
1) a model with prosodic symbols representing the syntactic dependency distanceat the phrase boundaries
2) a model with prosodic symbols that reflect a prosodic generation control mechanism[Results]→ Both proposed models could successfully synthesize speech sounds thatreflect syntactic structures
1) pause insertion that indicates the phrase boundary
2) F0 resetting at the right-branching boundaries→ Subjective evaluation
Synthesize speech from two proposed models were more natural than the baseline2021©︎Nobuyoshi Kaiki AHC-Lab, IS, NAIST

8/8