Phoneme Level Speaking Rate
Variation on Waveform
Generation using GAN-TTS
Mayuko Okamoto1, Sakriani Sakti1,2, and Satoshi Nakamura1,2
1 Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan
2 RIKEN, Center for Advanced Intelligence Project AIP (RIKEN AIP), Japan

Background• Text-to-speech Synthesis (TTS)→ Development of TTS continues to advance→ Able to produce speech with high degree of intelligibility→ Naturalness of generated speech has improved
However, for spoken dialogue applications:→ The generated speech is still too monotonous→ Lacks variety and liveliness found in natural speech• Humans→ Vary their speaking rate
→ Tend to slow down to emphasize words• For more natural spoken dialogue,TTS that can control speaking rate is better
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

1

Related Work• Several studies have addressed speaking stylevariations[Yoshimura et al., 1999, Yamagishi et al., 2004]→ Most existing studies were based on HMM-based TTS→ Few studies have addressed the speaking rate issue• Recent studies with seq2seq deep learning[Wang et al., 2018] Global style tokens for Tacotron→ Varying speed and speaking style→ Information is stored globally→ Controlling at phoneme level is difficult[Park et al., 2019] Phoneme level duration model in seq2seq→ Required phoneme input instead of text
→ Only evaluated using simulated data
→ Did not consider the effect on listeners
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST2

Proposed Approach
Considering speech synthesis that can control utterance features
within a conversation
TextTTSfastslowfast• Construct natural speech corpora• Analyze differences in speaking rates• Use TTS based on generative adversarial networks (GAN-TTS)
to improve quality of synthesized speech• Enable GAN-TTS to generate speech waveform with phonemelevel speaking rate variations• Investigate the effect of speaking rate variation on listeners
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

3

Overall Architecture

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

4

Overall Architecture

GAN consists of 2 networks:• Generator (G)Learns to create speech output that causes the Discriminator D tomisrecognize the generated result as natural speech• Discriminator (D)Learns to accurately distinguish between natural and synthetic speech
produced by the Generator G

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

5Overall Architecture

Text

Good morning.

Phonemeg uh d …

Triphonesil-g+uh g-uh+d …

Full-context pentaphone context + syllable context + word level context +labelphrase level context + utterance level context
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

6

Full-context LabelText
Good morning.
Phoneme g uh d …
pentaphone context

sil ˆ sil - g + uh = dPrevious

Current
phonemeNext+ syllable context (position of “g” in syl, syl stress, #phonemes)+ word level context (position syl in wrd, part of speech, #syllables)+ phrase level context (#syllables and #words in phrase)+ utterance level context (#syllables, #words and #phrase in utterance)

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

7

Full-context Label
Text
Good morning.
Phoneme g uh d …
pentaphone context

sil ˆ g - uh + d = mPrevious

Current
phonemeNext+ syllable context (position of “g” in syl, syl stress, #phonemes)+ word level context (position syl in wrd, part of speech, #syllables)+ phrase level context (#syllables and #words in phrase)+ utterance level context (#syllables, #words and #phrase in utterance)

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

8

Proposed Method
1. Phonetic symbol expansion• Add speaking rate information to the phoneme itself

silˆsil-g+uh=d……

SSS
silˆsil-g N +uh N =d N ……
FF
F
S: Slow, N: Normal, F: Fast
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

9Proposed Method
2. Add ratio of speaking rate
• Specify the speaking rate ratio in the part newly added[g] silˆsil-g+uh=d... + /K:100[uh] silˆg-uh+d=m... + /K:100.
.
.

2019/10/29/K:75 Slow
/K:100 Normal/K:125 Fast

Mayuko Okamoto, AHC lab, IS, NAIST

10

Data Construction
Based on the CMU ARCTIC database (1132 utterance)A) Original utterancesB) 0.75 x Speaking rateC) 1.25 x Speaking rate

→ Sample Data
Recorded natural speech, spoken by one male andone female speaker.
Speakers were asked to record each utterance threetimes at different speaking rates, in as natural amanner as possible.
→ normal, slow, fast
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

11

Data Sample
femalemaleslownormalfast
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

12Analysis
We analyzed our dataset regarding threecharacteristics:ØUtterance lengthØVowel and consonant lengthØPower

All results are displayed as a ratio with Normal as 1
Reference data→ light color
Øa_slow = normal speaking rate0.75
Øa_fast = normal speaking rate1.25

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

13

Analysis: Utterance length
Comparing average utterance lengthof natural and reference speech
2.5female2natural is longer

male

length/normal length

1.5
1natural is shorter

0.502019/10/29

normalslow[natural][natural]a_slow

normal0.75[reference]

Mayuko Okamoto, AHC lab, IS, NAIST

fast[natural]a_fastnormal1.25[reference]14

Analysis: Vowel and consonant length
Comparing average vowel and consonant length when thespeaking rate changed
3
2.5
length/normal length

2vowel (female)consonant (female)vowel (male)consonant (male)

There seems to be no difference
between vowels and consonants

1.5
10.50

2019/10/29

normalslow[natural][natural]a_slow
normal0.75[reference]

Mayuko Okamoto, AHC lab, IS, NAIST

fast[natural]a_fastnormal1.25[reference]15

Analysis: Power
Comparing average power when the speaking rate changed
1.4
1.2

femalemale

No change in power occurs due
to change in speaking rate

power/normal power

10.8
0.6
0.4
0.20

2019/10/29

normalslow[natural][natural]a_slow
normal0.75[reference]

Mayuko Okamoto, AHC lab, IS, NAIST

fast[natural]a_fastnormal1.25[reference]16

Experimental Set-up
We conducted two experiments to confirm theusefulness of the proposed method• Subjective assessment of the naturalness of generated
speechØ Does speech generated using the proposed method sound morenatural than post-processed speech?

• Effectiveness of the phoneme level speaking rate variation
Ø Does the proposed method change the speaking rate at thephoneme level?

Subjective preference test
• 11 subjects (7 men, 4 women)• TOEIC score 700 or higher
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

17

Experiments 1: Methods• Subjective assessment of the naturalness ofgenerated speechØ Does speech generated using the proposed method sound morenatural than post-processed speech?

• Generated 3 types of speech
fastnormalslow

Proposed Method 1 (phonetic symbol
expansion)
Proposed Method 2 (add ratio ofspeaking rate)
Speaking rate changed through postprocessing (using Sox)
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

18

Experiments 1: Methods• Subjects were randomly presented with twoutterances and asked which sounded more natural

A sounds morenatural than BAB
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

19

Experiments 1: Results
Method
1SOX0%50%

Method10%

No diff20%

2019/10/29

Method240%60%

Method20%

100%• Proposed methods areperceived as more naturalthan post-processing• No difference in quality
between the proposed
methods

Mayuko Okamoto, AHC lab, IS, NAIST

50%

No diff

100%

No diff80%

SOX

100%20Experiments 2: Methods• Effectiveness of the phoneme level speaking ratevariation
Ø Does the proposed method change the speaking rate at thephoneme level?

• Generated 3 types of speech[Normal]“You were making them talk shop, Ruth charged him.”[fast]“You were making them talk shop, Ruth charged him.”[proposed]“You were making them talk shop, Ruth charged him.”
normal: black, fast: blue, slow: red
2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST

21

Experiments 2: Methods• Subjects were presented with text containingemphasis markers on certain words (bold print),
and two randomly selected utterances• They were asked which sample featured strongeremphasis
The bold part is moreemphasized in A than in B

AB
2019/10/29[Display]
You were making them talk shop,Ruth charged him.
Mayuko Okamoto, AHC lab, IS, NAIST

22

Experiments 2: Results• The results showed that the proposed method canchange the speaking rate appropriately.
• There was no difference between the proposed
method and the “normal” speech• The recorded “normal” speech was slower than regularnatural reading speech

Fast No diff0%2019/10/29

50%

100%

Proposed0%

Mayuko Okamoto, AHC lab, IS, NAIST

Normal50%

No diff

100%

23

Conclusion• GAN-TTS that controls the speaking rate variation at
the phoneme level• Methods• Phonetic symbol expansion• Add ratio of speaking rate

• Experiments• Subjective assessment of the naturalness of speech• Effectiveness of the phoneme level speaking rate variation• Results• The proposed method was perceived as more natural thanmanipulating the waveform of synthetic speech with postprocessing• It successfully performed speaking rate variation at thephoneme level

2019/10/29

Mayuko Okamoto, AHC lab, IS, NAIST24