Feature Articles: Communication Science as aCompass for the Future

Generative Modeling of Voice
Fundamental Frequency Contours
for Prosody Analysis, Synthesis, andConversion
Hirokazu Kameoka
Abstract

This article introduces a state-of-the-art technique that makes it possible to convert speech to different
speaking styles through the manipulation of the fundamental frequency (F0) contour without destroying
the naturalness of the speech. This technique can be used, for instance, to convert non-native speech tonative-like speech, and to convert normal speech to speech with a more lively intonation similar to theway broadcasters speak. It can also be incorporated into text-to-speech systems to improve the naturalness of computer-generated speech.
Keywords: prosody, intonation, accent, voice fundamental frequency contour, generative model

1. Introduction
The fundamental frequency (F0) contour refers to
the time course of the frequency of the vocal foldvibration of speech. We usually use not only words
and sentences to convey messages to the listener indaily communication, but also F0 contours to add
extra flavor to speech such as the identity, intention,attitude, and mood of the speaker. It is also important
to note that the naturalness of F0 contours is one of
the most significant factors that affect the perceived
naturalness of speech as a whole. In fact, synthesized(artificially created) speech with an unnatural F0 contour often sounds robotic, lifeless, or emotionless.
This article introduces a technique that makes it possible to convert the speaking style of an input utterance into different speaking styles by controlling theF0 contour while retaining its naturalness.
The proposed technique can be used, for example,
to convert the intonation of the speech uttered by anon-native speaker to a more fluent intonation similar
to the way native speakers speak, to convert theaccents of speech to those with different dialects, and

to convert the intonation of normal speech to a more
lively intonation similar to the way broadcasters
speak. It would also allow us to modify the intonationor accents of the acted speech by actors or actresses
as desired without the need to retake the scene. It canalso be incorporated into text-to-speech (TTS) systems to improve the naturalness of computer-generated speech. Furthermore, we can build a self-training
system to assist students in improving their presentation and language skills. We are also interested inapplying the proposed technique to develop a speaking-aid system that makes it possible to convert electrolaryngeal speech to normal speech, which can beused to assist people with vocal disabilities.
2. Fundamental frequency contour(intonation and accent)
The F0 contour of speech consists of intonation andaccent components. The intonation component corresponds to the relatively slow F0 variation over theduration of a prosodic unit, and the accent component
corresponds to the relatively fast F0 variation in anNTT Technical Review

Feature Articles

accented syllable. Both of these components are characterized by a fast rise followed by a slower fall. The
former usually contributes to phrasing, while the latter contributes to accentuation during an utterance. InJapanese, for example, changing the positions ofaccents results in speech with different dialects ormeanings. The magnitudes of these components correspond to how much emphasis the speaker intends toplace on the associated phrase or accent. Thus, themagnitudes and positions of these components assist
the listener in interpreting an utterance and draw
attention to specific words. The F0 contour also plays
an important role in conveying to the listener various
types of non-linguistic information such as the identity, intention, attitude, and mood of the speaker.
3. Generative modeling of F0 contours
3.1 F0 control mechanism
F0 contours are controlled by the thyroid cartilage,
which sits in front of the larynx. The assumption that
the F0 contour of speech consists of intonation andaccent components is justified by the fact that thethyroid cartilage involves two mutually independent
types of movement with different muscular reaction
times. Specifically, the intonation and accent components respectively correspond to contributions associated with the translation and rotation movements ofthe thyroid cartilage. In the late 1960s, Fujisaki proposed a well-founded mathematical model that
describes an F0 contour as the sum of these two contributions [1, 2] (Fig. 1). This model approximates
actual F0 contours of speech fairly well when themodel parameters are appropriately chosen, and its
validity has been demonstrated for many typologically diverse languages. If we can estimate the movements of the thyroid cartilage automatically from araw F0 contour, we can simulate or predict the F0
contour that we may observe when the thyroid cartilage moves differently, by simply modifying the values of the motion parameters. Since the movementsof the thyroid cartilage are characterized by the levels
and timings of the intonation and accent components,
one important challenge is to solve the inverse problem of estimating these components directly fromspeech. However, this problem has proved difficult tosolve. This is because it is difficult to determine aunique intonation and accent component pair only
from their mixture (namely an F0 contour), in thesame way that it is impossible to determine a unique
X and Y pair only from X + Y = 10. Several techniques have already been developed but so far with
Vol. 13 No. 11 Nov. 2015

Intonation parameterup(t )
Accent parameterua(t )

tGp(t )

Ga(t )
txp(t )y(t )xa(t )

tFig. 1. F0 control model (Fujisaki model).

limited success.
3.2 Statistical formulation
There are some clues that can possibly help solve
this problem. That is, we can use the fact that thelevels and timings of intonation and accent components are statistically biased in normal speech. The
author has proposed constructing a stochastic counterpart of the Fujisaki model that makes it possible touse statistical inference techniques to accurately andefficiently estimate the underlying parameters of theFujisaki model [3, 4]. The problem of estimating theintonation and accent components from a raw F0 contour is somewhat similar to the audio source separation problem. Audio source separation refers to theproblem of separating the underlying source signals
from mixed signals. Even though it looks as difficult
as solving the X + Y = 10 problem, a statistical
approach utilizing the statistical properties and thestatistical distribution of the waveforms of audio signals has proved effective in solving it. The idea for theproposed method was inspired by this idea (Fig. 2).
An example of the estimated parameters related tothe intonation and accent components (blue andgreen lines) along with the estimated F0 contours (redline) plotted on the spectrogram of the input speech isshown in Fig. 3(a). An example of converted speechwith magnified accent components is shown inFig. 3(b), and an example of converted speech withshifted positions of accent components is in Fig. 3(c).
It is important to note that in these examples we wereable to convert the speaking style of the input speech
into different styles (one with prominent accents andthe other with a different dialect) while retaining thenaturalness as if the same speaker were uttering thesame sentence in a different way. This was made possible because the proposed framework allows us tomodify F0 contours in such a way as never to violate
the physical constraint of the actual F0 control mechanism.

Feature Articles

F0 control model (Fujisaki model)
r1

r0

a0

Intonation

Accenta1
Path-restricted
hidden Markova2
model
For k = 1,...,K:Sk Sk −1 ~ πSk −1,Sk (state sequence)up[k ]
ua[k ]Sk ~ Nµp,Skµa,Sk,

σ2p 0up[k ]
Gp[k ]

ua[k ] kGa[k ]kChange ofvariables

p0

xp[k ]y [k ]
kxa[k ]

Generative model of F0 contours
y ~ N (µ, Σ)µ = Gpµp + Gaµa + µb 10 σ2aΣ = σ2pGpGTp + σ2aGaGTa + Σb

Fig. 2. Proposed model (a stochastic counterpart of the Fujisaki model, described by a discrete-time stochastic process).

(c)
6400
3200
1600
800
400
200
100

Spectrogram

0.5

1

1.5

3

3.5

0.6
0.4
0.2
0

0.5

1

1.5

2
2.5
Time (sec)

33.5

4

Spectrogram

0.5

11.5

1Parameters

0.8

6400
3200
1600
800
400
2001004Magnitude

Magnitude

12
2.5
Time (sec)Frequency (Hz)Frequency (Hz)

(a)

22.5
Time (sec)

3

3.5

4
Parameters

0.8
0.6
0.4
0.2
00.5

11.5

22.5
Time (sec)

3

3.5

4

Frequency (Hz)

(b)
6400
3200
1600
800
400
200
100

Spectrogram

0.5

1

1.5

Magnitude

22
2.5
Time (sec)

3

3.5

4
Parameters

1.5
1
0.5
0

0.5

1

1.5

2
2.5
Time (sec)

3

3.5

4

Fig. 3. (a) Example of the estimated parameters related to intonation and accent components (blue and green lines) alongwith the estimated F0 contours (red line) plotted on the spectrogram of the input speech; (b) example of converted
speech with magnified accent components; (c) example of converted speech with shifted positions of accentcomponents.

NTT Technical Review

Feature Articles

4. Future perspective
While linear predictive coding (LPC), proposed in
the late 1960s, led to the development of a speech
analysis/synthesis system that provides a powerfulparameter estimation framework for the vocal tract
model, the proposed technique enables a new systemthat does the same for the F0 control model (i.e., theFujisaki model). In a way similar to the developmentof LPC, which has given rise to modern speech analysis/synthesis and become the cornerstone module fortoday’s mobile and voice-over-IP (Internet protocol)
communication, this work can also potentially open
the door to a brand new speech analysis/synthesis
framework.
Acknowledgment
This work was carried out in collaboration with themembers of Sagayama/Moriya/Kameoka laboratoryof the University of Tokyo. I thank all the people who
contributed to this work. This work was supportedby JSPS KAKENHI Grant Number 26730100,26280060.
References
[1]

[2]

[3]

[4]

H. Fujisaki and S. Nagashima, “A Model for the Synthesis of Pitch
Contours of Connected Speech,” Annual Report of the Engineering
Research Institute, The University of Tokyo, Vol. 28, pp. 53–60,
1969.
H. Fujisaki, “A Note on the Physiological and Physical Basis for thePhrase and Accent Components in the Voice Fundamental Frequency
Contour,” Vocal Physiology: Voice Production, Mechanisms andFunctions, pp. 347–355, Raven Press, New York, USA, 1988.
H. Kameoka, J. Le Roux, and Y. Ohishi, “A Statistical Model ofSpeech F0 Contours,” Proc. of SAPA 2010 (the 2010 Workshop on
Statistical and Perceptual Audition), pp. 43–48, Makuhari, Japan,Sept. 2010.
H. Kameoka, K. Yoshizato, T. Ishihara, K. Kadowaki, Y. Ohishi, andK. Kashino, “Generative Modeling of Voice Fundamental Frequency
Contours,” IEEE/ACM Trans. Audio, Speech and Language Processing, Vol. 23, No. 6, pp. 1042–1053, 2015.

Hirokazu Kameoka

Distinguished Researcher, NTT Communication Science Laboratories.
He received his B.E., M.S., and Ph.D. from theUniversity of Tokyo in 2002, 2004, and 2007. His
research interests include audio, speech, andmusic signal processing and machine learning.
He received 13 awards over the past 10 years,including the IEEE (Institute of Electrical andElectronics Engineers) Signal Processing Society 2008 SPS Young Author Best Paper Award.
He is the author or co-author of about 90 articles
in journal papers and peer-reviewed conference
proceedings. He is currently also an Adjunct
Associate Professor at the University of Tokyo.

Vol. 13 No. 11 Nov. 2015