Generative modeling of speech F0 contours

1,211221*
H. Kameoka , K. Yoshizato , T. Ishihara , Y. Ohishi , K. Kashino , S. Sagayama

1 The University of Tokyo, 2 NTT Communication Science Laboratories (*Currently with National Institute of Informatics)

Abstract: This paper reviews our ongoing work on generative modelingof speech fundamental frequency (F0) contours for estimating prosodic
features from raw speech data [3]-[6]. Proposed F0 contour model isformulated by translating “Fujisaki model” into probabilistic model
described as discrete-time stochastic process. The motivation behind
this formulation is:① to derive a general parameter estimation framework for Fujisaki
model, allowing for introduction of powerful statistical methods② to construct an automatically trainable version of Fujisaki model sothat in future it can be incorporated into statistical-model-based
text-to-speech synthesis systems

 Generative modeling of phrase and accent commands
Requirements• Phrase command is impulse• Accent command is rectangular pulse• Two commands do not overlap each other

p1

r1

r0

a1

F0 contour typically consists of two components:• phrase components(long term pitch variations over the duration of prosodic units), and• accent components (short term pitch variations in accented syllables).
“Fujisaki model” is a well-founded mathematical model that describes
F0 contour as the sum of these two components (See below for details).
 A notable feature of this model is that the model parameters areassociated with physiologically and linguistically meaningful quantities.
 However, automatic estimation of Fujisaki model parameters from raw
F0 contour has been a difficult task...
andPhrase commandPhrase component

Profile of Gaussian mean

2nd order critically damped filter (Laplace domain):Inverse filter (Laplace domain):

Backward difference
approximation
 Relationship between

a2Time “Fujisaki model” [1] Relationship between

 Discretization of phrase control mechanism

modeling phrase & accent command sequence

F0time course of frequency of vocal fold vibration(Manifestation of physical movement of thyroid cartilage)
 contains various types of non-linguistic information• Speaker’s identity, intention, attitude, mood, etc.
 modeling and analyzing F0 contours can be potentially
useful for many speech applications in which prosodic
information plays a significant role.
(In speech synthesis, one challenge is to create a natural-sounding pitch contour for the utterance as a whole.) Generative modeling of F0 contour

Propose introducing path-restricted HMM for

1. Introduction
 Fundamental frequency (F0) contour

Discretized Fujisaki model

3. Generative model of speech F0 contours

and


a3

 Probability density function ofSampling
periodInverse filter (z-domain):and(discrete-time index)

r0 p1 r1 a1 a1 r1 a2 a2 r0 p1 r1state sequence

state transition:

where Probability density function ofConstrained all-pole model governed by single parameter

state emission: Discretization of accent control mechanism
 follows exactly the same as above

4. Singing voice F0 contour modeling [6]5. Evaluation using real/synthetic data

 Propose describing singing voice F0 contour based on a model structurally[Experiment 1]

similar to Fujisaki model[Experiment 2]

 Aim of this papertranslate Fujisaki model into a probabilistic model (stochastic process) motivation for this is twofold:• to provide general parameter estimation framework utilizing powerfulstatistical inference methods, and• to construct an automatically trainable version of Fujisaki model thatcan potentially be combined with statistical model for speech synthesis.
 F0 contour (log scale) =lower bound for F0

Phrase component + Accent component + Base value• contributions associated with forward-backward translation androtation of thyroid cartilage, respectively• outputs of different 2nd order critically damped systems“Phrase command”

Dirac deltas“Accent command”
rectangular pulses

5. MAP estimation
 Maximum A Posteriori estimation of Fujisaki model parameters(namely, state sequence of present HMM)

observed F0 contour

 Prior distribution:

 Developed Iterative algorithm for estimating See [3]-[6] for details.

DR (%)
Conventional [2] 68.8
Proposed1 [4]69.7
Proposed2 [5]69.5

based on EM algorithm

RMSE
0.17190.1372
0.0611

Conventional [2]Proposed1 [4]Proposed2 [5]DR (%)
72.6
76.8
83.4

DR: Detection rate
“RMSE”: RMSE between observed F0 contour and optmized model

References Likelihood function:[Experiment 2][Experiment 1]

F0 (Hz) Fujisaki model assumes:

 This model can be translated into a probabilistic model in the same way as above.

 Results

102(2) Estimated phrase and accent commands
1

Magnitude

2. Original Fujisaki model [1](1) Observed and estimated F0s

Example of parameter estimatesobtained with [5]

0.5
001

2
Time (s)34[1] H. Fujisaki, “A note on the physiological and physical basis for the phrase and accent components in the voice fundamental frequency contour,” in Vocal
Physiology: Voice Production, Mechanisms and Functions, O. Fujimura, Ed. Raven Press, 1988.
[2] S. Narusawa, N. Minematsu, K. Hirose, and H. Fujisaki, “A method for automatic extraction of model parameters from fundamental frequency contours ofspeech,” in Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2002), 2002, pp. 509–512.
[3] H. Kameoka, J. Le Roux, and Y. Ohishi, “A statistical model of speech F0 contours,” in Proceedings of the 2010 ISCA Workshop on Statistical and PerceptualAudition (SAPA 2010), 2010, pp. 43–48.
[4] K. Yoshizato, H. Kameoka, D. Saito, and S. Sagayama, “Statistical approach to fujisaki-model parameter estimation from speech signals and its quantitativeevaluation,” in Proceedings of Speech Prosody 2012, 2012, pp. 175–178.
[5] K. Yoshizato, H. Kameoka, D. Saito, and S. Sagayama, “Hidden markov convolutive mixture model for pitch contour analysis of speech,” in Proceedings ofthe 13th Annual Conference of the International Speech Communication Association (Interspeech 2012), 2012.
[6] Y. Ohishi, H. Kameoka, D. Mochihashi, K. Kashino, “A stochastic model of singing voice F0 contours for characterizing expressive dynamic components,” inProc. Interspeech 2012, 2012.