VAE-SPACE: DEEP GENERATIVE MODEL OF
VOICE FUNDAMENTAL FREQUENCY CONTOURS
Kou Tanaka1 , Hirokazu Kameoka1 , Kazuho Morikawa2
1

NTT Communication Science Laboratories, NTT Corporation, Japan,
2
Graduate School of informatics, Nagoya University, Japan{tanaka.ko, kameoka.hirokazu}@lab.ntt.co.jp, morikawa.kazuho@g.sp.m.is.nagoya-u.ac.jp

ABSTRACT
Modeling the speech generation process can provide flexible and interpretable ways to generate intended synthetic
speech. In this paper, we present a deep generative model offundamental frequency (F0 ) contours of normal speech andsinging voices. The generative model we propose in this paper 1) is able to accurately decompose an F0 contour into thesum of phrase and accent components of the Fujisaki model,a mathematical model describing the control mechanism of
vocal fold vibration, without an iterative algorithm, and 2)
can represent/generate F0 contours of both normal speechand singing voices reasonably well.
Index Terms— Deep generative model, voice F0 contour,
singing voice, variational autoencoder, gated convolutional
network
1. INTRODUCTION
The fundamental frequency (F0 ) contours in normal speech
contain linguistic and para/non-linguistic information. For
example, they are usually used to convert a regular phrase toa question. They also indicate intonation in pitch accent languages. Furthermore, they play the role of adding extra flavor
to speech such as the identity, intention, attitude, and mood of
the speaker to convey para/non-linguistic information to thelistener. In singing voice, they are used to express the melodyof the song and the singing style of the singer. If we can build
a physically or musically interpretable generative model, it
may provide flexible ways to synthesize expressive speech or
singing voices. This paper is concerned with developing agenerative model of voice F0 contours, which allows us togenerate natural sounding F0 contours conditioned on a contextual input such as a phrase/accent command sequence anda musical score.
Conventionally, several attempts have been made tomodel F0 contours of speaking and singing voices. One
well-known model is called the Fujisaki model [1], which
describes the control mechanism of vocal fold vibration ina physically interpretable way. This model assumes that theF0 contour on a logarithmic scale is the superposition of aphrase component, an accent component, and a base value.
The phrase and accent components are considered to be associated with mutually independent types of movement of the
thyroid cartilage with different degrees of freedom and muscular reaction times. Another example is F0 control modelsof singing voices [2, 3]. Similar to the Fujisaki model, these978-1-5386-4658-8/18/$31.00 ©2018 IEEE

models assume that a singing voice F0 contour is described
as a mixture of several types of F0 fluctuations such as overshoot, vibrato, and preparation. The Fujisaki model and thesinging voice F0 control models share in common that there
is a need to solve an inverse problem to obtain the underlying
parameters. Although the models reported in [2, 3, 4] haveprovided a tractable way of estimating the underlying parameters by using statistical inference techniques, shortcomingsof these models are that parameter estimation algorithms typically require many iterations, which can be computationallydemanding, and parameter estimation accuracy is limited dueto the inherent difficulties in the inverse problem.
Recently, several types of generative model described bya neural network have been proposed, such as a variational
autoencoder (VAE) [5, 6]. As the name implies, VAEs area stochastic counterpart of autoencoders, consisting of encoder and decoder networks. The encoder network generates
a set of parameters of the conditional distribution Q(z|x)
of a latent space variable z given an input data vector x
whereas the decoder network generates a set of parametersof the conditional distribution P (x|z) of the data vector x
given the latent space variable z. Given a training data set
X = {xn }N
n=1 , VAEs learn the entire network parameters sothat the encoder distribution Q(z|x) becomes consistent withthe posterior P (z|x) ∝ P (x|z)p(z). If we can associate thelatent space variables with a set of interpretable parameters
governing the data of interest, the decoder can be seen as a
generative model (like the Fujisaki model) that relates theunderlying parameters to observed data and the encoder canbe seen as an inverse problem solver. While the Fujisaki
model, for example, is a hand-crafted or manually designed
model, an interesting point of view would be that throughtraining of VAE, we would be able to discover the structureof a generating process model in a data-driven manner as wellas an inverse process of estimating the underlying parameters.
Furthermore, since VAEs provide a principled and convenient
way of handling semi-supervised learning tasks [6, 7], they
can be very useful especially when it takes a lot of timeand effort to collect a large amount of labeled data. For ourtask, while collecting a complete pair of F0 contours andthe underlying parameters can be a demanding process, we
can easily collect a large amount of unlabeled F0 contours.
Indeed, VAEs have been applied to various supervised/semisupervised tasks with notable success [8, 9, 10].
In this paper, we propose a generative model of F0 contours based on a VAE with a fully convolutional architecture.
In particular, we adopt a gated CNN architecture [11] to be

5779

ICASSP 2018

able to capture and reflect long- and short-term dependencies
in F0 contours. Experimental results showed that our proposed framework successfully achieved higher performance
than a conventional method in terms of the subjective pairwised comparison for singing voice quality, the generation error of the underlying parameters of the F0 contours in speaking voice, and processing time required to solve the inverse
problem.
2. F0 CONTOUR AND ITS UNDERLYINGPARAMETERS
Here, we briefly review conventional work on voice F0 contour modeling for speaking and singing voices.
2.1. Fujisaki model
The Fujisaki model [1] is one of well-known models describing the control mechanism of vocal fold vibration in a physically interpretable way. This model assumes that the F0
contour x[t] on a logarithmic scale is given as the sum of
three components x[t] = xp [t] + xa [t] + µb where xp [t] andxa [t] are a phrase component and an accent component attime frame t, and µb is a constant value, respectively. The
phrase and accent components are assumed to be the outputs of different second-order critically damped filters excitedwith Dirac deltas (phrase commands) and rectangular pulses(accent commands), respectively. These components respectively correspond to contributions associated with the translation and rotation movements of the thyroid cartilage. The
former usually contributes to phrasing, while the latter contributes to accentuation during an utterance. The magnitudesof these components correspond to how much emphasis the
speaker intends to place on the associated phrase or accent.
These parameters, which we call the Fujisaki model parameters, are thus physically and linguistically interpretable. If we
can estimate these parameters from raw F0 contours, we will
be able to flexibly control them as desired.
2.2. Singing voice F0 contour model
The F0 contour of a singing voice contains the melody contour of a song and an expression contour such as overshoot,preparation and vibrato. Compared with the F0 contours of
speaking voice, those of singing voice change more rapidly,
and their dynamic range is wider and so the Fujisaki modelcannot be directly applied to singing voices. In [3], a singing
voice version of the Fujisaki model is proposed.
3. VAE-SPACE: F0 CONTOUR REPRESENTATIONVIA DEEP GENERATIVE MODEL3.1. Concept
In [4], we proposed formulating a stochastic counterpart of
the Fujisaki model. The key idea of this model is that aphrase/accent command pair sequence, given by an impulse
train and a rectangular pulse train, is modeled as an output
sequence of a path-restricted hidden Markov model (HMM).
Similarly, [3] proposed introducing a stochastic counterpartof a singing voice version of the Fujisaki model. With thismodel, two sequences, one representing a melody contour andthe other representing an expression contour, are modeled aspiecewise constant functions generated by a path-restricted
HMM. These models have allowed us to utilize statisticalinference techniques to estimate the underlying parameters.
However, the parameter estimation algorithms must be run
for many iterations, which can be computationally demanding. Furthermore, parameter estimation accuracy is limited
due to the inherent difficulties in the ill-posed inverse problem. Another limitation of these models is the lack of flexibility needed to express a wide variety of voice F0 contours andneither of these models can be universally applied to all possible F0 contours. We may need to manually design different
models and algorithms according to languages, speakers andtypes of speech (e.g., singing voices and regular/emotional
speech) as long as we take a hand-engineering approach.
To overcome these limitations, we take a learning-based
approach. In particular, we focus on VAEs with a gated CNN
architecture for flexibly modeling voice F0 contours. As mentioned in Sec. 1, if we can associate the latent space variables
with a set of interpretable parameters (like the phrase/accent
components in the Fujisaki model) governing the data of interest, the decoder can be seen as a generative model (likethe Fujisaki model) that relates the underlying parameters to
observed data whereas the encoder can be seen as a parameter extractor. It would be interesting if we could automatically discover the structure of a generating process modelin a data-driven manner through training of VAE as well as
the parameter estimation process. Furthermore, since VAEs
provide a principled and convenient way of handling semisupervised learning tasks, they can be very useful especiallywhen it takes a lot of time and effort to collect a large amountof labeled data. For our task, even though collecting a complete pair of F0 contours and the underlying parameters canbe a painstaking process, we can easily collect a large amountof unlabeled F0 contours. These are the main reasons we havefocused on VAEs.
To realize the above-mentioned concept, an architecture
design is a key to success. Given the fact that both the Fujisaki model and the singing voice F0 contour model mentioned in Sec. 2 are described as a mixture of linear timeinvariant systems, we believe that convolutional architectures
can be a reasonable choice for our architecture design. In
particular, we focus on a convolutional architecture called thegated CNN. The gated CNN has recently been shown to be
powerful in modeling long-term sequential data. It was originally introduced to model word sequences for language modeling and was shown to outperform long short-term memory(LSTM) language models trained in a similar setting [11]. We
previously applied a gated CNN architecture for speech sequence modeling and its effectiveness has already been confirmed [12]. With a gated CNN, the output of a hidden layerof a network is described as a linear projection modulated byan output gate. Similar to an LSTM [13] and gated recurrent
unit (GRU) [14], the output gate controls what information
should be propagated through the hierarchy of layers and allows capturing long-term structures.
3.2. VAE-SPACELet us use z to denote a sequence of parameters governing thegenerating process of F0 contours. In the case of the Fujisaki

5780

model, this corresponds to a sequence of a phrase/accent component pair. Here, we consider a “decoder” network that generates the parameters of a conditional distribution Pθ (x|z) ofan F0 contour x. The posterior distribution Pθ (z|x) can beseen as an inverse process of generating z given x. Since obtaining the exact posterior is intractable, we introduce anothernetwork, i.e., “encoder”, that generates the parameters of aconditional distribution Qφ (z|x) and train both the decoderand encoder networks so that Qφ (z|x) becomes consistent
with the exact posterior Pθ (z|x) ∝ Pθ (x|z)P (z). We canshow that the log marginal distribution log Pθ (x) is given as

Table 1. Details of network architecture.
Input:
1ch. 1 × 1200
Encoder CNN 1 :4 ch. 1 × 200, GLU
Encoder CNN 2 :8 ch. 1 × 50,
GLU
Encoder CNN 3 :4 ch. 1 × 15,
GLU
Encoder CNN 4 :8 ch. 1 × 5,
GLU
Encoder CNN 5 : K ch. 1 × 50
Latent: K ch. 1 × 1200
Decoder CNN 1 : 16 ch. 1 × 10,
GLU
Decoder CNN 2 :1 ch. 1 × 10
Output:1 ch. 1 × 1200

log Pθ (x) = L(θ, φ; x) + DKL [Qφ (z|x)||Pθ (z|x)] ,(1)
L(θ, φ; x) = −DKL [Qφ (z|x)||P (z)] + EQφ (z|x) [log Pθ (x|z)] Table 2. Results of subjective evaluation for F0 contour sim|
{z
} |
{z
} ilarity (# of evaluated samples: 42, # of evaluators: 7).
Regularization term over z

Reconstruction term(2)
where DKL [·|·] denotes the Kullback-Leibler (KL) divergence. This implies we can minimize the KL divergence between Pθ (z|x) and Qφ (z|x) by maximizing L(θ, φ; x) withrespect to θ and φ. One typical way of modeling Qφ (z|x)
and Pθ (x|z) is to assume normal distributions. As for theprior distribution P (z), we can design its specific form according to the assumption we would like to make about z.
For example, if we associate z with a phrase/accent command
pair sequence, we can employ the path-restricted HMM with
Gaussian emission densities proposed in [4]. In this case,by using s to denote the P
state sequence of the HMM, P (z)is written as P (z) =s P (z|s)P (s). Since our VAE
is designed to perform statistical phrase/accent component
estimation (SPACE), we call it “VAE-SPACE”.
3.3. Sequential modeling with gated CNN
To capture long- and short-term dependencies in F0 contours, we use a gated CNN [11] to construct both the encoderand decoder networks of the VAE. Gated CNNs are CNNs
equipped with gated linear units (GLUs) as activation functions instead of regular rectified linear units (ReLUs) [15] orTanh activations. The output of the lth hidden layer of a gated
CNN is described as a linear projection H l−1 ∗ W l + bl
modulated by an output gate σ(H l−1 ∗ V l + cl )H l = (H l−1 ∗ W l + bl ) ⊗ σ(H l−1 ∗ V l + cl ),(3)

where W l , V l , bl and cl are the network parameters to
be trained, σ is the sigmoid function and ⊗ indicates theelement-wise product. Here, the input to the 1st layer is
H 0 = x for the encoder and H 0 = z for the decoder
whereas the output from the lth layer is H l = [µz ; log σ 2z ]for the encoder and H l = [µx ] for the decoder. Similar to LSTMs, the output gate multiplies each element of
H l−1 ∗ W l + bl and control what information should be
propagated through the hierarchy of layers in a data-driven
manner.
4. EXPERIMENTS
4.1. Experimental Conditions
Datasets: For the speaking voice F0 contours, we used the
ATR speech database [16], 429 sentences (around 0.5 hours)

VAE-SPACE
76.2 %Musical score or Fair
23.8 %

p-value
0.000312

uttered by one male speaker to train a deep generative model,
and the remaining 53 sentences (around 3 minutes) to evaluate
the performance. For singing voice F0 contours, we used real
singing data paired with the musical score, 42 songs (around
15 minutes) sung by 6 singers including female and male classical, pop, and amateur singers (namely, each singer recorded
7 songs). To evaluate the performance of singing voice F0
contour modeling, we adopted the leave one out cross validation strategy over singer dependent models.
F0 extraction: we adopted TEMPO [17] as an F0 analyzer.
Based on the label data, the frame shifts were set to 8 and5 ms for the speaking and singing voices, respectively. Notethat we carefully checked the actual extracted F0 contours andexcluded data that have failed to analysis. The final numberof data after excluding was described in Datasets.
Model architecture: The model setting of the conventional
stochastic model of F0 generative process (SPACE) was thesame as reported in [4]. Table 1 details the network architectures of our proposed model (VAE-SPACE) for speaking
voice F0 contour. The stride of each convolution was set to
1. For speaking voice, we set the number of channels overlatent space, K, to 4 indicating the mean and variance values of the phrase and accent components. We canceled thebaseline value 60 Hz of the speaking voice F0 contour, in advance. For singing voice, we set K to 2 indicating the mean
and variance values of the musical score. We normalized thesinging voice F0 contour, its musical score, and its backward
difference to zero-mean and unit-variance using their training
sets, respectively. We optimized the model parameters using
the Adam optimizer [18] with a mini-batch of size 32. The
learning parameters α, β1 , β2 were set to 0.0001, 0.9, and0.99, respectively.
4.2. Experimental Results
4.2.1. F0 Contour Similarity Over Singing Voice
We subjectively evaluate singing voice synthesized with two
types of F0 contours. One is F0 contour corresponding to themusic score, and another is F0 contour generated by using theonly decoder part our proposed framework given the musicalscore after training both of the encoder and decoder parts of
the VAE.

5781

Frequency
Frequency[Hz][Hz] Magnitude Magnitude
Magnitude Magnitude

Frequency [Hz]Frequency [Hz]Frequency [Hz]

600
500
400
300

MIDI
0

1

2

3

4

5
6
Time [sec]300

Singing
7

Proposed

8

9

10

11

0

1

2

3

4

5

6

7

300

8

9 10 11 12 13 14 15 16 17 18 19Time [sec]

200
100

0

1

2

3

4

5
6
Time [sec]

7

8

9

10

SPACE
0.08830.123
0.122

F0 contour
Phrase component
Accent component

0.01690.03220.03290.07900.131
0.110

Proposed

50

1.0 0

d) accent component 1

0.5
0.0
1.0

2
Time [sec]3e) phrase component

0.5f) F0 contour

100

Table 3. Generation errors of F0 contour and its underlying
parameters for real speaking data (top) and ”ideal” condition(bottom; # of evaluated samples: 53).
VAE-SPACE
0.0536
0.0947
0.0936

Conventional

0.0
1.0 b) phrase component
0.5

0.0
200

11

Fig. 1. Samples of F0 contour and MIDI note for real singing
data sung by female classical (top), male pop (middle), andmale amateur (bottom) singers.

F0 contour
Phrase component
Accent component

Speaking

0.0
200 b) F0 contour
100

200
100

1.0 a) accent component
0.5

50

0

1

4.2.2. Generation Error Over Speaking Voice
Calculating root mean square error (RMSE) between the reference and the generated one, we objectively evaluate the performance of our proposed model, VAE-SPACE. Note that
not only actual F0 contours observed in real speaking data
but also F0 contours reconstructed by the Fujisaki model parameters are used as the reference. Using F0 contours reconstructed by the Fujisaki model parameters means the ”ideal”
condition.
Table 3 shows the performances in the case of training
the model by using only actual F0 contours observed in real
speaking data and the ”ideal” condition. Both of the results
show that our proposed model successfully achieved higher
performance compared with the conventional method. The
major factors of getting high performance is the constraint of

3

Fig. 2. Samples of F0 contour and phrase and accent components for real speaking data (a ∼ c) and ”ideal” condition (d∼ f).
Table 4. Processing time to solve inverse mapping from F0
contour to its underlying parameters (Unit: [sec]).
53 sentences
average

Table 2 shows that our proposed model achieved to generate F0 contours which are similar to those of real singing
data. Note that the breakdown of ”Musical score or Fair” isthat ”Musical score” and ”Fair” are 7.14 % and 16.7 %, respectively. Considering the comments of evaluators, it seems
that they have felt the fluctuations even if F0 contours aremonotonic, namely the musical score. One possible reason isthat the singing style causes the fluctuations of acoustic features including not only F0 contours but also spectral featuresand power information of the waveform. As shown in samples (Fig. 1), we confirmed that our proposed model made it
possible to generate F0 contours with the fluctuations, suchas vibrato and overshoot.

2Time [sec]VAE-SPACE
0.0126 ± 0.0002—

SPACE
2712.080
5.67

VAE that is the training of the parametric encoder in combination with the generator network. As shown in Fig. 2, theF0 contours and their underlying parameters generated by our
proposed model are more closer to the reference comparedwith those generated by the conventional method. In particular the ”ideal” condition, the underlying parameters estimatedby our proposed framework are truly close to the references.
4.2.3. Processing Time to Solve Inverse Mapping
To demonstrate the use of CNN architecture, we measure theprocessing time to estimate the underlying parameters of F0
contours given actual obtained F0 contours. Although our
proposed model VAE-SPACE enables to work on a GPU, theconventional model SPACE works on only a CPU. The CPU
and GPU are ”Intel R Xeon R Processor E5-2699 v3” and”NVIDIA Corporation GK210GL [Tesla K80] (rev a1)”, respectively.
Table 4 shows that our proposed model makes it possible
to work in real time.
5. CONCLUSIONSIn this paper, we have presented a unified approach to modelboth of speaking and singing voice F0 contours. The keyrole of our approach is a learning-based mapping to realize
complex mapping, which is really difficult to fully elucidate,
between F0 contours and their underlying parameters. Experimental results revealed that the presented approach significantly outperforms the conventional stochastic model of F0
generative process.
Acknowledgements: This work was supported by JSPS
KAKENHI 17H01763.

5782

6. REFERENCES[1] Hiroya Fujisaki, “A note on the physiological and physical basis for the phrase and accent components in thevoice fundamental frequency contour,” Vocal physiology: Voice production, mechanisms and functions, pp.
347–355, 1988. 1, 2
[2] Siu Wa Lee, Shen Ting Ang, Minghui Dong, andHaizhou Li, “Generalized f0 modelling with absoluteand relative pitch features for singing voice synthesis,”in Acoustics, Speech and Signal Processing (ICASSP),
2012 IEEE International Conference on. IEEE, 2012,
pp. 429–432. 1[3] Yasunori Ohishi, Hirokazu Kameoka, Daichi Mochihashi, and Kunio Kashino, “A stochastic model ofsinging voice f0 contours for characterizing expressive
dynamic components,” in Thirteenth Annual Conference of the International Speech Communication Association, 2012. 1, 2
[4] Hirokazu Kameoka, Kota Yoshizato, Tatsuma Ishihara,
Kento Kadowaki, Yasunori Ohishi, and Kunio Kashino,“Generative modeling of voice fundamental frequency
contours,” IEEE/ACM Transactions on Audio, Speech,
and Language Processing, vol. 23, no. 6, pp. 1042–
1053, 2015. 1, 2, 3
[5] Diederik P Kingma and Max Welling, “Auto-encoding
variational bayes,” arXiv preprint arXiv:1312.6114,
2013. 1
[6] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe,
Søren Kaae Sønderby, and Ole Winther, “Ladder variational autoencoders,” in Advances in Neural Information
Processing Systems, 2016, pp. 3738–3746. 1[12] Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, and Kunio Kashino, “Sequence-to-sequence
voice conversion with similarity metric learned using
generative adversarial networks,” Proc. Interspeech
2017, pp. 1283–1287, 2017. 2[13] Sepp Hochreiter and Jürgen Schmidhuber, “Long shortterm memory,” Neural computation, vol. 9, no. 8, pp.
1735–1780, 1997. 2[14] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger
Schwenk, and Yoshua Bengio, “Learning phrase representations using rnn encoder-decoder for statisticalmachine translation,” arXiv preprint arXiv:1406.1078,
2014. 2[15] Vinod Nair and Geoffrey E Hinton, “Rectified linear
units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814. 3[16] Akira Kurematsu, Kazuya Takeda, Yoshinori Sagisaka, Shigeru Katagiri, Hisao Kuwabara, and Kiyohiro
Shikano, “Atr japanese speech database as a tool ofspeech recognition and synthesis,” Speech Communication, vol. 9, no. 4, pp. 357–363, 1990. 3[17] Hideki Kawahara, Ikuyo Masuda-Katsuse, and Alain
De Cheveigne, “Restructuring speech representations
using a pitch-adaptive time–frequency smoothing and
an instantaneous-frequency-based f0 extraction: Possible role of a repetitive structure in sounds,” Speech communication, vol. 27, no. 3, pp. 187–207, 1999. 3[18] Diederik Kingma and Jimmy Ba,
method for stochastic optimization,”
arXiv:1412.6980, 2014. 3[7] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez
Rezende, and Max Welling, “Semi-supervised learningwith deep generative models,” in Advances in Neural
Information Processing Systems, 2014, pp. 3581–3589.
1[8] Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin, “Variational autoencoder for deep learning of images, labels
and captions,” in Advances in Neural Information Processing Systems, 2016, pp. 2352–2360. 1[9] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio,“Generating sentences from a continuous space,” arXiv
preprint arXiv:1511.06349, 2015. 1[10] Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt,
Oriol Vinyals, Alex Graves, et al., “Conditional image generation with pixelcnn decoders,” in Advancesin Neural Information Processing Systems, 2016, pp.
4790–4798. 1[11] Yann N Dauphin, Angela Fan, Michael Auli, and David
Grangier, “Language modeling with gated convolutional networks,” arXiv preprint arXiv:1612.08083,
2016. 1, 2, 35783

“Adam: A
arXiv preprint