GENERATIVE MOMENT MATCHING NETWORK-BASED RANDOM MODULATION
POST-FILTER FOR DNN-BASED SINGING VOICE SYNTHESIS AND NEURALDOUBLE-TRACKING
Hiroki Tamaru1 , Yuki Saito1 , Shinnosuke Takamichi1 , Tomoki Koriyama2 , and Hiroshi Saruwatari1
1

Graduate School of Information Science and Technology, The University of Tokyo, Japan.
2
School of Engineering, Tokyo Institute of Technology, Japan.
ABSTRACTThis paper proposes a generative moment matching network(GMMN)-based post-filter that provides inter-utterance pitch variation for deep neural network (DNN)-based singing voice synthesis.
The natural pitch variation of a human singing voice leads to aricher musical experience and is used in double-tracking, a recording method in which two performances of the same phrase arerecorded and mixed to create a richer, layered sound. However,singing voices synthesized using conventional DNN-based methods never vary because the synthesis process is deterministic andonly one waveform is synthesized from one musical score. To address this problem, we use a GMMN to model the variation of themodulation spectrum of the pitch contour of natural singing voices
and add a randomized inter-utterance variation to the pitch contour
generated by conventional DNN-based singing voice synthesis. Experimental evaluations suggest that 1) our approach can provide
perceptible inter-utterance pitch variation while preserving speech
quality. We extend our approach to double-tracking, and the evaluation demonstrates that 2) GMMN-based neural double-tracking
is perceptually closer to natural double-tracking than conventionalsignal processing-based artificial double-tracking is.

Human singing voice

Musical
score

Human
singerSynthesized singing voice

Variouswaveforms

Musicalscore

SynthesisOnewaveform

Fig. 1. Comparison of human and synthesized singing voices.
Double-tracking
(DT)

Sing
twice
Mix

Naturally layered voice

Artificial Doubletracking (ADT)Neural doubletracking (proposed)

Synthesize
once

Synthesize
once

Signalprocessing
Mix

GMMN
postfilter
Mix

Artifacts in voice
due to unnatural
signal processing

Naturally layered voice
due to modeling ofinter-utterance variation

Index Terms— DNN-based singing voice synthesis, moment
matching network, inter-utterance pitch variation, artificial doubletracking, modulation spectrum

Fig. 2. Double-tracking (DT), artificial double-tracking (ADT), andproposed neural double-tracking. This figure shows ADT performedon synthesized voice, but it can also be performed on natural voice.

1. INTRODUCTION

but also the ability to take advantage of double-tracking (DT) [6, 7](Fig. 2). DT involves two or more vocal performances uttered by one
human singer that are combined in a mix. It gives layeredness andrichness to the resulting voices, thanks to the singer’s inter-utterance
variation. An alternative way is signal processing-based artificial
double-tracking (ADT). Instead of recording multiple performances,
ADT modulates one voice (e.g., through a delay and chorus effect)
and mixes the original and modulated voices. Since ADT does not
require multiple voices with inter-utterance variations, it can be easily applied to not only human voices but also synthesized voices.
However, the effects of ADT are known to result in unnatural soundartifacts, such as tonal alterations and timbre coloration [8].
To address these problems, we propose a post-filter that gives
inter-utterance pitch variations to a synthesized singing voice. Sincesuch variations can be assumed to follow a certain complicated distribution, we use deep generative models, which are known to becapable of modeling complicated distributions. Building on our previous work on spectrum generation in text-to-speech synthesis [9],we utilize a generative moment matching network (GMMN) [10, 11]
as the deep generative model because it is effective and easy to implement compared with other models. For example, generative ad-

These days, synthesized singing voices are being used for creatingmusic. In particular, there are a variety of singing voice synthesis systems that work on the basis of unit selection synthesis (e.g.,
Vocaloid [1]), hidden Markov models (HMMs) [2, 3], and deep neural networks (DNNs) [4, 5]. One of the aims of these systems is tocreate expressive singing voices and music regardless of the users’
gender or skill. Among them, DNN-based ones utilize a machinelearning-based synthesis process and have the potential to synthesize
high-quality expressive voices.
However, the conventional DNN-based singing voice synthesis
lacks inter-utterance variation, as shown in Fig. 1. Given one musical score, a human singer will sing it differently when asked to repeat
it. The inter-utterance variation contributes to a richer musical experience. For instance, it proves that a singer is actually singing, notlip-synching. It also provides a music producer with the opportunity
to choose a favorite from various recordings of the same song. In
contrast, only one voice is synthesized from one musical score inconventional DNN-based singing voice synthesis because the synthesis process is deterministic. The lack of inter-utterance variation loses not only the rich musical experiences mentioned above,versarial networks (GANs) [12] involve a difficult minimax problem
and variational auto-encoders [13] are subject to the degration of
the decoding quality due to over-regularization [14]. In our method,
given the synthesized pitch contour and a prior noise vector, the conditional GMMN is trained to represent the distribution of natural
pitch contours (i.e., the natural variation of the human singer’s pitch
contours). To capture the long-term pitch structure, the modulation
spectrum (MS) [15] of the pitch contours is used, and the conditional GMMN models the natural MS variation. In the synthesis,
given a prior noise vector, the GMMN randomly generates MSs thathave natural inter-utterance variations, and the filtered pitch contouris created by modulating the input contour by the randomly generated MSs. In this study, we extend this framework to ADT (i.e.,
NDT: neural double-tracking). Since the GMMN-based post-filter
provides natural inter-utterance variation, our NDT achieves naturally layered singing voices. The experimental evaluation demonstrates that our post-filtering approach can provide perceptible interutterance pitch variations while preserving speech quality and thatthe sound of our NDT is perceptually closer to that of natural DT
than conventional signal processing-based ADT is.
2. CONVENTIONAL METHODS
2.1. DNN-based singing voice synthesis
In DNN-based singing voice synthesis [4], the relation between amusical score and the speech parameters of the parallel singing voice
is modeled with DNNs. First, a musical score is converted into a sequence of vectors representing linguistic and musical contexts. Using DNNs, the parameters of the singing voices are predicted fromthe context sequence. Here, we minimize the mean square error(MSE) [4] between the natural and predicted speech parameters as
follows:
LMSE (y, ŷ) = ||y − ŷ||2 ,(1)
where y and ŷ are natural and predicted speech parameter sequences lasting T frames, respectively. Since we are focusing on
a 1-dimensional continuous fundamental frequency (F0 ) [16], we
define y as a scalar value sequence [y(1), · · · , y(t), · · · , y(T )]> ,where y(t) is the continuous log-scaled F0 value at frame t and >is the transpose. In the synthesis, ŷ = [ŷ(1), . . . , ŷ(t), . . . , ŷ(T )]>is used as the speech parameter of the synthesized singing voice.
Since this synthesis process is deterministic, the resulting sound has
no inter-utterance variation.
2.2. ADT
Fig. 2 shows the difference between DT and ADT. DT is a recording method in which multiple performances of the same phrase are
mixed in order to create a layered and rich sound [6, 7]. However,singing the same phrase twice in a similar fashion may be difficult
and tiresome. ADT, which is an alternative method and only requires one recording, was originally achieved by taking a vocal signal from the sync head of a multi-track, recording it to another loopof tape which was speed varied with a slow oscillation and recording
it back onto the multi-track [6]. More recently, signal processingbased methods have been used; the most common one uses a chorus effect [8]. In the chorus effect, a waveform is copied and its
pitch is modulated with a low-frequency oscillator; i.e., a sine wave
is added to the original pitch contour, and the modulated sound ismixed with the original sound with some temporal difference to increase the doubled-voice feeling. Although such methods can beapplied to DNN-based singing voice systems, they involve adding
two signals with similar phases, which results in comb-filtering andits subsequent tonal alterations and timbre coloration [8].

Generated
pitch contour

Naturalpitch contour
MS

Phase

STFT

STFT

MS
Conditional
MMD
training
Segment-wise noise
Filtered
ISTFT
pitch contour
MS

Fig. 3. Schematic diagram of our post-filter.
3. PROPOSED GMMN-BASED POST-FILTER AND ITSAPPLICATION TO NDTHere, we describe the proposed GMMN-based post-filter and how
NDT is performed on MSE-based singing voices. First, the MSs of
the log-scaled continuous F0 of natural and generated pitch contours
are extracted in order to capture the temporal structure of the contours. Second, A GMMN is trained to sample the MS randomly sothat naturally varied pitch contours can be produced. Finally, NDT
is performed using the non-filtered and filtered voices.
3.1. MS extraction
The MS is defined as the log-scaled power spectrum of a speechparameter sequence [15]. Namely, it captures temporal structures
through the Fourier transform. We calculate the MS S y of y througha short-time Fourier transform (STFT), as follows:
S y = [sy (1), · · · , sy (τ ), · · · , sy (T 0 )],(2)sy (τ ) = [sy (τ, 0), · · · , sy (τ, m), · · · , sy (τ, M )]> ,(3)
where τ is the segment index (one segment corresponds to one windowed continuous F0 contour) and m is the modulation frequency
index. sy (τ, m) indicates the MS of the modulation frequency mat segment τ . T 0 and M correspond to the total number of segments and one-half the number of segments of the STFT, respectively. S ŷ , the MS of ŷ, is calculated in the same manner. Here,the zero-mean continuous F0 sequence [15] is used to deal witherrors caused by zero padding. We reconstruct the continuous F0
sequence from the inverse STFT using the MS and original phase
information and use the analysis settings (e.g., windowing length) toachieve a perfect reconstruction from the STFT and inverse STFT.
We use only the lower modulation frequency components (i.e., components corresponding to slowly changing temporal structures) for
post-filtering, because post-filtering the higher ones causes unnatural temporal fluctuations in the F0 contours.
3.2. GMMN-based post-filtering
Here, we describe the GMMN-based post-filtering method to randomly modulate the generated continuous F0 sequence ŷ. Fig. 3
shows the schematic diagram of the post-filter. The GMMN [10] is a
deep generative model, and it enables stabler training compared witha GAN [12]. The training criterion is called maximum mean discrepancy (MMD), which is a moment-based discrepancy between two
distributions. The DNN takes a prior noise vector as input; this vector is the source of the inter-utterance variation in the post-filtering
stage. In the training stage of the post-filter, given the MS of the generated continuous F0 and segment-wise prior noise, the conditional
distribution of the MS of natural continuous F0 is modeled with the

− 2 · tr(LS ŷ · K S y ,Ŝ ŷ )},
−1

0

−1

(4)

LS ŷ = H̃ S ŷ H S ŷ H̃ S ŷ ,(5)H̃ S ŷ = H S ŷ + λI T 0 ,(6)

Pit ch (MIDI not e num ber)

DNN. Let n(τ ) ∼ U [−1, 1) be the prior noise vector at segment
τ , and G(·) be a DNN for post-filtering. The input of the DNN atsegment τ is the joint vector [sŷ (τ )> , n(τ )> ]> , and the output isthe filtered MS ŝŷ (τ ), i.e., ŝŷ (τ ) = G([sŷ (τ )> , n(τ )> ]> ). LetŜ ŷ = [ŝŷ (1), · · · , ŝŷ (τ ), · · · , ŝŷ (T 0 )]; the following conditional
MMD (CMMD) [11] is minimized in training:
1
LCMMD (S ŷ , S y , Ŝ ŷ ) = 02 {tr(LS ŷ · K S y ,S y )T+ tr(LS ŷ · K Ŝ ŷ ,Ŝ ŷ )

72
71
70
69
68
67
66
65
64
63
620.0
0.0

0.5

1.0

1.5

2.0

2.5

Tim e [ s]Fig. 4. Example of generated pitch contours. We used proposed
method to sample four contours. Value of unity on vertical axis isequal to semitone.

0

where I T 0 is the T -by-T identity matrix and λ is a regularization
coefficient. K Ŝ ŷ ,S y is the T 0 -by-T 0 Gram matrix between Ŝ ŷ andS y ; i.e., its i, jth component is k(ŝŷ (i), sy (j)), where k(·) is an arbitrary kernel function between two vectors. Similarly, H S ŷ is theGram matrix for S ŷ ; i.e., its i, jth component is h(sŷ (i), sŷ (j)),
where h(·) is an arbitrary kernel function between two vectors. Notethat we can choose different kernel functions between k(·) and h(·).
After training, the model represents the natural MS distribution (i.e.,variation of the pitch contours) given the generated MS. The synthesis stage first generates ŷ from DNN-based singing voice synthesis
and calculates its MS and phase. It then filters the MS, for it to havenatural variation, by using the trained GMMN and randomly sampled prior noise. The final F0 contour is generated by performing
the inverse STFT on the filtered MS and the non-filtered phase.
3.3. Application to NDTHere, we describe how to give natural layeredness to a synthesized
singing voice. After the speech parameter generation, one waveform
is synthesized in the standard vocoding process. Another waveform
is synthesized using F0 values modulated by our post-filter. The final
double-tracked voice is obtained by mixing the modulated waveformwith the non-filtered one with some temporal difference.
3.4. Discussion
In [9], we built a GMMN-based spectrum generator by using framewise noise vectors in text-to-speech synthesis. However, the method
had two problems: 1) such frame-wise noise and variation modeling caused unpleasant sounds in the case of F0 contours and 2) the
GMMN conditioned with linguistic features could not provide perceptible variations because of the sparseness of the linguistic features. This paper’s method effectively solves these problems. This isbecause the MS of F0 contours is a lower-dimensional and effective
representation with which to capture segment-wise temporal structure, and filtering the lower modulation frequency components canmodulate the F0 contour and at the same time preserve the continuityof the contour. The other reason is that the use of GMMNs as a postfilter can avoid the sparseness problem and can provide perceptible
inter-utterance variations, as shown in Section 4.2.
Since our method considers the distribution of natural MSs, thevariation of the post-filtered voices should be in the natural range.
Fig. 4 shows MSE-based (i.e., deterministic and non-filtered) andpost-filtered (i.e., randomly modulated) pitch contours. We can seethat our method samples random, but continuous pitch contours.
This is the first study to 1) provide natural inter-utterance pitch variations by using GMMNs to model MSs and to 2) introduce such asystem as a post-filter for singing voices. As described in Section
1, inter-utterance variation enables one to pick a favorite from various voices. Since the pitch contour of each segment can be saved by

fixing the input noise, users can choose their favorite contour phraseby-phrase and concatenate phrases to make the whole song.
Conventional ADT involves a waveform made by modulating
the original sound deterministically without considering the natural
distribution of pitch variations. NDT is a new approach in that thesound is modulated considering the natural pitch variations and thatit results in more doubled-voice feeling as shown in Section 4.4.
Data augmentation is a powerful method for improving the accuracy of DNN modeling. In this study, we utilized data augmentation in relation to the STFT. Since the MS utilizes an STFT, the valuechanges significantly with the segment position (i.e., the frame index
of the beginning of the segmentation). To cover such perturbations,
we added all possible offsets to the first frame of the STFT analysis.
This method effectively augments the training data of our post-filter
and improves its training accuracy.
Since our method utilizes pitch contours as the input and theoutput of the neural network, we expect that it can be extended to
NDT for human singers, i.e., where a human singer sings only once
and the extended NDT synthesizes a naturally layered voice. We will
pursue this idea in the future (see Section 5).
4. EXPERIMENTAL EVALUATION
4.1. Experimental conditions
We used Japanese singing voice data of 31 songs from a singingvoice-synthesis demo of HTS [17], 26 songs from the JSUT-songcorpus [18], and nine in-house songs sung by the same singer as in
the JSUT-song corpus. From these corpora, we used 58 songs fortraining the DNN for singing voice synthesis, 28 songs of the HTS
demo for training our post-filter, and three songs of the HTS demothat were not included in the training data for the evaluation. Labeldata were augmented 3-fold by transposing the songs up and down
a semitone [5]. The speech signals were sampled at a rate of 16
kHz. The WORLD analysis-synthesis system [19] was used to extract the speech features and to synthesize the waveforms. The frame
shift was 5 ms. The DNN used to predict the MSE-based speech parameters was a feed-forward network including a 705-dimensional
input layer, 3 × 256-unit gated linear unit (GLU) [20] hidden layers,
and a 127-unit linear output layer. The learning rate was 0.005, andthe batch size was 500. The learning process was repeated for 50
epochs by using AdaGrad [21]. The 705-dimensional input features
included 688-dimensional linguistic and musical features, a one-hotsong code and a one-hot singer code [22]. The DNN predicted a 127dimensional vector consisting of 40-dimensional mel-cepstral coefficients, log-scaled continuous F0 , band-aperiodicity [23, 24], dynamic (delta- and delta-delta-) features [25] of those 42-dimensionalparameters, and a binary unvoiced/voiced label. The DNN used

Table 1. Answer rate of perceived inter-utterance difference
Proposed
0.276

MSE
0.176

p-value
7.45×10−3in the conditional GMMN was a feed-forward network including
an 11-dimensional input layer, 3 × 128-unit GLU hidden layers,
and input-to-output residual net [26]. The learning process was repeated ten times by using AdaGrad. The learning rate was 0.005, andthe batch size was 13000. We used an approximation using 1024dimensional random Fourier features [27] to calculate LS ŷ in Eq.
(5) because calculating the inverse matrix was computationally infeasible. The 11-dimensional input vector consisted of the first-order
MS (m = 1) of the singing voice synthesized based on MSE and aten-dimensional noise vector generated from a uniform distribution
U [−1, 1). For stable training, noise vectors were first generated foreach segment and were then fixed during training. The regularization coefficient λ was set to 0.01. The kernel function was Gaussian,i.e., exp{−||sy (i) − ŝŷ (j)||2 /σ 2 }. A Gaussian kernel was used forthe input features as well. The σ for the input features was 100.0,
and the σ for the output features was 1.0; these values were empirically chosen. The natural MS was normalized so that all data fellwithin the range [0.01, 0.99]. For the STFT, a 96-frame (480 ms)
Hanning window and 48-frame (240 ms) segment shift were used.
To clarify the effect of pitch modulation, we used spectral parameters, band-aperiodicity, and the unvoiced/voiced label of the natural
singing voices in the vocoding process.
We conducted three subjective evaluations for determining 1)
whether our post-filter provided perceptible inter-utterance variations, 2) whether the post-filter degraded the naturalness of thesynthesized voices, and 3) whether the sound of NDT was perceptually closer to that of natural DT than ADT was. The evaluations
were performed using the Lancers crowdsourcing platform [28]. To
make it easier for listeners to judge, we manually split the songs
into segments in correspondence with three conditions: short (onephrase), middle, and long (several phrases). The average lengths forthe three conditions were 3.01 s, 4.88 s, and 10.24 s, respectively.
4.2. Perception of inter-utterance pitch variation
To determine whether the inter-utterance pitch variation could beperceived by human listeners, we asked 25 listeners whether they felt
there was a difference between a pair of singing voices. The shortduration voices were used because it is easier to remember subtle
differences if shorter sounds are presented. Each participant listened
to 20 pairs of singing voices consisting of ten pairs of randomly
post-filtered voices (proposed) and ten pairs of identical MSE-based
voices. We used Welch’s t test to calculate the p-value.
Table 1 shows the results. The perception rate of MSE was
17.6% despite that the same voices were presented. On the other
hand, the rate of our method was statistically significantly higher
than that of MSE. This suggests that our method is capable of producing perceptible inter-utterance variations.
4.3. Naturalness of post-filtered voice
We tried to determine whether our post-filter degraded the quality ofthe synthesized voices. We asked 25 participants to listen to ten pairsof post-filtered and MSE-based voices and to choose the more natural one. The middle- and long-duration voices were used to make iteasy to judge the overall naturalness.
Table 2 shows the results. There was no statistically significant
difference for either the middle-duration voices or the long ones.
This implies that the post-filter did not degrade the naturalness ofthe singing voices.

Table 2. Preference scores of singing voice naturalness and theirp-values for the middle and long conditions
Length condition
MiddleLong

Proposed
0.504
0.480

MSE
0.496
0.520

p-value
8.58 × 10−1
3.72 × 10−1

Table 3. Preference scores of double-trackedness and their p-values
for the middle and long conditions
Length condition
Middle
Long

NDT
0.724
0.736

ADT
0.276
0.264

p-value< 10−10< 10−104.4. Evaluation of NDT
We evaluated the double-trackedness (i.e., the perceptual similarity
to naturally double-tracked sound) of the proposed NDT and conventional ADT:NDT: Modulating the log-scaled F0 sequence using our post-filter
and mixing its vocoded speech waveform with the MSEbased waveform.
ADT: Modulating the log-scaled F0 sequence by using a lowfrequency oscillator and mixing its vocoded speech waveform with the MSE-based waveform. The shape, rate, anddepth of the oscillation were a sine wave, 0.775 Hz, and 10%of a semitone, respectively. The parameters were determinedby referring to [8]. The modulation was performed in the
vocoder parameter domain, not in the waveform domain,
because doing so is considered to produce fewer artifacts.
In both methods, the modulated wave was delayed by 20 ms and thevolume was reduced by 3 dB to produce the usual ADT setting [8].
We asked 25 participants to listen to ten pairs of sounds generated
under the two conditions above and to choose the one that sounded
more like a naturally double-tracked sound. As before, the middleand long-duration voices were used.
Table 3 shows the results. The score of our NDT was significantly higher than that of conventional ADT for both the middle
and long conditions. This suggests that our deep-generative method
gives a more double-tracked feeling than the conventional signal
processing-based approach does.
5. CONCLUSION
This paper described a GMMN-based post-filter for generating interutterance pitch variation and its application to ADT. Although ahuman singer never sings the same song in the same way twice,
conventional singing voice synthesis systems only generate a single waveform that is considered to be the most natural one. Our
post-filter models inter-utterance pitch variation by using a GMMN
to match the statistical moments of the MS of synthesized voices
and those of natural voices. Experimental results suggest that the
post-filter can generate pitch variations that are perceptible by human listeners without degrading the naturalness of the synthesized
singing voices. We also discussed the effectiveness of applying thepost-filter to ADT. Experimental results suggest that the proposed
NDT is perceptually closer to natural DT than conventional ADT is.
Future work will include modeling the inter-utterance variationof the duration and spectral parameters and combining them withthat of pitch in order to reproduce more natural variation. Anothertopic is formulating a post-filter that can be used to modulate the MSof natural singing voices.
Acknowledgements: Part of this work was supported by
SECOM Science and Technology Foundation.

6. REFERENCES[1] H. Kenmochi and H. Ohshita, “Vocaloid - commercial singing
synthesizer based on sample concatenation,” in Proc. INTERSPEECH, Antwerp, Belgium, Aug. 2007, pp. 4011–4012.
[2] K. Saino, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda, “AnHMM-based singing voice synthesis system,” in Proc. ICSLP,
Pittsburgh, U.S.A., Sep. 2006, pp. 2274–2277.
[3] K. Oura, A. Mase, T. Yamada, S. Muto, Y. Nankaku, andK. Tokuda, “Recent development of the HMM-based singing
voice synthesis system - Sinsy,” in Proc. SSW7, Kyoto, Japan,
Sep. 2010, pp. 211–216.
[4] M. Nishimura, K. Hashimoto, K. Oura, Y. Nankaku, andK. Tokuda, “Singing voice synthesis based on deep neural networks,” in Proc. INTERSPEECH, San Francisco, U.S.A., Sep.
2016, pp. 2478–2482.
[5] M. Blaauw and J. Bonada, “A neural parametric singing synthesizer modeling timbre and expression from natural songs,”
Applied Sciences, vol. 7, no. 12, Dec. 2017.
[6] R. Brice, Music Engineering, Elsevier Science, Oct. 2001.
[7] K. Womack, The Beatles Encyclopedia: Everything Fab Four,
ABC-CLIO, Jun. 2014.
[8] R. Izhaki, Mixing Audio: Concepts, Practices, and Tools, Taylor & Francis, Oct. 2017.
[9] S. Takamichi, K. Tomoki, and H. Saruwatari, “Samplingbased speech parameter generation using moment-matching
network,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug.
2017, pp. 3961–3965.

[19] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoderbased high-quality speech synthesis system for real-time applications,” IEICE transactions on information and systems, vol.
E99-D, no. 7, pp. 1877–1884, Jul. 2016.
[20] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” arXiv,
vol. abs/1612.08083, 2016.
[21] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” EURASIP
Journal on Applied Signal Processing, vol. 12, pp. 2121–2159,
Jul. 2011.
[22] N. Hojo, Y. Ijima, and H. Mizuno, “An investigation of DNNbased speech synthesis using speaker codes,” in Proc. INTERSPEECH, San Francisco, U.S.A., Sep. 2016, pp. 2278–2282.
[23] H. Kawahara, Jo Estill, and O. Fujimura, “Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system STRAIGHT,” in MAVEBA, Firentze,
Italy, Sep. 2001, pp. 1–6.
[24] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation,”in Proc. INTERSPEECH, Pittsburgh,
U.S.A., Sep. 2006, pp. 2266–2269.
[25] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMMbased speech synthesis,” in Proc. ICASSP, Istanbul, Turkey,
Jun. 2000, pp. 1315–1318.

[10] Y. Li, K. Swersky, and R. Zemel, “Generative moment matching networks,” in Proc. ICML, Lille, France, Jul. 2015, pp.
1718–1727.

[26] T. Kaneko, S. Takaki, H. Kameoka, and J. Yamagishi, “Generative adversarial network-based postfilter for STFT spectrograms,” in Proc. INTERSPEECH, Stockholm, Sweden, Aug.
2017, pp. 3389–3393.

[11] Y. Ren, J. Li, Y. Luo, and J. Zhu, “Conditional generative
moment-matching networks,” in Proc. NIPS, Barcelona, Spain,
Dec. 2016, pp. 2928–2936.

[27] A. Rahimi and B. Recht, “Random features for large-scale
kernel machines,” in Proc. NIPS, Vancouver, Canada, Dec.
2008, pp. 1177–1184.

[12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, Montreal, Canada, Dec. 2014,
pp. 2672–2680.

[28] “Lancers,” https://www.lancers.jp/.

[13] D. P. Kingma and M. Welling, “Auto-encoding variational
bayes,” arXiv, vol. abs/1312.6114, 2013.
[14] Y. Saito, Y. Ijima, K. Nishida, and S. Takamichi, “Non-parallelvoice conversion using variational autoencoders conditionedby phonetic posteriorgrams and d-vectors,” in Proc. ICASSP,
Calgary, Canada, Apr. 2018, pp. 5274–5278.
[15] S. Takamichi, T. Toda, A. W. Black, G. Neubig, S. Sakti, and
S. Nakamura, “Postfilters to modify the modulation spectrum
for statistical parametric speech synthesis,” IEEE Transactions
on Audio, Speech, and Language Processing, vol. 24, no. 4, pp.
755–767, Apr. 2016.
[16] K. Yu and S. Young, “Continuous F0 modeling for HMM
based statistical parametric speech synthesis,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no.
5, pp. 1071–1079, Jul. 2011.
[17] “HMM-based
speechhttp://hts.sp.nitech.ac.jp/.

synthesis

system(HTS),”[18] “JSUT-song,” https://sites.google.com/site/shinnosuketakami
chi/publication/jsut-song.