JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

1

Post-Filters to Modify the Modulation Spectrum
for Statistical Parametric Speech Synthesis
Shinnosuke Takamichi, Student Member, IEEE, Tomoki Toda, Member, IEEE, Alan W. Black, Member, IEEE
Graham Neubig, Nonmember, IEEE Sakriani Sakti, Member, IEEE Satoshi Nakamura, Fellow, IEEE

Abstract—This paper presents novel approaches based on
Modulation Spectrum (MS) for high-quality statistical parametric speech synthesis, including Text-To-Speech (TTS) and
Voice Conversion (VC). Although statistical parametric speechsynthesis offers various advantages over concatenative speechsynthesis, the synthetic speech quality is still not as good asthat of concatenative speech synthesis or the quality of natural
speech. One of the biggest issues causing the quality degradationis the over-smoothing effect often observed in the generated
speech parameter trajectories. Global Variance (GV) is known
as a feature well correlated with the over-smoothing effect, andthe effectiveness of keeping the GV of the generated speechparameter trajectories similar to those of natural speech has
been confirmed. However, the quality gap between natural speechand synthetic speech is still large. In this paper, we propose
using the MS of the generated speech parameter trajectories asa new feature to effectively quantify the over-smoothing effect.
Moreover, we propose post-filters to modify the MS utterance byutterance or segment by segment to make the MS of synthetic
speech close to that of natural speech. The proposed postfilters are applicable to various synthesizers based on statisticalparametric speech synthesis. We first perform an evaluation of
the proposed method in the framework of Hidden Markov Model(HMM)-based TTS, examining its properties from different perspectives. Furthermore, effectiveness of the proposed post-filters
are also evaluated in Gaussian Mixture Model (GMM)-based
VC and Classification And Regression Trees (CART)-based TTS(a.k.a., CLUSTERGEN). The experimental results demonstratethat (1) the proposed utterance-level post-filter achieves quality
comparable to the conventional generation algorithm considering
the GV, and yields significant improvements by applying to
the GV-based generation algorithm in HMM-based TTS. (2)the proposed segment-level post-filter capable of achieving lowdelay synthesis also yields significant improvements in synthetic
speech quality, and (3) the proposed post-filters are also effective
in not only HMM-based TTS but also GMM-based VC and
CLUSTERGEN.
Index Terms—Statistical parametric speech synthesis, oversmoothing, post-filter, global variance, modulation spectrum,
HMM-based text-to-speech, GMM-based voice conversion,
CLUSTERGEN

I. I NTRODUCTIONLTHOUGH human beings naturally utilize their own
speech as a communication tool, there exist many barriers in speech communication, such as vocal disorders [1], [2],A

S. Takamichi, S. Sakti, G. Neubig and S. Nakamura are with the Graduate
School of Information Science, Nara Institute of Science and Technology,
Japan.
T. Toda is with Information Technology Center, Nagoya University, Japan.
A. W. Black is with Language Technologies Institute, Carnegie Mellon
University, United States.
Manuscript received XXX XX, 20XX; revised XXX XX, 20XX.

[3], language differences [4], [5], [6], and physical constraints[3], [7]. Many speech technologies have been studied to break
down these barriers. One of the promising technologies isparametric speech generation [8], including Text-To-Speech(TTS) synthesis [9] and Voice Conversion (VC) [10]. TTS
is a technique to synthesize speech corresponding to a given
text, and VC is a technique to convert non-/para-linguistic
characteristics of input speech while preserving linguistic
characteristics.
In parametric speech generation, statistical parametricspeech synthesis [11] was established in the 1990s [12], [10],and has gained popularity in this decade. Nowadays, manytechnologies have been studied within this basic framework,
including speech synthesis using Hidden Markov Models(HMM) [13], Gaussian Mixture Model (GMM) [14], Classification And Regression Trees (CART) [15], kernel regression [16], [17], and Deep Neural Nets (DNN) [18], [19].
Whereas concatenative speech synthesis [9], [20] directly uses
waveform segments or natural speech parameter segments togenerate a speech waveform, statistical parametric speech synthesis collects statistics from the speech parameter segments
and utilizes these to generate speech parameters to be usedin speech waveform generation. This statistical modeling andgeneration framework make it possible to build small footprint
synthesizers [21], adapt existing voices to other target voices
using only a small amount of speech data [22], and flexibly
control voice characteristics of synthetic speech [23], [24].
On the other hand, a serious drawback of statistical parametric speech synthesis compared to concatenative speechsynthesis is the lower quality of synthetic speech [25], [26].
There are three main reasons causing the quality degradation[11]: parameterization errors in the speech analysis/synthesis
stage [27], [28], [29], inaccurate modeling in the training
stage [30], [18], and over-smoothness of the generated speechparameters in the synthesis stage [25], [26]. In particular, thelast factor, the over-smoothing effect usually makes synthetic
speech sound muffled compared to concatenative speech synthesis or natural speech. One promising approach to alleviate
the over-smoothing effect is to extract a specific feature toquantify the over-smoothing effect and to generate speechparameters so that their corresponding features become moresimilar to those of natural speech parameters. One widely
known example of such a feature is Global Variance (GV) [31],[14], which is a second order moment of the speech parametersequence. Considering the GV during parameter generation
effectively works to alleviate the over-smoothing effect and to
significantly improve synthetic speech quality. Currently, theJOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

GV-based parameter generation has been applied in a numberof ways [32], [33], [34], [35]. However, the use of this metricin the parameter generation tends to additionally generate
artificial sounds [32], [34] and the quality gap between naturaland synthetic speech is still large.
In this paper, we propose a new feature more sensitively
correlated to the over-smoothing effect than the GV, theModulation Spectrum (MS). The linear-scaled MS of a speechparameter sequence is defined as the power spectrum of thesequence. The linear-scaled MS, like GV, is a second ordermoments of the parameter sequence. The effectiveness of the
MS in capturing speech properties has been noted in otherresearch areas, such as spectral cues of speech perception [36],the use as acoustic features in HMM-based speech recognition[37] and acoustic signal classification [38], and as a countermeasure to discriminate synthetic speech from natural speechin speaker verification [39]. Related to the perceptual effect,[40], [41] investigated the effect of the MS (especially, lower
modulation frequency band) on the perceptual intelligibility.
Because generated speech parameter sequences tend to be
temporally smoothed by the parameter generation process, theMS of synthetic speech tends to be degraded compared to thatof natural speech. This MS degradation is still observed evenwhen GV is used in parameter generation. The post-filtering
approach proposed in this paper remedies this problem bymodifying the generated speech parameter sequence so thatits MS becomes more similar to that of natural speech. The
proposed post-filter modifies the MS utterance by utterance
and can be automatically constructed using natural speechand synthetic speech as training data. This utterance-levelpost-filter is further extended to a segment-level post-filter tomodify the MS segment by segment in order to achieve lowdelay parameter generation [42], [43].
We first evaluate the proposed post-filters in HMM-based
TTS from various perspectives. Then, we evaluate them inother speech synthesizers: the utterance-level post-filter in
GMM-based VC [14] and the segment-level post-filter in
CART-based TTS (a.k.a., CLUSTERGEN) [15]. The experimental results show that (1) the proposed post-filters effectively improve the naturalness of the spectrum, F0 , and HMMstate duration, yielding significant quality improvements in
HMM-based TTS (as also shown in [44], [45]), (2) the
proposed segment-level post-filter is also effective (as also
shown in [45]), and (3) the proposed post-filters are effective
in not only HMM-based TTS but also GMM-based VC (asalso shown in [46]) and CLUSTERGEN.
The rest of this paper is organized as follows. Section
II briefly reviews HMM-based TTS, GMM-based VC, and
CLUSTERGEN. Section III presents the concept of our study,
the MS, and Section IV proposes the MS-based post-filter.
Section V and Section VI are the experimental evaluation andconclusion, respectively.
II. S PEECH PARAMETER G ENERATION IN S TATISTICAL
PARAMETRIC S PEECH S YNTHESISThis section describes the speech parameter generation procedures in statistical parametric speech synthesis frameworks,

2

Fig. 1. An overview of HMM-based TTS.

such as HMM-based TTS, GMM-based VC, and CLUSTERGEN.

A. HMM-Based TTS [13]Figure 1 illustrates an overview of HMM-based TTS. The
HMMh models the T -frame output
i> speech feature sequence>>>
given the contextual factor
Y = Y 1 ,··· ,Y t ,··· ,Y T
sequence X of the input text, where Y t is a speech feature
vector at frame t. Speech features and HMM-state duration
are simultaneously modeled in a unified framework [47].
In synthesis, given the contextual feature sequence X, thecorresponding HMM is first constructed, then the HMM-state
sequence q̂ = [q̂1 , · · · , q̂t , · · · , q̂T ] is determined by maximizing the duration probability density function as follows:q̂ = argmax P ( q| X, λ) ,(1)qwhere q̂t is a HMM-state at frame t, and λ is the parameterset of the HMM. The synthetic speech parameter sequence isgenerated by the following algorithms.
Using HMMs [48]: The speech parameter sequence isgenerated by maximizing the following output probability
density function under the explicit relationship between staticand dynamic features.
ŷ = argmax P ( W y| q, X, λ) ,

(2)

y


 >
> >
where ŷ=y1 , · · · , y>is a speecht , · · · , yT
parameter vector sequence of T frames, y t=>[yt (1) , · · · , yt (d) , · · · , yt (D)] is a D-dimensional speechfeature vector at frame t, d is a dimension index, and W
is the weighting matrix for calculating the dynamic features[11], where Y = W y.
Synthetic speech parameter sequences generated by Eq. (2)tend to be over-smoothed, and the synthetic speech sounds
muffled compared to the natural speech.
Using HMMs and a GV model [31]:JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

3The GV is defined as a second order moment of theparameter trajectory, which is calculated as
v (y)

=

v (d)

=

>

[v (1) , · · · , v (d) , · · · , v (D)] ,
!2
T
T
1X
1 X
yt (d) −yτ (d) .
T t=1
T τ =1(3)(4)

The speech parameter sequence is generated by maximizing aweighted combination of the two probability density functions.
wŷ = argmax P ( W y| q, X, λ) P ( v (y)| λv ) v ,(5)yFig. 2. Graphic representation of how to derive the MS s (y) from the speechparameter sequence y.

where λv is the parameter set of the GV, and wv is theweight of the GV probability density function. The probability
density function of the GV is trained from the natural speechparameters in the training data.

53
1

In GMM-based VC, a GMM performs frame-level modelingusing input and output speech feature sequences X and Y .
The Dynamic Time Warping (DTW) algorithm is used totemporally align X and Y . In synthesis, mixture component
sequence q̂ is determined similarly to Eq. (1), where qt is a
mixture component at frame t. The synthetic speech parametersequence ŷ is determined in the same manner as Eq. (5), whereP ( W y| q, X, λ) is derived from the GMM.

Whereas HMM-based TTS ties the probability density functions over multiple frames with the HMM-state-level probability density function, which is usually determined by decision
tree clustering based on the Minimum Description Length(MDL) criterion [49], CLUSTERGEN predicts the probability
density functions frame by frame in the CART framework.
The output probability density function in Eq. (2) is calculatedusing the contextual factor sequence X. The synthetic speechparameter sequence ŷ is determined in the same manner as
Eq. (2).
III. M ODULATION S PECTRUM A NALYSIS
Though the MS is traditionally defined as a value calculatedusing the Fourier transform of the parameter sequence [50],this paper defines the MS as its log-scaled power spectrum.
The temporal fluctuation of the parameter sequence is modeled
as power values of individual modulation frequency components of the parameter sequence. The MS s (y) of theparameter sequence y is calculated as:
h
i>
>
>
>
s (y) = s (1) , · · · , s (d) , · · · , s (D), (6)sd (f )>

[sd (0) , · · · , sd (f ) , · · · , sd (Ds )] ,

!2
T
X

= log
yt (d) cos mt
=

(7)

t=1

+

−1
4

T
X
t=1

!2 
yt (d) sin mt

,

(8)

1st

2
0

−2
3

9th
natural1−1

C. CLUSTERGEN [15]s (d)Modulation spectrum

B. GMM-Based VC [14]

−31

15th

HMM

10
Modulation frequenc [Hz]HMM+GV100

Fig. 3. Averaged MSs of the 1st, 9th and 15th mel-cepstral coefficientsequences from above in HMM-based TTS.

where f is a modulation frequency index, m = −πf /Ds is
a modulation frequency, and Ds is one half of the Discrete
Fourier Transform (DFT) length. The MS is calculated fromzero-padded parameter sequences so its length is 2Ds . As
shown in Fig. 2, s (y) is given as a super vector consisting of
the MSs corresponding to individual feature dimensions.
To demonstrate how the MS allows us to capture relevant
frequency characteristics, we first demonstrate some characteristics of the MS of natural and synthetic speech. Figure 3
shows the MS mean of the mel-cepstral coefficient sequences
generated using Eq. (2) (“HMM”) and Eq. (5) (“HMM+GV”)
in HMM-based TTS. Additionally, the MS mean of a natural
speech parameter sequence (“natural”) is shown in the same
figure for comparison. It can be observed that the MS of“HMM” is markedly degraded compared to that of “natural.”This is because temporal fluctuation observed in the natural
speech parameter sequences is lost in the HMM framework.
We can also find that the MS of “HMM+GV” is closer tonatural speech in lower modulation frequency bands but there
is still a large gap between the MSs of “HMM+GV” and“natural speech” in higher modulation frequency bands (morethan 10 Hz). From these results, we can expect that further

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

4

Parameter sequence

Training

Modulation spectrumPowercalculation

Training

Statistical modelPower
calculation

Generation
SynthesisPowercalculation

Generation

Fig. 4. Averaged MSs of log-F0 contours in HMM-based TTS. Note that the
Nyquist frequency is 100 Hz similarly to the spectral parameters, but only
< 10 Hz components are shown.

Fig. 5. Averaged MSs of phoneme-level duration in HMM-based TTS. Notethat the pseudo Nyquist frequency is set to 100 Hz because we cannot define
the Nyquist frequency for duration.

quality improvements will be yielded by compensating forthese differences in the MS.
In addition, we consider the spectral tilt of the MS (definedas “MS tilt”) which indicates the power difference between thelower and the higher modulation frequency components in Fig.
3. We can observe that the MS tilt of the natural mel-cepstrum
tends to increase in the higher order mel-cepstral coefficients.
On the other hand, the MS tilt of the generated mel-cepstrum
is similar among different order mel-cepstral coefficients. Even
when using the GV in the parameter generation “HMM+GV,”
the MS is just shifted and the MS tilt is not changed. These
results show that the parameter generation process shown by
Eq. (2) or Eq. (5) tends to constrain the MS tilt of the generated
speech parameter sequence to be unnatural.
In addition to the cepstral coefficients, we can also calculate
the MSs of the other features. as described in the following
section. The MS of the F0 contour shown in Fig. 4 is also degraded by the statistical process. Higher modulation frequency
components of the generated MS are almost the same as thoseof natural speech, but lower components are slightly different.
HMM-state duration determined by Eq. (1) is also affectedby the over-smoothing effect due to the statistical averaging
process implicit in conventional parameter generation, as in
the spectrum and F0 components [47], [51]. Figure 5 shows
the MS mean of phoneme duration sequences. We can seethat the generated MS is generally smaller than that of natural
speech.

Postfilter

Phase

Fig. 6. A schematic diagram of the proposed MS-based post-filter to modify
the MS of the generated parameter sequence in the case of HMM-based TTS.

Fig. 7. An example of the MS conversion in the synthesis stage. Note that
the MS envelope (“Generated MS” and “Filtered MS”) is drawn instead of
the MS itself for clear illustration. The MS envelope is calculated by low-passliftering the cepstrum of MS.

IV. P ROPOSED MS-BASED P OST-F ILTER
This section proposes post-filters to modify the MS of thegenerated parameter sequence. Figure 6 shows a schematic
diagram of the proposed method. Parameters of the proposed post-filter are automatically trained using natural andgenerated speech parameter sequences in the training data.
The speech parameters are generated by an individual speechsynthesizer. First, the utterance-level MS-based post-filter is
described for spectrum, F0 , and HMM-state duration. Then,the segment-level MS-based post-filter is derived by localizing
the utterance-level post-filtering process.

A. Utterance-level MS-Based Post-Filter
The post-filter described in this section performs utterancelevel MS filtering. The MS is calculated from a parametersequence that is zero-padded to set its sequence length to 2Ds ,and it is assumed that the sequence length in the training andsynthesis is less than 2Ds .
1) Training Stage: The following probability distributionfunction is estimated from natural speech parameter sequences:(N)
P (s|λs ) = N s; µ(N),s , Σs(9)JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

5TABLE IT HE DETAILED PROCEDURE OF THE PROPOSED POST- FILTERING PROCESS .
1
2
3
4
5
6
7

Zero-pad the original parameter sequence.
Take the DFT and store the phase characteristics.
Calculate the log-scaled power spectrum (= MS).
Apply the post-filter to the MS.
Compute the power and add the original phase.
Take the inverse DFT.
Truncate the resulting signal to have an appropriate length.

(N)

where µs and Σ(N)are the mean vector and the diagonalscovariance matrix of the MS s,

>
(N) >
(N) >
(N) >
, (10)
µ(N)
=
µ
,
·
·
·
,
µ
,
·
·
·
,
µ
s
1
D
d
h
i
(N)
(N)
(N)
Σ(N)
= diag Σ1 , · · · , Σd , · · · , ΣD ,
(11)
s
h
i>
(N)
(N)
(N)
(N)
µd
= µd,0 , · · · , µd,f , · · · , µd,Ds ,
(12)
h
i
(N)
(N) 2
(N) 2
(N) 2
Σd
= diag σd,0 , · · · , σd,f , · · · , σd,Ds , (13)

s

(N) 2

(N)

where µd,f and σd,f are the mean and the variance ofsd (f ), respectively.
 λs is the parameter set of the MS.
(G)(G)
N ·; µs , Σsis also estimated in the same manner using
the speech parameter sequences generated as described in
Section II. To avoid the effect of the duration difference
between natural and generated speech parameter sequences in
HMM-based TTS, the parameter sequence is generated using
the natural speech duration. In the case of GMM-based VC,
temporally-aligned input speech parameter sequence X is used
to generated the speech parameter sequence y.
2) Synthesis Process: The following filter is applied to thegenerated speech parameter sequence y (see Fig. 7.):s0d (f )Fig. 8. An illustration of the pre-processing procedures to calculate thecontinuous F0 contour from the original F0 contour.

(1 − k)sd (f )
#
" (N)

σd,f (G)(N)
sd (f ) − µd,f + µd,f ,
+ k
(G)
σd,f
=

(14)

where k is a post-filter emphasis coefficient between 0 and
1. If k = 1, the MS will be modified to be close to the MSof natural speech parameter sequences. On the other hand,
if k = 0, the filtered sequence will be the same as the nonfiltered sequence. The filtered parameter sequence is calculated
from the modified MS and original phase characteristics of theparameter sequence before filtering. The detailed procedure isshown in Table I.
We implemented the simple time-invariant filter as the yet
another implementation to recover the MS, but the synthesized
speech sounded discontinuous as reported in [45].
B. Application to Various Features
1) F0 Contour: While the proposed post-filter can be directly applied to the spectral component, additional processing
is required for its application to the F0 component because
observed F0 contours are not a continuous sequence. To solve
this problem, we use continuous F0 modeling [52] which also
estimates F0 values at the unvoiced frames. Following [53],sFig. 9. Procedures of the segment-level MS-based post-filter in HMM-based
TTS.

F0 values of the unvoiced frames are estimated with splinebased interpolation. Because the effect of micro prosody on
speech quality is small [54] but the effect on the MS is not
negligible, we remove it with a Low Pass Filter (LPF). Moreover, the utterance-level F0 mean is subtracted from original
F0 values before estimating continuous F0 contours to avoid
discontinuous transitions in the zero-padding process. These
procedures are shown in Fig. 8. Because spline-based methods
are inappropriate for extrapolation, i.e., silence frames, we
calculate the MS from the non-silence frames1 .
In synthesis, the utterance-level mean and unvoiced/voiced
regions of the generated F0 contour are extracted before
applying the proposed post-filter. First, the filtered continuous
F0 contour is calculated in the same manner as the spectral
component. Then, the filtered F0 contour is calculated byadding the mean to the filtered continuous F0 contour andrestoring the unvoiced/voiced regions.
2) HMM-state duration: The proposed utterance-level postfilter modifies the MS of the phoneme-level duration calculated
from the state-level duration determined by Eq. (1). The
phoneme-level duration sequence is filtered after excluding
silence and its mean value is normalized as with the F0 parameters. After restoring the utterance-level mean, the phoneme1 We also considered simple approaches to estimate F of silence such as0
the use of the utterance-level mean of F0 or the use of the F0 value in thenearest voiced frame. However, we have confirmed that the current method
is better to model the MS.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

6level duration is revised if it is smaller than the number ofstates of the phoneme HMM. Finally, the HMM-state duration
is updated by maximizing the state duration while fixing thephoneme duration to the filtered values.
C. Segment-Level Post-Filter
Because the proposed utterance-level MS-based post-filter
calculates the MS utterance by utterance, the DFT length needs
to be set large enough to cover various lengths of utterances.
This MS calculation causes some problems: if the length ofan utterance to be synthesized is longer than the previously
determined DFT length, the MS can not be calculated accurately, and thus it is difficult to apply the utterance-levelfiltering process to a low-latency speech synthesis framework[42], [43] where frame-level or segment-level processing basedon the recursive parameter generation is essential [12].
In order to handle these cases, we propose a segmentlevel post-filter that is effective on shorter segments. The
segment-level post-filter is derived by localizing the postfiltering process as illustrated in Fig. 9. A part of the speechparameter sequence that is windowed by a triangular window
with constant length is used as a segment to calculate theMS and its statistics. The window shift length is set to a halfof the window length. The MS-based post-filtering process
is performed segment by segment in the same manner as
the trajectory-level post-filtering process. The filtered speechparameter sequence is generated by overlapping and adding
the filtered segments. The Hanning window may also be usedinstead of the triangular window. Note that for the spectrum
parameters, silence frames are removed in calculating the
MS statistics to alleviate the over-fitting problem [18]. The
segment-level post-filtering can be applicable to low-delay
speech waveform generation. Moreover, it is possible to further
implement context-dependent post-filtering.

Fig. 10. An example of the 1st, 9th, and 15th mel-cepstral coefficientsequences from above in HMM-based TTS.

D. Discussion
The proposed post-filters can be automatically constructedin a data-driven manner. Whereas conventional post-filtering
processes [47], [55], [56], [57] requires the rule-based design[47], or manual tuning [55], the proposed post-filters enable
automatic design and tuning.
Another data-driven approach is the post-filtering process
to maintain the GV of the generated parameter sequence [35].
The generated speech parameters are linearly converted asfollows:v
u (GV,N)uµd{yt (d) − hyt (d)i} + hyt (d)i,(15)
ŷt (d) = t (GV,G)µd(GV,N)(GV,G)

where µd
and µdare the GV mean of the dth dimension of the natural and synthetic speech parametersin the training data, respectively, and hyt (d)i is the meanof the d-th dimension of the synthetic speech parameters.
In this method, since only the variance of the sequence isconsidered, the MS degradation is not completely recovered.
Thus, temporal fluctuation of the generated speech parameters
after filtering is still very different from that of natural speech.

Fig. 11. An example of the spectrograms in HMM-based TTS.

On the other hand, the proposed post-filters can recover thisfluctuation because we directly consider the MS itself.
According to the Parseval’s theorem, the power of a temporal sequence is preserved during a DFT. The GV definedin Eq. (4) represents the power of the sequence excluding thebias component. Because the utterance-level MS is definedas the power spectrum of the sequence, the sum of the MS
over all modulation frequencies excluding the bias component(frequency zero) is equal to the GV2 . In the GV-based postfiltering process, MSs of all modulation frequencies other thanthe bias are converted in the same way. Namely, the GV-based
post-filtering process is a special case of the proposed MS-

2 Properly described, the sum of linear-scaled MS excluding the bias is
proportional to GV.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

7

they sometimes result in audible warbling in the synthesized
speech.
V. E XPERIMENTAL E VALUATIONFirst, we investigate the effects of the proposed utterancelevel and segment-level post-filters from various perspectives
in HMM-based TTS. Then, we evaluate them in other statistical parametric speech synthesis frameworks: the effect of theutterance-level post-filter in GMM-based VC and the effect of
the segment-level post-filter in CLUSTERGEN.
Fig. 12. An example of the F0 contour in HMM-based TTS.

A. Experimental Conditions for Evaluation in HMM-Based
TTS

Fig. 13. An example of the phoneme-level duration in HMM-based TTS.

based post-filtering process under the following conditions:
(·)

µd,f
(N)
µd,f
(N)
σd,f

=

log µ(GV,·) (f > 0),

(16)

=

(G)
µd,f (f = 0),(G)
σd,f ,

(17)

=

(18)

in which the post-filter emphasis coefficient is set to 1.
Namely, the GV-based post-filtering process only causes theconstant MS shift as shown in Fig. 33 . On the other hand, theproposed methods can directly convert the MS components at
individual modulation frequencies.
Figure 10 draws an example of the filtered/non-filtered melcepstral coefficient sequences. It is observed that the proposed
post-filter generates the fluctuated parameter sequence, andthe effect is larger in the higher order of the mel-cepstralcoefficients. This is because the MS difference between naturaland generated parameter sequences is larger in higher-ordermel-cepstral coefficients as shown in Fig. 3. The effect ofthe proposed post-filter is also observed in the spectrogram
shown in Fig. 11. We can find that the proposed post-filter
produces a more fluctuated spectral sequence compared to theconventional approaches. Similarly, Fig. 12 and Fig. 13 show
the F0 contour and duration. We can also find the fluctuatedparameter sequences are generated by the proposed post-filter.
Note that although these fluctuated parameter sequences
are effective for improving naturalness of synthetic speech,
3 In Fig. 3, the parameter generation algorithm considering the GV rather
than the GV-based post-filter is used. Although it tends to make the GV ofthe generated speech parameter sequence almost equal to the GV mean µv[58], [59], it still roughly results a MS shift in practical effect, although theamount of the MS shift changes utterance by utterance.

We trained a context-dependent phoneme Hidden SemiMarkov Model (HSMM) [60] for a Japanese female speakerfor evaluation in HMM-based TTS. We used 450 sentences
for training and 53 sentences for evaluation from the 503
phonetically balanced sentences included in the ATR Japanese
speech database [61]. Speech signals were sampled at 16
kHz. The shift length was set to 5 ms. The 0th-through-24th
mel-cepstral coefficients were extracted as spectral parameters
and log-scaled F0 and 5 band-aperiodicity [62], [63] wereextracted as excitation parameters. The STRAIGHT analysissynthesis system [27] was employed for parameter extraction
and waveform generation. The feature vector consisted ofspectral and excitation parameters and their delta and deltadelta features. Five-state left-to-right HSMMs were used.
The proposed post-filter was trained in a context-independent
manner. A 10 Hz-cutoff LPF was used to remove the micro
prosody from the continuous F0 contours4 .
We conducted evaluation with the following systems:
HMM: The spectrum and F0 are generated with Eq. (2), andthe HMM-state duration is determined with Eq. (1).
HMM+MS: The proposed post-filter is applied to “HMM.”
HMM+GV: The spectrum and F0 are generated with Eq.
(5).
HMM+GV+MS: The proposed post-filter is applied to“HMM+GV.”
Note that the post-filter of “HMM+GV+MS” was trained usingparameter sequences generated with the GV. The “HMM”
system was used for the components that the proposed methods were not applied to. The post-filters were not applied tothe aperiodicity component because there is no quality gain
achieved by the post-filters5 . Sections V-B and V-C adopt
the utterance-level and the segment-level MS-based post-filter,
respectively.
B. Evaluation of Utterance-Level MS-Based Post-Filter
We investigate the effectiveness of the proposed utterancelevel post-filter in HMM-based TTS. The filter emphasis
coefficients for spectrum, F0 and duration are first tuned by
the likelihoods. The synthetic speech quality is then evaluated
4 We evaluated training accuracy of MS likelihood for various cutoff
frequencies, and confirmed that this setting was the best.
5 The same tendency is reported in the parameter generation algorithm
considering the GV [63].

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

Fig. 14. HMM, GV, and MS likelihoods for the spectral parameter sequences
filtered by the proposed utterance-level post-filter in HMM-based TTS.

Fig. 15. HMM, GV, and MS likelihoods for the F0 contours filtered by the
proposed utterance-level post-filter in HMM-based TTS.

using the tuned emphasis coefficients. The DFT length tocalculate MS (= 2Ds ) was set to 4096, which is over the
maximum frame length in training and evaluation data.
1) Tuning of the Emphasis Coefficients: In order to determine the filter emphasis coefficients, we calculated the HMM
likelihood, GV likelihood, and MS likelihood for filteredspectrum, F0 , and HMM-state duration for settings of theemphasis coefficient from 0 to 1. The duration likelihood was
calculated instead of the HMM likelihood when tuning thecoefficient for duration. For comparison, the likelihood fornatural speech parameter sequences was calculated, which waslabeled as “natural.” Note that the HMM likelihood and theMS likelihood were normalized by the total number of frames
T and one half of the DFT length Ds , respectively.
Figure 14 shows the likelihoods for the filtered spectral
parameters. It is observed that the HMM likelihoods of“HMM+MS” and “HMM+GV+MS” decrease as the emphasis
coefficient increases. Nevertheless, their values are always
higher than that of “natural.” In the GV likelihood, we cansee that these likelihoods cross that of “natural speech” at
k = 0.85. On the other hand, MS likelihoods increase as
the coefficient increases but their values always lower than“natural speech.” Considering these results, we determined thefilter emphasis coefficient for spectral component to be 0.85.
Figure 15 shows the likelihoods for the filtered F0 contour.
The change of these likelihoods as the coefficient varies show
the same tendency as those for the spectral components except
the relation with the likelihoods of “natural speech.” We can

8Fig. 16. HMM, GV, and MS likelihoods for the phoneme-level duration
sequences filtered by the proposed utterance-level post-filter in HMM-based
TTS.

Fig. 17. Preference scores on speech quality with 95% confidence interval(proposed utterance-level post-filter).

find that all likelihoods of “HMM+MS” and “HMM+GV+MS”
are higher than “natural speech” when setting the emphasis
coefficient over k = 0.75, and we can also find that thecoefficient k = 1.0 is the highest point of MS likelihood.
From these results, we set the coefficient to 1.0.
Figure 16 shows the likelihoods for the filtered phonemelevel duration. The tendency of the likelihood change is similar
to those of the spectrum and F0 , and the MS likelihood is thehighest at k = 1.0. Therefore, we set the coefficient k = 1.0.
We can also see discontinuous transitions of the MS likelihood.
We expect that this was caused by the effect of rounding thefiltered duration values into integer values after filtering.
2) Subjective Evaluation on Speech Quality: To investigate
whether or not quality improvements are yielded by applying
the proposed post-filter to the spectrum, F0 , and duration
components, we conducted a preference AB test on speech
quality. Every pair of these types of synthetic speech waspresented to listeners in random order. Listeners were asked
which sample sounded better in terms of speech quality.
Evaluation for spectrum, F0 , and duration was conducted by
8, 8, and 6 listeners, respectively.
Figure 17 shows the preference test for the spectrum, F0 ,and duration. For spectrum, we can see that the score of the“HMM+MS” system dramatically increases over the “HMM”
system, and achieves a similar score to the ”HMM+GV”
system. Additionally, further improvement can be observed
by applying the proposed method to “HMM+GV.” From these

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

Fig. 18. HMM, GV, and MS likelihoods for the spectral parameter sequences
filtered by the proposed segment-level post-filter in HMM-based TTS.

results, the effectiveness of the proposed method for the
spectral component is confirmed. For F0 , “HMM+MS” and“HMM+GV+MS” achieve a better score than “HMM,” butthere are not additional gains over when GV is considered.
The reason why the score differences among conventional
and proposed methods are smaller than those in the spectral
components is that the MS of the generated F0 contours isquite close to that of the natural F0 contours, as shown in Fig.
4, even if not applying the proposed post-filter. Finally, we canalso see a slight improvement in quality for duration. These
results demonstrate a quality gains by the proposed utterancelevel post-filter for spectrum, F0 and duration.
We have explained that the MS involves the GV, but it isshown that combining the GV and MS (“HMM+GV+MS”)
yields improvements compared to HMM+MS. This is because
the post-filtering-based approach ignores the HMM probability
density function. We expect that quality of parameter generation considering the MS will be comparable to that considering
the GV and MS.
C. Evaluation of Segment-Level MS-Based Post-Filter
We evaluate the effectiveness of the segment-level post-filter
in HMM-based TTS. The window length and window shift
length were set to 125 ms (25 samples) and 60 ms (12 samples)[64]. A 64-taps DFT was used to calculate the MS. The tuning
step and evaluation step were conducted in the same way as
the evaluation of the proposed utterance-level post-filter. Notethat the post-filter was not applied to the duration because
we could not observe a large difference between filtered andnon-filtered sequences.
1) Tuning the Emphasis Coefficients: The HMM likelihood,
GV likelihood, and MS likelihood for the filtered spectral
parameters and F0 contours were calculated. The results areshown in Fig. 18 and Fig. 19. Their tendencies are similar to
those of the utterance-level post-filter. Although the segmentlevel post-filtering process causes a degradation of the HMM
likelihoods, they are still greater than those of natural parameters. Almost all likelihoods tend to increase as the filter
coefficient approaches 1. We observed a degradation of theMS likelihood for F0 , but it is always greater than that ofnatural parameters. From these results, we tuned the emphasis
coefficient to 1.0 for both spectrum and F0 . As the general

9Fig. 19. HMM, GV, and MS likelihoods for the F0 contours filtered by the
proposed segment-level post-filter in HMM-based TTS.

Fig. 20. Preference scores on speech quality with 95% confidence interval(proposed segment-level post-filter in HMM-based TTS).

tendency, the change of the MS likelihoods is smaller thanthat in the utterance-level post-filter.
2) Subjective Evaluation on Speech Quality: The preference AB test on speech quality by 7 listeners was conducted in
the same manner as in the previous section. The post-filtering
was applied to both spectrum and F0 .
The preference score is shown in Fig. 20. It is observedthat a significant quality gain is yielded by “HMM+MS”
compared to “HMM,” and it is comparable to that yielded by“HMM+GV.” Furthermore, we can see that an additional gain
is yielded by “HMM+GV+MS” compared to “HMM+GV.”
This tendency is similar to that observed in the utterance-levelpost-filter. Note that the segment-level post-filter is applicableto speech parameter sequences of various lengths but theutterance-level post-filter is not.
3) Comparison of Utterance-Level and Segment-Level
Post-Filters: We compare the proposed utterance-level andsegment-level post-filters that are applied to “HMM+GV” forspectrum and F0 . We used the emphasis coefficients tunedin this and the previous section. The preference AB test on
speech quality by 8 listeners was conducted.
Fig. 21 shows the result. Because there is no significant
difference between two post-filters, we can find that the
proposed post-filters have the same capability in the speech
quality improvement.
D. Evaluation in Various Synthesizers
We confirm the effectiveness of the proposed post-filters in
GMM-based VC and CLUSTERGEN.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

10

1.0

Preference score

0.8
0.6
0.4
0.2
0.0

HMM+GV+MS(utterance-level)

HMM+GV+MS(segment-level)

Fig. 21. Preference scores on speech quality with 95% confidence interval(proposed utterance-level and segment-level post-filters in HMM-based TTS).
Fig. 23. Preference scores on speech quality with 95% confidence interval
in GMM-based VC and CLUSTERGEN

at the emphasis coefficient k = 0.90. Therefore, the emphasis
coefficient is set to 0.90.

Fig. 22. GMM, GV, and MS likelihoods for the spectral parameters filteredby the proposed utterance-level post-filter in GMM-based VC.

1) GMM-Based VC: The proposed utterance-level postfilter was applied to GMM-based VC. Because this framework
has a similar synthesis criterion as that described in Section
II, the tuning step and evaluation step are conducted in
the same manner as the evaluation for HMM-based TTS.
Here, “HMM+GV” and “HMM+GV+MS” were relabeled as“GMM+GV” and “GMM+GV+MS,” respectively. The systems corresponding to “HMM” and “HMM+MS” were notused in the evaluation.
We prepared speech from two Japanese male and female
speakers6 . We selected 50 parallel sentences of subset A fromthe 503 phonetically balanced sentences included in the ATR
Japanese speech database [61] for training, and 50 sentencesof subset B for evaluation. We trained female-to-male GMMs.
The speech features were the same as in the evaluationsfor HMM-based TTS. The spectral parameters and aperiodic
components were converted with a 64-mixture GMM and a
16-mixture GMM, respectively. The log-scaled F0 was linearly
converted. The DFT length to calculate MS was set to 2048,
which is over the maximum frame length in the training andevaluation data. The proposed utterance-level post-filter wasapplied to the spectral parameters.
The GMM likelihood, GV likelihood, and MS likelihoodfor the filtered spectral parameters were shown in Fig. 22.
From this result, we can see that the tendency of the likelihoodchanges is almost the same as that in Fig. 14, but the GV
likelihood of “GMM+GV+MS” starts to fall below “natural”
6 The female speaker here is a different person from the speaker we usedin the evaluation for HMM-based TTS.

We conducted a preference AB test on speech quality,
and a preference XAB test on speaker individuality. We
first presented an analysis-synthesized reference speech as”X”, then we presented random-ordered synthesized speech.
7 listeners participated in each evaluation. Fig. 23 shows theresults. In term of speech quality, a significant quality gain
is observed. However, there is no significant difference in thepreference score on speaker individuality. We expect that nocues for individuality are at higher modulation frequencies thatare recovered by the MS-based post-filter.
2) CLUSTERGEN: The proposed segment-level post-filter
was also applied to CLUSTERGEN. We also tuned the emphasis coefficient as in the previous experiments. We observedthat the likelihoods didn’t vary very much as shown in Figs. 18
and 19. We also confirmed that a quality gain was yielded by
setting k to 1.0. Here, the methods corresponding to “HMM”
and “HMM+MS” were relabeled as “CNV” and “CNV+MS,”
respectively.
We prepared an English female speaker. 418 and 46 sentences of news reader speech were used for training andevaluation, respectively. The speech features were the sameas those in the evaluation for HMM-based TTS, but they wereextracted by Speech signal Processing ToolKit (SPTK) [65]and the aperiodicity component was not used. The window
length and window shift length of the segment-level post-filter
were set to 125 ms (25 samples) and 60 ms (12 samples). A
64-taps DFT was used to calculate the MS. The segment-levelpost-filter was applied to both spectrum and F0 . parameters.
A preference AB test on speech quality was conductedby 6 listeners on the Amazon Mechanical Turk service [66].
Because many listening environments are expected, a nopreference option was prepared. The right side of Fig. 23
shows the result. We can see that large improvements areyielded by the segment-level post-filter.
The results presented in this section suggest that the proposed MS-based post-filters are effective for a variety ofstatistical parametric speech synthesis frameworks.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

VI. C ONCLUSION
This paper introduced the Modulation Spectrum (MS) of
speech parameter trajectory as a new feature to effectively
quantify the over-smoothing effect, which is the main causeof the synthetic speech quality degradation in statisticalparametric speech synthesis. Moreover, this paper also proposed the MS-based post-filters on the utterance level andthe segment level to improve the synthetic speech quality.
Experimental evaluation was conducted using various statistical parametric speech synthesis methods, such as Hidden
Markov Model (HMM)-based Text-To-Speech (TTS), Gaussian Mixture Model (GMM)-based Voice Conversion (VC),
and Classification And Regression Trees (CART)-based TTS(a.k.a., CLUSTERGEN). The experimental results demonstrated that (1) the proposed utterance-level post-filter achieves
better quality for spectrum, F0 , and HMM-state duration in
HMM-based TTS, (2) the proposed segment-level post-filter
capable of achieving low-delay synthesis also yields significant
improvements in synthetic speech quality, and (3) the proposed
post-filters are also effective in not only HMM-based TTS but
also GMM-based VC and CLUSTERGEN. As future work,
we plan to investigate which modulation frequency bands
significantly affect synthetic speech quality, and integrate the
MS into the speech synthesis metric.
Acknowledgements: Part of this work was supported by
JSPS KAKENHI Grant Number 26280060 and Grant-in-Aid
for JSPS Fellows Grant Number 26 · 10354, and part of thiswork was executed under “JSPS Strategic Young Researcher
Overseas Visits Program for Accelerating Brain Circulation.”

Shinnosuke Takamichi received his B.E. from Nagaoka University of Technology, Japan, in 2011and his M.E. degree from the Graduate School ofInformation Science, Nara Institute of Science andTechnology (NAIST), Japan, in 2013. He was a
short-time researcher at the NICT, Kyoto, Japan in
2013, and a visiting researcher of Carnegie Mellon
University (CMU), in United States, from 2014 to2015. He is currently a Ph.D. student of NAIST,
and Research Fellow (DC2) of Japan Society for
the Promotion of Science, Japan. He received the
7th Student Presentation Award from ASJ, the 35th Awaya Prize Young
Researcher Award from ASJ, the 8th Outstanding Student Paper Award fromIEEE Japan Chapter SPS, the Best Paper Award from APSIPA ASC 2014, theStudent Paper Award from IEEE Kansai Section, the 30th TELECOM System
Technology Award from TAF, and the 2014 ISS Young Researcher’s Award in
Speech Field from the IEICE. His research interests include electroacoustics,
signal processing, and speech synthesis. He is a student member of ASJ andIEEE SPS, and a member of ISCA.

11

Tomoki Toda earned his B.E. degree from Nagoya
University, Aichi, Japan, in 1999 and his M.E. and
D.E. degrees from the Graduate School of Information Science, NAIST, Nara, Japan, in 2001 and 2003,
respectively. He was a Research Fellow of JSPS in
the Graduate School of Engineering, Nitech, Aichi,
Japan, from 2003 to 2005. He was an AssistantProfessor of the Graduate School of Information
Science, NAIST from 2005 to 2011, where he iscurrently an Associate Professor. He has also been a
Visting Researcher at the NICT, Kyoto, Japan, since
May 2006. From March 2001 to March 2003, he was an Intern Researcher at
the ATR SLC Research Laboratories, Kyoto, Japan, and then he was a Visiting
Researcher until March 2006. He was also a Visiting Researcher at the LTI,
CMU, Pittsburgh, USA, from October 2003 to September 2004 and at the
CUED, Cambridge, UK, from March to August 2008. His research interests
include statistical approaches to speech processing. He received the 18th
TELECOM System Technology Award for Students and the 23rd TELECOM
System Technology Award from the TAF, the 2007 ISS Best Paper Award andthe 2010 ISS Young Researcher’s Award in Speech Field from the IEICE,
the 10th Ericsson Young Scientist Award from Nippon Ericsson K.K., the
4th Itakura Prize Innovative Young Researcher Award and the 26th Awaya
Prize Young Researcher Award from the ASJ, the 2009 Young Author Best
Paper Award from the IEEE SPS, the Best Paper Award (Short Paper inRegular Session Category) from APSIPA ASC 2012, the 2012 Kiyasu Special
Industrial Achievement Award from the IPSJ, and the 2013 Best Paper Award(Speech Communication Journal) from EURASIP-ISCA. He was a memberof the Speech and Language Technical Committee of the IEEE SPS from2007 to 2009.

Alan W. Black is a Professor in the Language Technologies Institute at Carnegie Mellon University. Before joining the faculty at CMU in 1999, he worked
in the Centre for Speech Technology Research at
the University of Edinburgh, and before that at ATR
in Japan. He is one of the principal authors of thefree software Festival Speech Synthesis System, the
FestVox voice building tools and CMU Flite, a small
footprint speech synthesis engine, that is the basis
for many research and commercial systems aroundthe world. He also works in spoken dialog systems,
the LetsGo Bus Information project and mobile speech-to-speech translation
systems. Prof Black is an elected member of ISCA board (20072015). He hasover 200 refereed publications and is one of the highest cited authors in his
field.

Graham Neubig received his B.E. from University
of Illinois, Urbana-Champaign in 2005, and his M.S.
and Ph.D. in informatics from Kyoto University in2010 and 2012 respectively. From 2012, he has been
an assistant professor at the Nara Institute of Scienceand Technology, where he is pursuing research inmachine translation and spoken language processing.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

Sakriani Sakti received her B.E degree in Informatics (cum laude) received her B.E degree inInformatics (cum laude) from Bandung Instituteof Technology, Indonesia, in 1999. In 2000, she
received ”DAAD-Siemens Program Asia 21st Century” Award to study in Communication Technology,
University of Ulm, Germany, and received her MSc
degree in 2002. During her thesis work, she worked
with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between
2003-2009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006-2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005-2008) with Dialog Systems Group University ofUlm, Germany, and received her PhD degree in 2008. She actively involved incollaboration activities such as Asian Pacific Telecommunity Project (20032007), A-STAR and U-STAR (2006-2011). She also served as a visiting
professor of Computer Science Department, University of Indonesia (UI)
in 2009-2011. Currently, she is an assistant professor of the Augmented
Human Communication Lab, NAIST, Japan. She is a member of JNS,
SFN, ASJ, ISCA, IEICE amd IEEE. Her research interests include statistical
pattern recognition, speech recognition, spoken language translation, cognitive
communication, and graphical modeling framework.

Satoshi Nakamura is Professor of Graduate Schoolof Information Science, Nara Institute of Science andTechnology, Japan, Honorarprofessor of Karlsruhe
Institute of Technology, Germany, and ATR Fellow.
He received his B.S. from Kyoto Institute of Technology in 1981 and Ph.D. from Kyoto University
in 1992. He was Associate Professor of Graduate
School of Information Science at Nara Instituteof Science and Technology in 1994-2000. He was
Director of ATR Spoken Language Communication
Research Laboratories in 2000-2008 and Vice president of ATR in 2007-2008. He was Director General of Keihanna ResearchLaboratories and the Executive Director of Knowledge Creating Communication Research Center, National Institute of Information and Communications
Technology, Japan in 2009-2010. He is currently Director of Augmented
Human Communication laboratory and a full professor of Graduate Schoolof Information Science at Nara Institute of Science and Technology. He also
serves as a visiting professor of Collaborative Research Unit, National Instituteof Informatics. He is interested in modeling and systems of speech-to-speechtranslation and speech recognition. He is one of the leaders of speech-tospeech translation research and has been serving for various speech-to-speechtranslation research projects in the world including C-STAR, IWSLT and
A-STAR. He was a project leader of the world first network-based commercial speech-to-speech translation service for 3-G mobile phones in 2007
and VoiceTra project for iPhone in 2010. He received Yamashita ResearchAward, Kiyasu Award from the Information Processing Society of Japan,
Telecom System Award, AAMT Nagao Award, Docomo Mobile ScienceAward in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Ministerof Education, Science and Technology, and the Commendation for Scienceand Technology by the Minister of Internal Affair and Communications.
He also received LREC Antonio Zampoli Award 2012. He organized theInternational Workshop of Spoken Language Translation (IWSLT 2006) and
Oriental Cocosda 2008 as a general chair. He also served as the program chairof INTERSPEECH 2010. He has been Elected Board Member of International
Speech Communication Association, ISCA, since June 2011 and IEEE Signal
Processing Magazine Editorial Board Member since April 2012.

12

R EFERENCES[1] J. Yamagishi, C. Veaux, S. King, and S. Renals, “Speech synthesis
technologies for individuals with vocal diabilities: Voice banking andreconstruction,” Acoust. Sci. technol., vol. 33, pp. 1–5, 2012.
[2] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Anevaluation of excitation feature prediction in a hybrid approach toelectrolaryngeal speech enhancement,” in Proc. ICASSP, Florence, Italy,
May 2014, pp. 4521–4525.
[3] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing voice
conversion method based on many-to-many eigenvoice conversion andtraining data generation using a singing-to-singing synthesis system,” inProc. APSIPA ASC, Hollywood, U.S.A., Nov. 2012.
[4] N. Hattori, T. Toda, H. Kawai, H. Saruwatari, and K. Shikano, “Speakeradaptive speech synthesis based on eigenvoice conversion and languagedependent prosodic conversion in speech-to-speech translation,” in Proc.
INTERSPEECH, Florence, Italy, Aug. 2011, pp. 2769–2772.
[5] P. K. Muthukumar and A. W. Black, “Automatic discovery of a phonetic
inventory for unwritten languages for statistical speech synthesis,” inProc. ICASSP, Florence, Italy, May 2014.
[6] S. Sitaram, G. Anumanchipalli, J. Chiu, A. Parlikar, and A. W. Black,“Text to speech in new languages without a standardized orthography,”in Proc. SSW8, Barcelona, Spain, Aug. 2013.
[7] K. Kobayashi, T. Toda, T. Nakano, M. Goto, G. Neubig, S. Sakti,and S. Nakamura, “Regression approaches to perceptual age control insinging voice conversion,” in Proc. ICASSP, Florence, Italy, May 2014,
pp. 7954–7958.
[8] Z.-H. Ling, S.-Y. Kang, H. Zen, A. Senior, M. Schuster, X.-J. Qian,
H. Meng, and L. Deng, “Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques
and future trends,” IEEE Signal Processing Magazine, vol. 32, no. 3,
pp. 35–52, 2015.
[9] Y. Sagisaka, “Speech synthesis by rule using an optimal selection of
non-uniform synthesis units,” in Proc. ICASSP, New York, U.S.A., Apr.
1988, pp. 679–682.
[10] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. Speech and Audio Processing, vol. 6, no. 2, pp. 131–142, Mar. 1988.
[11] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speech synthesis,” Speech Commun., vol. 51, no. 11, pp. 1039–1064, 2009.
[12] K. Tokuda, T. Kobayashi, and S. Imai, “Speech parameter generationfrom HMM using dynamic features,” in Proc. ICASSP, Detroit, U.S.A.,
May 1995, pp. 660–663.
[13] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura,“Speech synthesis based on hidden Markov models,” Proceedings of theIEEE, vol. 101, no. 5, pp. 1234–1252, 2013.
[14] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trajectory,” IEEE
Transactions on Audio, Speech and Language Processing, vol. 15, no. 8,
pp. 2222–2235, 2007.
[15] A. W. Black, “CLUSTERGEN: A statistical parametric synthesizer using
trajectory modeling,” in Proc. INTERSPEECH, Pittsburgh, U.S.A., Sep.
2006.
[16] T. Koriyama, T. Nose, and T. Kobayashi, “Statistical parametric speechsynthesis based on gaussian process regression,” IEEE Journal of
Selected Topics in Signal Processing, vol. 8, no. 2, pp. 173–183, Apr.
2014.
[17] E. Helander, T. V. H. Silen, and M. Gabbouj, “Voice conversion using
dynamic kernel partial least squares regression,” IEEE Transactions on
Audio, Speech, and Language Processing, vol. 20, no. 3, pp. 806–817,
Mar. 2012.
[18] H. Zen and A. Senior, “Deep mixture density networks for acoustic
modeling in statistical parametric speech synthesis,” in Proc. ICASSP,
Florence, Italy, May 2014, pp. 3872–3876.
[19] T. Nakashika, R. Takashima, T. Takiguchi, and Y. Ariki, “Voice conversion in high-order eigen space using deep belief nets,” in Proc.
INTERSPEECH, Lyon, France, Aug. 2013, pp. 369–372.
[20] A. J. Hunt and A. Black, “Unit selection in a concatenative speechsynthesis system using a large speech database,” in Proc. ICASSP,
Atlanta, U.S.A., May 1996, pp. 373–376.
[21] K. Oura, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda, “An investigationof implementation performance analysis of DNN based speech synthesis
system,” in Proc. INTERSPEECH, Brighton, U. K., 2014, pp. 577–582.
[22] J. Yamagishi and T. Kobayashi., “Average-voice-based speech synthesis
using HSMM-based speaker adaptation and adaptive training,” IEICE
Trans., Inf. and Syst., vol. E90-D, no. 2, pp. 533–543, 2007.

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, XXX 20XX

[23] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi, “A style control
technique for HMM-based expressive speech synthesis,” IEICE Trans.,
Inf. and Syst., vol. E90-D, no. 9, pp. 1406–1413, 2007.
[24] L. Chen, M. J. F. Gales, L. Chen, K. Chin, K. Knull, and M. Akamine,“Exploring rich expressive information from audiobook data using
cluster adaptive training,” in Proc. INTERSPEECH, Portland, U.S.A.,
Sep. 2012.
[25] S. King and V. Karaiskos, “The blizzard challenge 2011,” in Proc.
Blizzard Challenge workshop, Turin, Italy, Sept. 2011.
[26] Y. Stylianou, “Voice transformation: A survey,” in Proc. ICASSP, Taipei,
Taiwan, Apr. 2009, pp. 3585–3588.
[27] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne, “Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based F0 extraction: Possible role of arepetitive structure in sounds,” Speech Commun., vol. 27, no. 3–4, pp.
187–207, 1999.
[28] M. Morise, “An attempt to develop a singing synthesizer by collaborative
creation,” in Proc. SMAC, Stockholm, Aug. 2013.
[29] Y. Agiomyrgiannakis, “Vocaine the vocoder and applications in speechsynthesis,” in Proc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4230–
4234.
[30] P. K. Muthukumar, A. W. Black, and H. T. Bunnell, “Optimizations
and fitting procedures for the Liljencrants-Fant model for statisticalparametric speech synthesis,” in Proc. ICASSP, Vancouver, Canada, May
2013.
[31] T. Toda and K. Tokuda, “A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,” IEICE
Trans., vol. E90-D, no. 5, pp. 816–824, 2007.
[32] S. Takamichi, T. Toda, Y. Shiga, S. Sakti, G. Neubig, and S. Nakamura,“Parameter generation methods with rich context models for high-quality
and flexible text-to-speech synthesis,” IEEE Journal of Selected Topics
in Signal Processing, vol. 8, no. 2, pp. 239–250, 2014.
[33] T. Nose, V. Chunwijitra, and T. Kobayashi, “A parameter generation
algorithm using local variance for HMM-based speech synthesis,” IEEE
Journal of Selected Topics in Signal Processing, vol. 8, no. 2, pp. 221–
228, 2014.
[34] M. Shannon and W. Byrne, “Fast, low-artifact speech synthesis considering global variance,” in Proc. ICASSP, Vancouver, Canada, May.
2013, pp. 7869–7873.
[35] T. Toda, T. Muramatsu, and H. Banno, “Implementation of computationally efficient real-time voice conversion,” in Proc. INTERSPEECH,
Portland, U.S.A., Sept. 2012.
[36] R. Drullman, J. M. Festen, and R. Plomp, “Effect of reducing slow
temporal modulations on speech reception,” J. Acoust. Soc. of America,
vol. 95, pp. 2670–2680, 1994.
[37] S. Thomas, S. Ganapathy, and H. Hermansky, “Phoneme recgnition usng
spectral envelop and modulation frequency features,” in Proc. ICASSP,
Taipei, Taiwan, April 2009, pp. 4453–4456.
[38] S. Gergen, A. Nagathil, and R. Martin, “Reduction of reverberation
effects in the MFCC modulation spectrum for improved classificationof acoustic signals,” in Proc. INTERSPEECH, Dresden, Germany, Sep.
2015, pp. 1992–1995.
[39] Z. Wu, X. Xiao, E. S. Chng, and H. Li, “Synthetic speech detection using
temporal modulation feature,” in Proc. ICASSP, Vancouver, Canada,
May. 2013, pp. 7234–7238.
[40] R. Drullman, J. M. Festen, and R. Plomp, “Effect of temporal envelope
smearing on speech perception,” J. Acoust. Soc. of America, vol. 95, pp.
1053–1064, 1994.
[41] T. Arai, M. Pavel, H. Hermansky, and C. Avendano, “Intelligibility ofspeech with filtered time trajectories of spectral envelopes,” in Proc.
ICSLP, vol. 4, 1996, pp. 2490–2493.
[42] T. Muramatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano,“Low-delay voice conversion based on maximum likelihood estimationof spectral parameter trajectory,” in Proc. INTERSPEECH, Brisbane,
Australia, Sep. 2008, pp. 1076–1079.
[43] H. Zen and H. Sak, “Unidirectional long short-term memory recurrent
neural network with recurrent output layer for low-latency speechsynthesis,” in Proc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4470–
4474.
[44] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “A postfilter to modify modulation spectrum in HMM-based speech synthesis,”in Proc. ICASSP, Florence, Italy, May 2014, pp. 290–294.
[45] S. Takamichi, T. Toda, A. W. Black, and S. Nakamura, “Modified modulation spectrum-based post-filter for HMM-based speech synthesis,” inProc. GlobalSIP, Atlanta, United States, Dec. 2014, pp. 710–714.
[46] ——, “Modulation spectrum-based post-filter for GMM-based voice
conversion,” in Proc. APSIPA ASC, Siem Reap, Cambodia, Dec. 2014.

13[47] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,“Simultaneous modeling of spectrum, pitch and duration in HMM-based
speech synthesis,” in Proc. EUROSPEECH, Budapest, Hungary, Apr.
1999, pp. 2347–2350.
[48] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura,“Speech parameter generation algorithms for HMM-based speech synthesis,” in Proc. ICASSP, Istanbul, Turkey, June 2000, pp. 1315–1318.
[49] K. Shinoda and T. Watanabe, “MDL-based context-dependent subword
modeling for speech recognition,” J. Acoust. Soc. Jpn.(E), vol. 28, no. 3,
pp. 140–146, 2007.
[50] L. Atlas and S. A.Shamma, “Joint acoustic and modulation frequency,”
EURASIP Journal on Applied Signal Processing, vol. 7, pp. 668–675,
2003.
[51] S. Pan, J. Tao, and Y. Wang, “A state duration generation algorithm
considering global variance for HMM-based speech synthesis,” in Proc.
APSIPA ASC, Xi’an, China, 2011.
[52] K. Yu and S. Young, “Continuous F0 modeling for HMM based
statistical parametric speech synthesis,” IEEE Trans. Audio, Speech andLanguage, vol. 19, no. 5, pp. 1071–1079, 2011.
[53] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “A hybrid
approach to electrolaryngeal speech enhansement based on spectral
subtraction and statistical voice conversion,” in Proc. INTERSPEECH,
Lyon, France, Sep. 2013, pp. 3067–3071.
[54] P. Taylor, Text-To-Speech Synthesis. Cambridge Univ. Press, 2009.
[55] F. Eyben and Y. Agiomyrgiannakis, “A frequency-weighted post-filteringtransform for compensation of the over-smoothing effect in HMM-based
speech synthesis,” in Proc. ICASSP, Florence, Italy, May 2014, pp. 275–
279.
[56] K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, “The effect ofneural networks in statistical parametric speech synthesis,” in Proc.
ICASSP, Brisbane, Australia, Apr. 2015, pp. 4455–4459.
[57] L.-H. Chen, T. Raitio, C. V.-Botinhao, J. Yamagishi, and Z.-H. Ling,“DNN-based stochastic postfilter for HMM-based speech synthesis,” inProc. INTERSPEECH, MAX Atria, Singapore, May 2014, pp. 1954–
1958.
[58] H. Zen, M. J. F. Gales, Y. Nankaku, and K. Tokuda, “Product ofexperts for statistical parametric speech synthesis,” IEEE Trans. on
Audio, Speech, and Language processing, vol. 20, no. 3, pp. 794–805,
Mar. 2011.
[59] T. Nose and A. Ito, “Analysis of spectral enhancement using global
variance in HMM-based speech synthesis,” in Proc. INTERSPEECH,
MAX Atria, Singapore, May 2014, pp. 2917–2921.
[60] H. Zen, K. Tokuda, T. K. T. Masuko, and T. Kitamura, “Hidden semiMarkov model based speech synthesis system,” IEICE Trans., Inf. andSyst., E90-D, no. 5, pp. 825–834, 2007.
[61] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara, “A large-scale Japanese speech database,” in ICSLP90,
Kobe, Japan, Nov. 1990, pp. 1089–1092.
[62] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation
for a high quality speech analysis, modification and synthesis system
STRAIGHT,” in MAVEBA 2001, Firentze, Italy, Sept. 2001, pp. 1–6.
[63] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood voice conversion based on GMM with STRAIGHT mixed excitation,” in Proc. INTERSPEECH, Pittsburgh, U.S.A., Sep. 2006, pp.
2266–2269.
[64] V. Tyagi, I. McCowan, H. Misra, and H. Bourlard, “Mel-cepstrum
modulation spectrum (MCMS) features for robust ASR,” in Proc. ASRU,
MAX Atria, Singapore, Nov. 2003, pp. 399–404.
[65] “Speech signal processing toolkit (SPTK) http://sp-tk.sourceforge.net/.”[66] “Amazon mechanical turk https://www.mturk.com/.”