2014 IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP)A POSTFILTER TO MODIFY THE MODULATION SPECTRUM
IN HMM-BASED SPEECH SYNTHESIS
Shinnosuke Takamichi, Tomoki Toda, Graham Neubig, Sakriani Sakti and Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology,
8916–5 Takayama-cho, Ikoma, Nara, 630-0192, Japan
ABSTRACTIn this paper, we propose a postfilter to compensate modulation spectrum in HMM-based speech synthesis. In order to alleviate oversmoothing effects which is a main cause of quality degradation in
HMM-based speech synthesis, it is necessary to consider featuresthat can capture over-smoothing. Global Variance (GV) is one wellknown example of such a feature, and the effectiveness of parameter
generation algorithm considering GV have been confirmed. However, the quality gap between natural speech and synthetic speech
is still large. In this paper, we introduce the Modulation Spectrum(MS) of speech parameter trajectory as a new feature to effectively
capture the over-smoothing effect, and we propose a postfilter basedon the MS. The MS is represented as a power spectrum of the parameter trajectory. The generated speech parameter sequence is filtered
to ensure that its MS has a pattern similar to natural speech. Experimental results show quality improvements when the proposed
methods are applied to spectral and F0 components, compared withconventional methods considering GV.
Index Terms— HMM-based speech synthesis, over-smoothing,global variance, modulation spectrum, postfilter
1. INTRODUCTION
Text-To-Speech (TTS) is a technology that can convert any text into
speech, and it plays an important role in many speech applications.
The demand of synthesis techniques that can synthesize naturalsounding speech is rapidly growing. One of the major reasons that
HMM-based speech synthesis [1] have been an active research target
is its voice control capability [2, 3, 4] based on the elegant framework of HMMs. On the other hand, the quality in synthetic speech
is strongly degraded compared with natural speech and the synthetic
speech sounds muffled [5]. This is because traditional generation
frameworks generate over-smoothed parameter trajectories.
Global Variance (GV) [6] is one of the well-known features tocapture this over-smoothing effect. Despite the fact that the GV iscalculated in a simple form according to the second moment of parameters, generation algorithms considering the GV can efficiently
alleviate the over-smoothing effect. However, the quality gap between natural speech and synthetic speech is still large.
In this paper, we introduce the Modulation Spectrum (MS) of
parameter trajectory as a new feature to effectively capture the oversmoothing effect, and propose a postfilter to compensate MS. The
MS is represented as the power spectrum of the temporal parameter sequence. The effectiveness of the MS in capturing the sound of
speech has been noted in other research area, such as spectral cuesof speech perception [7], and the use as acoustic features in HMMbased speech recognition [8]. Because the generated sequence is
temporally smoothed, the MS of the synthetic speech tends to be

978-1-4799-2893-4/14/$31.00 ©2014 IEEE290

degraded compared with that of the natural speech even when ageneration algorithm considering GV is used. Therefore, the proposed method filters to fluctuate the generated parameter sequence.
The postfilter is trained using training data consisting of natural andsynthetic speech. Experimental results show quality improvements
when the proposed methods is applied to spectral and F0 components, compared with conventional method considering GV.
2. PARAMETER GENERATION IN HMM-BASED SPEECHSYNTHESIS
2.1. Maximizing HMM Likelihood [9]In HMM-based speech synthesis, context-dependent HMMs aretrained using natural speech parameters. In synthesis, sentence
HMMs corresponding to input text to be synthesized are constructed,
and speech parameter trajectory is generated to maximize HMM
likelihood under a constraint on the relationship between static anddynamic features, which is as follows:ĉ = argmax P (W c|λ) ,

(1)

c

[
]
⊤
⊤ ⊤
where c = c⊤
is a speech parameter vector
1 , · · · , ct , · · · , cT
sequence of T frames, ct = [ct (1) , · · · , ct (d) , · · · , ct (D)]⊤ is a
D-dimensional parameter vector at frame t, d is a dimensional index,
W is the weighting matrix for calculating the dynamic features [9],λ is a HMM parameter set, respectively.
Parameter sequences generated with Eq. (1) tend to be oversmoothed, and the synthetic speech sounds muffled compared with
the natural speech.
2.2. Maximizing HMM and GV Likelihood [6]The GV is defined as second moment of the parameter trajectory,
and is calculated as:
v (c)

=

v (d)

=

[v (1) , · · · , v (d) , · · · , v (D)]⊤ ,
(
)2
T
T
1 ∑
1 ∑
ct (d) −cτ (d) .
T t=1
T τ =1(2)(3)

Speech parameter trajectory is generated to maximize both HMM
and GV likelihoods.
ĉ = argmax P (W c|λ) P (v (c) |λv )w ,(4)cwhere λv is a parameter set of GV and w is a weight of the GV likelihood. The statistics of the GV are trained from the natural speechparameters.
The GV generated using Eq. (1) is usually smaller than that of
the natural speech parameters. Compensation of GV by this method
improves speech quality, but the improvements are still limited.

Modulation spectrum [dB]5

Training

Standard deviation

4

Parameter sequence32

Natural speech

1

HMM+GV

0
-1
-2
-3
0.1

1
10
Modulation frequency [Hz]100

Statistical modelPower
calculation

Generation
Synthesis

HMM

Modulation spectrumPowercalculation

TrainingPowercalculation

Postfilter
Phase

Generation

Fig. 2. Schematic diagram of the proposed method.
Fig. 1. Modulation spectrum of 9-th mel-cepstral coefficient sequence.

3. MODULATION SPECTRUM ANALYSIS
Though a Modulation Spectrum (MS) is traditionally defined as avalue calculated using the Fourier transform of the parameter sequence [10], in this paper we define the MS as its log-scaled power
spectrum. This feature, of course, can represent the temporal fluctuation of the parameter sequence. The MS s (c) of parameter sequencec is calculated as:[]⊤s (c) =s (1)⊤ , · · · , s (d)⊤ , · · · , s (D)⊤ ,
(5)
s (d)

=

[sd (0) , · · · , sd (m) , · · · , sd (M )]⊤ ,(6)

where sd (m) is the m-th MS of the d-th dimension of the parametersequence [c1 (d) , · · · , ct (d) , · · · , cT (d)]⊤ , m is a modulation frequency index, M is one half number of the Discrete Fourier Transform (DFT) length. In this paper, the MS is calculated from a parameter sequence that is zero-padded to set its sequence length to2M .
Here, we analyze the MS of natural and synthetic speech. The
MS mean of the 9-th mel-cepstral coefficient sequence generated using Eq. (1)（“HMM”） and Eq. (4)（“HMM+GV”） are shownin Fig. 1. Additionally, the MS of natural speech parameter sequence (“natural speech”) is shown in same figure for comparison.
It is observed that the MS of “HMM” is markedly degraded compared with that of “natural speech.” This is because temporal fluctuation observed in the natural speech parameter sequence is lost in the
HMM frameworks. We can also find that the MS of “HMM+GV” iscloser to natural one but there is still a big gap between the MSs of“HMM+GV” and “natural speech.”
From these result, we can expect further improvements in quality by directly accounting for this difference in the MS.
4. POSTFILTER BASED ON MODULATION SPECTRUM
In this section, we propose a postfilter to compensate the MS of thegenerated parameter sequence. The schematic diagram of the proposed method is shown in Fig. 2. Parameters of the proposed postfilter are trained using natural and generated parameter sequences inthe training data.
4.1. Training Process
The following probability distribution function is estimated fromnatural speech parameter sequences:()
P (s (c) |λs ) = N s (c) ; µ(N) , Σ(N) ,
(7)

291

(
)
where N ·; µ(N) , Σ(N) is a Gaussian distribution of mean vec]⊤
[
(N)
(N)
tor µ(N) = µ1,0 , · · · , µD,M
and diagonal covariance matrix
[(
(
)2
)2
(
)2 ]
(N)
(N)
(N)
(N)
, · · · , σD,M
, µd,m and σd,m is aΣ(N) = diag σ1,0mean and a variance of sd (m) and( λs is a parameterset of MS.
)

Probability distribution function N ·; µ(G) , Σ(G) is estimated in
the same manner using the speech parameter sequences generated
with the generation method described in Section 2. To avoid the effect of the duration difference between natural and generated speechparameter sequence, the parameter sequence is generated under thenatural speech duration.
4.2. Synthesis Process
The following filter is applied to the generated speech parameter sequence c:s′d (m)=+(1 − k)sd (m)
[ (N)
]
)
σd,m ((G)(N)
k
sd (m) − µd,m + µd,m ,
(G)
σd,m

(8)

where k is a postfilter emphasis coefficient valued between 0 and 1.
If k = 1, the MS will be modified to be close to the MS of natural
speech parameter sequences. On the other hand, if k = 0, the filteredsequence will be the same as the non-filtered sequence. The filteredparameter sequence is calculated from the MS and frequency phasecharacteristics of the parameter sequence, which are calculated before filtering.
4.3. Application to F0 Component
While the proposed postfilter can be directly applied to the spectral component, additional process is required for application to theF0 component because observed F0 contours are not a continuous
sequence. To solve this problem, we use continuous F0 modeling[11] which can estimate F0 values at the unvoiced frames. According to [12], we estimate the F0 values of the unvoiced frames with
spline-based interpolation. To avoid the MS’s fluctuation of the continuous F0 contour, we removed micro prosody by Low Pass Filter(LPF). We believe that the effect of micro prosody on speech quality
is small as referring in [13]. Moreover, we subtracted utterancelevel F0 from original F0 values before estimating continuous F0
contours to avoid discontinuous transition in zero-padding process.
Since the F0 estimation quality is degraded by spline-based extrap-

85

3.2

HMM+MS

80
75
70

in which the postfilter emphasis coefficient is set to 1. Conversely,
the proposed method can convert MSs in each modulation spectrum
individually.
5. EXPERIMENTAL EVALUATION
5.1. Experimental Conditions
We trained a context-dependent phoneme Hidden Semi-Markov
Model (HSMM) [15] for a Japanese female speaker. We used 450
sentences for training and 53 sentences for evaluation from phonetically balanced 503 sentences included in the ATR Japanese speech
database [16]. Speech signals were sampled at 16 kHz. The shift
length was set to 5 ms. The 0th-through-24th mel-cepstral coefficients were extracted as spectral parameters and log-scaled F0 and
5 band-aperiodicity [17, 18] were extracted as excitation parameters by the STRAIGHT analysis system [19]. The feature vector
1 We also considered simple approaches to estimate F of silence such as0
the use of utterance-level mean of F0 or the use of the F0 value in the nearest
voiced frame. However, we have confirmed that current method is better tomodel the MS.
2 Properly describing, the sum of linear-scaled MS except bias is equivalent to GV.

292

0.2 0.4 0.6 0.8
Filter coefficient

2.8
2.6
2.4
Natural speech

2.2

Natural speech

0

HMM+GV+MS

30

1

0.2 0.4 0.6 0.8
Filter coefficient

1

Fig. 3. HMM likelihood for the Fig. 4. HMM likelihood for thefiltered F0 contours.
filtered spectral parameters.
100

3.6

HMM+GV+MS
Natural speech

HMM+GV+MS

3.4

50

Log-scaled likelihood

Log-scaled likelihood(GV,G)

are the GV mean of d-th dimension of the，µdwhere µd
natural and synthetic speech parameters in the training data, respectively, and ⟨ct (d)⟩ is the mean of d-th dimension of the synthetic
speech parameters. In this method, since only the variance of thesequence is considered, the MS degradation is not completely recovered, thus the generation of parameters including temporal fluctuation of natural speech parameters is difficult. On the other hand,the proposed method can recover this fluctuation because we directly
consider the MS itself. Therefore, the proposed method can be expected to yield quality improvements.
According to the Perceval’s theorem, the power of a temporalsequence is preserved during a DFT. the GV defined in Eq. (3) represents the power of the sequence except the bias component. Because
the MS is defined as the power spectrum of the sequence, the sumof the MS over all modulation spectra except the bias component isequivalent to the GV2 . In the GV-based postfiltering process, MSs of
all modulation frequency except bias is converted in the same way.
Namely, the GV-based conversion process is special case of the proposed MS-based conversion process under the following conditions:
{
1
m=0
(·)
(·)
µd,m = 0, σd,m =,(10)(GV,·)µd
otherwise

60
55

HMM+MS

HMM+GV+MS

65

A postfiltering process to ensure the GV of the generated parametersequence is proposed in [14]. The generated speech parameter are
linearly converted as follows:vu (GV,N)uµdĉt (d) = t (GV,G){ct (d) − ⟨ct (d)⟩} + ⟨ct (d)⟩,(9)µd(GV,N)

90

Log-scaled likelihood

4.4. Relationship to GV-based Postfilter

95

Log-scaled likelihood

olation, we calculate the MS from the non-silence frames1 .
In synthesis, the utterance-level mean and unvoiced/voiced regions of the generated F0 contour are extracted before applying the
proposed filter. After filtering, first, the filtered continuous F0 contour is calculated in the same manner as the spectral component.
Then, the filtered F0 contour is calculated by adding the mean to
the filtered continuous F0 contour and restoring the unvoiced/voiced
regions.

0
HMM+MS

-50

-100
-150
0

3.2
3
2.8

HMM+MS

2.6

Natural speech

2.4

0.2 0.4 0.6 0.8
Filter coefficient

1

2.2

0

0.2 0.4 0.6 0.8
Filter coefficient

1

Fig. 5. GV likelihood for the fil- Fig. 6. GV likelihood for the filtered F0 contours.
tered spectral parameters.
consisted of spectral and excitation parameters and their delta anddelta-delta features. Five-state left-to-right HSMMs were used. The
DFT length to calculate MS is set to 4096, which is over the maximum frame length in training and evaluation data. A 10 Hz-cutoff
LPF is used to remove the micro prosody3 .
We conducted some evaluations with the following systems:
HMM： generated with Eq. (1)
HMM+MS： applied the proposed postfilter to “HMM”
HMM+GV： generated with Eq. (4)HMM+GV+MS： applied the proposed postfilter to “HMM+GV”
Note that the postfilter of “HMM+GV+MS” is trained using parameter sequences generated with GV.
5.2. Objective Evaluation for Emphasis Coefficient
In order to determine the filter emphasis coefficient for the spectraland F0 components, we calculate the HMM likelihood, GV likelihood, and MS likelihood for filtered parameter sequence for settingsof the emphasis coefficient from 0 to 1. For comparison, the likelihood for natural speech parameter sequence is calculated.
3 We evaluated training accuracy of MS likelihood in various cutoff frequencies, and we have confirmed that this setting is the best.

-4 HMM+GV+MS

-1.62

-6

-1.64 Natural speech

HMM+MS

-8

-1.66

-10

-1.68

HMM+GV+MS

-1.72

-14
0

0.2 0.4 0.6 0.8
Filter coefficient

1

-1.74
0

0.2 0.4 0.6 0.8
Filter coefficient

1

Fig. 7. MS likelihood for the fil- Fig. 8. MS likelihood for the filtered continuous F0 contours.
tered spectral parameters.

1

1

0.8

Preference score

Preference score

0.8

0.6

0.6

0.4

0.4
0.2
0

0.2

HMM HMM HMM HMM+MS +GV +GV
+MS

0

0.4
0.2
0

HMM
HMM+GV
HMM+GV+MS
Natural speech

280

300

320
340
Frame index

360

380

Fig. 11. Examples of the natural and generated 4th mel-cepstral coefficient sequence.

we set the coefficient to 1.0.

5.3. Subjective Evaluation for Speech Quality

95% confidence interval

95% confidence interval

0.6

-0.2

HMM+MS

-1.70

-12

-16

4th mel-cepstral coefficient-1.60

Natural speechLog-scaled likelihood (x 0.001)

Log-scaled likelihood (x 0.001)-2

HMM HMM HMM HMM+MS +GV +GV+MS

Fig. 9. Preference score for Fig. 10. Preference score for F0
spectral component.
component.
The HMM likelihood, GV likelihood, and MS likelihood for filtered spectral parameters are shown in Fig. 3, Fig. 5 and Fig. 7,
respectively. It is observed in Fig. 3 that the HMM likelihoods of“HMM+MS” and “HMM+GV+MS” decrease as the emphasis coefficient increases, and their values are always higher than that of“natural speech.” In the GV likelihood shown in Fig. 5, we can seethat these likelihoods cross that of “natural speech” at 0.85. On theother hand, MS likelihoods increase as the coefficient increases but
their values always lower than “natural speech.” From these results,
we determined filter emphasis coefficient for spectral component tobe 0.85.
Those likelihoods for the filtered F0 contour are shown in Fig.
4, Fig. 6 and Fig. 8, respectively. The transition of these likelihoods
as the coefficient changes show the same tendency as those for the
spectral components except the relation with the likelihoods of “natural speech.” We can find that all likelihoods of “HMM+MS” and“HMM+GV+MS” are higher than “natural speech” when setting theemphasis coefficient over 0.75, and we can also find that the coefficient 1.0 is the highest point of MS likelihood. From these results,

293

To evaluate the quality improvements yielded by the proposed postfilter, we conducted a preference test (AB test) on speech quality
by eight listeners for the spectral and F0 components4 . Every pairof these four types of synthetic speech was presented to listeners in
random order. Listeners were asked which sample sounds better interms of speech quality. The “HMM” system is used for the component that the proposed methods were not applied to.
The result of the preference test for spectral component is shownin Fig. 9, and an example of the spectral parameter sequence isshown in Fig. 11. We can see that the score of “HMM+MS” system dramatically increases over the “HMM” system, and achieves asimilar score to the ”HMM+GV” system. Additionally, further improvement by applying the proposed method to “HMM+GV” canbe observed. From these result, the effectiveness of the proposed
method in quality in the spectral component was yielded.
Similarly, the preference score for the F0 component is shown in
Fig. 10. Again, “HMM+MS” and “HMM+GV+MS” achieve a better score than “HMM,” but there are not additional gains over when
GV is considered. The reason why the score differences among conventional and proposed methods are smaller than those in the spectral components is that both natural and generated F0 contour transitions are smoothly and these MSs are closer than those in spectral
parameter sequence.

6. SUMMARY
In this paper, we proposed a postfilter to compensate the modulation spectrum of the generated parameter trajectory in HMM-based
speech synthesis. The experimental results demonstrated that thequality improvements by the proposed method are yielded for both
spectral and F0 components. As future work, we will incorporate
the modulation spectrum to the parameter generation algorithm.
Acknowledgements: Part of this work was supported by JSPS
KAKENHI Grant Number 22680016.
4 Some samples are available from http://isw3.naist.jp/
˜shinnosuke-t/sample_mspf.html.

7. REFERENCES[1] H. Zen, K. Tokuda, and A. Black. Statistical parametric speechsynthesis. Speech Commun., Vol. 51, No. 11, pp. 1039–1064,
2009.
[2] T. Yoshimura, T. Masuko, K. Tokuda, T. Kobayashi, and T. Kitamura. Speaker interpolation for HMM-based speech synthesis system. J. Acoust. Soc. Jpn. (E), Vol. 21, No. 4, pp. 199–
206, 2000.
[3] J. Yamagishi and T. Kobayashi. Average-voice-based speechsynthesis using HSMM-based speaker adaptation and adaptive
training. IEICE Trans., Inf. and Syst., Vol. E90-D, No. 2, pp.
533–543, 2007.
[4] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi. A stylecontrol technique for HMM-based expressive speech synthesis. IEICE Trans., Inf. and Syst., Vol. E90-D, No. 9, pp. 1406–
1413, 2007.
[5] S. King and V. Karaiskos. The blizzard challenge 2011. In
Proc. Blizzard Challenge workshop, Turin, Italy, Sept. 2011.
[6] T. Toda and K. Tokuda. A speech parameter generation algorithm considering global variance for HMM-based speech synthesis. IEICE Trans., Vol. E90-D, No. 5, pp. 816–824, 2007.
[7] R. Drullman, J .M. Festen, and R. Plomp. Effect of reducing
slow temporal modulations on speech reception. J. Acoust. Soc.
of America, Vol. 95, pp. 2670–2680, 1994.
[8] S. Thomas, S. Ganapathy, and H. Hermansky. Phoneme recgnition usng spectral envelop and modulation frequency features.
In Proc. ICASSP, pp. 4453–4456, Taipei, Taiwan, April 2009.
[9] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura. Speech parameter generation algorithms for HMMbased speech synthesis. In Proc. ICASSP, pp. 1315–1318, Istanbul, Turkey, June 2000.
[10] L. Atlas and S. A.Shamma. Joint acoustic and modulation
frequency. EURASIP Journal on Applied Signal Processing,Vol. 7, pp. 668–675, 2003.
[11] K. Yu and S. Young. Continuous F0 modeling for HMM based
statistical parametric speech synthesis. IEEE Trans. Audio,
Speech and Language, Vol. 19, No. 5, pp. 1071–1079, 2011.
[12] K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura.
A hybrid approach to electrolaryngeal speech enhansement
based on spectral subtraction and statistical voice conversion.
In Proc. INTERSPEECH, pp. 3067–3071, Lyon, France, Sep.
2013.
[13] P. Taylor. Text-To-Speech synthesis. Cambridge Univ. Press,
2009.
[14] T. Toda, T. Muramatsu, and H. Banno. Implementation of conputationally efficient real-time voice conversion. In Proc. INTERSPEECH, Portland, Oregon, U.S., Sept. 2012.
[15] H. Zen, K. Tokuda, T. Kobayashi T. Masuko, and T. Kitamura.
Hidden semi-markov model based speech synthesis system.
IEICE Trans., Inf. and Syst., E90-D, No. 5, pp. 825–834, 2007.
[16] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara. A large-scale Japanese speech database. In ICSLP90, pp. 1089–1092, Kobe, Japan, Nov. 1990.
[17] H. Kawahara, Jo Estill, and O. Fujimura. Aperiodicity extraction and control using mixed mode excitation and group delay

294

manipulation for a high quality speech analysis, modificationand synthesis system STRAIGHT . In MAVEBA 2001, pp.
1–6, Firentze, Italy, Sept. 2001.
[18] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano. Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation. In Proc. INTERSPEECH, pp. 2266–2269,
Pittsburgh, U.S.A., Sep. 2006.
[19] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne. Restructuring speech representations using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based
F0 extraction: Possible role of a repetitive structure in sounds.
Speech Commun., Vol. 27, No. 3–4, pp. 187–207, 1999.