GlobalSIP 2014: Machine Learning Applications in Speech Processing

Modified Post-filter to Recover Modulation
Spectrum for HMM-based Speech Synthesis
Shinnosuke Takamichi∗† , Tomoki Toda∗ , Alan W Black† , and Satoshi Nakamura∗∗ Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Japan

Email: shinnosuke-t@is.naist.jp† Language Technologies Institute, Carnegie Mellon University (CMU), U.S.A

Abstract—This paper proposes a modified post-filter to recovera Modulation Spectrum (MS) in HMM-based speech synthesis.
To alleviate the over-smoothing effect which is one of the major
problems in HMM-based speech synthesis, the MS-based postfilter has been proposed. It recovers the utterance-level MSof the generated speech trajectory, and we have reported its
benefit to the quality improvement. However, this post-filter is
not applicable to various lengths of speech parameter trajectories,
such as phrases or segments, which are shorter than an utterance.
To address this problem, we propose two modified post-filters,(1) the time-invariant filter with a simplified conversion form and(2) the segment-level post-filter which applicable to a short-term
parameter sequence. Furthermore, we also propose (3) the postfilter to recover the phoneme-level MS of HMM-state duration.
Experimental results show that the modified post-filters also yieldsignificant quality improvements in synthetic speech as yieldedby the conventional post-filter.
Index Terms—HMM-based speech synthesis, modulation spectrum, post-filter, over-smoothing

I. I NTRODUCTION
Parametric speech synthesis based on Hidden Markov Models (HMMs) [1] is an effective framework for generating
diverse synthetic speech. In HMM-based speech synthesis,
speech parameters, i.e., spectral and excitation features and
HMM-state duration are simultaneously modeled with contextdependent HMMs in a unified framework. This approachallows us not only to produce smooth speech parameter
trajectories under a small footprint [2] but also to apply several
techniques for flexibly controlling them [3], [4], [5] to various
speech-based systems [6], [7].
One of the critical problems of HMM-based speech synthesis is that the trajectories generated from the trained HMMs areoften over-smoothed. This phenomenon causes the degradationof perceptual quality, and synthetic speech sounds muffled[8]. One approach to addressing this problem is to combine
a unit selection framework [9], [10], and the other approachis to enhance specific features not well reproduced from thetraditional HMMs due to the over-smoothing effect [11], [12].
The latter approach can achieve the production of high-quality
speech while preserving its small footprint. As one of themethods based on the latter approach, we have proposed theModulation Spectrum (MS)-based post-filter [13]. The MS is
known as a perceptual cues [14], [15], and the proposed postfilter can improve the quality by recovering the utterancelevel MS of the generated speech parameters. However the
post-filtering process needs to calculate the MS of the fixed978-1-4799-7088-9/14/$31.00 ©2014 IEEE

length of speech parameter trajectories, and therefore, it is notapplicable to various lengths of speech parameter trajectories,
such as phrases or segments. This constraint causes some
limitations; e.g., it prevents a recursive speech parameter generation algorithm [16] from being used for low-delay speech
waveform generation.
In this paper, we propose two modified post-filters capableof being widely used by relaxing the constraint: (1) the timeinvariant filter and (2) the segment-level post-filter. The timeinvariant filter makes the filtering process independent of thelength of the generated trajectories. The segment-level filter
achieves a segment-by-segment filtering process to recover
the MS of a shorter length of speech parameter trajectories
compared to the conventional utterance-level filter. Furthermore, to further improve naturalness of synthetic speech, (3)we propose the post-filter for HMM-state duration to recover
the MS of a phone-level duration sequence in a similar manner
to in the conventional post-filter. We evaluate performance of
the individual proposed methods separately to investigate theeffect of them on naturalness of synthetic speech.
II. PARAMETER G ENERATION
In synthesis, HMMs corresponding to input text are constructed from context-dependent HMMs build using natural
speech parameters in training. . After determining the HMMstate sequence q = [q1 , · · · , qT ] to maximize the duration
likelihood, the parameter trajectory is generated to maximize
HMM likelihood under a constraint on the relationship between static and dynamic features as follows:ĉ = argmax P (W c|q, λ) ,

(1)

c

]
[
⊤ ⊤
where c = c⊤
is a speech parameter vector
1 , · · · , cT⊤
sequence of T frames, ct = [ct (1) , · · · , ct (d) , · · · , ct (D)]is a D-dimensional parameter vector at frame t, d is adimensional index, W is the weighting matrix for calculating
the dynamic features [17], qt is a HMM-state index at framet, and λ is a HMM parameter set. To alleviate the oversmoothness of the generated parameters, Global Variance (GV)[11] can be also considered in parameter generation.
III. C ONVENTIONAL MS- BASED P OST-F ILTER [13]A. MS-based Post-Filtering Process
The MS s (c) is defined as a log-scaled power spectrum of
the temporal sequence c, which is calculated as

710

GlobalSIP 2014: Machine Learning Applications in Speech Processing

[

⊤

⊤

⊤

]⊤

s (c)

=

s (1) , · · · , s (d) , · · · , s (D)

s (d)

=

[sd (0) , · · · , sd (f ) , · · · , sd (Fs )] ,

,

⊤

(3)

where sd (f ) is the f -th MS of the d-th dimension of the⊤parameter sequence [c1 (d) , · · · , cT (d)] , f is a modulation
frequency index, Fs is a half number of the DFT length.
In synthesis, the speech parameter sequence generated fromthe HMM is transformed to the modulation frequency domain.
Then, its MS is converted as follows:s′d (f ) =(1 − k)sd (f )
[ (N)
]
)
σd,f ((G)(N)+ k
sd (f ) − µd,f + µd,f ,
(G)
σd,f

(·)

s′d (f )

(2)

(4)

(·)

where µd,f and σd,f are mean and standard deviation of sd (f ),N and G indicate of MS of the natural parameter and thegenerated speech parameter sequence, respectively. The MS
statistics are estimated in advance from natural and generated
speech parameter sequences for training data. The coefficient
k is a parameter to control the degree of emphasis, which
is determined manually. Finally, the filtered speech parametersequence is generated from the converted MS and its original
phase.
B. Problems
In [13], the MS is calculated utterance by utterance, The
DFT length for the MS calculation needs to be set large enoughto cover various lengths of utterances. This MS calculation
causes some problems: if the length of an utterance to besynthesized is longer than the previously determined DFTlength, the MS can not be calculated accurately; the utterancelevel filtering process is hard to be applied to a low-latency
speech synthesis frame work [18] where a frame-level orsegment-level processing based on the recursive parameter
generation [16] is essential.
Moreover, it has been reported that post-processing toenhance speech parameters, such as the GV-based parameter
generation, is also effective for not only spectral and F0
parameters but also HMM-state duration [19]. Although wehave applied the MS-based post-filter to only spectrum andF0 , it is worthwhile to also apply the MS-based post-filter tothe HMM-state duration and investigate its effectiveness.
IV. P ROPOSED M ODIFICATION M ETHODS FOR MS-BASEDP OST-F ILTER
To address the problems in the conventional MS-based postfilter, we propose two modification methods for the MS-based
post-filter. Moreover, we also propose the MS-based post-filter
for the HMM-state duration.
A. Method 1: Time-Invariant Post-Filter(N)
A time-invariant post-filter is derived by assuming that σd,f(G)
is equal to σd,f in Eq. (4) as follows:

[
]
(G)
(N)
(1 − k)sd (f ) + k sd (f ) − µd,f + µd,f
[
]
(N)
(G)
= sd (f ) + k µd,f − µd,f .
(5)=

Because the second term in R.H.S. is independent of sd (f ),this conversion process can be represented as a filtering
process for the generated speech parameter sequence with atime-invariant FIR filter.
B. Method 2: Segment-Level Post-Filter
A segment-level post-filter is derived by localizing the postfiltering process as illustrated in the left-hand side of Figure
1. A part of the speech parameter sequence that is windowed
by a triangular window with constant length is used as asegment to calculate the MS and its statistics. The window
shift length is set to a half of the window length. The MSbased post-filtering process is performed segment by segmentin the same manner as the conventional filtering. The filteredspeech parameter sequence is generated by overlapping andadding the filtered segments. The hanning window may also
be used instead of the triangular window. Note that for the
spectrum parameter, silence frames are removed in calculating
the MS statistics to alleviate the over-fitting problem [20]. For
F0 , continuous F0 pattern [21] is used [13]. The segment-levelpost-filtering can be applicable to the low-delay speech waveform generation. Moreover, it is possible to further implement
context-dependent post-filtering.
C. Method 3: MS-Based Post-Filter for Duration
Although the state duration is not an actual parameter
trajectory, it is affected by the over-smoothing effect due to astatistical averaging process as in spectrum and F0 parameters[22]. As illustrated in Figure 2, we can interestingly find the
MS degradation of the modulation frequency1 of phonemelevel duration sequences. Therefore, it is expected that quality
improvements in synthetic speech are yielded by recovering
their MS.
The overview of the proposed method is illustrated in
the right side of Figure 1. First, phoneme-level duration is
calculated from the determined state-level duration. Then, aphoneme-level duration sequence over an utterance is constructed by excluding the silence parts and its mean value isnormalized as in F0 parameters [13]. The resulting sequenceis used to calculate the MS and is also filtered in the samemanner as the conventional post-filtering. After restoring theutterance-level mean, the phoneme-level duration is revised if
it is smaller than the number of states of the phoneme HMM.
Finally, the HMM-state duration is updated by maximizing thestate duration while fixing the phoneme duration to the filteredvalues.
V. E XPERIMENTAL E VALUATIONS
A. Experimental Conditions
We trained a context-dependent phoneme Hidden SemiMarkov Model (HSMM) [23] for a Japanese female speaker.

711

1 Nyquist frequency is set to 1.0.

GlobalSIP 2014: Machine Learning Applications in Speech Processing'

!?A@CB! D*+ , )

;=<=>

7 &
/ 65+
34
12
0/ ) %-.
*,+()We used 450 sentences for training and 53 sentences forevaluation from phonetically balanced 503 sentences includedin the ATR Japanese speech database [24]. Speech signals
were sampled at 16 kHz. The shift length was set to 5 ms.
The 0th-through-24th mel-cepstral coefficients were extracted
as a spectral parameter and log-scaled F0 and 5 bandaperiodicity [25], [26] were extracted as excitation parameters.
The STRAIGHT analysis-synthesis system [27] was employedfor parameter extraction and waveform generation. The feature
vector consisted of spectral and excitation parameters and theirdelta and delta-delta features. Five-state left-to-right HSMMs
were used.

$#"!" 


$


Fig. 1. An overview of the proposed methods (left: the segment-level postfilter, right: the post-filter for duration).

The window length and window shift length were set to
125 ms (25 samples) [28] and 60 ms (12 samples), respectively. 64-taps FFT was used. We compared the following
speech samples:
HMM: original parameter generated by Eq. (1)   !#"

& #





&
0

#
((

.13

#"

4657598;:<5;=#"4657598?>A@?8B:C5;=-

       # # ,+./- " + ,*() "
&'
$%
"

DEGFHJIKEJL    !"

123254768279
123254;:"<=4768279%% "
+.0/.120 %$ #. /+, $)*'(  #


C. Evaluation 2: Segment-Level Post-Filter



 

Preference scores with
Fig. 2. Averaged MSs of phone-level Fig. 3.
duration sequences (“DUR”: gener- 95% confidence intervals (the timeinvariant post-filter).
ated duration).

B. Evaluation 1: Time-Invariant Post-Filter
To confirm the effect by the time-invariant filter, we conducted the subjective evaluation to compare the following
speech samples:
HMM: original parameter by Eq. (1)
HMM+MS (ti): parameters filtered by the time-invariant
filter
HMM+MS: parameters filtered by the conventional filter
From [13], the emphasis coefficient and DFT length were set to0.85 and 4096, respectively. We applied the MS-based postfilter to both spectrum and F0 . We conducted a preference
test (AB test) on speech quality. Every pair of three types ofsynthetic speech was presented to listeners in random order. 6
listeners were asked which sample sounds better in terms ofspeech quality.
The preference result is shown in Figure 3. We can see thata significant quality improvement is yielded by applying thetime-invariant post-filter to the generated speech parameters.
Although the improved quality is not comparable to thatyielded by the conventional post-filter, the time-invariant postfilter is applicable to various lengths of speech parametersequences.

 

(& ' "#%!$ 

:

*89

-

"

>@?BADCFE?FG   !

Fig. 4.
HMM likelihoods for the Fig. 5.
HMM likelihoods for thefiltered spectrum.
filtered F0 .

HMM+LMS: “HMM” parameters filtered by the segmentlevel filter
HMM+GV: parameters generated by Eq. (1) with the GV
HMM+GV+LMS: “HMM+GV” parameters filtered by thesegment-level filter
1) Tuning Emphasis Coefficient: We calculated the HMM
likelihood, GV likelihood, and MS likelihood for the filteredboth spectral parameters and F0 contours while varying theemphasis coefficient from 0 to 1. For comparison, the likelihood for natural speech parameters was calculated, which islabeled as “Natural.”
The results are shown in Figs. 4 to 9. Their tendencies
are similar to those of the conventional post-filter as reported
in [13]. The degradation of HMM likelihoods by the postfiltering process, but they are sill greater than that of naturalparameters. Almost likelihoods tend to increase as the filter
coefficient is close to 1. we observed the degradation of theMS likelihood for F0 but it is always greater than that ofnatural parameters. From these results, we tuned the emphasis
coefficient to 1.0 for both spectrum and F0 .
2) Subjective Assessment on Speech Quality: AB test using
the above 4 methods on speech quality by 7 listeners was

712

GlobalSIP 2014: Machine Learning Applications in Speech Processing

$&
1
((

$#

0
/231

/ 0.

,/0.

# $&

,-

)+*

'(

#
# %$&

    !"

, -+ # )*'( #
%&#

243537698:3<;

#     !"



;
15
/24

77:
#
798
6



)) #

JKMLNPOKPQ

$#%
0 "''.13/ " $#%'

0/231 # &
/ 0.
,- &'*+ #()#%$  


 

 
  




 


           

. /-456587:9;5=<
.120




'

)*

'( $ &
$ %

Fig. 8. MS likelihoods for the filtered Fig. 9. MS likelihoods for the filteredspectrum.
F0 .

conducted in the same manner as in the previous section. The
post-filtering was applied to both spectrum and F0 .
The preference score is shown in Figure 10. It is observedthat the significant quality gain is yielded by “HMM+LMS”
compared to “HMM,” and its comparable to that yielded by“HMM+GV.” Furthermore, we can see that the additional gain
is yielded by “HMM+GV+LMS” compared to “HMM+GV.”
This tendency is similar to that observed in the conventional
post-filter as reported in [13]. Please note that the segmentlevel post-filter is applicable to various lengths of a speechparameter sequence but the conventional one cannot.
D. Evaluation 3: MS-Based Post-Filtering for Duration
We evaluated the effectiveness of the post-filter for duration.
64-taps FFT was used. The spectrum and F0 is not filtered.
Compared speech samples are below:
DUR: original duration
DUR+MS: duration filtered by the proposed the post-filter
The duration likelihood and MS likelihood are shown in Figure
12 and Figure 13, respectively. We can see that the MS
likelihood increases as the filter coefficient is close to 1 while
preserving the duration likelihood high enough. Therefore, theemphasis coefficient was set to 1.0 in the subjective evaluation.

# %&;=<=>?A@CB1

0 # %
/231/ 0. # %&,*+ %&() ##&D&FEGHD!%@BACD@C

)) # % 
/24

$
+,

ACBEDF GB H
    !



#

456587?>%@?7:9;5=<

&'





Fig. 10. Preference scores with 95% Fig. 11. Preference scores with 95%
confidence intervals (local MS-based confidence intervals (post-filter for dupost-filter).
ration) .

. /-" $#%
    !"







+, " $#%(*)<>=?=A@CBD=FE:9 $ 
687
504 $ 
((.13 $ /.120

<=?=A@HG&IH@CBD=FE




!





 

! 

@ACBDFEAFG" $#%

%"# $ !

%# $ 
"

Fig. 6. GV likelihoods for the filtered Fig. 7. GV likelihoods for the filteredspectrum.
F0 .

# '*'( ) &

2435376>=#?>698:3<;. #$ 
&&,/1 $#-567678:9;6=</24

 

$#

5 6>678@?BA@8=9;6:<
CDFEG HD I

57698;:=<?># %$&     !"#

# %$      !"



Fig. 12. Duration likelihoods for the Fig. 13.
MS likelihoods for thefiltered duration.
filtered duration.

We can also see discontinuous transition of the MS likelihood.
We expect that this was caused by the effect of rounding thefiltered duration values into integer values after filtering.
The result of AB test by 6 listeners is shown in Figure 11.
We can see that te MS-based post-filter for duration tends toslightly improve speech quality.
VI. S UMMARY
This paper have proposed the modified Modulation Spectrum (MS)-based post-filters in HMM-based speech synthesis.
We have reported that the post-filters can avoid the conventional limitation while preserving the quality gain. Furthermore, we have applied the MS-based post-filter to phone-level
duration, and have yielded the effectiveness on speech quality.
We will investigate the benefits of the post-filter and MS itself
on various situation.
Acknowledgements: Part of this work was supported by
JSPS KAKENHI Grant Number 26280060 and Grant-in-Aid
for JSPS Fellows Grant Number 26 · 10354, and part of thiswork was executed under ”JSPS Strategic Young Researcher
Overseas Visits Program for Accelerating Brain Circulation.”

713

GlobalSIP 2014: Machine Learning Applications in Speech Processing

R EFERENCES[1] K. Tokuda, Y Nankaku, T. Toda, H. Zen, J. Yamagishi, and K. Oura.
Speech synthesis based on hidden markov models. Proceedings of theIEEE, Vol. 101, No. 5, pp. 1234–1252, 2013.
[2] K. Oura, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda. Tying covariance matrices to reduce the footprint of HMM-based speech synthesis
systems. In Proc. INTERSPEECH, pp. 1759–1762, Brighton, U. K.,
2009.
[3] T. Yoshimura, T. Masuko, K. Tokuda, T. Kobayashi, and T. Kitamura.
Speaker interpolation for HMM-based speech synthesis system. J.
Acoust. Soc. Jpn. (E), Vol. 21, No. 4, pp. 199–206, 2000.
[4] J. Yamagishi and T. Kobayashi. Average-voice-based speech synthesis
using HSMM-based speaker adaptation and adaptive training. IEICE
Trans., Inf. and Syst., Vol. E90-D, No. 2, pp. 533–543, 2007.
[5] T. Nose, J. Yamagishi, T. Masuko, and T. Kobayashi. A style control
technique for HMM-based expressive speech synthesis. IEICE Trans.,
Inf. and Syst., Vol. E90-D, No. 9, pp. 1406–1413, 2007.
[6] K. Shirota, K. Nakamura, K. Hashimoto, K. Oura, Y. Nankaku, andK. Tokuda. Integration of speaker and pitch adaptive training for
HMM-based singing voice synthesis. In Proc. ICASSP, pp. 2578–2582,
Florence, Italy, May 2014.
[7] J. Yamagishi, C. Veaux, S. King, and S. Renals. Speech synthesis
technologies for individuals with vocal diabilities: Voice banking andreconstruction. Acoust. Sci. technol., Vol. 33, pp. 1–5, 2012.
[8] S. King and V. Karaiskos. The blizzard challenge 2011. In Proc. Blizzard
Challenge workshop, Turin, Italy, Sept. 2011.
[9] Z. Ling, L. Qin, H. Lu, Y. Gao, L. Dai, R. Wang, Y. Jiang, Z. Zhao,
J. Yang, J. Chen, and G. Hu. The USTC and iFlytek speech synthesis
systems for blizzard challenge 2007. In Proc. Blizzard Challenge
workshop, Bonn, Germany, Aug. 2007.
[10] S. Takamichi, T. Toda, Y. Shiga, S. Sakti, G. Neubig, and S. Nakamura.
Parameter generation methods with rich context models for high-quality
and flexible text-to-speech synthesis. IEEE Journal of Selected Topics
in Signal Processing, Vol. 8, No. 2, pp. 239–250, May 2014.
[11] T. Toda and K. Tokuda. A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis. IEICE
Trans., Vol. E90-D, No. 5, pp. 816–824, 2007.
[12] T. Nose, V. Chunwijitra, and T. Kobayashi. A parameter generation
algorithm using local variance for HMM-based speech synthesis. IEEE
Journal of Selected Topics in Signal Processing, Vol. 8, No. 2, pp. 221–
228, 2014.
[13] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura. A postfilter to modify modulation spectrum in HMM-based speech synthesis.
In Proc. ICASSP, pp. 290–294, Florence, Italy, May 2014.
[14] R. Drullman, J .M. Festen, and R. Plomp. Effect of reducing slow
temporal modulations on speech reception. J. Acoust. Soc. of America,
Vol. 95, pp. 2670–2680, 1994.
[15] S. Thomas, S. Ganapathy, and H. Hermansky. Phoneme recognition
using spectral envelop and modulation frequency features. In Proc.
ICASSP, pp. 4453–4456, Taipei, Taiwan, April 2009.
[16] K. Tokuda, T. Kobayashi, and S. Imai. Speech parameter generationfrom HMM using dynamic features. In Proc. ICASSP, pp. 660–663,
Detroit, U.S.A, May 1995.
[17] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura.
Speech parameter generation algorithms for HMM-based speech synthesis. In Proc. ICASSP, pp. 1315–1318, Istanbul, Turkey, June 2000.
[18] T. Baumann and D. Schlangen. INPRO iSS A component for just-intime incremental speech synthesis. Proc. ACL, pp. 103–108, Jul. 2012.
[19] S. Pan, Y. Nankaku, K. Tokuda, and J.Tao. Global variance modelinf
on the log power spectrum of lsps for HMM-based speech synthesis. In
Proc. ICASSP, pp. 4716–4719, Prague, Czech Republic, 2011.
[20] H. Zen and A. Senior. Deep mixture density networks for acoustic
modeling in statistical parametric speech synthesis. In Proc. ICASSP,
pp. 3872–3876, Florence, Italy, May 2014.
[21] K. Yu and S. Young. Continuous F0 modeling for HMM based statisticalparametric speech synthesis. IEEE Trans. Audio, Speech and Language,Vol. 19, No. 5, pp. 1071–1079, 2011.
[22] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura.
Simultaneous modeling of spectrum, pitch and duration in HMM-based
speech synthesis. In Proc. EUROSPEECH, pp. 2347–2350, Budapest,
Hungary, Apr. 1999.

[23] H. Zen, K. Tokuda, T. Kobayashi T. Masuko, and T. Kitamura. Hidden
semi-Markov model based speech synthesis system. IEICE Trans., Inf.
and Syst., E90-D, No. 5, pp. 825–834, 2007.
[24] Y. Sagisaka, K. Takeda, M. Abe, S. Katagiri, T. Umeda, andH. Kuawhara. A large-scale Japanese speech database. In ICSLP90,
pp. 1089–1092, Kobe, Japan, Nov. 1990.
[25] H. Kawahara, Jo Estill, and O. Fujimura. Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation
for a high quality speech analysis, modification and synthesis system
STRAIGHT . In MAVEBA 2001, pp. 1–6, Firentze, Italy, Sept. 2001.
[26] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano. Maximum likelihood
voice conversion based on GMM with STRAIGHT mixed excitation. In
Proc. INTERSPEECH, pp. 2266–2269, Pittsburgh, U.S.A., Sep. 2006.
[27] H. Kawahara, I. Masuda-Katsuse, and A. D. Cheveigne. Restructuring
speech representations using a pitch-adaptive time-frequency smoothing
and an instantaneous-frequency-based F0 extraction: Possible role of arepetitive structure in sounds. Speech Commun., Vol. 27, No. 3–4, pp.
187–207, 1999.
[28] V. Tyagi, I. McCowan, H. Misra, and H. Bourlard. Mel-cepstrum
modulation spectrum (MCMS) features for robust ASR. Proc. ASRU,
pp. 399–404, Nov. 2003.

714