2-2-8Articulatory Controllable Speech Modification based on Gaussian Mixture
Models with Direct Waveform Modification using Spectrum Diﬀerential ∗☆ Patrick Lumban Tobing，Kazuhiro Kobayashi, Tomoki Toda，Graham Neubig，Sakriani Sakti，Satoshi Nakamura (NAIST)

1Introduction

The mapping systems to model the relationshipbetween articulatory parameters and acoustic parameters with Gaussian mixture models (GMMs)
are capable of achieving acoustic-to-articulatory inversion mapping and articulatory-to-acoustic production mapping without any text information [1].
We have recently developed an articulatory controllable speech modification system by combining these
two mapping systems [2]. Our system enables tomodify input speech waveform by manipulating unobserved articulatory parameters. However, significant quality degradation in the generated speech
waveform is caused by using the vocoder-based
waveform generation framework, which easily suffers from errors in the spectral and source excitation
parameter extraction and modeling.
In this work, we propose the articulatory controllable speech modification system based on a direct
waveform modification technique [3] to avoid using
the vocoder-based waveform generation. We conduct subjective evaluations, demonstrating that theproposed system can significantly improve quality of
the generated speech waveform.

2GMM-based articulatory controllable speech modification

Our previously proposed articulatory controllable
speech modification system is shown in top of Fig.
1. First, an input speech waveform is analyzed into
its mel-cepstrum parameters and source excitation
parameters. The mel-cepstral segments are then extracted from the mel-cepstrum parameters at multiple frames. Then, the articulatory parameters are
estimated from the mel-cepstral segments by theGMM for the inversion mapping. These articulatory
parameters are then manipulated as we want while
also revising unmanipulated articulatory parameters
by considering inter-dimensional correlation among
the articulatory parameters [2]. The manipulated
articulatory parameters together with the source excitation parameters are then converted into the corresponding mel-cepstrum parameters by the GMM
for the production mapping. Finally, speech waveform is generated from the converted mel-cepstrum
and the natural source excitation parameters by using vocoder.

3Implementation of waveform modification with spectrum diﬀerential

The conventional framework using the vocoderbased waveform generation is sensitive to extraction∗Fig. 1 Flow of the conventional system and theproposed systems
and modeling errors of the excitation and spectral
parameters. To address this issue, the proposed
framework directly modifies the input speech waveform according to the spectrum diﬀerential between
the target and input speech waveforms through afiltering process. A sequence of spectral diﬀerentials then needs to be estimated. In this paper, we
propose three diﬀerent estimation methods.
Fig. 1 also illustrates the processing of the three
diﬀerent proposed systems. The first proposed system (Type 1) calculates the spectral diﬀerentialfrom the modified mel-cepstrum and the originalmel-ceptrum extracted from the input speech waveform. This method may suﬀer from mismatches between the statistically estimated mel-cepstrum andthe extracted mel-cepstrum. To avoid them, thesecond proposed system (Type 2) calculates thespectral diﬀerential from the modified mel-cepstrum
and the converted mel-cepstrum estimated from theunmodified articulatory parameters. On the other

混合正規分布モデルと波形変形処理に基づく調音制御を可能とする音声変換 by Patrick Lumban Tobing, 小林 和弘，戸田 智基，Graham Neubig，Sakriani Sakti，中村 哲（奈良先端大）日本音響学会講演論文集- 267 -

2015年3月

Fig. 2 Mean opinion scores (MOS) result for onefold scaling value (normal articulation)
hand, the third proposed method (Type 3) analytically determines a diﬀerential GMM on a joint
static and dynamic feature space from the GMM for
estimating the modified mel-cepstrum and that for
estimating the unmodified mel-cepstrum, and thenthe spectral diﬀerential is generated from the modified GMM. Both the second method and the third
method calculate the spectral diﬀerential from thestatistically estimated mel-cepstra without using theextracted mel-cepstrum, but the second method determines the diﬀerential GMM on a static feature
sequence space rather than the joint static and dynamic feature space.

4Experimental evaluation

4.1 Experimental conditions
We used a set of simultaneously recorded speech
and articulatory data set with one British malespeaker provided in MOCHA [4]. The speech data
was sampled at 16 kHz. STRAIGHT [5] was used
to extract spectral envelope at each frame. The 1stthrough 24th mel-cepstral coeﬃcients were used as
the spectral parameters. A current ± 10 frames
were used to extract mel-cepstral segments as thesource feature for the inversion mapping. As the articulatory parameters, we used the 14-dimensional
EMA data in Z-scores. The frame shift was 5 ms.
The MLSA filter [6] was used as the synthesis filter in both the vocoder-based waveform generationand the direct waveform modification. We used 350
sentences for training and the remaining 110 sentences for evaluation. Two GMMs were trained separately for the inversion and production mappings.
The number of mixture components was set to 64
for both mappings. Note that the global variance(GV) [7] is only considered in the baseline system.
A subjective evaluation was conducted to compare
the speech quality of the conventional and all three
proposed methods described in Section 3 by using
mean opinion score (MOS) test. All 14-dimensionsof the articulatory data were scaled with three diﬀerent values, one-fold (normal articulation), half-fold(hypo-articulated) and two-fold (hyper-articulated)
to evaluate the performance of individual methods
in various modification settings. The number of listeners was 12 and each listener evaluated 96 synthetic speech samples consisting of 8 diﬀerent sentences, which were generated by each system andeach scaling setting.

日本音響学会講演論文集

Fig. 3 Mean opinion scores (MOS) result for halffold scaling value (hypo-articulated)

Fig. 4 Mean opinion scores (MOS) result for twofold scaling value (hyper-articulated)
4.2 Experimental results
Figs. 2, 3, and 4 show the MOS results forone-fold scaling value, half-fold scaling value, andtwo-fold scaling value, respectively. These results
show that the Type 2 and Type 3 methods can significantly improve quality of the generated speechcompared to the conventional method. These twomethods can produce similar highest average scores
across scaling values.

5Conclusion

We have proposed the GMM-based articulatory
controllable speech modification systems with direct waveform modification using spectrum diﬀerential. The results of the subjective evaluations
have demonstrated that the proposed systems yieldsignificant quality improvements in the generated
speech waveform by avoiding using the vocoderbased waveform generation. We plan to conduct
further evaluations in terms of controllability ofphoneme sounds.
Acknowledgements This research was supported inpart by JSPS KAKENHI Grant Number 22680060.

References[1] T. Toda et al., Speech Communication, vol. 50,
No. 3, pp. 215-227, 2008.
[2] P. L. Tobing et al., Proc. INTERSPEECH, pp.
2298-2302, 2014.
[3] K. Kobayashi et al., Proc. INTERSPEECH, pp.
2514-2518, 2014.
[4] A. Wrench, http://www.cstr.ed.ac.uk/artic/mocha.html, 1999.
[5] H. Kawahara et al., Speech Communication, Vol.
27, No. 3–4, pp. 187–207, 1999.
[6] S. Imai et al., Electron. Comm. Jpn., Vol. 66,
No. 2, pp. 10–18, 1983.
[7] T. Toda et al., IEEE Trans. ASLP, vol. 15, No.
8, pp. 2222-2235, 2007

- 268 -2015年3月