2-1-3
An Evaluation of Articulatory Controllable Speech Modification based onGaussian Mixture Models with Direct Waveform Modification ∗◎ Patrick Lumban Tobing, Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig,
Sakriani Sakti, Satoshi Nakamura (NAIST)

1

Introduction

Statistical inversion and production mapping systems based on Gaussian mixture models (GMMs) [1]have a great potential to be developed into various
new applications of speech modification. As a firststep, we have developed an articulatory controllablespeech modification system based on GMMs [2] bysequentially performing the acoustic-to-articulatory
inversion mapping and articulatory-to-acoustic production mapping. This system makes it possible
to modify an input speech signal by manipulating unobserved articulatory parameters, with several implementations of direct waveform modification method [4]. In this paper, in order to continue
our previous evaluation for a male speaker [4], we
conduct additional experiments to evaluate both thespeech quality and the controllability of articulatory
parameters for a female speaker.

2Articulatory controllable speech
modification based on GMMs withdirect waveform modification

In the GMM-based articulatory controllable
speech modification system [2], given an input
speech signal, spectrum parameters are extracted
and articulatory parameters are estimated using theGMM-based inversion mapping. Then, the articulatory parameters are modified as we want while
also revising unmodified parts of them by considering their inter-dimensional correlation [2]. Next,
corresponding spectrum parameters are estimatedfrom the modified articulatory parameters using theGMM-based production mapping. Finally, the modified speech waveform is synthesized with a vocoderbased waveform generation process in the conventional system (Conv). However, in the conventional
system, the quality of the modified speech is degraded compared to the input natural speech due toparameterization errors of spectral and source excitation features in the vocoder-based framework.
To address this issue, we have developed an articulatory controllable speech modification system basedon direct waveform modification method [3], capableof avoiding the use of vocoder-based excitation generation by directly filtering the input speech waveform according to spectrum diﬀerential parameters[4]. We have proposed three kinds of implementation to estimate the spectrum diﬀerential parameters composed by diﬀerence values between the spectrum parameters of the modified speech waveform
and that of the input speech waveform. In the basic∗method (DiﬀBM), spectrum diﬀerences are calculated between estimated spectrum of the modified
speech and natural spectrum of the input speech.
Thus, in the filtering process, the natural spectrum
is eliminated, giving only the oversmoothed spectrum of the modified speech. In the refined method(DiﬀRM), the diﬀerences are calculated between
estimated spectrum of the modified speech and estimated spectrum of the input speech waveform,
which is determined by performing the inversion andproduction mappings without any modification of
the articulatory parameters. Because both the estimated spectra are oversmoothed, the resulting spectral diﬀerentials are also oversmoothed. Hence, inthe filtering process, fine structure of spectrum of
the input speech is still kept, which alleviates theover-smoothing problem. The third method (DiffGMM) is essentially similar to the DiﬀRM but it
directly estimates the over-smoothed spectrum differentials using a diﬀerential GMM instead of performing the production mapping twice.

3

Investigation of eﬀectiveness ofproposed system in typical applications

In this paper, we investigate two typical applications of the proposed articulatory controllable
speech modification system: control of articulation
conditions and control of phonetic sounds. In thefirst application, we investigate the speech quality ofmodified speech signal in hypo and hyper articulation conditions. In the second application, we investigate the capability of modifying phonetic sounds
through manipulation of articulatory parameters.
3.1 Control of hypo and hyper articulated
speechHypo and hyper articulated speeches are produced with little eﬀort of articulations (e.g. talk in avery close distance to someone) and with more eﬀort(e.g. in noisy environment), respectively [5]. Therefore, it is expected that hypo articulated speech isgenerated by reducing the articulatory movements
and hyper articulated speech is generated by increasing them. We have reported that the proposedsystem is capable of generating high speech quality of hypo and hyper articulated speech for a malespeaker by scaling the degree of articulation. In thispaper, we further evaluate it for a female speaker.
3.2 Control of phonetic sounds
Capability of the proposed system in modifying phonetic sounds through articulatory manipu-

時間波形変形処理と混合正規分布モデルに基づく調音制御機能付き音声変換法の評価 by Patrick LumbanTobing, 小林 和弘，戸田 智基， Graham Neubig， Sakriani Sakti，中村 哲（奈良先端大）日本音響学会講演論文集- 221 -2015年9月

Mean Opinion Score

5595% Confidence Interval

595% Confidence Interval

95% Confidence Interval

44433322211Conv

DiffBM DiffRM DiffGMM

1Conv

DiffBM

DiffRM DiffGMM

Conv

DiffBM

DiffRM DiffGMM

Perception percentage (%)

Fig. 1 Mean Opinion Scores (MOS) on three diﬀerent degree of articulations for female speaker, hypoarticulation (left), normal articulation (centre), and hyper-articulation (right)
100/I//e//ae/

80
60
40
200+1.0 +0.5 0 -0.5 -1.0
Tongue height relative position (cm)Fig. 2 Perception percentage of the phoneme modification results for the female speakerlation makes it possible to be implemented in manyapplications, e.g. language learning, speech recovery, phonetic research, etc. We have reported
that the proposed system is capable of modifying
vowel sounds of a male speaker by manipulating thetongue movements [4]. In this paper, we also further
evaluate it for a female speaker.

4

Experimental evaluation

4.1 Experimental conditions
We used a set of simultaneously recorded speech
and articulatory data of one British female speaker,
which is used also in [1]. STRAIGHT analysis [6]was used to extract spectral envelopes which wereconverted into the 1–24th mel-cepstral coeﬃcientsas the spectrum parameters. As the articulatory
parameters, 14-dimensional EMA data in Z-scores
were used. Two GMMs were trained separately forthe inversion and production mappings.
We conducted a speech quality evaluation using
the female speaker’s data to compare the conventional vocoder-based system and three direct waveform modification-based systems described in Section 2. Three degree of articulations were employed by scaling the movements of all articulators:
normal (1.0x), hypo-articulation (0.5x), and hyperarticulation (2.0x). 12 listeners were asked to evaluate quality of 96 speech samples each, consisted of
8 diﬀerent utterances for all four systems and threespeaking conditions, in an opinion test.
We also conducted a controllability evaluation by
performing modification of three English front vowels, by shifting only the tongue height position.
Original vowel /E/ has the middle position, vowel/I/ is in the highest position, and /æ/ has the lowest
tongue position. 12 words containing vowel /E/ werechosen from evaluation data. Five modifications ofthe tongue height position were employed, ranged

日本音響学会講演論文集

from -1.0 cm to +1.0 cm. 10 listeners were asked toevaluate phonetic sounds of the modified vowel partsby selecting the most similar vowel among /I/, /E/,
and /æ/. Note that time frames corresponding to
the target vowels /I/ and /æ/ were removed fromthe training data set of the GMMs. DiﬀGMM system was evaluated using the female speaker’s data.
4.2 Experimental results
Figure 1 and Fig. 2 show the results of the speech
quality evaluation and the phoneme modification,
respectively. The speech quality evaluation result
indicates that hypo and hyper articulated speeches
are perceived with a reasonably high quality withthe use of DiﬀRM and DiﬀGMM. Moreover, thephoneme modification result also demonstrates atendency of vowel transition, as the tongue height
position is changed, which validates the controllability of articulatory movements of the system. Theseresults, which are quite important for this evaluation paper, suggest the capability of the system toperform suitable practices in speech applications.

5Conclusion

We have conducted experimental evaluations toinvestigate the eﬀectiveness of the proposed articulatory controllable speech modification system basedon direct waveform modification. The experimental results have showed that the proposed system
is capable of yielding high speech quality in various degree of articulations and also modifying vowel
sounds by simply manipulating articulatory movements. We will further investigate a way of manipulating articulatory parameters to develop various
speech modification applications.
Acknowledgements Part of this work wassupported by JSPS KAKENHI Grant Number
26280060.

References[1] T. Toda, et al., Speech Communication, Vol. 50,No. 3, pp. 215–227, 2008.
[2] P. L. Tobing, et al., Proc. INTERSPEECH, pp.
2298–2302, 2014.
[3] K. Kobayashi, et al., Proc. INTERSPEECH, pp.
2514–2518, 2014.
[4] P. L. Tobing, et al., Proc. INTERSPEECH,2015.
[5] B. Picart, et al., Neurocomputing, Vol. 132, pp.
142-147, 2014.
[6] H. Kawahara et al., Speech Communication, Vol.
27, No. 3–4, pp. 187–207, 1999.

- 222 -2015年9月