INTERSPEECH 2014

Articulatory Controllable Speech Modification Based on Statistical Feature
Mapping with Gaussian Mixture Models
Patrick Lumban Tobing1,2 , Tomoki Toda1 , Graham Neubig1 , Sakriani Sakti1 ,Satoshi Nakamura1 , Ayu Purwarianti2
1

Graduate School of Information Science, Nara Institute of Science and Technology, Japan
2
STEI, Institut Teknologi Bandung, Indonesia

13510013@std.stei.itb.ac.id, tomoki@is.naist.jp, neubig@is.naist.jp, ssakti@is.naist.jp,
s-nakamura@is.naist.jp, ayu@stei.itb.ac.id

Abstract

speech production mechanism is too complex to be mathematically modeled without some approximations.
Recently,some research has examinedstatistical approaches
that do not mathematically model the speech production mechanism. These mapping systems between articulatory parametersand speech acoustics are developed in a data-driven manner using parallel acoustic-articulatory data.There have been proposedseveral statistical methods, e.g.,the mapping system using codebooks [8, 9], hidden Markov models (HMMs) [10, 11], neuralnetworks [12, 13], and Gaussian mixture models (GMMs) [1],and their effectiveness has been confirmed in both the inversion
and production mapping. Moreover, it has been reported thatphoneme sounds of synthetic speech are effectively modified
by manipulating the articulatory parameters in articulatory controllable HMM-based text-to-speech synthesis, where the articulatory parameters are modeled as intermediate features [6].
Inspired by the conventional work [1, 6], we propose
a novel articulatory controllable speech modification system.
Specifically we do so by developing a new speech analysis/synthesis framework by combining the inversion and production mapping to make it possible to modify speech signals
by manipulating the unobserved articulatory parameters. Sucha framework has a great potential to develop various new speech
applications, such as speech recovery for vocally disabled people, pronunciation enhancement in speaking foreign languages,
and concealing messages by modifying phonemes/words.
In this paper, we focus on the GMM-based inversion/production mapping methods [1] as one of the promising methods capable of easily being applied to any language.
Thanks to its independence of text/language specification input,
which is needed by [6], as only speech signals are needed as
the input of this system. In the proposed system, the articulatory parameters are first estimated from a given input speech
signal using a GMM-based inversion mapping system. These
articulatory parameters are manipulated, and then the acoustic
parameters are estimated from the manipulated articulatory parameters using a GMM-based production mapping system. Finally, a modified speech signal is generated from the estimatedacoustic parameters. We also propose an articulatory manipulation method for refining unmodified parts of the articulatory
parameters according to the modified parts by considering theirinter-dimensional correlation.

This paper presents a novel speech modification method capable of controlling unobservable articulatory parameters basedon a statistical feature mapping technique with Gaussian Mixture Models (GMMs). In previous work [1], the GMM-based
statistical feature mapping was successfully applied to acousticto-articulatory inversion mapping and articulatory-to-acoustic
production mapping separately. In this paper, these two mapping frameworks are integrated to a unified framework to develop a novel speech modification system. The proposed systemsequentially performs the inversion and the production mapping, making it possible to modify phonemic sounds of an input
speech signal by intuitively manipulating articulatory parameters estimated from the input speech signal. We also propose amanipulation method to automatically compensate for unmodified articulatory movements considering inter-dimensional correlation of the articulatory parameters. The proposed system isimplemented for a single English speaker and its effectivenessis evaluated experimentally. The experimental results demonstrate that the proposed system is capable of modifying phonemic sounds by manipulating the estimated articulatory movements and higher speech quality is achieved by considering theinter-dimensional correlation in the manipulation.
Index Terms: speech modification, acoustic-to-articulatory inversion mapping, articulatory-to-acoustic production mapping,Gaussian mixture model, inter-dimensional correlation

1. Introduction
Articulators are a set of human speech organs that are used ina unified way to control the resonance characteristics of the vocal tract. Therefore, speech can be characterized by articulatory
parameters, such as movements of the articulators. Because thearticulatory parameters vary much more slowly than the acoustic parameters of speech [2], they have a potential to yield betterparameterization of speech in many applications such as speechcoding [3], speech recognition [4], and speech synthesis [5].
Furthermore, speech is more easily modified in an understandable way by manipulating articulatory parameters rather thanacoustic parameters [1, 6].
There have been many attempts at developing mapping systems between the speech acoustic parameters and the articulatory parameters [1, 3, 6, 7, 8, 9, 10, 11, 12, 13]. There aremainly two mapping systems: one is an acoustic-to-articulatory
inversion mapping system to estimate the articulatory parameters from the given acoustic parameters and the other is anarticulatory-to-acoustic production mapping system to estimate
the acoustic parameters from the given articulatory parameters[1]. One of the typical approaches to these mapping systems isbased on mathematical production models [3, 7]. However, theCopyright © 2014 ISCA

2. GMM-based Inversion and Production
Mapping [1]A simultaneously recorded speech and articulatory data set isused as training data to construct the GMMs for the inversion and production mapping. In this paper, we use speakerdependent GMMs. For articulatory parameters, we use 14-2298

14- 18 September 2014, Singapore

function using a single mixture component sequence m ={m1 , · · · , mT } [15] is employed, where mt shows the mixture component index at frame t. First, the suboptimum mixture
component sequence m̂(O) is determined as follows:m̂(O) = arg max P m|O, λ(O,X) .
(3)

dimensional Electromagnetic articulograph (EMA) data, which
are provided in MOCHA [14]. Locations of seven articulators(top lip, bottom lip, bottom incisor, tongue tip, tongue body,tongue dorsum, and velum) are measured in x- and y- coordinates on the midsagittal plane.
Let ct , st , and xt be spectral envelope parameters (i.e.,
mel-cepstrum in this paper), source excitation parameters (i.e.,
log-scaled F0 and log-scaled waveform power in this paper),and the articulatory parameters. Time sequence vectors of> >these parameters over an utterance are c = c>,
1 , · · · , cT
 >



> >
>
> >
s = s1 , · · · , sT , and x = x1 , · · · , xT , respectively,
where T is the number of frames and > denotes the transposition of the vector.

mThen, the converted articulatory parameter sequence vector x̂ isdetermined as follows:x̂ = arg max P X|O, m̂(O) , λ(O,X) ,(4)xsubject to X = W (x) x,

where Wis a linear transform to expand the articulatory parameter sequence vector x into its joint static and dynamic feature sequence vector X.

2.1. Acoustic-to-articulatory inversion mapping
In the inversion mapping, spectral envelope parameters of an
input speech signal (the source features) are converted to thecorresponding articulatory parameters (the target features).

2.2. Articulatory-to-acoustic production mapping
In the production mapping, the spectral envelope parameters are
determined from both the articulatory parameters and the excitation parameters.

2.1.1. Source and target features in inversion mapping
The source features consist of a mel-cepstral segment feature vector extracted from mel-cepstrum parameters at multiple frames around the current frame. The mel-cepstral segment
feature vector at frame t is denoted as O t , which is given by
Ot

=

h
i>
>
>
A c>
+ b,
t−L , · · · , ct , · · · , ct+L

2.2.1. Source and target features in production mapping
As the source features, a joint static and dynamic feature vector including not only the articulatory parameters but also thesource excitation parameters is used, which is given by Y t =

 > >
> >
at frame t. On the other hand, as thext , st , ∆x>t , ∆sttarget features, a joint static and dynamic feature vector of the> >mel-cepstrum C t = c>is used at frame t.
t , ∆ct(1)

where linear transformation parameters A and b are determinedwith principal component analysis for the training data in advance. On the other hand, a joint static and dynamic feature
vector of the articulatory parameters is used as the target fea
> >, where ∆xt is theture, which is given by X t = x>t , ∆xt
dynamic feature vector of the articulatory parameters at framet.

2.2.2. Training process
The training process is basically the same as described in Section 2.1.2. After constructing the joint source and target feature
vectors in the training data, the joint probability density function of the source and target features is modeled with the GMM
for the production mapping as follows:
P Y t , C t |λ(Y,C)

2.1.2. Training process in inversion mapping> >A joint source and target feature vector [O >is cont , Xt ]structed at each frame in the training data. Then, the joint probability density function of the source and target features is modeled with the GMM for the inversion mapping as follows:
P O t , X t |λ(O,X)

=

M
X

(O,X)
αm
N

h

>
O>
t , Xt

i>

(5)

(x)

=

M
X

(Y,C)
αm
N

h

>
Y>
t , Cti>; µ(Y,C), Σ(Y,C)
,
m
m

(6)

m=1

where λ(Y,C) denotes a parameter set of the GMM for the
production mapping, which consists of mixture-component(Y,C), and full covariance maweights αm , mean vectors µ(Y,C)m(Y,C)
of individual mixture components.
trices Σm


(O,X)
, (2)
; µ(O,X)
,
Σ
m
m

m=1

2.2.3. Conversion process

where N (·; µ, Σ) is a Gaussian distribution with a mean vector µ and a covariance matrix Σ. λ(O,X) denotes a parameterset of the GMM for the inversion mapping, which consists of
(O,X)
(O,X)
mixture-component weights αm, mean vectors µm
, and
(O,X)
full covariance matrices Σmof individual mixture components. The mixture component index is m. The total number of
mixture components is M .

The conversion process is also basically the same as describedin Section 2.1.3. Given a time sequence of the source feature
vectors Y , that of the converted mel-cepstrum parameters ĉ isdetermined as follows:m̂(Y ) = arg max P m|Y , λ(Y,C) ,
(7)
m


ĉ = arg max P C|Y , m̂(Y ) , λ(Y,C)(8)

2.1.3. Conversion processcsubject to C = W (c) c,

Given a time sequence of the mel-cepstral segment feature
vectors O, a time sequence of the articulatory parameters x
is determined bymaximizing the conditional probability density function P X|O, λ(O,X) , which is analytically derived from the GMM for the inversion mapping. In this paper, an approximation of the conditional probability density

(c)

(9)

where Wis a linear transform to expand the static melcepstrum sequence vector c into its joint static and dynamic
feature sequence vector C. Note that the global variance (GV)[15] is also considered in the production mapping to improvethe converted speech quality.

2299

3. Articulatory Controllable Speech
Modification
The proposed articulatory controllable speech modification process is shown in Figure 1. First a given input speech signal is analyzed to extract speech acoustic parameters, such as
mel-cepstrum parameters c and the source excitation parameters s including waveform power and F0 . Then, the inversion mapping is performed to determine the estimated articulatory parameters x̂ corresponding to the given input speech
signal from the mel-cepstral segment features O as describedin Section 2.1. Next, the estimated articulatory parameters
are modified manually, e.g., scaling movements of some articulators or changing positions of some articulators to modify
phoneme sounds. After that, the production mapping is performed to determine the estimated mel-cepstrum parameters ĉ
corresponding to the modified articulatory parameters x̂0 andthe extracted source excitation parameters s in the manner described in Section 2.2. Finally, the modified speech signal isgenerated from the estimated mel-cepstrum parameters ĉ andthe extracted source excitation parameters s using a vocoder.
In the manipulation of articulatory parameters, it is convenient to manually control movements of a limited number ofarticulators, e.g., only the movement of the tongue tip, rather
than to manually control all articulators simultaneously. In thispaper, we implement two manipulation methods to do so.

Figure 1: Proposed speech modification process(u)that of xt is equivalent to D. In the second stage inversion
mapping, the unmodified articulatory parameter sequence vector is determined as follows:(m)0x̂(u) = arg max P X (u) |O, X̂
, m̂(O) , λ(O,X) ,(10)
x(u)
(u)

subject to X (u) = W (x)x(u) ,(11)(u)

where W (x ) is a linear transform to expand the unmodified
articulatory parameter sequence vector x(u) into its joint staticand dynamic feature vector sequence X (u) . The mixture component sequence m̂(O) is given by Eq. (3).
The conditional probability density function used in the inversion mapping effectively models inter-dimensional correlation of the articulatory parameters with the mixture-dependent
full covariance matrices. Therefore, the unmodified articulatory
parameters are automatically revised in Eq. (10) according to
the modified articulatory parameters. Note that the inter-frame
correlation of the articulatory parameters is also considered inthis revision due to the trajectory-based conversion framework[1] using an explicit relationship between the static and dynamic
features shown in Eq. (11). Consequently, it is expected that
this manipulation method will yield more natural movements of
the articulatory parameters compared to the simple manipulation method.

3.1. Simple manipulation method
The articulatory parameters at frame t estimated by the inversion mapping are denoted as the D-dimensional vector x̂t =[x̂t (1) , · · · , x̂t (D)]> . Then, the manipulated articulatory parameters x̂0t are defined by changing only components corresponding to the movements of target articulators to be manipulated; e.g., if only the first and second dimensional components
are changed to x̂0t (1) and x̂0t (2), respectively, x̂0t is given by>[x̂0t (1) , x̂0t (2) , x̂t (3) , · · · , x̂t (D)] .
This method is capable of easily manipulating only the
movements of the target articulators. However, because movements of some articulators are strongly correlated to each other[16], e.g., the movements of the tongue tip affects those of the
tongue body, this method possibly causes unnatural movementsof the articulators.

4. Experimental Evaluation
4.1. Experimental conditions
As a simultaneously recorded speech and articulatory data set,
we used one British male speaker’s data in MOCHA [14].
Speech data was sampled at 16 kHz. EMA data was used as
the articulatory data.
In speech acoustic parameter extraction, we used the
STRAIGHT analysis method [17] to calculate the spectral envelope at each frame. It was then converted into the 1st through24th mel-cepstral coefficients as the spectral envelop parameters. The current ± 10 frames were used to extract the melcepstral segments for the inversion mapping as described in
Section 2.1.1. For the source excitation parameters, we used
log-scaled F0 values also including an unvoiced/voiced binary
decision feature and log-scaled power values extracted fromthe STRAIGHT spectrum. The fixed-point analysis [18] in
STRAIGHT was employed to extract F0 values. For the articulatory parameters, we used 14-dimensional EMA data as briefly
explained in Section 2. These 14-dimensional articulatory feature vectors were converted to a Z-score (zero mean and unit
variance). Frame shift was set to 5 ms.
We used 350 sentences for training and the remaining
110 sentences for evaluation. The silence frames were removed using phonetic segmentation information included in the
MOCHA. We trained two GMMs separately for the inversion
mapping and the production mapping, as described in Sections
2.1 and 2.2. The number of mixture components was optimized

3.2. Manipulation method considering inter-dimensionalcorrelation of articulatory parameters
To consider the inter-dimensional correlation of the articulatory
parameters, we propose a second manipulation method based on
two stage inversion. After the inversion mapping and the simple manipulation of the articulatory parameters as mentioned
above, the modified components of the articulatory parameters
are appended to the source features. Then, the second stage inversion mapping is performed to refine the other componentsof the articulatory parameters using the conditional probability
density function derived from the GMM for the inversion mapping.
The modified articulatory parameter vector consisting of
only the manually modified components at frame t is given(m)0
by x̂t . A time sequence vector of the joint static and dy(m)0

namic feature vectors is given by X̂
. On the other hand,the unmodified articulatory parameter vector consisting of the(u)other components at frame t is given by xt and a time sequence vector of the joint static, and dynamic features is given(m)0
by X (u) . The sum of the number of dimensions of x̂t
and

2300

Figure 2: Mean Opinion Score (MOS) test result of the qualityof modified synthetic speech from both manipulation methods

Figure 3: Perception percentage of vowel in modified words
resulting from manipulation of tongue’s height position

so that estimation accuracy in each mapping was maximized
for the test data. The resulting numbers of mixture components were 64 for both the inversion mapping and the production
mapping. In the inversion mapping, the correlation coefficient
between the estimated and natural articulatory parameters was
0.79. In the production mapping, mel-cepstral distortion between the estimated and natural mel-cepstra was 4.70 dB. In
the proposed sequential inversion and production mapping system, we also evaluated the estimation accuracy of mel-cepstrum
without performing any modifications of the estimated articulatory parameters. The resulting mel-cepstral distortion between
the estimated and natural mel-cepstral was 4.45 dB.

4.3. Evaluation of phoneme sound modification
We investigated the feasibility of intentionally modifying certain phonemes with manual articulatory manipulations. We
picked three front vowels in English, /æ/, /E/ and /I/, where thetongue tip’s height is the obvious difference between them [6];i.e., /æ/ has the lowest position, /E/ has the middle position, andI/ has the highest position. We selected 10 distinct words fromthe test data, where each of them contained vowel /E/. Then, thevalue of the tongue tip’s height at the center frame of the vowel/E/ was shifted from -1.5 cm to +1.5 cm in 0.5 cm intervals,
relative to the originally observed position. A negative shifting value means the tongue tip is shifted down, whilst a positive
value means it is shifted up. Spline interpolation was performed
to generate continuous movements of the tongue tip. The manipulation method considering sthe inter-dimensional correlation was employed.
The result is shown in Figure 3. We can observe a clear
transition between /E/ and /I/, as the tongue tip’s height getshigher. On the other hand, the transition is not so clear from
/E/ to /æ/, as the tongue tip’s height gets lower. However, we
can still observe a reasonable tendency that the perception rateof /æ/ sound increases. Although further improvements will benecessary, the proposed system has a great potential to achieve
manual modification of phoneme sounds of input speech by intuitively manipulating unobservable articulatory parameters.

Two subjective evaluations were conducted to measure theperformance of the proposed speech modification system. In
the first evaluation, we compared two articulatory modification
methods described in Sections 3.1 and 3.2 in terms of speech
quality by scaling the movements of an articulator. In the second evaluation, we investigate whether the proposed system areable to modify a specific phoneme by means of manipulating
the articulatory positions. For both of the evaluations, there
were ten listeners. Two of them were German native speakers,
seven of them were Indonesian native speakers, and one wasVietnamese native speakers.
4.2. Speech quality evaluation for comparison of articulatory manipulation methods

5. Conclusions

We evaluated the quality of synthetic speech modified by the
proposed system. The tongue tip’s movement in y-coordinate
was scaled to several values over an utterance from one-fold(i.e., without modification) to 5-fold. An opinion test was conducted, setting the opinion score to a 5-point scale (5: excellent,
4: good, 3: fair, 2: poor, 1: bad). Every listener evaluated 15
distinct sentences, randomly selected from the test data, where
each of them were modified with two manipulation methods using five scaling settings.

In this paper, we proposed a novel speech modification method
to make it possible to modify speech signals by manipulating
unobserved articulatory movements in a sequential flow of statistical inversion and production mapping with Gaussian mixture models (GMMs). We also proposed a new articulatory
manipulation method to automatically refine unmodified articulatory movements according to the modified ones by considering the inter-dimensional correlation between individual articulators. The experimental results showed that higher quality of the modified speech is yielded by considering the interdimensional correlation in the articulatory manipulation andthat the proposed system is also capable of intentionally modifying phoneme sounds by manipulating the articulatory parameters. We plan to further improve quality of the modified speechand controllability of the articulatory parameters.

Figure 2 shows the result. Speech quality degrades more as
the scaling value grows larger. The manipulation method considering the inter-dimensional correlation effectively reduces
this quality degradation. We can see that it enables the system to keep speech quality of the modified speech equivalent tothat of the unmodified speech, when the scaling value is set to 2.
This result shows that the manipulation method considering theinter-dimensional correlation is more effective than the simple
manipulation method.

6. Acknowledgements
This research was supported in part by JSPS KAKENHI Grant
Number 22680016 and JASSO (Japan Student Services Organization) scholarship.

2301

7. References[1] Toda, T., Black, A. W., and Tokuda, K., “Statistical mapping
between articulatory movements and acoustic spectrum using a
gaussian mixture model,” Speech Communication, Vol. 50, No. 3,
pp. 215–227, Mar. 2008.
[2] Parthasarathy, S., Schroeter, J., Coker, C. and Sondhi, M. M., “Articulatory analysis and synthesis of speech,” Fourth IEEE region
10 international conference, pp. 760–764, Nov. 1989.
[3] Schroeter, J. and Sondhi, M. M., “Speech coding based on physiological models of speech production,” Advances in Speech Signal
Processing, S. Furui and M. M. Sondhi, Marcel Dekker New York,
pp. 231–267, 1992.
[4] Wrench, A. A. and Richmond, K., “Continuous speech recognition using articulatory data,” Proc. ICSLP, Beijing, China, pp.
145–148, Oct. 2000.
[5] Bollepali, B., Black, A., and Prahallad, K., “Modeling a noisychannel for voice conversion using articulatory features,” Proc.
INTERSPEECH, Portland, USA, Sep. 2012.
[6] Ling, Z., Richmond, K., Yamagishi, J., and Wang, R., “Articulatory control of HMM-based parametric speech synthesis driven
by phonetic knowledge,” IEEE Trans. Speech, Audio, and Lang.
Process., Vol. 7, No. 6, pp. 697–708, 2008.
[7] Schroeter J. and Sondhi M. M., “Techniques for estimating vocaltract shapes from the speech signal,” IEEE Trans. Speech and Audio Process., Vol. 2, pp. 133–150, 1994.
[8] Hogden, J., Lofqvist, A., Gracco, V., Zlokarnik, I., Rubin, P.,
and Saltzman, E., “Accurate recovery of articulator positions fromacoustics: new conclusions based on human data,” J. Acoust. Soc.
Am., Vol. 100, pp. 1819–1834, 1996.
[9] Kaburagi, T. and Honda M., “Determination of the vocal tract
spectrum from the articulatory movements based on the searchof an articulatory-acoustic database,” Proc. ICSLP, pp. 433–436,
Sydney, Australia, Dec. 1998.
[10] Hiroya, S. and Honda, M., “Estimation of articulatory movementsfrom speech acoustics using an HMM-based speech production
model,” IEEE Trans. Speech and Audio Process., Vol. 12, No. 2,
pp. 175–185, 2004.
[11] Hiroya, S. and Honda, M., “Speaker adaptation method foracoustic-to-articulatory inversion using an HMM-based speech
production model,” IEICE Trans. Inf. and Syst., Vol. E87-D, No.
5, pp. 1071–1078, 2004.
[12] Kello, C. T. and Plaut, D. C., “A neural network model of thearticulatory-acoustic forward mapping trained on recordings of articulatory parameters,” J. Acoust. Soc. Am., Vol. 116, No. 4, pp.
2354–2364, 2004.
[13] Richmond, K., King, S., and Taylor, P., “Modelling the uncertainty in recovering articulation from acoustics,” Computer
Speech and Language, Vol. 17, No. 2, pp. 153–172, 2003.
[14] Wrench, A., “The MOCHA-TIMIT articulatory database”,
http://www.cstr.ed.ac.uk/artic/mocha.html, Queen Margaret University College, 1999.
[15] Toda, T., Black, A. W., and Tokuda, K., “Voice conversion basedon maximum likelihood estimation of spectral parameter trajectory”, IEEE Trans. Audio, Speech and Lang. Process., Vol. 15,
No. 8, pp. 2222–2235, 2007.
[16] Ben Youssef, A., Badin, P., and Bailly, G., “Can tongue be recovered from face? The answer of data-driven statistical models”, Proc. INTERSPEECH, pp.2002-2005, Makuhari, Japan,
Sep. 2010.
[17] Kawahara, H., Masuda-Katsuse, I., and de Cheveigne, A., “Restructuring speech representation using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based F0
extraction: possible role of a repetitive structure in sounds”,
Speech Communication, Vol. 27, No. 3-4, pp. 187-207, 1999.
[18] Kawahara, H., Katayose, H., de Cheveigne, A., and Patterson, R.
D., “Fixed point analysis of frequency to instantaneous frequency
mapping for accurate estimation of F0 and periodicity”, Proc. EUROSPEECH, pp. 2781-2784, Budapest, Hungary, Sep. 1999.

2302