1-2-8
An Investigation of Acoustic-to-Articulatory Inversion Mapping with Latent
Trajectory Gaussian Mixture Model ∗◎ Patrick Lumban Tobing (NAIST), Tomoki Toda (Nagoya University/NAIST),
Hirokazu Kameoka (NTT), Satoshi Nakamura (NAIST)

1

Introduction

An acoustic-to-articulatory inversion mapping using a Gaussian mixture model [1] is eﬀective for developing a new speech modification framework [2].
Smoothly varying articulatory parameter trajectoryis well estimated from a given acoustic parametersconsidering interframe correlation by imposing anexplicit relationship between static and dynamic features on a maping process. However, this constraint
is not taken into accout in a training process of theGMM. The trajectory training method [3] was proposed to address this issue, but it makes the training process too complicated to analytically optimize
model parameters.
In this paper, as an alternative method to address this issue, we propose a latent trajectory
GMM (LTGMM)-based inversion mapping method
inspired by the latent trajectory HMM [4], whichmakes it possible to use EM algorithm to optimize
the model parameters. We conduct an experimental evaluation using a single speaker’s articulatoryacoustic data, demonstrating that higher mapping
accuracy is achieved using the LTGMM than thetraditional GMM.

2Conventional GMM for acousticto-articulatory inversion mapping⊤ ⊤be a time sequence ofLet x = [x⊤1 , · · · , xT ]Dx -dimensional static acoustic feature vectors and⊤ ⊤be that of Dy -dimensionaly = [y ⊤1 , · · · , yT ]static articulatory feature vectors. At frame t,
2Dx /2Dy -dimensional acoustic/articulatory feature⊤ ⊤vectors are denoted as X t = [x⊤andt , ∆xt ]⊤ ⊤⊤Y t = [y t , ∆y t ] , consisting of Dx /Dy -dimensional
joint static and dynamic features. Their joint vec⊤ ⊤tor is denoted as Z t = [X ⊤Moreover,t ,Y t ] .
their time sequences are written respectively as⊤ ⊤⊤⊤ ⊤X = [X ⊤1 , · · · , X T ] , Y = [Y 1 , · · · , Y T ] , and⊤⊤ ⊤Z = [Z 1 , · · · , Z T ] .
The joint probability density of the acoustic andarticulatory feature vectors is modeled by a GMMas follows:
T ∑M
∏
(Z)
P (Z|λ(Z) ) =αm N (Z t ; µ(Z)(1)m , Σm ),t=1 m=1

where λ(Z) is a set of model parameters consisting(Z)of a mixture weight αm , a mean vector µm and a(Z)covariance matrix Σm for the mth mixture component with M total number of mixture components.
These parameters are optimized for training datawith EM algorithm [1].
∗In the mapping process, given an acoustic feature
sequence X, the estimated articulatory feature sequence ŷ is determined as follows:ŷ = argmax P (Y |X, λ(Z) ) s.t. Y = W y y,(2)ywhere W y is a linear transform to append dynamicfeatures to a static feature sequence. This ML estimate can be determined with EM algorithm [1].
In this paper, an approximation of the above conditional p.d.f. is employed using a single mixture
component sequence m = {m1 , · · · , mT }. First, thesub-optimum mixture component sequence m̂ is determined. Then, the estimated articulatory feature
sequence ŷ is generated from the approximated conditional p.d.f., where its ML estimate can be analytically determined with EM algorithm [5].
Note that the interframe correlation is explicitly
considered with the constraint (Y = W y y) in themapping process. In contrast, it is ignored while optimizing GMM parameters in the training process.

3

Proposed latent trajectory GMM
for the inversion mapping

Let the observed variable be a time sequence⊤ ⊤of joint static feature vectors z = [z ⊤1 , · · · , zT ] ,⊤⊤ ⊤where z t = [xt , y t ] . The following soft constraint
is used in the LTGMM:
Z ≃ W z z = [W x , W y ][x⊤ , y ⊤ ]⊤ .

(3)

The joint probability density of the acoustic andarticulatory feature vector sequences is modeledwith an LTGMM∫as follows:
P (z|λ(z) ) =P (z|Z, Σ)P (Z|λ(Z) )dZ,(4)

where
−1−1
P (z|Z, Σ) = N (z;(W ⊤W z )−1 W ⊤Z,z Σz Σ(5)−1(W ⊤W z )−1 ).
z ΣThe covariance matrix Σ depends on only dimension of z t , i.e., independent of both time frames andmixture components. The model parameters can be
optimized with variational EM algorithm [4]. In thispaper, an approximation of the above joint p.d.f. isemployed using the sub-optimum mixture component sequence m̂. The approximated joint p.d.f. isgiven by:∫P (z|λ(z) ) = P (z|Z, Σ)P (Z|m̂, λ(Z) )dZ. (6)In this case, the model parameters can be optimizedwith EM algorithm.
In the inversion mapping process, given an acous-

潜在トラジェクトリ混合正規分布モデルによる調音運動逆推定の検討 by Patrick Lumban Tobing (NAIST)，戸田 智基 (名古屋大学/奈良先端大)， 亀岡弘和 (NTT)，中村 哲（奈良先端大）日本音響学会講演論文集- 227 -2016年3月

GMM
LTGMM

Average RMSE [mm]

1.9
1.85

Average Correlation Coefficient1.95(16)(32)

1.8(32)1.75(64)1.7(64)

(64)

(64)
(128)

1.65
1.6
1.55

50

100 150 200 250 300 350
Number of training utterances

400

0.78
GMM
LTGMM

0.76(64)(128)

100 150 200 250 300 350
Number of training utterances

400

0.74

(64)

0.72

(64)

(64)

(32)

0.7
0.68
0.66

(32)
(16)

50

Fig. 1 Average root-mean-square error of testingdata and optimum numbers of mixture components.

Fig. 2 Average correlation coeﬃcient of testingdata and optimum numbers of mixture components.

tic feature sequence x, the estimated articulatory
feature sequence ŷ is determined as follows:ŷ = argmax P (y|x, λ(z) ),(7)correlation coeﬃcients between the estimated articulatory parameters and the measured ones. The
number of training utterances was varied to 50, 100,150, 200, 250, 300, 350, and 400. The number of
mixture components was optimized for each numberof training utterances, as given in both Fig. 1 andFig. 2, using the conventional GMM. The numberof testing utterances was 20.

ywhereP (y|x, λ(z) ) =∫P (y|Y , Σ)∫(8)(z)P (X, Y |x, λ )dXdY .

This ML estimate can be determined with variational EM algorithm. In this paper, an approximation of the above conditional p.d.f. is employed byfirst determining the sub-optimum mixture component sequence m̂. Then, the estimated articulatory
parameter sequence ŷ is determined as follows:ŷ = argmax P (y|x, m̂, λ(z) ),(9)ywhere∫P (y|x, m̂, λ(z) ) = P (y|Y , Σ)∫(10)(Z)(z)P (Y |X, m̂, λ )P (X|x, m̂, λ )dXdY .

The ML estimate can be determined analytically.
Note that the interframe correlation is considered,both in the training and conversion process, by imposing explicitly the constraint in Eq. (3).

4Experimental evaluation

4.1 Experimental conditions
A set of speech and articulatory data of a single British male speaker in MOCHA [6] was used.
As the acoustic parameters, we used the 1st-to-24th
mel-cepstral coeﬃcients extracted with STRAIGHT
analysis [7] from 16 kHz sampled speech data. As
the articulatory parameters, we used 14-dimensional
EMA data converted to z-score, which represented
the movements of 7 articulators, as used in [1].
Frame shift was set to 5 ms.
The constant positive-definite matrix Σ in Eq. (5)was set to the diagonal matrix of global variances.
In the training process, the sub-optimum mixture
component sequence m̂ was initialized beforehandand held fixed. The trained conventional GMM wasused as an initial model for the LTGMM training.
We conducted an objective evaluation by calculating the root-mean-square errors (RMSEs) and the

日本音響学会講演論文集

4.2 Experimental results
Figures 1 and 2 show the RMSE and the correlation coeﬃcient averaged over all 14 dimensions
articulatory parameters through all 20 testing utterances. Higher accuracy of inversion mapping isachieved with the LTGMM. This is because the LTGMM can be optimized while considering the interframe correlation, which is considered in only themapping process if using the conventional GMM.

5Conclusion

We have proposed an inversion mapping method
based on the latent trajectory GMM (LTGMM).
The experimental results have demonstrated thathigher accuracy of inversion mapping with LTGMM
is achieved than the traditional GMM. We will further investigate its performance with the use ofan acoustic segment feature consisting of multiple
frames of input features and by taking into accountall possible mixture component sequences.
Acknowledgements Part of this work wassupported by JSPS KAKENHI Grant Number
26280060.

References[1] T. Toda, et al., Speech Communication, Vol. 50,
No. 3, pp. 215–227, 2008.
[2] P. L. Tobing, et al., Proc. INTERSPEECH, pp.
3350–3354, 2015.
[3] H. Zen, et al., Computer Speech and Language,Vol. 21, pp. 153–173, 2007.
[4] H. Kameoka, IEEE 25th International Workshop
on MLSP, pp. 1–6, 2015.
[5] T. Toda, et al., IEEE Trans. ASLP, Vol. 15, No.
8, pp. 2222–2235, 2007.
[6] A. Wrench, http://www.cstr.ed.ac.uk/artic/mocha.html, 1999.
[7] H. Kawahara, et al., Speech Communication,
Vol. 27, No. 3–4, pp. 187–207, 1999.

- 228 -2016年3月