1-6-10
Acoustic-to-Articulatory Inversion Mapping with Variational Latent
Trajectory Gaussian Mixture Model ∗◎ Patrick Lumban Tobing (Nagoya Univ.), Hirokazu Kameoka (NTT),
Tomoki Toda (Nagoya Univ.)

1Introduction

Gaussian mixture model (GMM)-based statistical feature mapping method, using trajectory-based
conversion process [1], has been reported to perform eﬀectively in an acoustic-to-articulatory inversion mapping task [2]. However, in the training, themodel parameters are optimized with diﬀerent likelihood criterion compared to in the conversion.
To address this inconsistency issue, the trajectory
training method has been proposed [3]. Nonetheless,it still has several limitations in terms of parameter optimization due to the diﬃculties in finding
analytical solution. Based on the latent trajectorymodel [4], in our previous work [5], we have proposed a latent trajectory Gaussian mixture model(LT-GMM) that is capable of addresing the inconsistency issue while allowing convenient parameter
optimization with EM algorithm. However, we still
use an approximation of a fixed suboptimum mixture component sequence to train the model.
In this report, we propose a variational LT-GMM
to allow the inclusion of all possible mixture component sequences in training the model. To investigate
the eﬀectiveness of the approximation in our previous work, we compare the accuracy on the inversion mapping task between the proposed variational
LT-GMM and the LT-GMMs with a suboptimum
mixture component sequence. The experimental result demonstrates that such approximation resembles the performance of the variational method.

2

Conventional GMM for acousticto-articulatory inversion mapping⊤ ⊤Let x = [x⊤be a time sequence of
1 , · · · , xT ]Dx -dimensional static acoustic feature vectors and⊤ ⊤be that of Dy -dimensionaly = [y ⊤1 , · · · , yT ]static articulatory feature vectors. At frame t,2Dx /2Dy -dimensional acoustic/articulatory feature⊤ ⊤vectors are denoted as X t = [x⊤andt , ∆xt ]⊤⊤ ⊤Y t = [y t , ∆y t ] , consisting of Dx /Dy -dimensional
joint static and dynamic features. Their joint vec⊤ ⊤tor is denoted as Z t = [X ⊤Moreover,t ,Y t ] .
their time sequences are written respectively as⊤ ⊤⊤⊤ ⊤X = [X ⊤1 , · · · , X T ] , Y = [Y 1 , · · · , Y T ] , and⊤⊤ ⊤Z = [Z 1 , · · · , Z T ] .
The joint probability density of the acoustic andarticulatory feature vectors is modeled by a GMM
as follows:T ∑M
∏
(Z)
(Z)
P (Z|λ ) =αm N (Z t ; µ(Z)(1)m , Σm ),where λ(Z) is a set of model parameters consisting(Z)of a mixture weight αm , a mean vector µm and a(Z)covariance matrix Σm for the mth mixture component with M total number of mixture components.
These parameters are optimized for training datawith EM algorithm.
In this report, in the conversion process, given an
acoustic feature sequence X, the estimated articulatory feature sequence ŷ is determined byŷ = argmax P (Y |X, m̂, λ(Z) ) s.t. Y = W y y, (2)ywhere W y is a linear transform to append dynamic features to a static feature sequence andm̂ = {m̂1 , . . . , m̂T } is a suboptimum mixture component sequence determined as follows:m̂ = argmaxm3

P (mt |X t , λ(Z) ).

(3)

t=1

Variational LT-GMM for the inversion mapping

Let the observed variable be a time sequence⊤ ⊤of joint static feature vectors z = [z ⊤1 , · · · , zT ] ,⊤ ⊤⊤where z t = [xt , y t ] . The following soft constraint
is used in the LT-GMM:
Z ≃ W z z = [W x , W y ][x⊤ , y ⊤ ]⊤ .

(4)

The joint probability density of the acoustic andarticulatory feature vector sequences is modeledwith an LT-GMM as follows:∫ ∑P (z|λ) =P (z|Z, Σ)P (Z, m|λ(Z) )dZ, (5)all mwhere m = {m1 , . . . , mT } is a mixture component
sequence, andP (z|Z, Σ) = N (z; HZ, Λ−1 )−1−1
H = Λ W⊤z Σ−1
Λ = W⊤W z.
z Σ(6)(7)(8)

A functional lower bound F(q, λ), corresponding
to the log-likelihood of Eq. (5), by following theJensen’s inequality, can be written as follows:∫∑P (z, Z, m|λ)F(q, λ) =dZ, (9)q(Z, m) log
q(Z, m)all mwhere q(Z, m) is a variational posterior distribution, which is further approximated as follows:

t=1 m=1∗T∏

P (Z, m) ≃ P (Z)P (m).

(10)

変分潜在トラジェクトリ混合正規分布モデルによる調音運動逆推定 by Patrick Lumban Tobing (名古屋大学)，亀岡弘和 (NTT)，戸田 智基 (名古屋大学)

日本音響学会講演論文集- 227 -2017年3月

In the expectation step (E-step), the variational
posterior distribution P (Z) is calculated as follows:

Table 1 Average of root-mean-square error(RMSE) and correlation coeﬃcient for inversion
mapping using GMM and three models of LT-GMM

1E[Z] = µ(Z) + Σ(Zz) Σ(zz) (z − Hµ(Z) ) (11)−1

E[ZZ ⊤ ] = Σ(Z) − Σ(Zz) Σ(zz) Σ(zZ) ,(12)GMM
LT-GMM fixseq
LT-GMM updseq
LT-GMM allseqs

and the variational posterior distribution P (m) as:(Z)

αm N (E[Z]t ; µm , Σ(Z)m )Cm,tγm,t = ∑M(Z)(Z)
n=1 αn N (E[Z]t ; µn , Σn )Cn,t{ 1−1 }
Cm,t = exp − Tr(E[ZZ ⊤ ]t Σ(Z)) .
m2(14)(15)

ywhere the suboptimum mixture component sequence m̂ is given by Eq. (3).

4Experimental evaluation

4.1 Experimental conditions
A set of speech and articulatory data of a British
male speaker in MOCHA [6] was used. As the acoustic parameters, we used the first through 24th melcepstral coeﬃcients extracted using STRAIGHT [7].
As the articulatory parameters, we used the 14dimensional EMA data.
The error-covariance matrix Σ was fixed to theinitial values of weighted covariance over all mixture
components. We trained three models of LT-GMM:
with a fixed suboptimum mixture component sequence m̂ (LT-GMM fixseq), as in [5]; by updating
the suboptimum mixture component sequence in theM-step of each iteration (LT-GMM updseq); andwith all possible mixture component sequences, i.e.,
using the variational method, (LT-GMM allseqs).
The trained conventional GMM was used as the initial model for training the LT-GMMs.
4.2 Experimental results
The result of the experiment is shown in Table 1.
It can be observed that the three LT-GMMs gives
higher accuracy for the inversion mapping comparedto the conventional GMM by yielding lower valuesof RMSE and higher values of correlation coeﬃcient. Furthermore, the LT-GMM with variational

日本音響学会講演論文集

Avg. Corr.

1.609
1.554
1.554
1.548

0.756
0.756
0.756
0.758

(13)

Note that, an overline in Eqs. (11) and (12) denotea sequence of mean vectors or covariance matrices,where at each time-frame t, they are marginalized
over all mixture components using the occupancy
γm,t . Hence, to optimize these distributions, i.e.,
P (Z) and P (m), the values in one must be kept
while refining the values in the other one.
Then, in the maximization step (M-step), themodel parameters λ are optimized by using the functional lower-bound in Eq. (9) and the approximatedvariational posterior distribution q(Z, m).
In the conversion, given an acoustic feature sequence x, the estimated articulatory parameter sequence ŷ is determined as in [5] byŷ = argmax P (y|x, m̂, λ),Avg. RMSE

method, i.e., to consider all possible mixture component sequences (LT-GMM allseqs), gives slightly
better accuracy than its two counterparts, i.e., byusing a fixed suboptimum mixture component sequence (LT-GMM fixseq) and by updating the suboptimum sequence (LT-GMM updseq). Specifically,
the observed diﬀerence of values between the LTGMM allseqs and the other two are 0.006 for theaverage RMSE and 0.002 for the average correlation coeﬃcient. This result suggests that, eventhough the variational technique is capable of giving the best accuracy through exhaustive consideration of all possible mixture component sequences,
the approximation of a suboptimum mixture component sequence can still rival that performance witha marginal diﬀerence in the accuracy.

5Conclusion

In this report, we have presented a variationalmethod for the latent trajectory Gaussian mixture
model (LT-GMM). The variational technique allows
us to optimize the coupled latent variables, i.e., thesequence of joint static and dynamic features andthe mixture component sequence. The experimental
result demonstrated that by considering all possible
mixture component sequences, i.e., using the variational method, a marginal improvement was yieldedcompared to using only a suboptimum sequence.
This implies that, through the low-cost approximation of a suboptimum mixture component sequence,
similar performance can be achieved to avoid thehigh-cost variational LT-GMM procedure.
Acknowledgment Part of this work was supported by JSPS KAKENHI Grant Number
26280060.

References[1] T. Toda, et al., IEEE/ACM Trans. Audio SpeechLang. Process, vol. 15, no. 8, pp. 2222–2235,
2007.
[2] P. L. Tobing, et al., in Proc. INTERSPEECH,pp. 3350–3354, 2015.
[3] H. Zen, et al., Comput. Speech Lang., vol. 21,pp. 153–173, 2007.
[4] H. Kameoka, in Proc. MLSP, pp. 1–6, 2015.
[5] P. L. Tobing, et al., Proc. INTERSPEECH, pp.
953–957, 2016.
[6] A. Wrench, http://www.cstr.ed.ac.uk/artic/mocha.html, 1999.
[7] H. Kawahara, et al., Speech Commun., vol. 27,
no. 3–4, pp. 187–207, 1999.

- 228 -2017年3月