Proceedings of APSIPA Annual Summit and Conference 201712 - 15 December 2017, Malaysia

Deep Acoustic-to-Articulatory Inversion Mappingwith Latent Trajectory Modeling
Patrick Lumban Tobing∗ , Hirokazu Kameoka† and Tomoki Toda‡∗ Graduate School of Information Science, Nagoya University, Japan† NTT Communication Science Laboratories, NTT Corporation, Japan‡ Information Technology Center, Nagoya University, Japan

E-mail: patrick.lumbantobing@g.sp.m.is.nagoya-u.ac.jp, kameoka.hirokazu@lab.ntt.co.jp, tomoki@icts.nagoya-u.ac.jp

Abstract—This paper presents a novel implementation of latent
trajectory modeling in a deep acoustic-to-articulatory inversion
mapping framework. In the conventional methods, i.e., the Gaussian mixture model (GMM)- and the deep neural network (DNN)based inversion mappings, the frame interdependency can beconsidered while generating articulatory parameter trajectories
with the use of an explicit constraint between static and dynamic
features. However, in training these models, such a constraint is
not considered, and therefore, the trained model is not optimum
for the mapping procedure. In this paper, we address thisproblem by introducing a latent trajectory modeling into the
DNN-based inversion mapping. In the latent trajectory model, theframe interdependency can be well considered, in both training
and mapping, by using a soft-constraint between static anddynamic features. The experimental results demonstrate thatthe proposed latent trajectory DNN (LTDNN)-based inversion
mapping outperforms the conventional and the state-of-the-art
inversion mapping systems.

I. I NTRODUCTION
Articulators, such as tongue and lips, play a dominantrole in determining the phonetic quality of a speech sound.
Indeed, representations of the articulatory movements, e.g.,
articulatory parameters, have been used in various works,
such as for speech recognition enhancement [1], for speechtherapy/pronunciation learning [2], and for speech production/modification systems [3]. This infers that there is a risingneed for a reliable acoustic-articulatory mapping framework.
In the recent years, in fact, there have been many notableworks for developing robust statistical data-driven acousticto-articulatory inversion mapping frameworks. These include
the codebook-based inversion mapping [4], the hidden Markov
model (HMM)-based mapping [5], the Gaussian mixture
model (GMM)-based mapping [6], and the artificial neuralnetwork (ANN)-based mapping [7]. In this paper, due to its
powerful predictive capability, we focus on the use of the deepneural-network (DNN)-based inversion mapping system.
In estimating articulatory parameter trajectories in the inversion mapping, it was reported that the use of temporalconstraints reduces the errors of the estimated trajectory [8].
Furthermore, in the GMM-based [6] and the state-of-the-art
mixture density network (MDN)-based inversion mappings[7], the use of maximum likelihood parameter generation(MLPG) [9], which explicitly uses a constraint between staticand dynamic features, significantly improves the mapping accuracy. However, this constraint, which allows a consideration

978-1-5386-1542-3@2017 APSIPA

of frame interdependency, is not taken into account while
training the corresponding models. In our previous work [10],based on the latent trajectory concept [11], we have proposeda latent trajectory training for the GMM-based inversion
mapping, where the frame interdependency is considered inboth training and mapping with a soft-constraint between astatic feature sequence (regarded as an observed variable)
and a static-dynamic feature sequence (regarded as a latentvariable). Note that, this framework capable of using a wellformulated algorithm to optimize model parameters, such as
the variational EM algorithm, is different from [12], in whichsuch an algorithm is difficult to be used to optimize some of
the model parameters.
In this paper, to make it possible to consider the frameinterdependency in training while keeping the model parameter optimization as easy as in the conventional DNN-based
inversion mapping, we propose a novel implementation of
the latent trajectory modeling for the deep inversion mapping
framework. Consistency between training and mapping ispreserved by considering the frame interdependency withina latent space through the use of a soft-constraint between
static and dynamic features. Moreover, almost the same model
optimization procedure as in the conventional DNN training
is available in the proposed latent trajectory DNN (LTDNN).
The experimental results demonstrate the effectiveness of the
proposed LTDNN-based inversion mapping, yielding superior
mapping accuracy compared to the conventional methods.
II. C ONVENTIONAL D EEP ACOUSTIC - TO -A RTICULATORYI NVERSION M APPINGSIn this section, we describe two conventional deep architectures for the use of acoustic-to-articulatory inversion mapping,i.e., the deep neural network (DNN) and the minimum generation error DNN (MGEDNN) [13].
A. Feature description
Let xt and y t be the Dx - and Dy -dimensional acoustic and articulatory feature vector at frame t, respectively.
The segmental acoustic feature vector is denoted as X t =
⊤
⊤
⊤
[x⊤
t−C , . . . , xt , . . . , xt+C ] , while the joint static-dynamic⊤ ⊤articulatory feature vector is denoted as Y t = [y ⊤t , ∆y t ]at frame t. The number of contextual frames is 2C + 1.

APSIPA ASC 2017

Proceedings of APSIPA Annual Summit and Conference 201712 - 15 December 2017, Malaysia

B. Deep neural network (DNN)-based inversion mapping
In the DNN-based inversion mapping, the conditional probability density function (pdf) of the joint static-dynamic articulatory feature vector Y t , at frame t, is defined as follows:
P (Y t |X t , D, λ) = N (Y t ; fλ (X t ), D),(1)where fλ (·) is a nonlinear function given by the network, i.e.,
the network output. The network parameters, i.e., weights andbiases are given in a set of the parameters λ. The diagonalcovariance matrix of the articulatory training data is denoted
as D.
In the training procedure, the updated network parameters
λ̂ are determined with
λ̂ = argmax
λ= argminλTX

log P (Y t |X t , D, λ)

t=1
T
X

1
(Y t −fλ (X t ))⊤ D −1 (Y t −fλ (X t )). (2)2 t=1

A graphical representation of the training process is givenin the left diagram of Fig. 1. Following the MLPG procedure [9], given a segmental acoustic feature vector sequence⊤⊤ ⊤X = [X ⊤1 , . . . , X t , . . . , X T ] , the corresponding articula(λ)
tory parameter trajectory ŷ X is given by(λ)

ŷ X = argmax
λTXlog P (Y t |X t , D, λ) s.t. Y = W y

t=1

1
(λ)
(λ)
= argmin (W y − M X )⊤ U −1 (W y − M X )2λ(λ)= (W ⊤ U −1 W )−1 W ⊤ U −1 M X ,

(3)

where
U = I T ×T ⊗ D,

(4)


⊤
(λ)
M X = fλ (X 1 )⊤ . . . , fλ (X t )⊤ , . . . , fλ (X T )⊤ ,(5)

and W is a transformation matrix to develop a joint
static-dynamic articulatory feature vector sequence Y =
⊤
⊤ ⊤
⊤
⊤ ⊤
[Y ⊤
from y = [y ⊤1 ,...,Y t ,...,Y T]
1 , . . . , yt , . . . , yT ] .
The Kronecker product is denoted as ⊗.
C. Minimum generation error DNN (MGEDNN)-based inversion mapping
In the MGEDNN-based [13] inversion mapping, the conditional pdf of the articulatory feature vector sequence y isdefined as follows:(λ)
P (y|X, D, λ) = N (y; ŷ X , I),

(6)

(λ)
where ŷ X is given in (3).

In the training procedure, the updated network parameters
λ̂ are determined with
1
(λ)
(λ)
λ̂ = argmin (y − ŷ X )⊤ (y − ŷ X )2λ(λ)= argmax N (y; ŷ X , I).
λ

978-1-5386-1542-3@2017 APSIPA

(7)

D

DNN

MGEDNNLTDNN

t = 1:Tt = 1:Tt = 1:T

Xt

Xt

Xt
D

Yt

A-1

y

Yt
y

I

Fig. 1. Graphical representations for the training procedure of the conventional
DNN, MGEDNN, and the proposed LTDNN. Unobserved variables areshaded.

A graphical representation of the training process is given in
the middle diagram of Fig. 1. The network parameters need
to be updated utterance by utterance, which is different fromthe frame-by-frame update available in the conventional DNN.
Note that, due to the use of identity covariance matrix, it
is difficult to apply articulatory parameter modification withcovariance compensation as has been done in our previous
work [10].
III. P ROPOSED D EEP I NVERSION M APPING WITH L ATENTT RAJECTORY M ODELINGIn this section, we describe the proposed latent trajectorymodeling [11] for the DNN-based inversion mapping, i.e., thelatent trajectory DNN (LTDNN).
A. Latent trajectory model
Let the articulatory feature vector sequence y be the observed variable and the joint static-dynamic articulatory feature
vector sequence Y be the latent variable. The following
soft-constraint is used between the observed and the latentvariables:
Y ≃ W y.
(8)Considering an error-covariance matrix Σ, the conditional
pdf of the latent variable Y can be defined as follows:


1
⊤ −1
P (Y |y, Σ) ∝ exp − (Y −W y) Σ (Y −W y) . (9)2
By completing the square of the exponential part of the above
pdf, the conditional pdf of the observed articulatory feature
sequence y is then defined as follows:
P (y|Y , Σ) = N (y; HY , Λ−1 ),(10)

H = Λ−1 W ⊤ Σ−1 ,

(11)

⊤

(12)

where
Λ=W Σ

−1

W.

Then, by marginalizing out the latent variable Y , the likelihood function of the observed articulatory feature vector
sequence y can be written asZ
P (y|Σ, λ) = P (y|Y , Σ)P (Y |λ)dY ,(13)
where we can simplifya pdf of the latent variable P (Y |λ),Qe.g., P (Y |λ) = t P (Y t |λ).

APSIPA ASC 2017

Proceedings of APSIPA Annual Summit and Conference 201712 - 15 December 2017, Malaysia

B. Proposed latent trajectory DNN (LTDNN)-based inversion
mapping

In the mapping procedure, the estimated articulatory feature(λ)
vector sequence ŷ X is given by

In the proposed LTDNN, following the likelihood functionin (13), given an input segmental acoustic feature vector
sequence X, the conditional pdf of the articulatory feature
vector sequence y is defined as follows:ZTY
P (y|X, Σ, D, λ) = P (y|Y , Σ)P (Y t |X t , D, λ)dYŷ X = argmax P (y|X, Σ, D, λ) = HM X

t=1

Z

−1

(λ)

N (y; HY , Λ )N (Y ; M X , U )dY
# 
  "
!
Z
(λ)
Λ−1 + HU H ⊤ HU
y
HM X
= N
;
,
dY
(λ)
Y
U H⊤
U
MX
=

(λ)

= N (y; HM X , Λ−1 + HU H ⊤ ).

(14)

In the training procedure, the updated network parameters λ̂
are determined with
λ̂ = argmax log P (y|X, Σ, D, λ).

(15)

λ

A graphical representation of the training process is given in
the right diagram of Fig. 1.
An auxiliary function to assist in finding the updated parameters λ̂ can be defined as follows:Z
Q(λ̂, λ) = P (Y |y, X, Σ, D, λ) logP (Y |X, D, λ̂)dY,(16)

where

(λ)

(λ)

P (Y |y, X, Σ, D, λ) = N (Y ; Ŷ y,X , Σ̂y,X )
(λ)

(17)
(λ)

(λ)

Ŷ y,X = M X +U H ⊤ (Λ−1 +HU −1 H ⊤ )−1 (y−HM X )
(18)
(λ)

Σ̂y,X = U − U H ⊤ (Λ−1 + HU −1 H ⊤ )−1 HU .

(19)

In this paper, an approximation of the posterior pdf of theabove auxiliary function is employed with delta function as
follows:Z(λ)
Q(λ̂, λ) ≈ δ(Y = Ŷ y,X ) log P (Y |X, D, λ̂)dY(λ)= log P (Ŷ y,X |X, D, λ̂).

(20)

Therefore, the updated network parameters λ̂ are determined
with
(λ)

λ̂ = argmax log P (Ŷ y,X |X, D, λ̂)
λT= argminλ(λ)
1 X (λ)(Ŷ−fλ (X t ))⊤ D −1 (Ŷ y,X,t −fλ (X t )).
2 t=1 y,X,t(21)

Note that this equation is almost the same as (2). Therefore,
the frame-by-frame network parameter update is still available
although a mean vector of the posterior pdf needs to be
estimated utterance by utterance as shown in (18).

978-1-5386-1542-3@2017 APSIPA

(λ)

(λ)

y

(λ)

= (W ⊤ Σ−1 W )−1 W ⊤ Σ−1 M X .

(22)Thanks to the use of the temporal covariance matrix, it would
be straightforward to apply articulatory parameter modification
as in [10] with the proposed LTDNN framework.
IV. E XPERIMENTAL EVALUATION
A. Experimental conditions
We used the multichannel articulatory (MOCHA) data
[14] provided by CSTR, University of Edinburgh. From thisdataset, the speech and the articulatory data, recorded withelectromagnetic articulograph (EMA), were used. In this experiment, we used the female speaker dataset (fsew0). The
total number of utterances was 460.
As the spectral envelope parameters, we used the firstthrough 24th mel-cepstral coefficients converted from thespectral envelope, which was extracted frame-by-frame using
STRAIGHT [15] analysis. As the articulatory parameters, weused the 14-dimensional EMA data, consisting of the timevarying positions of seven articulators: lower incisor, upper
lip, lower lip, tongue tip, tongue body, tongue dorsum, andvelum; on x- and y-coordinate in the mid-sagittal plane. These
articulatory parameters were converted into Z-scores. The
speech data were sampled at 16 kHz. The frame shift was set
to 5 ms. Starting and ending silence frames were included, 20
frames for each side, which were smoothed toward starting or
ending zero values, respectively, using half-hanning windows.
We conducted experiments to evaluate the accuracy ofthe proposed inversion mapping system by comparing thegenerated articulatory trajectory with the measured ones. The
proposed LTDNN-based inversion mapping system was compared to three baseline systems, i.e., GMM [6], DNN, and
MGEDNN. The inversion mapping accuracy was measured bycomputing the root-mean-square error (RMSE). The numberof training uterrances was varied to 46, 92, 184, and 368. The
validation set contained 46 utterances, consisting of the files
ending with 2, e.g.,“fsew0 002”. The evaluation set contained
also 46 utterances, but consisting of the files ending with 6.
The hyperparameters of the network were set as follows: thelearning rate was set to 0.0006; the number of hidden units
was set to 1024; the rectified linear unit (ReLU) was usedas the non-linear activation function for the hidden units; thenumber of hidden layers was set to 5; the number of iteration
limit for early-stopping was set to 20; and utterance mini-batch
was used. The Adam [16] optimization algorithm was used
to train the network parameters. The weights were randomly
initialized with Xavier [17] initialization method. The biases
were initialized with zero values. The trained DNN were usedas the initial model for both the MGEDNN and the LTDNN.
As for the GMM, the number of mixture components was set
to 256, 512, 1024, and 2048, with respect to each number oftraining utterances, using a tied-covariance matrix.

APSIPA ASC 2017

Proceedings of APSIPA Annual Summit and Conference 2017
TABLE IAVERAGE OF ROOT- MEAN - SQUARE ERROR (RMSE) [ MM ] FOR ALL
MODELS AND NUMBER OF TRAINING UTTERANCES

Model
GMM
DNN
MGEDNN
LTDNN

Number of Training Utterances
46
92
184
368
1.655
1.576
1.589
1.575

1.558
1.464
1.462
1.450

1.498
1.391
1.379
1.371

1.423
1.326
1.318
1.302

12 - 15 December 2017, Malaysia
TABLE IIRMSE [ MM ] FOR FIVE IMPORTANT ARTICULATORY DIMENSIONS : LOWERINCISOR (LI), LOWER LIP (LL), TONGUE TIP (TT), TONGUE BODY (TB),
AND TONGUE DORSUM (TD) ON y- COORDINATE IN THE MID - SAGITTALPLANE ( USING THE OPTIMUM MODEL FOR EACH SYSTEM )
Model
GMM
DNN
MGEDNNLTDNN

LI yArticulatory Dimension
LL y
TT y
TB y

TD y

1.085
1.078
1.124
1.067

2.258
2.121
2.095
2.040

1.910
1.886
1.872
1.843

2.068
1.831
1.831
1.823

1.890
1.773
1.791
1.738

B. Experimental results
The results of the average RMSE from 14 articulatory
dimensions for all combinations of models and number oftraining utterances are shown in Table I. The lowest RMSEs
for all number of training utterances were achieved by the
proposed LTDNN, yielding 1.575 mm, 1.450 mm, 1.371 mm,
and 1.302 mm, for the 46, 92, 184, and 368 training utterances,respectively. Note that, as a related work, the best result with asimilar data configuration was achieved using the state-of-theart mixture density network (MDN)-based inversion mappingin [7] with 1.370 mm RMSE using 368 training utterances.
Considering that the movements of tongue and jaw areones of the most dominant in speech production mechanism,
we present also the RMSE for five important articulators,i.e., lower incisor (LI), lower lip (LL), tongue tip (TT),tongue body (TB), and tongue dorsum (TD), on y-coordinate
in the mid-sagittal plane. The RMSEs computed from eachof the optimum models are given in Table II. The lowest
RMSEs for all articulatory dimensions, i.e., LI y, LL y, TT y,
TB y, and TD y, are achieved by the proposed LTDNN
yielding 1.067 mm, 2.040 mm, 1.823mm, 1.738 mm, and1.843 mm, respectively. Note that in the related work [7], thelowest achieved RMSEs are respectively given as 1.030 mm,
2.200 mm, 1.940 mm, 1.730 mm, and 1.850 mm.
These experimental results show that the proposed latent
trajectory modeling for the deep acoustic-to-articulatory inversion mapping improves the mapping accuracy compared to theconventional baseline systems. Moreover, although it cannot
be compared directly, the proposed LTDNN achieves higher
accuracy than that of the state-of-the-art MDN-based inversion
mapping method with the same dataset configuration.
V. C ONCLUSIONS
We have proposed the latent trajectory modeling in deep
acoustic-to-articulatory inversion mapping systems. The latent
trajectory model allows frame-interdependency to be considered in training the model by utilizing a soft-constraint
between static and dynamic features in the latent space. The
experimental results show that the proposed LTDNN yields
higher inversion mapping accuracy compared to the conventional inversion mapping systems. In the future, we would liketo combine the latent trajectory modeling with deep mixture
density network.
ACKNOWLEDGMENT
Part of this work was supported by JST, PRESTO Grant
Number JPMJPR1657 and JSPS KAKENHI Grant Number

978-1-5386-1542-3@2017 APSIPA

17H01763.
R EFERENCES[1] K. Kirchhoff, G. A. Fink, and G. Sagerer., “Combining acoustic andarticulatory feature information for robust speech recognition,” SpeechCommun., vol. 37, no. 3, pp. 303–319, 2002.
[2] B. J. Kröger, V. Graf-Borttscheller, and A. Lowit, “Two- and threedimensional visual articulatory models for pronunciation training andfor treatment of speech disorders,” in Proc. INTERSPEECH, Brisbane,
Australia, Sep. 2008, pp. 2639–2642.
[3] P. L. Tobing, K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “Articulatory controllable speech modification based on Gaussian
mixture models with direct waveform modification using spectrum
differential,” in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015,
pp. 3350–3354.
[4] J. Hogden, A. Lofqvist, V. Gracco, I. Zlokarnik, P. Rubin, and E. Saltzman, “Accurate recovery of articulator positions from acoustics: New
conclusions based on human data,” J. Acoust. Soc. Am., vol. 100, no. 3,
pp. 1819–1834, 1996.
[5] S. Hiroya and M. Honda, “Estimation of articulatory movements fromspeech acoustics using an HMM-based speech production model,” IEEE
Speech Audio Process., vol. 12, no. 2, pp. 175–185, 2004.
[6] T. Toda, A. W. Black, and K. Tokuda, “Statistical mapping between
articulatory movements and acoustic spectrum using a Gaussian mixture
model,” Speech Commun., vol. 50, no. 3, pp. 215–227, 2008.
[7] K. Richmond, “Trajectory mixture density networks with multiple
mixtures for acoustic-articulatory inversion,” in Proc. NOLISP, Paris,
France, May 2007, pp. 263–272.
[8] S. Suzuki, T. Okadome, and M. Honda, “Determination of articulatory
positions from speech acoustics by applying dynamic articulatory constraints,” in Proc. ICSLP, Sydney, Australia, 1998, pp. 2251–2254.
[9] K. Tokuda, T. Kobayashi, and S. Imai, “Speech parameter generationfrom HMM using dynamic features,” in Proc. ICASSP, Detroit, USA,May 1995, pp. 660–663.
[10] P. L. Tobing, T. Toda, H. Kameoka, and S. Nakamura, “Acousticto-articulatory inversion mapping based on latent trajectory Gaussian
mixture model,” in Proc. INTERSPEECH, San Francisco, USA, Sep.
2016, pp. 953–957.
[11] H. Kameoka, “Modeling speech parameter sequences with latent trajectory hidden Markov model,” in Proc. MLSP, Boston, USA, Sep. 2015,
pp. 1–6.
[12] L. Zhang and S. Renals, “Acoustic-articulatory modeling with the
trajectory HMM,” IEEE Signal Processing Letters, vol. 15, pp. 245–
248, 2008.
[13] Z. Wu and S. King, “Minimum trajectory error training for deepneural networks, combined with stacked bottleneck features,” in Proc.
INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 309–313.
[14] A. Wrench. (1999) The MOCHA-TIMIT articulatory database.
Queen
Margaret
University
College.
[Online].
Available:
http://www.cstr.ed.ac.uk/artic/mocha.html[15] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveigné, “Restructuring
speech representation using a pitch-adaptive time frequency smoothing
and an instantaneous-frequency-based F0 extraction: Possible role of arepetitive structure in sounds,” Speech Commun., vol. 27, no. 3–4, pp.
187–207, 1999.
[16] D. P. Kingma and J. Ba, “Adam: A method for stochastic
optimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available:
http://arxiv.org/abs/1412.6980[17] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep
feedforward neural networks,” in AISTATS, vol. 9, 2010, pp. 249–256.

APSIPA ASC 2017