NORMALIZED TRAINING FOR HMM-BASED VISUAL SPEECH RECOGNITION
Yoshihiko Nankaku†, Keiichi Tokuda†, Tadashi Kitamura† and Takao Kobayashi‡†Department of Computer Science
Nagoya Institute of Technology, Nagoya 466-8555, Japan‡Interdisciplinary Graduate School of Science and Engineering
Tokyo Institute of Technology, Yokohama, 226-8502 Japan

ABSTRACTThis paper presents an approach to estimating the parameters of continuous density HMMs for visual speech
recognition. One of the key issues of image-based visual
speech recognition is normalization of lip location and lighting condition prior to estimating the parameters of HMMs.
We presented a normalized training method in which thenormalization process is integrated in the model training.
This paper extends it for contrast normalization in addition
to average-intensity and location normalization. The proposed method provides a theoretically-well-defined algorithm based on a maximum likelihood formulation, hence
the likelihood for the training data is guaranteed to increaseat each iteration of the normalized training. Experiments
on M2VTS database show that the recognition performancecan be significantly improved by the normalized training.

Although the idea of [5] is similar to that of this paper, theproposed method provides a theoretically-well-defined algorithm based on the ML (maximum likelihood) criterion,and can normalize average-intensity, contrast and location
simultaneously within an ML formulation. In our previous
work [6], we proposed the average-intensity and location
normalized training. In this paper, the normalized training algorithm is extended for contrast normalization, andthe M2VTS database [7] was used instead of the Tulips1
database [8] for larger task experiments.
This paper is organized as follows. The next section defined the normalization transformation used in thiswork. An ML-based normalization framework and the reestimation algorithm for the normalized training are described in section 3 and section 4, respectively. Section 5
presents experimental results. Concluding remarks and ourplans for future work are presented in the final section.
2.

1. INTRODUCTION
In visual speech recognition, one of the difficulties is theextraction of feature parameters from the image sequenceof lips. Methods to extract speech information from image sequences are largely categorized into two approaches:model-based approach [1]–[3] and image- or pixel-based
approach[3]–[5]. In the model-based approach, a contour
model of lips is first constructed, and it is represented by asmall number of parameters. Although the advantage of thisapproach is that the parameters have less influence of variability of lighting condition, lip location, rotation and scaling, it has a difficulty in the construction of a robust and efficient lip contour model which can locate and track lips. On
the other hand, in the image-based approach, pixel valuesof the image are preprocessed and then used as the feature
vector. However, this process must take account of the variety of lighting condition, lip location, rotation, and scaling.
The statistical models (e.g., HMMs) are trained with such avariation, the distributions of different classes overlap eachother, and the discriminatory capabilities of the statisticalmodels may be reduced. Therefore, the training data must
be normalized prior to model training.
This paper proposes an approach to estimating the parameters of continuous density HMMs for visual speech
recognition, in which normalization of average-intensity,
contrast and lip location is integrated in the model training.

NORMALIZATION TRANSFORMATION
2.1. Geometric Transformation
In image-based visual speech recognition, we extract mouthpart from the original image by a linear transformation:ô(r) (t) = A(r) o(r) (t)(1)

where o(r) (t) and ô(r) (t) are the original image vector andthe extracted lip image vector, respectively associated withutterance r at time t. Note that the intensity values of all
pixels in the image are collected in a long one-dimensional
vector. A rectangular matrix A(r) extracts mouth part fromthe original image and sub-samples the extracted image. Although (1) can normalize geometric transformations such as
lip location, rotation and scaling, we considered only location normalization in this work. If the size of o(r) (t) and theblock size of sub-sampling are L and k, respectively, eachrow of A(r) consists of k elements of 1/k and L − k elements of 0, and the arrangement of the non-zero elements
depends on the location of extracted mouth part.
2.2. Intensity Transformation
In our method, HMM parameters are transformed by a linear transformation for Intensity normalization:
µ̂(r)
m

=

Č(r) µm + b̌(r)

(2)

(r)
Σ̂m

=

Č(r) Σm Č(r)T(3)

where µm and Σm are the mean vector and the covariancematrix of the m-th Gaussian component, respectively. Thistransformation framework can represent various normalizations, e.g., color normalization. In this paper, we assume
that the transformation consists of additive coefficients foraverage-intensity normalization and multiplicative coefficients for contrast normalization:
b̌(r)
Č

(r)

=

b̌(r) 1

=

(r)

č

I

(4)
(5)

where 1 = [1 . . . 1]T , and b̌(r) is an average-intensity
normalization vector, Č(r) is a matrix for contrast normalization. We also assume that the lip location and averageintensity do not change very much during one utterance, one
transformation is prepared for each utterance.
3.

ML-BASED NORMALIZATION FRAMEWORK

4. RE-ESTIMATION ALGORITHM
4.1. Q-function
To solve the above optimization problem, we adopt the EM(Expectation-Maximization) algorithm, which is the iterative procedure of approximating ML estimates. The procedure consists of maximizing at each iteration the auxiliary function so called Q-function. The likelihood for the
training data is guaranteed to increase by increasing thevalue of the Q-function. Hence the maximization of the
Q-function value at each iteration maximizes the likelihoodfor the training data:Q(λ, λ0 ) ≥ Q(λ, λ) ⇒P (Ô0 | M0 , (b̌, Č)0 ) ≥ P (Ô | M, (b̌, Č))where Ô0 is the training data normalized by the updated
transformation set A0 , which is included in the updated parameters λ0 . The Q-function with respect to the HMM parameters and the transformations can be written as(r)= argmax P (Ô(r) | M, (b̌(r) , Č(r) ))

(6)

T (r)

where Ô(r) = [ô(r) (1), ô(r) (2), . . . , ô(r) (Tr )] is the lip image sequence associated with an utterance r. However, to(r)
determine T 0 , we need the HMM parameters M whichcannot be trained unless transformations for all the training
utterances, T (r) , r = 1, 2, ..., R are determined. Therefore
the optimum set of transformations T (r) , r = 1, 2, ..., R andthe set of HMM parameters M are jointly estimated so as
to maximize the likelihood:0

λ = argmax P (Ô | M, (b̌, Č))
λ(7)where Ô = {Ô(1) , Ô(2) , ..., Ô(R) } is all training utterances, and λ consists of the set of transformations for allutterances and HMM parameters:
λ = {A, (b̌, Č), M}(8)

R,M,Tr

1 X (r)γ (t)Q(λ, λ ) = K −2 r,m,t m
× Km + log (|Σm |) − 2 log (|C(r) |)(r)(t) − µm )+(õ(r) (t) − µm )T Σ−1m (õIn the conventional normalization approach, lip images
must be normalized prior to estimating the parametersof HMM. In our method, the transformation T (r) =(A(r) , b̌(r) , Č(r) ) is determined so as to maximize the likelihood of the HMM parameters M, that is, the optimal(r)transformation T 0 is derived asT00A(b̌, Č)= {A(r) | r = 1, 2, . . . , R}(r)

= {(b̌

, Č

(r)

(9)

) | r = 1, 2, . . . , R} (10)are all transformations for training HMM parameters. The
fundamental idea of the proposed method is to determine
the transformations {A, (b̌, Č)} which normalize training
utterances and HMM parameters M simultaneously.

(12)

where K is a constant dependent only on the transition
probabilities, Km is the normalization constant associated(r)with Gaussian m, and γm (t) is the posterior probabilityof Gaussian m at time t, that can be computed through theforward-backward algorithm:(r)(t) = P (qt = m | O(r) , T (r) , M)
γm

(13)

and õ(r) (t) is the normalized training data defined byõ(r) (t) = C(r) ô(r) (t) + b(r)

(14)

b(r)

(15)

where
C

(r)

=
=

−{Č(r)}−1 b̌(r) = b(r) 1(r) −1

{Č

}

=c

(r)

I

(16)

Thus, the intensity transformation in our method can be implemented as a transformation of the feature vector and asimple addition of the term log (|C(r) |). Therefore the loglikelihoods are calculated aslog (P (ô(r) (t) | µm , Σm , b̌(r) , Č(r) ))= log (N (õ(r) (t) ; µm , Σm )) + log (|C(r) |)

where

(11)

(17)

and there is no need to change the original HMM parameters.
The iterative approach using Q-function is adopted
in which one of the parameter sets (transformations{A, (b, C)} and HMM parameters M) is estimated at eachstage and the maximum likelihood re-estimation is used individually for each of the parameter sets keeping the otherparameters are fixed.

4.2. Maximization of Q-function
We first maximize the Q-function with respect to{A, (b, C)} while keeping model parameters M fixed tocurrent values. We cannot derive A(r) which maximize thevalue of Q-function in closed form since A(r) must satisfy
the constraints described in section 2 (i.e., lip area extraction
and sub-sampling). By giving a location, however, A(r) is
completely determined. Therefore, we adopt direct search
for the optimum location. To avoid a large amount of computation required for the exhaustive search, we adopted a
gradient search of Q-function value for the optimal locationin the experiment.
Then, we compute the transformation set (b(r) , C(r) )while keeping A(r) fixed to the current values. The solutionof∂Q(λ, λ0 )= 0, r = 1, 2, . . . , R,
(18)
∂c0 (r)gives the updated contrast normalization coefficient aspξ (r) + ξ (r)2 + 4nT ψ (r)
0 (r)
=
(19)
c
2ψ (r)
where
ξ (r)

=

M,T
X
m,t

ψ

(r)

=

M,T
X

(r)
(r)
γm
(t)ô(r) (t)Σ−1
(t),
m ô

(r)
(r)
γm
(t)(µm − b(r) )T Σ−1
(t)
m ô

(20)

By inspection, these equations are the same as thestandard estimation using the training data õ0(r) (t)obtained by the updated normalization transformation(r){A0 , (b(r) , C(r) )0 }The normalized training procedure is summarized asfollows:
Step 0. Give an initial transformation set {A, (b, C)} andconstruct an initial model M.
(r)Step 1. Compute the values of γm (t), and estimate the0transformations (b, C) , and this step is iterated until
the change of likelihood is small.
(r)
Step 2. Compute the values of γm (t), and estimate the0transformations A .
(r)
Step 3. Compute the values of γm (t), and estimate the0model parameters M .
Step 4. If the change of the likelihood after the reestimation is small, Stop. Otherwise go to Step 1.
It is easily verified that at each stage of the update process
the value of the Q-function is guaranteed to increase:
Q(λ, λ0 ) ≥ Q(λ, λ)(25)Hence the likelihood for the training data is also guaranteed
to increase, based on the properties of the Q-function statedearlier:
P (Ô | M, (b̌, Č))≤ P (Ô0 | M, (b̌, Č)0 )(21)≤ P (Ô0 | M0 , (b̌, Č)0 ) (26)and n is the size of the feature vector ô(r) (t). Similarly,(r)using the updated c0 , the updated average-intensity nor(r)malization coefficient b0 are given as

where Ô0 is the training data normalized by the transformation set A0 updated in Step 2.

m,t

M,T
X

b0

(r)

=

m,t

(r)
γm
(t)(µm − C0(r) 0(r)ô(t))T Σ−1
m 1
(22)

M,T
X

(r)
γm
(t)1T Σ−1m 1m,t

The estimation of the means and covariance matrices of
the Gaussian densities conditioned on the updated transformation set {A0 , (b, C)0 } is expressed as
R,T
Xr

µ0m =

r,t

(r)
γm
(t)õ0(r) (t)

R,T
Xr
r,t

R,T
Xr

Σ0m =

r,t

(23)
(r)
γm
(t)

(r)
γm
(t)(õ0(r) (t) − µ0m )(õ0(r) (t) − µ0m )T
R,T
Xr
r,t

(24)
(r)
γm
(t)

5. EXPERIMENT
Visual word recognition experiments were performed. Eachword model was represented by one HMM which is left-toright model of 8 states with two Gaussian distributions of
diagonal covariance. The size of extracted lip image was
80×40 and the block size of sub-sampling was 5×5. The
image vector (static feature vector), the temporal difference
vector (delta feature vector) and second order time derivatives (delta-delta feature vector) were combined to form thefeature vector o(r) (t).
For the experiment, the M2VTS database was usedwhich is a bimodal database comprising of face image sequences and speech signals of 37 speakers. This databaseprovides 5 shots for each speaker and during each shot
speakers pronounce French numbers from “0” to “9” continuously. Shots were taken at one week intervals to account for minor face changes like beards. For each speaker,
the most difficult shot to recognize is the fifth, because ofsome face and voice variations have been included. This
study was carried out by using the first four shots. The
visual frame rate is 25 frame/s and each frame is a fullcolor 350×286 pixel image. They were converted into greylevel images for the experiments in this study. We performed speaker independent word recognition tests using
the “leave-one-out method”. In the method, one of 37 subjects was used for testing and the remaining 36 subjects

x102
-134

Log-likelihood

-136
-138
-140
-142

L+C+I
L+I
C+I
I

-144
-146
w/o

5

10

15

20

25

location is also normalized prior to the model training since
the lip areas were extracted manually. Notice that the normalized training of “I” and “C+I” are better than the conventional normalization approach using the same location data.
In addition, the normalized training which include the location normalization (“L+I” and “L+C+I”) further improved
the performance. These results suggest that the normalization process should be integrated in the model training
to reduce the overlap among the distributions of different
HMMs, and it improves the recognition performance significantly.

iteration

Figure 1. Log likelihoods of HMMs for training data.
75

x102
74.1

74
73.0

recognition rate (%)

73

72.6

72
71

70.1

70
69

L+C+I
L+I
C+I
I

68
67
66
65

65.6
w/o

5

10

15

20

25

iteration

Figure 2. Recognition rate for each iteration of normalized training.

6. CONCLUSION
We proposed an approach to simultaneous intensity, contrast and location normalized training of HMMs for imagebased visual speech recognition. Experimental results showthat by integrating the normalization process into the modeltraining the recognition performance is significantly improved: a word recognition rate of 74.1% and an errorreduction of 24.7% were achieved. In the proposed algorithm, the likelihood for the training data is guaranteed toincrease at each iteration of parameter re-estimation. Extension to continuous visual speech recognition and integrationof the visual information to auditory information will be future works.
ACKNOWLEDGMENT
This work was partially supported by the Ministry of Education, Science, Sports and Culture Japan, Grant-in-Aid for
Encouragement of Young Scientists, 10780226.
REFERENCES

were used for training. This was repeated 37 times, leaving out a different subject each time.
Figure 1 shows the change of the likelihood with respect to the iteration of the normalized training. In the figure, “w/o” means that HMMs were trained without both theprior normalization and the normalized training, and they
were used as the initial models for normalized training. We
investigate the performance of the various normalized training described in this paper: the combination of the averageIntensity “I”, Contrast “C”and Location “L” normalization.
From the figure, it can be confirmed that the normalized
training monotonically increases the likelihood for the training data.
The word recognition rate for each iteration of the normalized training is shown in Figure 2. The transformation
for the testing data was obtained after three iterations of the
update (Step 1-2). The normalization process was applied
to all the testing data except for “w/o”. It can be seen that asignificant error reduction is achieved by the proposed technique. A recognition rate of 74.1% and error reduction of
24.7% were achieved in the case of “L+C+I”normalization.
When we used the conventional normalization approach, inwhich the average-intensity and the contrast of the training
and testing data are normalized independently of HMM, arecognition rate of 70.8% was obtained. In the case of only
the intensity normalization, a recognition rate of 69.9% wasachieved. These results can be regarded as those obtainedby the conventional normalization approach in which the[1] J. Luettin, N. Thacker, S. Beet, “Speechreading using
Shape and Intensity Information,” Proc. ICSLP, pp. 58–
61, 1996.
[2] J. Luettin, “Towards Speaker Independent continuous Speechreading,” Proc. Eurospeech, pp. 1991–1994,
1997.
[3] G. Potamianos and A. Potamianos, “Speaker Adaptation for Audio-Visual Speech Recognition” Proc. Eurospeech, pp. 1291–1294, 1999.
[4] J. R. Movellan, “Visual Speech Recognition with
Stochastic Networks,” G. Tesauro, D. Touretzky, T.
Leen (eds. ) , Advances in Neural Information Processing Systems 7, MIT Press Cambridge, 1995.
[5] O. Vanegas, A. Tanaka, K. Tokuda and T. Kitamura,“HMM-based Visual Speech Recognition using Intensity and Location Normalization,” Proc. ICSLP, pp.
289–292, 1998.
[6] Y. Nankaku, K. Tokuda and T. Kitamura, “Intensityand Location-Normalized Training for HMM-Based
Visual Speech Recognition,” Proc. Eurospeech, pp.
1287–1290, 1999.
[7] http://www.uk.infowin.org/ACTS/RUS/PROJECTS/ac102.htm[8] J. R. Movellan, “Visual Speech Recognition with
Stochastic Networks,” G. Tesauro, D. Touretzky, T.
Leen (eds. ) , Advances in Neural Information Processing Systems 7, MIT Press Cambridge, 1995.