FACE RECOGNITION BASED ON SEPARABLE LATTICE 2-D HMMS
USING VARIATIONAL BAYESIAN METHOD
Kei Sawada, Akira Tamamori, Kei Hashimoto, Yoshihiko Nankaku, Keiichi Tokuda
Department of Scientific and Engineering Simulation, Nagoya Institute of Technology, Nagoya, Japan
ABSTRACTThis paper proposes an image recognition technique based on separable lattice 2-D HMMs (SL2D-HMMs) using the variational
Bayesian method. SL2D-HMMs have been proposed to reduce theeffect of geometric variations, e.g., size and location. The maximum
likelihood criterion had previously been used in training SL2DHMMs. However, in many image recognition tasks, it is difficult touse sufficient training data, and it suffers from the over-fitting problem. A higher generalization ability based on model marginalization
is expected by applying the Bayesian criterion and useful prior information on model parameters can be utilized as prior distributions.
Experiments on face recognition indicated that the proposed method
improved image recognition.
Index Terms— face recognition, hidden Markov model, separable lattice 2-D HMMs, Bayesian criterion, variational Bayesian
method
1. INTRODUCTION
Statistical approaches have been successfully applied in image
recognition. Especially, principal component analysis (PCA) based
approaches such as the eigenface method [1] and subspace method
show good recognition performance in many applications. Although
there are many significant classifiers and feature representations,some pre-processing is usually applied to input images prior to feature extraction and classifier training. The aim of pre-processing
is to normalize image variations, e.g., geometric variations suchas size, location, and rotation. These normalization processes areimportant because many classifiers cannot absorb such image variations and the accuracy of normalization significantly affects image
recognition. However, task dependent heuristic techniques are applied independently of classifiers. Therefore, it is necessary todevelop normalization techniques for each task. Furthermore, thefinal objective in image recognition is not to accurately normalize images for human perception but to achieve better recognition.
Therefore, it is a good idea to integrate the normalization processes
into classifiers and optimize them based on a consistent criterion to
improve recognition.
Hidden Markov model (HMM) based techniques have been
proposed to reduce the influence of geometric variations. Geometricmatching between input images and model parameters is represented by discrete hidden variables and the normalization process
is included in calculating probabilities. However, the extension of
HMMs to multi-dimensions generally leads to an exponential increase in the amount of computation for the training algorithm. Separable lattice 2-D HMMs (SL2D-HMMs) have been proposed [2]to reduce computational complexity while retaining outstanding
properties that model multi-dimensional data. SL2D-HMMs canperform elastic matching both horizontally and vertically, whichmakes it possible to model not only invariances to the size andlocation of an object but also nonlinear warping in all dimensions.
In many image recognition tasks, only a small amount of training data is available and the efforts to achieve high generalization
ability are required. The maximum likelihood (ML) criterion has
typically been used in image recognition using SL2D-HMMs. However, since the ML criterion produces a point estimate of model parameters, the estimation accuracy may be degraded due to the overfitting problem when the amount of training data is insufficient. The
Bayesian criterion, on the other hand, assumes that model parameters are random variables, and a high generalization ability can beobtained by marginalizing all model parameters in estimating predictive distributions. Moreover, the Bayesian criterion can utilize priordistributions representing useful prior information on model parameters. However, the Bayesian criterion requires complicated integral
and expectation computations to obtain posterior distributions whenmodels have hidden variables. To overcome this problem, maximum a posteriori (MAP) method [3] and variational Bayesian (VB)
method [4] have been proposed as approximation methods. We applied the Bayesian criterion to image recognition based on SL2DHMMs, and derived the training algorithm based on the VB method,both of which are explained in this paper.
The rest of the paper is organized as follows. Section 2 briefly
explains the structure of SL2D-HMMs, and Section 3 describes the
VB method for SL2D-HMMs. Section 4 presents face recognition
experiments we did on the XM2VTS database and we finally conclude the paper in Section 5.
2. SEPARABLE LATTICE 2-D HMMS
Separable lattice 2-D hidden Markov models (SL2D-HMMs) are defined for modeling two-dimensional data. The observations of twodimensional data, e.g., the pixel values of an image and image sequence, are assumed to be given on a two-dimensional lattice:O = {O t | t = (t(1) , t(2) ) ∈ T },(1)

where t denotes the coordinates of the lattice in two-dimensional
space T and t(m) = 1, . . . , T (m) is the coordinate of the m-th dimension for m ∈ {1, 2}. Observation O t is emitted from the state
indicated by hidden variable S t ∈ K. Hidden variables S t ∈ K
can take one of K = K (1) K (2) states, which are assumed to be arranged on two-dimensional state lattice K = {1, . . . , K}. In otherwords, a set of hidden variables {S t | t ∈ T } represents a segmentation of observations into K states, and each state corresponds to asegmented region in which the observation vectors are assumed tobe generated from the same local deformation. Since observation
O t is only dependent on state S t as in ordinary HMMs, dependencies among hidden variables determine the properties and modelingabilities of two-dimensional HMMs.
To reduce the number of possible state sequences, the hidden

(2)

S1

(2)

S2

S1

(1)

S2

(1)

ST (1)

O 1,1

O 2,1

OT (1),1

O 1,2

O 2,2

OT (1),2

The predictive distribution of testing data X in the testing stage is
given by P (X | ΛML ). However, since the ML criterion produces apoint estimate of model parameters, the estimation accuracy may bedecreased due to the over-fitting problem when there are insufficientnumbers of training data.
On the other hand, the predictive distribution of the Bayesian
criterion is given by:Z
P (X | O) = P (X | Λ) P (Λ | O) dΛ.
(7)

(1)

Posterior distribution P (Λ | O) for a set of model parameters Λ canbe written with the Bayes theorem.
(2)
ST (2)

O 1,T (2)P (Λ | O) =O 2,T (2)

O T (1),T (2)Fig. 1. Graphical representation of SL2D-HMMsvariables to be composed of two Markov chains by constraint are:S = {S (1) , S (2) },
S

(m)

(2)

(m)
= {St(m) | 1 ≤ t(m) ≤ T (m) },(3)

where S (m) is the Markov chain along with the m-th coordinate(m)and St(m) ∈ {1, . . . , K (m) }. The composite structure of hidden
variables in SL2D-HMMs is defined as the product of hidden state
sequences:(4)

This means that the segmented regions of observations are constrained to rectangles, which allows an observation lattice to beelastic both horizontally and vertically. The number of possible state sequences can be reduced by using this structure from
Q
Q
Q
(m)
(m)
{ m K (m) } m T
to m {K (m) }T.
Figure 1 shows the graphical representation of SL2D-HMMs.
The joint probability of observation vectors O and hidden variables
S can be written as:
P (O, S | Λ) = P (O | S, Λ)P (S (m) | Λ).

(5)

m=1

When it is assumed that the state output probability distributions
are a single Gaussian distribution, a set of model parameters Λ is(m)(m)
represented by {π (m) , a(m) , µk , Σk }, where {πi }K
is thei=1(m)(m)
initial state probability, {aij }K
is
the
state
transition
probabili,j=1
ity, and µk and Σk are the mean vector and the covariance matrixof the Gaussian distribution at state k on 2-D state space K, i.e.,
(m)
(m)
(m)
(m)
(m)
P (S1 = i | Λ) = πi , P (St(m) = j | St(m) −1 = i, Λ) = aij ,and P (O t | S t = k, Λ) = N (O t | µk , Σk ).

The maximum likelihood (ML) criterion has typically been used totrain SL2D-HMMs in image recognition. The optimal model parameters are estimated in the ML criterion by maximizing the likelihoodof training data as:
ΛML = arg max P (O | Λ).

(6)

(9)

The MAP method can utilize prior distribution P (Λ), and can beseen as an extension of the ML criterion. Testing in the MAP method
is conducted using predictive distribution P (X | ΛMAP ). However,it still suffers from the over-fitting problem because of point estimates, when there are insufficient numbers of training data.
3.2. Variational Bayesian method
The VB method is an approximate version of the Bayesian approach.
Since the VB method does not use an asymptotic assumption for
the amount of data, it is possible to overcome the problem in the
MAP method. An approximate posterior distribution is estimatedin the VB method by maximizing a lower bound for log marginal
likelihood F instead of the true likelihood. A lower bound for log
marginal likelihood is defined by using Jensen’s inequality:
XZ
ln P (O) = ln
P (O, S | Λ) P (Λ)dΛ
S

= ln

XZ

Q (S, Λ)P (O, S | Λ) P (Λ)
dΛ
Q (S, Λ)Q (S, Λ) ln

P (O, S | Λ) P (Λ)
dΛ
Q (S, Λ)

S

≥

XZ

= F,

3.1. Bayesian criterion

ΛΛMAP = arg max P (O | Λ)P (Λ).

S

3. SEPARABLE LATTICE 2-D HMMS USING
VARIATIONAL BAYESIAN METHOD(8)

where P (Λ) is a prior distribution for Λ and P (O) is evidence.
The model parameters are integrated out in Eq. (7) so that the effect of over-fitting is mitigated. That is, the Bayesian criterion has
a higher generalization ability than the ML criterion when there areinsufficient numbers of training data. However, the Bayesian criterion requires complicated integral and expectation computations to
obtain posterior distributions when models have hidden variables.
Maximum a posteriori (MAP) and variational Bayesian (VB) methods have been proposed as approaches to approximation to overcome
this problem.
The optimal model parameters in the MAP method are estimatedby maximizing the posterior probability for given training data as:
Λ

(1)
(2)
S t = (St(1) , St(2) ).

2
Y

P (O | Λ)P (Λ)
,
P (O)

(10)

where Q (S, Λ) is an arbitrary distribution. The relation between
the log marginal likelihood and the lower bound F is represented byusing the Kullback-Leibler (KL) divergence between Q (S, Λ) andtrue posterior distribution P (S, Λ | O):
F = ln P (O) − KL(Q (S, Λ) || P (S, Λ | O)).

(11)

Therefore, maximizing F with respect to Q (S, Λ) provides a good
approximation of posterior distribution P (S, Λ | O) in terms ofminimizing the KL divergence. The solution can be obtained byfunctional approximation based on the variational method.
To obtain approximate posterior distribution (VB posterior distribution) Q (S, Λ), we assumed that random variables were conditionally independent of one another, i.e.,
Q (S, Λ) = Q(S (1) )Q(S (2) )Q (Λ) ,
(12)
R
(m)
where S (m) Q(S ) = 1 and Q (Λ) dΛ = 1. Under thisassumption, the optimal VB posterior distributions that maximize
the objective function F are given by the variational method as:
"
X Z
0
(m)
Q(S ) = CS (m) exp
Q(S (m ) )Q(Λ)

(a)

P

(b)

VB

UBM

τ = 100

τ = 1000

τ = 10000

τ = 100000

ML0

S (m )

× ln P (O, S(1)

Q (Λ) = CΛ P (Λ) exp

,S

"

(2)

#

| Λ)dΛ ,XX
S (1) S (2)× ln P (O, S

(1)

,S

(2)

(13)

Q(S (1) )Q(S (2) )#| Λ) ,(14)

where CS (m) is the normalization term for Q(S (m) ) and CΛ is thatfor Q (Λ), and m0 is the m0 -th dimension different from the m-th dimension. Since the VB posterior distributions, Q(S (m) ) and Q (Λ),
that are obtained are dependent on each other, these updates need to
be iterated as the EM algorithm. The update equations increase thevalue of the objective function F at each iteration until convergence.
Predictive distribution P (X | O) is estimated using Eq. (7) inthe testing stage of the VB method. Since Q(Λ) is an approximation of posterior distribution P (Λ | O), Q(Λ) can be substituted for
P (Λ | O) in Eq. (7). Although Eq. (7) includes a complicated expectation calculation, the same approximation as that in training canbe applied. In face recognition using SL2D-HMMs, SL2D-HMMs
are trained for each class, i.e., subject, separately, and the likelihoodof testing data, which is calculated by the predictive distribution of
SL2D-HMMs, is compared among all subjects and the class whichobtains the highest likelihood is chosen as the result.
3.3. Prior distribution
The Bayesian criterion has an advantage in that it can utilize priordistributions representing useful prior information on model parameters. Although arbitrary distributions can be used as prior distributions, conjugate prior distributions are widely used as prior distributions. A conjugate prior distribution is a distribution where theresulting posterior distribution belongs to the same distribution family as the prior distribution. The conjugate prior distribution of an
SL2D-HMM is defined as:
"
#
(m)
2
KY
Y
(m)
(m)
P (Λ) =D(π (m) | φ(m) )D(ai | αi )
m=1

×

Y

i=1
−1

N (µk | ν k , ξk Σk )W(Σ−1k | ηk , B k ), (15)kwhere D(·) is a Dirichlet distribution, and N (·)W(·) is a GaussWishart distribution. These distributions can be represented by a set(m)of hyper-parameters {φ(m) , αi , ν k , ξk , ηk , B k }. The posterior
distributions can also be represented by the same set of parameters(m)(m){φ̄ , ᾱi , ν̄ k , ξ¯k , η̄k , B̄ k } by using a conjugate prior distribution.

(c)

(d)

(e)

Fig. 2. (a) training images of different subjects, (b) all training images of one subject, (c) mean vector of UBM, (d) mean vector of
posterior distribution, (e) mean vector of model obtained with ML
methodSince the prior distributions of model parameters affect the estimation of posterior distributions in the Bayesian criterion, determining prior distributions is a serious problem in estimating appropriate
models. We set the prior distribution to P (Λ) ∝ P (Õ | Λ) by using data Õ given in advance (we called this prior data). We used
all training samples for all subjects as prior data in the research discussed in this paper. This is the same idea as that in the universal
background model (UBM). The hyper-parameters based on a UBM
are given as:
T̃ij
T̃0i
(m)
+ 1, αij =+ 1, ν k = Õ k ,
τ
τ
T̃
T̃
T̃
ξk = k , ηk = k + D, B k = k C̃ k ,
(16)
τ
τ
τ
where D is the dimension of a feature vector and τ is the tuning
parameter. Statistics T̃0i , T̃ij , and T̃k correspond to the occupancy
probabilities of initial state i, state transition from i to j, and state
k with respect to the prior data, respectively. Statistics Õ k and C̃ kcorrespond to the mean vector and the covariance matrix of priordata in the k-th state. We can control the degree of influence theprior distribution has on the posterior distribution by adjusting tuning
parameter τ .
(m)

φi

=

4. EXPERIMENTS
Face recognition experiments on the XM2VTS database [5] wereconducted to evaluate the effectiveness of the proposed method. We
prepared eight images of 100 subjects; six images were used fortraining and there were two images for testing. Face images of 64 ×
64 grayscale pixels were extracted from the original images. SL2DHMMs with 8 × 8, 16 × 16, 24 × 24, 32 × 32, 40 × 40, 48 × 48,
56 × 56, and 64 × 64 states were used in these experiments. The
hyper-parameters of the prior distribution were determined by using
statistics on UBM, which was trained from all training data. The ML(conventional), MAP, and VB methods (proposed) were compared toseparately evaluate the two advantages of the Bayesian approaches,i.e., the use of the prior distribution and marginalization of model
parameters.
Examples of training images and mean vectors have been given
in Figure 2 to demonstrate what effect prior distributions had in theBayesian approach. Figure 2(a) presents training images of different
subjects, and Figure 2(b) presents all training images for one subject.

Recognition rate (%)

Recognition rate (%)

60

50

Recognition rate (%)

50

ML
MAP
70
VB

40
30
20

ML
MAP
VB

10
6

40

70
60
50
40
30

ML
MAP
VB

20
10

5
4
3
Number of training images(a) 8 x 8

2

6

5
4
3
Number of training images(b) 24 x 24

2

8x8

16x16

24x24

32x32 40x40 48x48
Number of states

56x56

64x64

Fig. 3. Recognition rate (training data are six images)
Figures 2(c) and 2(e) present the mean vector of the model obtainedwith the UBM and ML method. Figure 2(d) has the mean vectors of
the posterior distributions obtained with the VB methods by varying
tuning parameter τ . The number of states is 40×40 for all models in
Figure 2. Although we can see that UBM roughly represents a facial
shape from Figure 2(c), it is difficult to identify the characteristics
of a particular subject. As tuning parameter τ is increased in Figure 2(d), the mean vector gradually changes from UBM to the imageof the subject in Figure 2(b). Using UBM as the prior distribution
with appropriate tuning parameter τ , similar state alignments are expected to be obtained for all subject models and this therefore avoids
the over-fitting problem. It can actually be seen in Figure 2(d) that
the VB method preserved the shape of the face using UBM, eventhough the shape of the ML criterion in Figure 2(e) had collapsed
due to over-fitting. However, tuning parameter τ needs to be carefully determined, because the optimal value depends on the numberof training data and the number of states. Although a method of
automatically determining the tuning parameter based on cross validation [6] and empirical Bayes have been proposed, we adjusted thevalue in this experiment to obtain the best recognition rate and theadjusted values were obtained in a range from τ = 100 to τ = 6000.
Figure 3 shows the recognition rates for ML, MAP and VB
methods using six images as training data for each subject. We cansee from the results that the Bayesian criterion achieved significantly
better recognition rates than the ML criterion. Similar performance
was obtained under all conditions by comparing the MAP and VB
methods. However, the VB method was slightly better than MAP
when the appropriate number of states was selected. The highest
recognition rates for VB and MAP methods were obtained at 40×40
states (VB: 77.5%, MAP: 75.5%), and ML obtained the best results
at 32 × 32 states (ML: 66.5%). Although the recognition rate for
ML significantly decreased with the increasing number of states, the
VB and MAP methods retained higher performance than the ML
method. This implies that the prior distribution for the VB and MAP
methods effectively avoided the over-fitting problem.
Figure 4 shows the recognition rates for ML, MAP, and VB
methods while the numbers of training data were changed. The VB
and MAP methods achieved higher recognition rates than ML for
all numbers of training images. The difference between ML and
VB/MAP especially became larger when small numbers of training images were used. Although the MAP and the VB methods
had almost the same recognition rates, the VB method obtainedslightly better recognition rates when two training images wereused. These results suggest that the Bayesian approach mitigatedthe over-fitting problem and achieved higher generalization ability
than the ML method. In addition, we confirmed that the use of a

Recognition rate (%)

30

Recognition rate (%)

70
70
60
50
40
30
20
10
0

ML
MAP
VB
6

5
4
3
Number of training images(c) 40 x 40

2

60
50
40
30
ML
MAP
VB

20
10
6

5
4
3
Number of training images(d) 56 x 56

2

Fig. 4. Recognition rate (training data are six to two images)(a) 8 × 8 states, (b) 24 × 24 states, (c) 40 × 40 states, and(d) 56 × 56 statesprior distribution was more effective than the marginalization ofmodel parameters in this task.
5. CONCLUSION
This paper proposed image recognition based on SL2D-HMMs using the VB method. The face recognition experiments were performed on the XM2VTS database. The Bayesian criterion demonstrated better recognition than the ML criterion in the experiments.
These results suggest that the Bayesian criterion is useful for applications of image recognition based on SL2D-HMMs. In addition,
we confirmed from these results that the use of prior distributions
was more effective than the marginalization of model parameters inthis task. We intend to apply the Bayesian criterion to image recognition based on hidden Markov eigenface models, which integrate
SL2D-HMMs and factor analyzers, in future work.
6. ACKNOWLEDGEMENTS
This work was partially supported by the Hori Sciences & Arts Foundation and the Artificial Intelligence Research Promotion Foundation.
7. REFERENCES[1] M. Turk and A. Pentland, “Face recognition using eigenfaces,”
IEEE Computer Society Conference, pp. 586–591, 1991.
[2] D. Kurata, Y. Nankaku, K. Tokuda, T. Kitamura, and Z. Ghahramani, “Face recognition based on separable lattice HMMs,”ICASSP, vol. 5, pp. 737–740, 2006.
[3] J. L. Gauvain and C. H. Lee, “Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov
chains,” IEEE Trans. SAP, vol. 2, pp. 291–298, 1994.
[4] H. Attias, “Inferring parameters and structure of latent variable
models by variational Bayes,” UAI, pp. 21–30, 1999.
[5] K. Messer, J. Mates, J. Kitter, J. Luettin, and G. Maitre,“XM2VTSDB: The extended M2VTS database,” AVBPA, pp.
72–77, 1999.
[6] K. Hashimoto, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda,“Bayesian context clustering using cross validation for speech
recognition,” IEICE Trans. Inf. & Syst., vol. E94-D, pp. 668–
678, 2011.