IMAGE RECOGNITION BASED ON
HIDDEN MARKOV EIGEN-IMAGE MODELSUSING VARIATIONAL BAYESIAN METHOD

Kei Sawada, Kei Hashimoto,
Yoshihiko Nankaku, Keiichi Tokuda
Nagoya Institute of Technology
APSIPA ASC 10/30/2013	 

OUTLINE	 
¢ Background & introduction¢ Models

Probabilistic eigen-image models (PEMs)u Separable lattice hidden Markov models (SL-HMMs)u Hidden Markov eigen-image models (HMEMs)u¢ Training criterion

Maximum likelihood (ML) criterionu Bayesian criterionul HEMEs using variational Bayesian (VB) method (proposed)¢ ExperimentsFace recognition experimentsu Conclusion & future work	 uBACKGROUND	 ¢ Image recognitionu

Assignment of a label to a given input imagel Biometrics, OCR, video recognition, etc.

uIncrease in demand in various fieldsl Security, industrial inspection, entertainment, etc.

¢ Approach to geometric variation in image recognitionu

Heuristic normalization techniquesl Development of task-dependent techniques require high costu

Local features (HOG, SIFT, etc.)l Global information can’t useu

Classifiersl Subspace method : corresponding to only pattern variationl Characteristic of geometric variation is not considered

Focus on techniques for modeling geometric variations explicitly	 INTRODUCTION	 
¢ Hidden Markov eigen-image models (HMEMs) [Nankaku,u

et al.; ’06]	 

Probabilistic PCA and factor analysisl Linear feature extraction

Integrating
two models

Separable lattice HMMs [Kurata, et al.; ’06]l Invariance size and locationu Over-fitting problem because of complex model structuresu¢ Training criterion of probabilistic modelsu

Maximum likelihood (ML) criterionl ML criterion produces point estimation of model parameters⇒ Over-fitting problem when amount of data is insufficientuBayesian criterionl Estimation of posterior distributions using prior information⇒ High generalization ability

Apply Bayesian criterion to HMEMs	 OUTLINE	 
¢ Background & introduction¢ Models

Probabilistic eigen-image models (PEMs)u Separable lattice hidden Markov models (SL-HMMs)u Hidden Markov eigen-image models (HMEMs)u¢ Training criterion

Maximum likelihood (ML) criterionu Bayesian criterionul HEMEs using variational Bayesian (VB) method (proposed)¢ Experiments

Face recognition experimentsu Conclusion & future work	 uPROBABILISTIC EIGEN-IMAGE MODELS (PEMS)	 
¢ Eigen-images are represented by probabilistic models

Probabilistic principal component analysis (PPCA)u Factor analysis (FA)uv

: Observations	
: Factor loading matrix	

wG

xG

wg : Eigen-image (base)	
: Factor vector	

v : Noise vector	
v

wG

xG

Linear feature extraction based on statistical analysis
Image normalization is required as pre-processing

PROBABILISTIC EIGEN-IMAGE MODELS (PEMS)	 
¢ Eigen-images are represented by probabilistic models

Probabilistic principal component analysis (PPCA)u Factor analysis (FA)uv Noise: observations	
vector variancewG

xG・Diagonalmatrix: factor loadingmatrix	
w
eigen-image (base)	
⇒g : FA: factor vector	
・Isotropicmatrix
noise vector	
v : PPCA⇒v

wGxG

Linear feature extraction based on statistical analysis
Image normalization is required as pre-processing

SEPARABLE LATTICE HMMS (SL-HMMS)	 
¢ SL-HMMs have horizontal and vertical Markov chains

State sequences of horizontal and vertical are independent

Vertical state sequence 	
z (2)

u

Horizontal state sequence	z (1)Each pixel isemitted from acorresponding
output PDF	Images are divided into
rectangular regions in the state	

Output PDFs	

Include size-and-location-normalization
Independence assumption of observations

SEPARABLE LATTICE HMMS (SL-HMMS)	 
¢ SL-HMMs have horizontal and vertical Markov chains

State sequences of horizontal and vertical are independent

Vertical state sequence 	
z (2)

u

Horizontal state sequence	z (1)Each pixel isemitted from acorresponding
output PDF	Images are divided into
rectangular regions in the state	

Output PDFs	

Include size-and-location-normalization
Independence assumption of observations

SEPARABLE LATTICE HMMS (SL-HMMS)	 
¢ SL-HMMs have horizontal and vertical Markov chains

State sequences of horizontal and vertical are independent

Vertical state sequence 	
z (2)

u

Horizontal state sequence	z (1)Each pixel isemitted from acorresponding
output PDF	Images are divided into
rectangular regions in the state	

Output PDFs	

Include size-and-location-normalization
Independence assumption of observations

HIDDEN MARKOV EIGEN-IMAGE MODELS (HMEMS)	 
¢ Integration of PEMs and SL-HMMs¢ Eigen-images and noise are generated from SL-HMMs	 v: Observations	
: Factor loading matrix	

wG

xG

wg : Eigen-image (base)	
: Factor vector	

v : Noise vector	
v

wG

xG

Linear feature extraction andinclude size-and-location-normalization

OUTLINE	 
¢ Background & introduction¢ Models

Probabilistic eigen-image models (PEMs)u Separable lattice hidden Markov models (SL-HMMs)u Hidden Markov eigen-image models (HMEMs)u¢ Training criterion

Maximum likelihood (ML) criterionu Bayesian criterionu

l HEMEs using variational Bayesian (VB) method (proposed)¢ Experiments

Face recognition experimentsu Conclusion & future work	 uMAXIMUM LIKELIHOOD (ML) CRITERION 	 ¢ Optimal model parameters

are estimated by

maximizing the likelihood
Training data	 

Test data	 Y

Amount oftraining data	 

Predictive dist.	 P (Y | ⇤ML )Parentpopulation	 

Large	 

Small	 

Training data	 
Estimateddistribution	 

Estimation accuracy is decreased by over-fitting problem

BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 Prior dist.	 

Training data	 

Posterior dist.	 

Test data	 YPredictive
P (Y | O) =dist.	 

Z

P (Y | ⇤)P (⇤ | O)d⇤

BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 
Large amountof face images 	 

Prior dist.	 

BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 Prior dist.	 

Training data	 
Face imagesof one subject	 

Posterior dist.	 	 	 BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 Prior dist.	 

Training data	 

Posterior dist.	 

Test data	 YPredictive
P (Y | O) =dist.	 

Z

P (Y | ⇤)P (⇤ | O)d⇤Using prior dist. and marginalization of model parameters
Complicated integral andexpectation computations

BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 Prior dist.	 

Training data	 

Posterior dist.	 

Test data	 YPredictive
P (Y | O) =dist.	 

ZP (Y | ⇤)P (⇤ | O)d⇤Using prior dist. and marginalization of model parameters
Complicated integral andX ZZ
P (O, x, z | ⇤)P (⇤)dxd⇤
expectation computations P (O) =zx, z : Hidden variable	

BAYESIAN CRITERION	 ¢ Estimation of posterior distribution	 Prior information	 Prior dist.	 

Training data	 

Posterior dist.	 

Test data	 YPredictive
P (Y | O) =dist.	 

ZP (Y | ⇤)P (⇤ | O)d⇤Using prior dist. and marginalization of model parameters
Complicated integral and⇒ ・ MCMC method [Gilks, et al.; ’96]
expectation computations⇒ ・ MAP method [Gauvain, et al.; ’94]
⇒ ・ VB method [Attias; ’99]	

VARIATIONAL BAYESIAN (VB) METHOD (1/2)	 
¢ Estimation of approximated posterior dist.
¢ Define a lower bound F(Q) of log marginal likelihoodln P (O)X X ZZ

Q(x, z(1),z(2)z (1) z (2)P (O, z (1) , z (2) , ⇤), ⇤) lndxd⇤(1)(2)
Q(x, z , z , ⇤)= F(Q)x : Factor vector z (1) : Horizontal state sequence	Q(x, z (1) , z (2) , ⇤) : Arbitrary dist.	 z (2) : Vertical state sequence	¢ KLD between arbitrary dist. Q and true posterior dist. P

KL(Q || P ) = ln P (O)F(Q)

Maximizing lower bound F(Q)⇔ Minimizing KLD
u Arbitrary dist. Q represents
approximated posterior dist.

ln P (O)KL(Q || P )uF(Q)Q

VARIATIONAL BAYESIAN (VB) METHOD (2/2)	 
¢ Assume the independency of random variables

P (x, z (1) , z (2) , ⇤ | O) ⇡ Q(x, z (1) , z (2) , ⇤)= Q(x)Q(z (1) )Q(z (2) )Q(⇤)

Q(x, z (1) , z (2) , ⇤) : Arbitrary dist.	

: VB posterior dist.	

¢ Updates of VB posterior dist. increase the value oflower bound F(Q) at each iteration until convergenceQ̄(z (1) ) = arg max F
Q(z (1) )VB E-step 	Q̄(z (2) ) = arg max F
Q(z (2) )Q̄(x) = arg max F

KL(Q|| P)
KL(Q||PPKL(QKL(Q||||P)) )KL(Q || P )

Q(x)

max F
VB M-step 	 Q̄(⇤) = arg Q(⇤)

ln P (O)

F(Q)

Q

Apply VB method to HMEMs	 OUTLINE	 
¢ Background & introduction¢ Models

Probabilistic eigen-image models (PEMs)u Separable lattice hidden Markov models (SL-HMMs)u Hidden Markov eigen-image models (HMEMs)u¢ Training criterion

Maximum likelihood (ML) criterionu Bayesian criterionul HEMEs using variational Bayesian (VB) method (proposed)¢ Experiments

Face recognition experimentsu Conclusion & future work	 uFACE RECOGNITION EXPERIMENTS	 
¢ Experimental conditions	 

Database	 

XM2VTS	 

Image size	 

64 × 64 pixel，Gray-scale	 

Training data	 

6 images per subject × 100 subjects

Test data	 

2 images per subject × 100 subjects

Model structure	 

SL-HMM, HMEM-PPCA, HMEM-FA	 

Number of states	 

32 × 32 states	 Estimate method	 ML method (ML criterion)，
VB method (Baysian criterion)	 

Prior distribution	 

Uniform distribution (flat)，
Universal background model (UBM)	 

RECOGNITION RATES (COMPARING ML AND VB)	 
Comparing ML and VB methods (prior dist. : uniform dist.)	 
80
Dotted lines : ML method
Solid lines : VB method
Colors	 : Model structures	 

Recognition rate (%)	 

7570

SLHMM-ML
SLHMM-VB
HMEM-PPCA-ML
HMEM-PPCA-VB (proposed)
HMEM-FA-ML
HMEM-FA-VB (proposed)

6560
55
50
1234
Number of factors	 

5VB method achieved higher recognition rates than ML method

PRIOR DISTRIBUTION	 ¢ Prior dist. affects the estimation of posterior dist.

Uniform distribution (flat)u Universal background model (UBM)	 
u

flat	 SLUBM	 UBM	 

Uniform dist.	 

All training samples
of all subjects	 

All training samples
of all subjects	 

Training UBM	 UBM (SL-HMMs)	 
Tuning the influenceof uniform dist.

Convert SL-HMMs
to HMEMs	 UBM (HMEMs)	 

Prior dist.	 

Training UBM	 UBM (HMEMs)	 

Tuning the influenceof UBM	 

Tuning the influenceof UBM	 

Prior dist.	 

Prior dist.	 

RECOGNITION RATES (COMPARING PRIOR DIST.)	 
HMEM-FA	 

84

84

82

82

Recognition rate (%)	 

Recognition rate (%)	 

HMEM-PPCA	 

80
78
7674
72

80
78
7674
72

70

70

68

68
12345
Number of factors	 

12345
Number of factors	 
MLflatSLUBMUBMRECOGNITION RATES (COMPARING PRIOR DIST.)	 
HMEM-FA	 

84848282

Recognition rate (%)	 

Recognition rate (%)	 HMEM-PPCA	 

80
78767472

8078767472707068

68
12345
Number of factors	 

12345
Number of factors	 
ML
HMEM-PPCA : flat outperformed SLUBM
flat
HMEM-FA : SLUBM outperformed flat
SLUBM
⇒ SLUBM is effective for FA (diagonal) structure	 
UBM

RECOGNITION RATES (COMPARING PRIOR DIST.)	 
HMEM-FA	 

84

84

82

82

Recognition rate (%)	 

Recognition rate (%)	 

HMEM-PPCA	 

80
78
7674
72

80
78
7674
72

70

70

68

68
12345
Number of factors	 

12345
Number of factors	 
MLNo significant difference between flat and UBMflat⇒ Prior dist. had tuned under the condition
SLUBM⇒ that the number of factor was one	 UBM

RECOGNITION RATES (COMPARING PRIOR DIST.)	 
HMEM-FA	 

8484

82

82

Recognition rate (%)	 

Recognition rate (%)	 

HMEM-PPCA	 

80
78
76
74
72

80
78
76
74
72

70

70

68

68
1

2
3
4
5
Number of factors	 

1Significant high recognition rate
⇒ High recognition rate can be expected⇒ if prior dist. can be set adequately	 2345
Number of factors	 
MLflatSLUBMUBM

CONCLUSION	 ¢ Focus on technique for modeling geometric variations	 ¢ Apply Bayesian criterion to HMEMs

Derive VB method for HMEMsu Face recognition experimentsul HMEMs based on VB method outperformed ML methodl Recognition rate is improved by using an appropriate priordistribution¢ Future work

Investigation of appropriate parameter sharing structuresof HMEMsu Experiments on various image recognition tasksu