「インタラクション2000」平成12年2月

自動リップリーディングのための位置正規化
OscarVanegas，徳田恵一，北村正名古屋工業大学

LocationNormalizationforAutomaticLipreading
OscarVaKitamura

1 Introduction

thelipstobringthemtothecenterplaceof丘ames．
Fbrtheexperiments，thefirst丘ameoforiginalut−

Some methods uslnglmageS ffom thelipsin motion

terances was taken and a image containing detected

areproposedforaudio−Visualspeechrecognition．How−edgeswasbuiltbysobelfi1tering．Ftomthisimage，anever problems arise when alargevariation ofliplo−
algorithmwascarriedoutinordertofindthegravita−
Cationis present onimages．Therefore，this paper
tionalcentercoordinates（xk，yl），SuChthat，thesum−presents a method to normalize theliplocation on
mation of・intensities to the right and to theleft are

images．The proposed methodis based on asearchalmostequalandalsothesummationofintensitiesto
thetopsideandtothebottomsidearealmostequal．

algorithm ofthelip positionin which aliplocation

normalizationisintegratedin the modeltraining［1］【4】・Inorderto studythe proposedmethod，eXperv

Afterthis，thelip−traCkingprocedureisfo1lowedin

imentsbasedonHMMs（HiddenMarkovModels）［2］ Ordertogetthecompleteset of鈷・ameSinwhichthe
OnSpeaker−independentisolatedwordrecognitionwere
Carried out uslng alip centeringapproach as apre−

lipsarealmostlocatedinthecenterplaceof致・ameS．

processlng and a good recognition performance was

3 Experiments

achievedandtheerrorratewasconsiderablyreduced．
3．1 ExperimentalConditions

Continuousspeechrecognitionuslngliplocationnor−
malizationalsowasimproved．

Fbr experiments，theM2VTS database【5】was used
andimagesfromthelipswereextractedafterconvert−

2 Location Normali2；ation

1ng COlorim早・geSinto gray−intensitiesimages・Eachframe consists of80×40pixels．The word bound−
2．1 LocationNormalizedTrainingariesofthetrainingdatawerefoundbyaHMMbased
SpeeChrecognitionsystemwhichwasusedtosegment
Thispaperisbasedonanormalizationmethodforthe
liplocationwhichwaspresente■dindetailin［4】・As andlabelthe sentences．
Subsamplingofdatawascarriedoutinordertore−
in［4］，gOOdrecognitionresultsforisolatedwordrecog−
ducethecomputationaltime，thatis，Orlglnalframes
Weredividedintoblocks，ノeaChblockhavingtheaver−nition were obtained．Itis shown that the proposed
methodisabletonormalizethepositionofthelips．

ageintensityoftheplXelsinsidethatblock．Fbrthis

Themethodbasicallyiscomposedoffo1lowlngtWO

Study，Subsamplingdatawith blocksof5×5pixels，

StepS：

thatis，tWO−dimensionalfeaturevectorsof128param−● Best Location Search

Fbrthetrainingdataset（I（k−1）），findthebest

liplocationdataset（I（k））inthesensethatthe
likelihood of each wordin the data setis thehighestforthecorrespondingHMMs（k−1），（k＝1，2，‥，m−1）．

●ModelUpdate

Updatethemodel＝MMs（k）bytheBaum→Welch
re−eStimation algorithm uslng alltraining dataset（Z（k））havingthebestlocation．
2．2 Lip GravitationalCent．ering

ThisexperimentwascarriedoutusinglmageSWithlip
trackingandanalgorithmtolocatethecenterplaceof

ー177−

eterswereused．InordertosoIvethelargevariability
Ofintensities，theaverageintensityoveranutterance

WaSSubtractedfromallpixelsonit．
Theliplocation normalized trainlng prOCeSS WaSapplied overthe complete set ofutterances．Exper−

iments of recognition by using a continuous Hidden
MarkovModels（HMMs）werecarriedout．Eachword
modelwasrepresentedbyoneHMMwhichisaleftto
rightmodelwith8statesandtwosingleGaussiandis−
tributionsofdiagonalcovariance．EachHMMmodel
COnSisted ofthe static，delta and acceleration coe侃−
Cients．
Fbrthetestingprocess，theleave−One−Outmethod
WaSuSed．Itmeansthat371eave−One−OutteStingswere

Carriedout，eaChonewith1440trainlngWOrds．Fbrthe

testingprocess，40testwordsperspeaker，prOducing
1480testingutterancesintotal．
Fig．1showstheexperimentalresults・Wordsbe−

Tablel：Contin11011SSpeeChrecognitionresults・

HMMs

datawithliptracking．

CDatameansthatthelipgravi−

wT wTI HO

tationalcenteringalgorithmhasbeenappliedoverthe
data．

Ⅰ

meansintensitynormalization．

kwN wI ‖0け

Original

glnningln HwH means data attributes）SuCh as？HTH

Data withmeans

Location

theiterationnumberofthelocationnormalizedtrain−

Nor叩alization

ingofHMMs（k）with㍑CM，‖TH and㍑Ⅰ門・Theliplo−CationnormalizedtrainlngprOCeSSisalsoappliedoverthetestlngdataset withintensitynormalization・InCaSeOfiteration

0，theliplocationnormalizationwas

appliedoveronlythetestingdata．

normalization method．The normalizedimages werepiledtomakethestring．Spacesbetweenutterances

TheapplicationofthelipgravitationalcenteringaトWerere−built aftereachutteranceuslngSameCOOrdi−natesofthelastimageofeachutterance・
gorithmisverye鮎ctive，theerrorrecognitionratewas
Fbrdatawithorlglnalmovement，areCOgnitionac−

decreasedby4％fordatawithliptrackingand4．3％

67∫

7CuraCyOf52．6％wasobtainedfor wI intablelfor
fordatawithliptrackingandintensitynormalization．
Originaldata．65．0％wasobtainedfor wTI incase
Intheexperiments，areCOgnitionrateof76．9％anda
ofdatawithlocationnormalization．Thelocationnor−
reductionerrorrateof18．5％wereobtainedatiteration
malizationmethodwasappliedforiteration
0 anda1 ．Althougharecognitionaccuracyof76，6％was
recognitionaccuracyof53・1％wasobtainedfororiginalobtainedatiteration 3 in［4］，thismethodgives adataand66．2％incaseofdatawithliplocationnor−
better result atiteration㍑1日．
malization．Theliplocation normalizationis shown
to be efkctive alsoin continuous speech recognition
comparingtheresultswiththeonesobtainedin［3］・
7■

54327

Inthisstudy，Wedescribesomeexperimentsofiso−

7

1

1atedwordrecognitionwithnormalizationofthelip

0location．Priortheexperiment，1ipgravitationalcen−

7

teringandaliptrackingalgorithmswereappliedon

7

imagesandbetterresultswereobtained・97︵ポ︶①ld正uO≡Uぎ00正5 Conclusions

7

Additionalexperimentsofcontinuousspeechrecog−
nitionwerecarriedollt andithasbeenshownthat a

6

WT wCT wTIwCTl O 1 2

applicationoflip−traCkingandnormalizedlipposition

1terationNumber

totheorlglnalmovementise鮎ctivealsoincontinuous

Figurel：E鮎ct ofliplocation normalizationforimages

SpeeChrecognition・

withsubsamplingdata；blocksof5×5withliptracking・

References［1］J・McDonough，T・Anastasakos andJ・Makhoul，

4 Continuous SpeechRecognition

Speaker

AdaptiveTraining：A Maximum Likelihood Approach to
SpeakerNormalization，

ICASSP97，pp．1043−1046，1997・

Fbrcontinuousspeechrecognitiontwoexperimentswere
Carriedout．First，reCOgnitionexperiments uslng H−［2］L．RabinerandB・Juang，tion， Prentice−Hal1，1993．

MMsobtainedftomthenaturalmovementandsecond，
experipentsuslngHMMsobtainedbytheliptracking
andliplodation normalized training．In both cases［3］J・Luettin， Tbwards SpeakerIndependent Continuous
Speechreading ，Eurvspeech97，pp・1991−1994，1997・

凡ndamentalsof郎eechReco9ni−［4］0・VanegaS，K・rIbkuda，T・Kitamura， Location normal−

Subsampled datawith5×5pixelswasused．HMMs

ization of HMM−Basedlip−reading：Experiments fbr theM2VTSdatabase ，ICIP99，26AP3．10．1999．

wereusedwith8statesand2singleGaussiandistribu−

tionswithdiagonalcovariance・Thesilencewasmod−【5］http＝／／www・tele・uCl・aC・be／M2VTS／m2vts−db・pS・Z・
eledwithlstateand2Gaussiandistribut，ionmixturesandcompletestringsofimages丘・OmZerOtOninewere
builtuslngliplmageS after applyingtheliplocation−178−