「インタラクション2000」平成12年2月

位置正規化学習を用いたバイモーダル話者認識

OscarVanegas，徳田恵一，北村正名古屋工業大学

Bimodal Speaker Recognition using Location Normalized Training
OscarVanegas，KeiichiTokuda，TadashiKitamura

NagoyaInstituteofTechnology
1 Introduction

datahaveintensitynormalizationinwhichtheaverage

Valueofintensitiesissubtractedh・Omallthepixelsin

Iswellknownthatinadverseconditions，theuseoflip
imagesforautomaticspeechrecognitioninadditionof

anutterance ．Thewordboundariesofthedatawere
foundbyanHMMbasedspeechrecognitionwhichwas
theacousticspeechsignal，improvestherecognitionre−
usedtosegmentandlabelthesentences．Inorderto
Sults・Duringthespeech，however）themouthreg10n
Suff6rssomedeformations，Changingintensitiesonthe
lipsareaandsometimesallowlngVisibilityoftheteeth

reducethecomputationaltime，imageswithsubsam−
plingdatawereused・Fbrexperiments，Subsampling

ictothespeakerandtothewaythatpersonspeaks，Therefore，WeareinterestedonlyinthemouthreglOnforspeakerverificationandidentification．

3 Lip Location Normalization

andtongue・Thischangecontainsinformationspecif− datawithblocksof5×5pixelsused128parameters．

In this study，We uSed theimage−based method，

Asiswellknown，iflipimageswithalargevariabil−

ityofliplocationareusedfortrainlng，HMMswith

WhichiswidelyusedduetoitssimplicityinitsmodelCOnStruCtion．However，thismethodhasthedisadvan−alargevariancecanbeobtained・Therefore，Wepre−

Sentedindetailourliplocationnormalizationmethod

tageofbeingaffectedverymuchfbrthevariabilityof
intensitiesandforthelargevariationoftheliploca−

in［3］・Thismethodisbasedonasearchalgorithmof

tions・Therefore，thispaperusesamethodtonormal− thelippositioninwhichaliplocationnormalization
izetheliplocationonimages・Inthispaper，eXperi− isintegratedinthemodeltraining［4］．

mentsbasedonHMMs（HiddenMarkovModels）［1］onSpeakerrecognitionwerecarriedout・TheM2VTS［2】Thenormalizationmethodisbasedonthetwosteps
explainedbelow：

databasewasusedforexperiments・Asaresultofap− ●BestIJOCationSearCh：Fbrthetrainlngdata
plyingthedescribedmethod，betterspeakerrecognl−set（I（k−1）），findthebestliplocationdataset（I（k））inthesensethatthelikelihoodofeach

tionperfbrmancewasachievedandtheerrorratewas
COnSiderablyreduced．

WOrdinthedatasetisthehighestforthecorre−

SpOndingHMMs（k−1），（k＝1，2，．．，n−1）．
2 Lip Extraction●ModelUpdate：UpdatethemodelHMMs（k）
bytheBaum−Welchre−eStimationalgorithmus−

ThemouthreglOn，nOrmallypresentsdeformationsand

thegrey−1evelchangesduetospeakerdependentinfor−
mation・Basedonthisfact，aSpatio−tempOralmodel

ingal1trainingdataset（I（k））havingthebest
location．

foreachspeaker．wasbuilt・Fbrtheexperiments，im−ages丘omthelipsinmotionwereextractedfromthe4 Speaker Recognition
M2VTS database・Theimageswereanalyzedanda
4・1 ExperimentalConditions
Centralpoint was visually calculated tryingtoleave
thelipsinthemostcenterplaceoftheframes．
Fbrexperiments，Orlglnalfu11colorimageswerecon−

Vertedintograylevelimages・Imagesof80×40pixels

FbrExperiments，theM2VTS databasewasused．In

thisdatabase，37peopleutterwordsfromzerotonine

intheirnativelanguage・Fiveshotswererecordedand

Ofthelipswerecuttedfromorlglnalimages・Twosetsthemostdi缶cultrecordingtorecognlZeisthenumber
Ofdatawereused，Oneisthe HorlglnaldataM andthe

fivebecauseofspecialcharacteristicsareincludedin

Other㍑datawithliptrackingandwithlocationnormal−this shot．

izedtraining［3］atiteration3，WhichwewillcallinFc）rtrainlngdata，thefirst4shotsoftheM2VTS

thispapertoabbreviate㍑locateddata門・Bothtypeof WereuSedandthefifthfortestingdata・OneHMM−41−

istrainedforeachdigitandspeaker．Fbrthetesting

Tablel：Identification acc11raCies，false acceptance anddata，per uSer，theloglikelihood was calculatedfor

falserqiectionratesglVenbyorlglnaldataandlocateddata，

eachdigituslngtheHMMdigitmodeloftheclaimedk−
thpersonandthesummationoflikelihoodswasfinallyCOmputed・Same calculationwascarriedoutuslngtheHMM
digit modelofeach speaker except the claimed per−SOn．Sothat，final1yweobtainedthelikelihoodforthe

Claimedk−thpersonandtheaveragelikelihoodforthe
remainlngperSOnS．Thedi鮎rencebetweenofthetwo

SCOreSisfinallycalculatedasanormalizedlikelihood．
AthresholdvalueOis calculatedinwhicht，henumber
Offalseacceptanceandfalserqjectionistheminimum．
Table2：Bimodalresults．

4．2 Acoust．ic and LabialSubsystem

The acoustic parameters arethe MelCepstrum coef−
ficientswith static，deltaand accelerationcoefBcients（39components）．LefttorightHMMswereusedwith
8statesandonlyonesingleGaussianmixturewithdi−
agonalcovariance matrix．For thelabialsubsystem，

eachmodelconsists of3states with one Gaussian dis−tributionofdiagonalcovarianCeandthesubsampling
datawasused．Parametersconsistof384components

5 Conclusions

Inthisstudy，eXperimentsofspeakerrecognitionus−
COmpOSedofstatic，deltaandaccelerationcoe用．cients．
1ngaCOuSticandimagesfromthelips，WereCarriedoutFc）r bimodalresults，Whichis considered to be a

Very neW domain［5］，anOrmalization ofscores wasuslngtheM2VTS database．Wedescribedamethod

to normalizetheliplocationonimages丘omthelips

donebyuslngtheslgmOidfunctionforbothcases，a−in motion．Experiments were carriedoutin orderto
COmpareSpeakerrecognitionfororlglnaldataanddata
Withliptrackingandliplocationnormalization．Ithasbeen shown that the described location normalization

coustic and visualdata．

（1）1＋e諾p（り＊（∬−β）））

isverye鮎ctiveforspeakerrecognition．

Where†7is equalto−0，005，Xis the calculatedlog
likelihood and Ois t，he threshold value．In order toCOmbineacousticandimagedata，SlgmOidfunctionsfbr
bothdatawerelinearlycombinedbyweightingfactor

Acknowledgment

αandl−α．

Thisworkwaspartia11ysupportedbytheSecomSci−
enceandTechnologyFbundation．

4．3 Results

Refbrences

Tablesland2showthespeakeridentification accur
racies，the false acceptance and false r匂ection rate
br acoustic，Originalandlocated data．Asin table［l］L・Rabiner．andB・Juang， 凡ndamentalsqf郎eechReco9n章二tion， Prentice−Hall，1993．

l，thelocateddatashowedbetterrecognitionresults． ［2］httpJ／www．tele．ucl．ac．be／M2VTS／m2vts＿db．ps．之Theidentificationratewasincreasedfrom70．3％upto［3］0・VanegaS，K・Tbkuda，T・Kitamura， Location normal−
91．9％．Asintable2，COmbiningboth，thespeechdataization of HMM−Basedlip−reading：Experiments for theandlipimagedata，the′sametendencywasshown．An
M2VTS database Proc．ICIP99，Paper26AP3．10，Octo−
ber，1999．
identificationaccuracyoflOO％wasobtainedforboth，
Originalandlocateddata．2．1％andO％offalseaccep−［4］J．McDonough，Ttanceandfalserejectionwereobtaineduslnglocateddatawhile3・3％and2．7％where▼Obtainedfororiginal

AdaptiveTraining：AMaximumLikelihoodApproachtoS−
peakerNormaliza．tion，Proc．ICASSP，pp．1043−1046，1997．

datarespectively．Datawithliptrackingandliploca− ［5］PierreJourlin，−JuergenLuettin，DominiqueGenoud，Hubert
Wassner， Integrating Acoustic瓦ndLabialInformationfor
SpeakerIdentification・aI！dVerification Proc，Eurosp
pp．1603−1606，1997・

tionnormalizationimprovedthefalseacceptanceand
r匂ectionrates．

−42−