Real-Time Active Human Tracking by Hierarchical Integration of Audition and Vision (2001)

Real-TimeActiveHumanTrackingbyHierarchicalIntegrationofAuditionandVisionKazuhiro Nakadaiy,Ken-ichi Hidai, Hiroshi G. Okunoyxand Hiroaki Kitanoy{yKitano Symbiotic Systems Pro ject, ERATO, Japan Science and Technology Corp.Mansion 31 Suite 6A, 6-31-15 Jingumae, Shibuya-ku, Tokyo 150-0001, JapanxDepartmentof Intelligence Science and Technology,Kyoto Universityoto 606-8501, Japan{Sony Computer Science Lab oratories, Inc., Tokyo 141-0022, Japanfnakadai, hidai, okuno, kitanog@symbio.jst.go.jpAbstractWepresentareal-timemultiplehumantrackingsystemforahumanoidwithpairofmicrophonesand a pair of cameras.Thekey idea forrobust recog-nitionandscalabilityofmulti-modalityisthehier-archicalintegrationofmultiplesensoryinformation.First,sounddirectionbyauditorylocalizationmod-ule,sp eakerIDbyspearidenti cationmodule,facelo cation,faceIDandob jectbystereovi-sion,andmotordirectionareextractedassepateevents.Next,eachstramisformedasatemporalsequenceofevents.Then,di erentstramsaras-sociatedaccordingtotheirproximity.Thisassocia-tion process consists oftwo stages.At the rst stage,anassociationinlo cationornameisperformed.Atthesecondstage,higherlevelassociationbetweenlo cation-andname-associatedstreamsmaybper-formed.Thehierarchicalassociationenablesintegra-tionofvariouskindsstreams and resolveambigui-ties of lowerlevelperceptions.Thefocus-of-attentionis mainly controled by hierarchical association streams.As a result, the humanoid demonstrates robust humantrackingevenwhenseveralpersonsspeaksimultane-ously.1Intro ductionRecently,rob otssuchasHONDAASIMO,SonyAIBO and SDR-3X receivemuch attention.They havecapabilitiesofbip ed-walkingandfriendlysimplein-teraction.These kinds of rob ots will b e deployed forwelfare work, housekeeping or as a p et in the future.Because such a rob ot is exp ected to b ehave as a part-nerofhumans,itshould have capabilities ofp ercep-tion and recognition.However, from the viewp ointofp erception and recognition, a lot of research remainstobedone.Forexample,rob otsshouldunderstandthe surrounding environment autonomously by usingvarious sensor information to select a prop er action ondemand.Suchacapabilityenrichestheso cialinter-action b etween a rob ot and a human.But, currently,arob otcannotunderstandthesurroundingveryro-bustly, so the so cial interaction is not rich enough tobehave as a partner of humans.Torealizerichso cialinteractionwithumans,arob otshouldatleasthavsimilarsensorstothosehumans have, and deal with information obtained bysuch sensors prop erly.In rob ots, vision is often used,while audition, which is one of ve senses for humans,is not p opular except for sp eech recognition.The rea-sons are as follows:1.Auditory pro cessing is less ac-curate.2.It is dicult to capture sound from a singlesource inrealworld,b ecauseamicrophonecapturesamixtureofsoundsevwhendirectionalmicro-phone is used.3.Auditory pro cessing is sensitivetoreverb eration andacousticchangeofaro om.Thesea ect sound source lo calization and sp eech recognitionbadly.Rob ots with auditory pro cessing suchasKismet[1]of MIT AI Lab andROBITA[2]of Waseda Univ.haveb een develop ed.They have a function of sp eech recog-nitionbyusingamicrophonewornnearthemouth.But they cannot use their own microphones for sp eechrecognition b ecause of a p o or signal-noise ratio.Theydo esnothavecapabilityofsoundsourceseparation.ROBITAcan lo calize sound byits own stereo micro-phonesb esidessp eechrecognition.Butitdo esnottakemotornoiseintoaccount,soithasdicultyauditory pro cessing with motion.Tosolve these problems and to make a rob ot withthecapabilityso cialinteraction,wedevelop edareal-timemultipleob jecttrackingsystembyusingfacerecognitionandactiveaudition[3].Activaudi-tion achieves auditory pro cessing with motion byusingsound source lo calization by auditory epip olar geom-Procedings of the 2001 IEEE{RAS International Conference on Humanoid RobotsCopyrightc2001