The JapanTheJapan Society
Societyof MechanicalEngineeis
Mechanical EngineersoE

1A1-L06

7 V( tiVX S >X coMza (ag84 tu)>x･mptvFOktoe7-+-e

JV ･ ,N ;iXco rcre2M LNkf

tiVZS

VlcoreEMme
Studyon Tlelexistence
LXXXIV-MutualCommunication forTelexistenceRobots using Virtuat
Hands Prejection-o\mE
E

V-7I D7 b,,w F -Vpt )i (tsdeXpaJk#)
7=fVt7F' fv V - (gems#ngJ<\)
Mme#JtsC(ptkeXesV<\)
ggpt(pats#mpd<\)"OMHD Yhmen SARAM, Keie University yarnen@kmd.keio.ac.jp
CharithLasanthaFERNANDO, Keie University;
charith@krnd.keio.ac.jp
kouta@kmd.keio.ac.jp
Keuta MINAMIZAIkQeL,Keio University;
Susumu TACHI, Keio University;tachi@tachilab.org
the lackofphys icalanns inTelexisteticerobots by prejecting ope[ator's handsimages
intoUser hands are captured frem the firstpoint of view (FPV)using an IR camera meunted on
en top efof robot vision.
user's HMD. and are segmented
frem the background.Those hands are then superinrposed
andmounted
en therobot head isused to project
thepresented to user's HMD. ln therobot place.a picoprQjector
hands after being corTected
intethe rernote environment
objects. Using thismethod,
itissarne imagesef user's
between the user and remote participants towards hand motien andpossibleto have a mutual understanding
intetactions
intheremote place.RemotelyOperatedRobot,Iletexistence

Ihispaperaddresses

the remoteenvironment.

Koj, IPbrds: MixedReality;

1.Introduction
have evolved in social activities andHowever, not all the telepresence
ofrobots previdea sense
presencete the user who iscentrolling.
Furthermore. these typeofrobots ptovidesminimum representationof the operatofs
body and state in the remete placeand usually
Ilelepresence systems

day-to-daycommunications,ef the userusing
a ffont display
providesa 1imited representation
showing
the head only. Though for social and cemmunication
an effective
cemmunication
applieations. our bcxlies
are considered
devices that erribodies {nternalrnenta1 states to the others[1].
of the body
Previousworks addTessed the physicalrepresentation
andtelexistence
for telcoperation[2]E5]. However, theseFig.1 Side-by-sideon Ternete object
arenot portable, too priceyand not ethcient
formobile
pointing
approaches
social contexts. Other approachisusing image based embodimeritfor collaborative screens has been proposed and used in [3][4], In thispaper,we proposea mutual body communication
methodhowever they do not apply forrnobile
telexistenceapp]ications,andfor lightweight telexistence robots that lacks physicalarms. As
were used fordesktopapplications.
shownin Fig. 1, operator]s hands are prejectedintothe Temote

A previously proposed
operatoTs

ownthem, and[61telexistencesystemprovideshands visualsinto his display (HMD) by capturing

segmenting

mobiletheir

ftom the baekground andcontour

te the renK)tethem onfound thatprovidingthe virtual hands improved the sense efbody
presencein the rerrpte place from the user perspectiveonly
however ittackedthe mutual hands representation in to the rernote
superimposingplace. As a result.

hands andsometimes
clues ofenvironnrent

the remoteimages. It wasparticipantswere not aware ofuser'sactions
withrespect to his avatar
body TherefoTe,
due to lack ofvisual
the remete participantsget confused

theeperator interactions.

pico pr{ijectormounted

place objects

usingaligned

head movement.

ontowithatable. remotea physicalto provide theclue

onhead andthe robot

Ihe virtual hands ean be projected
user,

or

to anyremotein ordersurfaces

hands interaction
and intendedactions.
These virtual hands alse providesthe awareness
forthe user about
for the sense ofbody
hisbody,which are necessarypresence.
ofuser's

IIhe proposed method
the arms

enhanceenvironment.
couldandcanbe usedin many

hands representation

ofthe

applications
operator

attoremotethe rernote participants
the intendedinteractions
clearly
though theWitb the proposed method,understandeperatorcannotperfoTmanyphysicatmaniputations.

No. 15-･2Preceedings
ef the2015 JSMECenference on Robotics

Mechatrenics,
Kyote,Japan,May 17-･19,
2015

IAI-L06(1)and
NII-Electronic
NII-Electionic

Library Service

The Japan Society
TheJapanSocietyof Mechanical
Mechanical Engineeis
EngineersoE

Table1Specit"tcations
ofthe' Stereodameres
Model

B;nocularAudioe'',J,-twt.,
Fig.2 Robot headdesign

The developed systern is dividedinto a typieal Master-Slave

ancl visuals

user'sis the contfollingsidemethodwill

Robot moi,ement(Leaprnotion),head movement,
follow.

wherefunherdetailsabout

theandmain

thchead design.FullHD stereo cameras

andareusedto enable

to the user

construction

from robotside.

enablesrobot

compenents

inusedcameras

The robot providesto the remotein the remote place via a LCD

and

1oog

thefront sideof

prejector

used

Lumex
(Model/usingprejectedhands

Bearrrpod)
the FPVa picepro.iector

headtheonmodule.

the remote

enFig. 3

showsvision,asrobot's

well

as

the

projectedhands.There isa slight positionmismatchingin
the pictuie,
to fixthis, a calibration step is required
to be applied
toextract projection
matrix and fixthe mapping.

One ofthe

goalofIhis

mainof keys to control.ieysticks

in the remotetherobet.

Thisin thiswork.

The captured Hands movement
and visuals are used to provide
visualfeedbackto user's side, as well as to be projected in therobot's side. The user observe
his own hands rnotion over robot'svision.To present user's hands in the remoteplace. the captured

isto freethehands ffom any

methodthemotion

intuitiyeinteractionswith the remoteThus it was important to eonsider

ofthe
enyironment

theapproachrobot.

andandto allow

anparticipants.

to contro1

robot"spLace.

communicatien

bee motion in remote places.AIso it has a pico projector for
displayinguser hands in the remote plaee.Ilabte1 liststhe
ofther,20gwithlenses

nios'ement

to see the user's top bedy and anns duringinteractions.
-iedesigned with fu11ywirelessand mobile
platformthatallows

specifications"'eight

2.1. MotionContrel

voice
on

65Lumensuperimposed

contrellingbinauralmicrophones

visualbidirectional
and auditery

participantsusefs video anddisplay and a speakeT
mounted640x48e(mative)t

is contrDlled
thein the robot side (slave).
a3
Degrees of Freedom (DOF)head isused to physically map
user's head rotational motion
at the rernete
place, Fig. 2 shews

1920xl080

Brightness

virtualtheuser islocated.
inthe user side. a set oftracking tools are used
to capture user's head rnevernent (Oculus
DK2) as well as handusing

Resolution

is mountedthatside34ego"zaithlens

side and2. Systemlmplementation

masterProjector

LogitechC615BeampodLumcxMX6S

imagesusing the mountedeameraon the HMD
are first
segmented
to isoLate
the hands from the backgreund. Then those segrriented
hands are superimposedon the visualstream from{he robot side, so
the user can have awareness ofhis handspresence.
The position
andsize of the eaptured
hands are presen'edinthe FPV withhishands,
so the pointing rernains
natural.
Those hands are ser)t to the robot

ditheprejector

andFleldofview

3Axis-iead'movement

camerasCameraPicePajectot

Ilelexistencesystems.

used[7]itwas

In a previous work

propesed to use the eiitire bodythe motionof a camera
in a virtualworld. We proposeinthis work
to use head motion as a method tocontrol the speed
and rotation of the mobile
robot in the remoteplace.Fig. 4 shows the motion mapping between user's head andthe applied directien
to the robot. The speed is measured
as themotion

asdiftlrence

ato control

rnethodwiththeoriginalhead. The advantages
ffom any controllers

calibrated

ofusingsopositienVorientationof theis:1) itfreesthe hands
use
his hands for otherthis method

the usercaninteractions,and 2) itprovidesa natural optical fiowwith the headmovernent.

---l1wardtt
tsicle St MowemeTrtStsltlst

Fig. 3 User's FPVwithhands being superirrrposed

No. 15-2 Proceedings

ofandF ig.4 Body

prejected

the 2015 JSMEConference

onRobetics-basedmotion

to control

robot rnovement

Kyote, Japan, May 17-19, 2015

IAI-L06(2)andMechatrenics,

NII-Electronic
NII-Electionic

Library Service

The Japan Society
TheJapanSocietyof Mechanical
Mechanical Engineeis
EngineersoE

2.2. HandsCapturingandSegmentation
the handsare captured using an IR camera
the HMD. The cameraprovides 110"fieldofview
whichcoyers HMD
FeV; and thusitispossibleto captureuser hands with no cropped
areas. Though the iesolutionof thecameras
are relatively low (640x240).
up sarnpl{ng step isnecessary
to smoothoutthe edges.
In the user

mountedonside,lk

the front of

The advantages

ofusing

IR camera

withcompared

RGB

rk

D
carneraobjects close to the camera
using theintensity,
inour case we capture hands visualseffectively.
However there isa resulting noise from {he background. We apply anonlinear
filtering
functionon the captured irnages.
this fimction
rernoves
the pixelswhichcolor intensity
are below a certainthresheld.
The results efapplying the filtercan be seen in Fig. 5.

DFig.6 TopfSideviewsis the possibitityto capture

Thiscalibration givesmatching

returned

FoV and pT{ijection
size

ofcarnerelprejectoT

for images prejected at theresultsit{saffected when theimages aTe
atdifferent
resultingmismatchingsize and shift ofprojected
plane,
hands position.
Ib iinprovethe results, a depth sensoT is necessary
the projected images.FoT ou[ spplicatien.
to be used to calibrate
withwithinthe range D ± 30cm. no significanceprejections
distortion
was observed,
though hands ghoststarted te appear (Asshown
previously inFig.3).
distance (D).
Howevercalibrated

3. Conclusion
image-basedapproach was used to provide
informationef operator hands infetexistence
robots.

In this paper,anmutualvisua1IIhehands are captured from the user side using an R camera
mountedon hisHMD,
and then superiirrposed on therernote images.
Those hands are also strearned to therobot side and projectedusing
Fig.5 Hands beforeand after segmentation
The captured

from the backgroundapico prejector mounted

onthe headofthe robot.

Using thisitispossible forthe participantste understandwheretheispeintingat or what the user isintendedto do with hishands
intheremote environment.
method,processedimagesare thenused inthe user side
as superimposod
irnagesover the rerrK)te fbedback,providing theabout
hisbody presence. Those imagesare also
operator awarenessstreamed
intothe robot side inorder te be piTljectedin the remoteanduserpiace.

lhiswork

wasAckuowledgments
by JSPS KAKENHI

supported

Grant #26240030

2.3. ProjectorCalibratien
References
between the prQjecter positionand theDue to thedisplacument
camerabetween both
positionin the robot side, and the differcnce
fieldsof vieve the projection of the hands directlywillresult
mismatch scale and positionwhen ebserved by the operator from
theFPV Thus itisnecessary to measuTe thisdisplacement
and scaleby calibrating
the prejector with respect
to one ofthe cameras.
The goalofthecalibration[3]

proeessisto determinethe amount of
the prejectedimage and camera's

displacernent (ttx,cly)
betweencapturedregion.

Also to extract the relative

projected image andthe field of

precess isdoneat a specificFig. 6viewofscale

tbe camera

betweenl4]the(R"IRh).This(D).
prejectiondistanceofthe top andside
views ofillustration
calibration
setup.
The parameters(PPP,HP).
{Mc,fib)represents thesize of prejection
and capture
for the prejecterand the camera
Ari automatedrespcctivcly
process is done to extract thesea chessboard
image intoa specific distanceparurnetersby projecting
whichasof hands reach. Theisset
te100cm
representatien(D)relativescale
ef the prejected images (Rvv;Rh)
iscalculated as theratio between {PP7,,HP)and (FPZr,Hlr).
Thisratio is used as a croppingis usedfacterforthe projected
hands,and the djsplacement Cdu,d.v)
to shift thecropping
region efhands
images.

shows

[s]

anNo. 15-2 Proceedings of the 2015 JSMEConference

onofPolhemus, T.
body as a mediumexpression."
Atlen Lane. 197S.
Fernando, C. L.,Furukawag M., Kuregi, T.,Karnum. S., Sato, K.,
LtDesign
Minamizawa, K., and Thchi,S.
of tclesar v fortransferring
intelexistence."
in inteUigent Robots andbodilyconsciousness
S.ystenrs
aROO, 20i2 iEEEtlZsuintemationaiConjlerenceon. IEEE,
pp. Sl12-5118, 2012.
"Clearboard:A scamlcss
mediumfor
Ishii,H,, and Kobayashi, M.
witheye contacl."in Prvceedings
sharcd drawing and conversationin ct)mpttn'ng
s.vstems,
ofthe SIGCHI cottference on Hiiman foctors
ACM, pp.S25-S32, I992.
tXrellaboard/a novcl
Kunz. A,, Nescher, T,, and Kuchler. M.
interact{veelectronic
whiteboard
for rernote
co11aboration
within Cvben,'o,'lds
t("ll),2010 international
people on conterit."
Couference on, IEEE. pp, 430-437, 201O.
Leithinger,D,, FollTrier;
S.. OlwaL A.. and lshli,H.
telepresence:
shapecapture
anddisplay for ernbodiea
computer-mediatedrumotecollaboration."in Pivceadingsofthe
27thBenthall. J., and[1][2]"Physicalannitat[6][7]'"TheAC-fsyrrLpositrm,ICM,pp. 46I-470,ondset' intetibee sqftvgate

andtechnolosv;

2014,
Saralj{,M, Y:, Fcrnande, C. L, MimLshina. Y/. Ksmiyama, Y.,'CEnforced
Minamizawa, K., and Thchi,S.
telexlstence, teleopei-ting
using
body andhaptic feedback." inphotorealisticvirtual
SIGGR.g2EIAsia2014Eineigingfechnolqgr'es,AaV2014.
Marchal, M., Pettnf,J..& Lecuyer, A.
A hurnan-scaleworlds."
fornaviguting in virtualin 3D User inteij2ices
joystiek
on. pp. 19-26. 2011
r3Dt-, 2011 IEEZi symposittm

Robotics`'JeyTTian:

Mechatrenics,
Kyete,Japan,May 17-19.2015

IAI-L06(3)andNII-Electronic
NII-Electionic

Library Service