The JapanSociety
TheJapan Societyof Mechanical Engineeis
EngineersoEMechanical

2Pl-D08

Studyon TelexistenceLXXXI- Effectivenessof Spatial
CoherentRemote DrivingExperiencewith
Telexistence
Backhoe Under Hazardous Conditions-aCharithLasanthaFERINANDOt, moYamen Saraljit,
Yoshio SEISHUt, Nobuo KURIOt, Kouta MINAIV{IZAWAt, and Susumu TACHIt

tGTaduateSchoo1ofMedia Design,Keio University
Machinery & Technology Development Division,ObayashiCorporation
Emailt{charith,yarnen,kout& tachi}@tachilab.org,
kuriu.nobuo}@obayashi.cojp

tt{seishu.yoshio,SpatialCognitien Telexistence Robot System (TORSO)
was mounted
on s driver's
seat
ofspatial Coherenteotperience of sitting on theBackhoe seat and manipulating themachne
was eyaluatedthroughsevera1 experienced and non-experienced
remoteconditions. Furthermore, theQperators by givingthem with a 8-shapedroad trickunder yarious hazardous
waseffectivenessof spatial
coherent driye
experience towards woricing at construction sites and real hazardousfields
discussed,ofa

in thisstudy, a 6 DOF
Unmanned Backhoe

The driveefficiency and effectiveness(BH).

KojTrvbivts:
Teleoperation,
OurdoorOperatiogTelexistenee1. Introduetion
Driving vehicles remotely could allow thehuman crperator toactions at a distantlocation.
Remote clriving may
perfbrrn
helpfulindailyactivities,however, in thispaperwe fbcuson
activities related to disasterand hazardoussituations. If ahuman operator could isolateffom the disastersite, while theoperators
human effectivenesscan be projected
into theremote locations
he could perfbrmoperations with safety,
effectively andmuchefficiently comparedto visiting
the site

During the past three decades, the majorityofphysically.
researchin vehicleteleoperation has centeredon
rate-controlled systems [1]where theoperator is looking attheremote environment
through multiple displays
showingthevideo feedfrom multiple carneras placedon the remoteSite.

More recent vehicle teleoperationsystems have emphasizedtheuse of multi-modal
operator
interfaces
andsupervisory
Fig.1 System Deseription(abRernoteBackhoe,fo)Qperator's
control [2],[3]where sensor fusiondisplayscombine
Cockpig(c)HMD view
inforrnationfrom multiple sensors or datasources into asingle, integrated
view [4],Under supervisory control, an
the leversof theindustrialvehicle
witha pneumaticrobotic
operator divides
a problemintoa sequence and resolves thesystem actuated by pneumatic rubber muscles (PARM) [7],systems provide
problem on a sequential basemanner. Tliese
distance information,collision avoidance
information [8]and CCD cameras for remote vision. These systems
provide technical capabilities of dnving a industrialvehicle
throughnotifications at thc display.Howeyer, it's not
remotely,
however, the visual fiow and spatial coherent
to use theseon a real disastersituation as thepractical
relationship betweenthe remote side and local side has notcomplex manipulations could not be perfbrrnedfaster,safelybeen .mapped,Therefore,the human operator could not feeland confidentially.
as ifheisinside
the machine while manipulation.
Therehasbeen several works
a humanoidrobot

to drive an

the effectiveness ofindusnialvehicle insteadofa

explaining

human operator [5],ffoma remote site. Itenables to use a[6]generaltype of vehicle safely in a dangerousfieldas thedecision is taken by the human operator and perfbrrns
manipulations
over the robot whosits on a cockpitof thebackhoe and manipulates
control
leversfor driving,Similar
workhasbeendonewith

onlyarm

actuators

forcontrvl!ing

Telexistence enables
real-time
actually

ahumanexist,

andto interactwith

These systems
[9],
[10],
visualexperiencespatialcoherent

betweentheremote-local
side andallow
to havehesensation of beingat a placeother than where

mappingsensation,

master-slave

robot

withsystemnotthe rernote

only

environment

provide acoustics andthe development of TELESAR
a combination[11],ofvision,

Ne. 15-2 Proceedingsof the 2015 JSME Conference on Roboticsand Mechatronics,Kyoto,Japan,May 17-19,2015
2Pl-De8(1)NII-Electronic Library Service

The Japan Society
TheJapanSocietyof Mechanical
Mechanical Engineeis
EngineersoE

operating

be very

complex,andrealisticvisualsmechanical

and

has

team

kinestheticsensation

the leversinsidethe backhoe.
However, this could

However,thesesystems are very complex
andnot economical
to be
used
forereating remote drivingexperience.
Our research

auditory

also"TORSO"

interactand

ina moreachievedawas

telexistence

minimalwith human-likeneck[l2]

systemto visually

movements

3-dimensionaldetailsin a remote

explorenaturalachieved.

andcomfortableobjectmanner.

Repeater

W{relessTranscdwet
StereoBaceckhoeg!trDller

hands werefu11HD

remotecontroller,

capturedusing

a dqpth carnera,angle

borderswith

respect

to the surroundings.

3. Implementation

Fig.2 Telexistence
Remete Backhee,SysternOverview

system,low latencyand photonot be thebestoption.

video

a wideRemoteBa[khoeAs shown in Fig.2,in thiswork,amightfeed.As shown in the Fig. 1(b)the operator
binocularhead mounted
displaythathas
motionsensors
to capture the positionand orientation of thehead.These datahas beenused to model the kinematics
fordriving the robot
torso andhead. With the above
configuration.
when
operatormoves
aroundto see somespecific thing around the vehicle, as shownin Fig.1(c)theaccordingly
androbot moves
provides spatial coherent
visuals
together with hisown hands superimposedso thathe
couldfeel the presence inside the remote backhoe
manipulating
the levers.Furtherrnore,with the awarenessof
the body in vehicle space, the operator
could
understandtheto remotevehicleOperatersCoclpit

arrns

from thebackgroundand provided as an overlay

segmented

wearsWirelessHM

Thus the operators

to maintain

TORSO
robet
head mounted
display(HMD), Backhoe
anda dedicated wireless
IP upldown
link
a modified

isused to provide theuser with kinesthetic
and remotevisualsensations at very low latency
over networked operations andto create the spatialcoherent remote driving
experience fora
remote backhoe.

Asslavein Fig, 2 the system
consistsof3sections,Athe operator manipulates
theremote backhoe,
data transinissionlre-transmission equipment
and thetelexistence backhoe. On the operator
side, commer-ciallyavailable

shown

cockpitwirelesswherewideangle

HMD

No: Oculus Rift
(Modelis used to track the users head motion
and provide
ffom the remote
processed video and audio datareceivedtelexistence backhoe,
A depthcamera placedon ffontside ofHMD
front captures the real time hands, contrvller and2. SpatialCoherentRemoteDriyingExperiencesuperimposed
onte the remote
video.
A proprietarywirelesstransceiver operating
at 2.4Ghz band wasused to transfer theWith theadvancementof technology, there'smany
driving 3D video data as well as robot cornmands.
aidssuch

as corner
techniquesvisual
positiondetectors.

cameras,

These drivingaids helps the userwith getting
farniliar
withthe vehicle boundaries
andhaveoutside presence.
However, they are static, and does notchangewhen
the drivertries to look intoa specific angle orsomethingoutside.
To maintainthe sarne spatial coherent
experienceof driving insidea remote
backhoe, the operator
navigation

shouldhe wouldetc.

be provided with thesame
expectwhen

sitsonDK2)

audie-visualfeedbackthat

A radio controlled
backhoe (ModelNo: Acera Geospec
SK135SR-2) is used as the main canier. It can remotely
controlled

forover a 2km distancewith the stocked remotecontroller, 6 DoF telexistence
robot system ismounted
on thebackhoe's driversseat adjusting to match the exact heightof
a typical operator.

the physicalbackhoe. ThisFig. 3 shows the6 DOF telexistenceslave robot used as thedelaysis important main component
to desigri and implement the spatial
when
doingcomplex tasks not becausehumans will not use
coherent remote drivesystem. Tlierebot not only has 3 rovision,
but they could perfbrm multiple 1imb movements
tational DOF dexterityacts as the neck, but also 2 rotational
without
thinkingand looking
at everything
at once.
Similarly,DOF as hip and 1 translational DOF as the spine. The
theuser issupposed to see hisown handsand arrns at the positionand postuTeof the operators headand body streamed
spatialsame

coherent

coordination

place thathe wouldwithout

expectthem to be.

from the cockpit

ln thissystem we achieve thespatial coherent correlation bycapturing
the stereo visualsfrom the backhoe and reconstmct
on theoperators cockpit with a headmounted display.
Whentheoperator changes thepointofview (PoV),theremote
has to captureside

the same PoV. As shownin Fig.
1(a)a 6 DoF robot system isused to mount thecamera head,
where
it fo11ows the user movements
synchrvnously.
If theoperators
hands could be modeled
and ftmctional
as a robotic
camerashand in the remotewasreceived

over

the network

andusedtothenecessary reference angles through kinematics
to drivethe slave robot's motors. At the same time, thecalculatepresent angles
encoders),

andof

each

is captured
joint

by a sensor(e.g.,

the positionand postureof the slave head

fo11ow those of the operator,

onrobot,

he wouldseehisownhands

has a DC motor,
anencoderanda photo
Each jointinterrupter,The photo interrapterdetermines the zeroof the encoder. The drivingperfbrmanceof eachposition
at a speed
is adequate becauseeach joint
can movejoint

No. 15-2 Proceedingsof the 2015 JSME Conferenceen Robotic$and Meehatronics,Kyoto,Japan,May 17-19,2015
2Pl-D08(2)NII-Electronic Library Service

The Japan Society
TheJapanSocietyof Mechanical
Mechanical Engineeis
EngineersoE

demuxed, H264 decoded and images were furtherprocessedfor lens distortioncorrection, color space correction, andcreate
the stereo camera
moclulefinally
converted to oculus ready data.A depth camera
No: Leap Motion) ismounted
on the ffontsurfaceofrequired by the system. To match the Oculusinterpupillary (Modeldistance(IPD),
two cameras were placed at a distance of
64mm. To match
the fieldof view (FOV)of theOculus,
higher than the maximum
Two Full HD carnera
wereused
to customcamera's
unit

stock

speed

modules

efthehuman head.
No:LogitechC615)(Modelofmotion

lenseswere replaced by wide conyersion lens(ModelNo: MA6ICA-WM)

to archive

a FOV

Sbercodemeras

100o.

ofGtls!g･lens unit isvery lightweightso thatthe(320g)
dynamicsof thehuman headcould be maintained.
The carnera

Fig,3 CAD Drawings of 6 DoF Telexistence
RobotHeed
RoMng.HeadYawing-

HeadPitching± 55'asx!'
BinoculerCemerasX,; ---"'1nxlnBend:----------'-tt---'-----j

L--t---.--------

1
jiilijL---

Opcrdn-loal

i111i,

RblmoteBecthoe-eevnvte

Fig.4 Data Flow Diagram of Spatial
CoherentDriyeSysternnpt6

the Oculus to capture

devioessuch
spi ej it hing'xx

spi'e Rliing

controller.

segrnented,

correctedandexperiencemotionof theremotex.xx

express/11/.!

and1i!!

1!

)

Theandmotorcontrol
grabbing, inverse kinematics
a miniPC (ModelNo; lnteIR NUC
Board
D54250WYB) isused, A PCIe to PCI bus expansion box(ModelNo: ECH-PCI-CE-H4B) is used to connect the DIA,AID and InterruptCounterboardsthathasbeen used in therobot. PCI expansion
box connects to mini PC via aMiniCard (mPCIe)to PCIe Xl busadaptor (ModelNo: KZB26-030) The AID and interrupt
counter boardsconnects toFor video

mannernetwork

andwherephotointerrupters

as motor

Fig. 4 shows

the data flow diagram ofthe spatialsystem,

outoperator

motionof hurnans in a moTe
to reflect the characteristics
oflatency wasend-to-end

isfromcould

naturalhumanscoherent

byitwas foundthatthelatency
is 130ms.SincetheOculus

DK2

has 20ms latency from input to display on the screen,
carculated
as 150ms.

total delay was

4. FieldExperiments
The spatial

coherent

backhoewas

remoteconstruetedthat the end-to-end

dnve system

withsuccessfu11y.

latencywas

atelcxistence

It has been proved

not an issueto construct
theHowcver, in order to fu11y
thecapability ofoperating the backhoeremotely,presence experience,

remotedrivers understandithasto be drivenby ordinary machine

drivesystem. Operators head positionand orientation iscaptured from the Oculus DK2 camera, sent to the remoteside via theUDP control bus.The received dataisused to
ealculate the InverseKiriematics
and resultant joint
angtesmeasured

testpattemthrough the remote cameraandalgorithms,

to the DLA board,maskedbinocular
parallax3D vision
backhoe.Sincethe6 DOF robot CIrORSO)
canfeedbacking a video

tntelNuC.-fietorDriversTtrtelVVeight:20kg
tntetNuC,MotorDrivets

connects

the hands werethan conventional robots. Thus, it can acquire motion
information
without moving itsentire body.
parallaxj

theencoders

selectedparallaxand

the upper-bodyaccumteotherCaptured IR data cloud''1'1'!'1'/''x'1
the other noise data.With the aboye'Nx'xxx'xhands andthe operator's

asoperators.

A 8 shaped testdriveroute was decidedto evaluate thetelexistence backhoe remotecones

weredriving experience.

3 red traffic

on theturningpoints
to indicate
thecenterplac)edwas
kept atdistance
is the standard
sdict
turning curve
wereused
to drive the 6 DOF robot's
motors.
Ibe robot
specification of the backhoe.
Operatorsubjects were chosen
moves according to the operator, and thestereo camera data
to be ordinary backhoedrivers(nonremote corrtrol),remote
was
at
720p
resolution
at
60fps
with
I420
contro1
based
backhoedrivers.
and ordinary users whodoesformat.
grabbedThe frarneswere then H264 encoded
andsteamedwith anot haveany driving
experience
ofbackhoe.
UDP payroad. On the operator'sside
the payload wasThe experiment was canied out with remote backhoeof turning point.The center-to-center

7000mrnwhich

No, 15-2 Proceeding$ of the 2015 JSME Conferenceon Rebotics and Meehatrenics,Kyeto,Japan,May 17-19,2015
2Pl-D08(3)NII-Electronic Library Service

The JapanSociety
TheJapan Society

operators
operatorsof Mechanical
Mechanical Engineeis
EngineersoE(2),ordinary backhoeoperators (3)non backhoeOrdinarybackhoedrivers
and ordinary users(35),

on how to use the remote controller togiving instructions
movefront,
back,curve and turn manipulations. Rernete
controller backhoeclrivers
skippedthisstep. Next, they weregivenHMD to wear, and a controller en theirhand. They
wereasked
te look straight front to initialize
the local
position and cormected to the remote backhoe. Once
connectod,
they could see as ifthey were insidethe backhoe

1(c)),
Also,beforemoving(Fig.

to the start point,when
theylook down, they could see theirown hands on the remote
backhoe and holding the remotecontroller.
Telexistence
backhoewas parkeclfew meters behindthe start point.They
wereasked
to moveforward,and time taken to fo11owthe
entire course ffom start point,6 turnsand return back to start

5. Conclusion
We proposeda spatial coherent remote driving
system where
the operator
can manipulate
a telexistence backhoe as ifhe isinsicle
the machine. The effectivenessof spatial coherent
remotednvingexperience

was evaluated with many

remotebackhoe, ordinary
backhoe,and non backhoe operatorsthem
theroadcourseofa8
shapedtestroad track,Itgiving
wasconfirmed that the completed
system can perfbrm thefu11operatienof a backhoe including navigation
andarmoperations under various environmenta1
conditions. in thisthe backhoe wasexperiment,

controlledon aarouncl1OOm wireless.However, as the next

expandthe remote

for evenreachability

distanceof

step

we

plan tolonger

muchdistances.

pointwasmeasured.
Acknowledgementi.,･//,
This projectis supportedby the New Energy and IndnsuialTechnology Development Organizationof Japan (NEDO)/t

the Unmanned Construction
TechnologyProjectandundercollaboratedwith

Obayashi Corporation.

References[1] D. E. McGevern,interfacesin rernote driving,"Sandia
Teeh. Rep,, 198g.
(USA),
teleoperation
intefues;'[2] T. Fong and C. Thorpe,
Autonomous robots, vol. 11,no. 1,pp. 9-18, 2001.
"HumanNational Labs., Albuquengue,

NM"Vehicle[3] T. Fong,C. Thorpe,and C. Baur,interfhces
fbrvehicre'`Adyanced

teleoperation: Collaboratiye
controL

fusiondptlays.and remotedriving
tools,"Autonorneus Robots,
voL 11,no, 1,pp, 77-85.
2001.
evaluation
ffamework for assessing
eperator[4] D. C. Feyle,displays,"in SPIEfiS&T 1992 Sympoperfbrrnancewith multisensor
sium on Electronicimaging: Science and Technology. internationa1
SecietyforOpticsand Photonics,1992, pp.51452S.
[5] H, Hasunum4 K. Nakastimq M. Kobayashi, F, Mifune, Y. Yanagibara,T. Uene, K. Ohyeg and K. Yekoi,
tele-operated humanoid
robot drivesa backhoe," in Roboticsand Automatio4 2003. Proceedings,
ICRA'03. IEEE lnternational
Conference ony vel. 3.MEE, 2003,
pp. 2998-3004.
[6] K.Yokoi,K.Nakashima,M,Kobayashi,H.Mihune,H,HasunumgY.
t`Atele-operatedYanagihaTa,T, Ueno, T. Gokyuu,and K. Endosensor'`ProposedFig,5Experimenta1ConditionsofTelexistenceBackhoeRemote
Drive SystemThe opeTator was situated on a separate cockpit room around
8m close to the startpointand the data communication
wasoccurred through the dedicatedwireless transceiverbeamedby two collinear antennas one on the cockpit room and otheron telexistence
backhoe.A helperpersonalwas kepton thecockpitroom
in case of if anything goes wrong to stop theoperation. The helperperson couldseethe remote"Ahumanoidrobet drivesa backhoeintheopen air," inIntelligent
Robotsandbackhoe inSystems, 2003.aROS2003), ftoceedings. 2003 IEEEIRSJ

intemationalConfuenceon,vol.2.IEEE,2003,pp.1117-1122.

contrelofbackhoe
his vicinity. Experimentswere carried out at Obayashi [7] T. Sasaki,T. Nagai, and K. Kawashimeq
for rescue activities using pneumatic robot system,"in Rebotics andCoorporation,
Tokyo MachineryPlant,
Kawagoe and Public
Automatien, 2006, ICRA 2006. Preceedings2006 IEEE internationa1"RetnoteWorks Research Institute,Tsukuba. Before assembling

DOF

the 6

the driversseat, remote backhoeoperatorsrobot on"Rernotecontrvl

tion site with a pneurnaticrobot system,"voL 17,no. 8, pp. 907-914,200g.

ofbackhoe

at censtruc-Automation in censtruction,

directlysitting on
the backhoe.The same subjects were asked to control the [9] S. Taehi.Telexistence.
Werld Scientific,
2010.
Tachi, K. Minamizawny M. Furukawa and C. L. Fernando.
[10]S.
backhoe in the route with remote controller while looking attTelexistenceffom
1980 to 2012," in In Proceedings of IEEEtRSJ
the backoe from outside, At this step the operators moved
intemationalConference on lnteltigent
Robots and Systems,IROS
with the backhoeas theycould not see thevisuals from first
MEE,2012,pp.5440-S441.
[11]S.Tachi,K Tanie,K.Komoriya,and M. Kaneko,(i):person perspective.Next, they we!e given l{MD and drive
Designand evaluation of a visual display
with sensation of presence,"undercompletely remote.
in Theeryand Practiceof Robotsand Manipulaters,Springer,
june
weregiven the same

Conferenceon.IEEE,2006,pp.3177-3182.

[8] T, Sasakiand K, Kawasbima

route

manipulating'12.

`firele-exister]ce

The telexistence backhoe wasexperiencedbackhoeoperators

backhoeoperators.

justunderstanding
cveryonewasableaswellasnonby ordinary

the remotecontrol commands,
to complete
the route successfu11y

hitting
the trathccones.

Withalmost
without

198S,vol, 1,Irp,245-2S4.

Watanabe, I. KawabuehL N. KawakamL T. Maeda,and S.Tachi,[12]K.
`"rorso:ofcgocennic
telegriosis
system," in in Proceedings
SIGGRAPH
2007 Ernerging Technolegie$, ser.
New York NY, USA: ACM, 2007.
Ernotion$ andto Presence:Cognition,[13]G. RiveqFrom Communication
Experience:Festschrift
CultureTowardstheUltimate
Communicativein Honer of Luigi Anelli.IosPress inc.,2006, vol, 9.
completion

ACM
SIGGRAPHof'07.

No, 15-2 Proceedingsof the 2015 JSME Conferenceon Roboticsend Mechatronics,
Kyoto,Japan,May 17-19,2015
2Pl-D08(4}NII-Electronic Library Service