Enforced Telexistence:
Teleoperating using photorealistic virtual body and haptic feedback
MHD Yamen Saraiji∗, Charith Lasantha Fernando, Yusuke Mizushina, Youichi Kamiyama,Kouta Minamizawa, Susumu Tachi
Graduate School of Media Design, Keio University, Japan

Figure 1: (a) Superimposed Virtual Arms Concept on a Remote Telexistence Robot, (b) Touch and Feel Remote Objects using virtual arms,(c) Grabbing a Functional Virtual Copy of TV Remote, (d) Local-Remote Collaborative Digital Whiteboard.

1Introduction

Telexistence [Tachi 2010] systems require physical limbs for remote object manipulation [Fernando et al. 2012]. Having arms andhands synchronized with voluntary movements allows the user tofeel robot’s body as his body through visual, and haptic sensation.
In this method, we introduce a novel technique that provides virtual arms for existing telexistence systems that does not have physical arms. Previous works [Mine et al. 1997; Poupyrev et al. 1998;
Nedel et al. 2003] involved the study of using virtual representationof user hands in virtual environments for interactions. In this work,
the virtual arms serves for several interactions in a physical remoteenvironment, and most importantly they provide the user the senseof existence in that remote environment. These superimposed virtual arms follows the user’s real-time arm movements and reacts
to the dynamic lighting of real environment providing photorealistic rendering adapting to remote place lighting. Thus, it allows theuser to have an experience of embodied enforcement towards theremote environment. Furthermore, these virtual arms can be extended to touch and feel unreachable remote objects, and to grab afunctional virtual copy of a physical instance where device control
is possible. This method does not only allow the user to experience a non-existing arm in telexistence, but also gives the ability toenforce remote environment in various ways.

2System Description

To realize this concept, we developed a Master - Slave Robot system, a human sized ubiquitous telexistence robot that represents
and avatar of the user in a different place. It synchronizes with theuser’s head and leg movements using a head tracking device (Oculus DK2) and a Pressure Pad (Wii Fit). The robot uses two binauralmicrophones and two Full-HD web cameras for audio and video
transmission to the user.
∗ e-mail: yamen@kmd.keio.ac.jp

HMDTracking
Markers

BinauralMicrophones

3 DOF HeadDepth
Sense
Data Glove

Full HD
CamerasVirtual Hands

Robot Base

Pressure
Pad

Figure 2: Master-Slave System Overview

As shown in Fig 2, the system tracks user’s body movement using OptiTrack and data gloves to calculate the posture of the user.
The measured posture is used to animate the virtual hands that act
as a visual representation of his body on the remote place. The
visual quality of the hand plays an important role in order to understand that virtual hand is your own. Image base lighting (IBL) isan efficient method to estimate lights of real environment images,
and to use it to illuminate virtual objects [Debevec 2002]. Here we
propose a method using the captured images from the telexistence
robot to apply it into the virtual arms in real-time. Thus the visual appearance and colour tones of the hands matches the ambient
colour of the remote place.
In this system we are using Oculus VR DK 2 to measure head rotations, and to display the robot vision with the virtual hands presented on it. Robot and the user are connected with a IP network
where the connection is initiated with a server-client network architecture. The robot head movement matches user’s head movement in order to maintain looking direction of the eyes, and robot
base movement is determined based on user’s feet interaction withthe pressure pad placed under. The user can drive the robot forward/backward and rotate the base sideways using the pressure pad.
The robot provides the user a stereo-image FullHD video stream in
realtime with latency less than 200 ms on a local wireless network.

Forces Estimation

Haptic Display (GG)

Old Finger
position

Motion Vector

Penetration
Surface Map

New position

3

SIGGRAPH Asia attendees can perform binaural communication
with other attendees, and experience their own arms and hands being represented as virtual hands merged in the remote side. Furthermore, they could touch and feel the real objects (Fig 1(b)) as if they
were touching them physically. Next, they can grab a functional
virtual copy of a real Television Remote Controller (Fig 1(c)) andturn on the real Television on the remote side. Finally, attendees candraw in a digital telexistecne whiteboard (Fig 1(d)) together with aremote participant where both parties can feel that they are drawing
in the same space with their own hands. The robot is completely
mobile, and safe to move along with attendees in the demo boothside-by-side.

4Figure 3: Calculating touch forces using the reconstructed 3D map
and user fingers motion vectors, and applying the forces to the haptic display (GG)

User Experience

Acknowledgement

This project is supported by JSPS-Grant-in-Aid for Scientific Research A (23240017)

References
D EBEVEC , P. 2002. Image-based lighting. IEEE Computer Graphics and Applications 22, 2, 26–34.

A depth sensor array (PrimeSense) mounted on the robots head isused to reconstruct the remote environment into a 3D representation(surface map) in real-time at the user side. The purpose of thisrepresentation is to estimate the touch forces on user’s finger tips
when he interacts with the remote place objects. Virtual hand’s
fingers are converted into the remote environment 3D space, andbased on the motion vectors of each finger the collision forces arecalculated (vertical and shearing vectors). Fig 3 shows how to themotion vectors of finger tips are used with the generated surface
map to calculate the touch forces. The estimated forces are applied
to the haptic displays Gravity Grabber (GG) [Minamizawa et al.
2007] that are mounted on user’s fingers.

User SideM INAMIZAWA , K., F UKAMACHI , S., K AJIMOTO , H.,
K AWAKAMI , N., AND TACHI , S. 2007. Gravity grabber:
wearable haptic display to present virtual mass sensation. In
ACM SIGGRAPH 2007 emerging technologies, ACM, 8.
M INE , M. R., B ROOKS J R , F. P., AND S EQUIN , C. H. 1997.
Moving objects in space: exploiting proprioception in virtualenvironment interaction. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, ACM
Press/Addison-Wesley Publishing Co., 19–26.

Robot SideN EDEL , L. P., F REITAS , C. M. D. S., JACOB , L. J., AND P I MENTA , M. S. 2003. Testing the use of egocentric interactivetechniques in immersive virtual environments. In INTERACT.

Lighting

RenderingForceDisplay (GG)

Surface GenerationForceestimation

Kinematics Generation

UDPStream

Network (Wired/ WiWireless)

IBL

Oculus DK2

F ERNANDO , C. L., F URUKAWA , M., K UROGI , T., H IROTA , K.,K AMURO , S., S ATO , K., M INAMIZAWA , K., AND TACHI , S.
2012. Telesar v: Telexistence surrogate anthropomorphic robot.
In ACM SIGGRAPH 2012 Emerging Technologies, ACM, Los
Angeles, CA, USA, SIGGRAPH ’12, 23:1–23:1.

UDP
Stream

H264
encoding

Stereo Camera
UDP Compress
StreamStream

Depth Sensor

Robot Base /Head Movements

Wii PadFigure 4: General system diagram describing the main components
used in this system
A general diagram of the system is shown in Fig 4. It highlights themain components used at the local and remote sites to control therobot, and the flow to generate the virtual arms at user site using
the calculated kinematics and IBL using the captured images fromthe remote site.

P OUPYREV, I., I CHIKAWA , T., W EGHORST, S., AND
B ILLINGHURST, M. 1998. Egocentric object manipulation in virtual environments: empirical evaluation of interaction
techniques. In Computer Graphics Forum, vol. 17, Wiley Online
Library, 41–52.
TACHI , S. 2010. Telexistence. World Scientific.