Mutual Hand Representation for Telexistence Robots using
Projected Virtual Hands
MHD Yamen Saraiji
yamen@kmd.keio.ac.jp

Charith Lasantha Fernando
charith@kmd.keio.ac.jp
Susumu Tachitachi@tachilab.org

Kouta Minamizawa
kouta@kmd.keio.ac.jp

ABSTRACTIn this paper, a mutual body representation for Telexistence
Robots that does not have physical arms were discussed. We
propose a method of projecting user’s hands as a virtual superimposition that not only the user sees through a HMD,
but also to the remote participants by projecting virtual hands
images into the remote environment with a small projectoraligned with robot’s eyes. These virtual hands are producedby capturing user’s hands from the first point of view (FPV),
and then segmented from the background. This method expands the physical body representation of the user, and allows
mutual body communication between the user and remoteparticipants while providing a better understanding user’s
hand motion and intended interactions in the remote place.

Figure 1. Projected User’s hands into remote objects seen from a remoteparticipant

Author Keywords

Mutual Body Representation; Projected Virtual Hands;
Human Augmentation; Communication; Telexistence
ACM Classification Keywords

H.5.1. Information Interfaces and Presentation (e.g. HCI):
Artificial, augmented, and virtual realities
INTRODUCTION

Telepresence systems have evolved in social activities andday-to-day communications. However, not all the telepresence robots provide a sense of presence to the user who iscontrolling. Furthermore, these type of robots provides minimum representation of the operator’s body and state in the remote place and usually limited to front display showing user’s
head only. Though for social and communication applications, our bodies are considered an effective communication
devices that embodies internal mental states to the others [1].
Previous works addressed the physical representation of thebody for teleoperation and telexistence [5, 2]. However, theseapproaches are not portable, too pricey and not efficient for
Graduate school of Media Design, Keio University.
Address: 4-1-1 Hiyoshi, Kohoku-ku,Yokohama, 223-8526 Japan

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
AH ’15, Mar 09-11, 2015, Singapore, Singapore
ACM 978-1-4503-3349-8/15/03.
http://dx.doi.org/10.1145/2735711.2735777

mobile social contexts. Image based embodiment for collaborative screens has been proposed previously [4, 3], however
they do not apply for mobile telexistence applications.
A previously proposed mobile telexistence system [6] provides operators own hands visuals by segmenting the themfrom the background and superimposing them on to the remote environment from the FPV. It was found out that providing the virtual hands improved the sense of body presencein the remote place, however it lacked the mutual hands representation in to the remote place. As a result the remoteparticipants were not aware of user’s hands and actions withrespect to his avatar body. Therefore, sometimes the remoteparticipants get confused due to lack of visual clues of theoperator interactions.
As shown in Figure 1, we propose a mutual embodied communication method for lightweight telexistence robots thatlacks physical arms. The virtual hands are projected into theremote place using a small projector mounted on the robot
head and aligned with head movement. The virtual hands canbe projected onto a physical table, remote user, or to any remote surfaces in order to provide the clue of user’s hands interaction and intended actions. These virtual hands also provides the awareness for the user about his body, which arenecessary for the sense of body presence.
The proposed method can be used in many applications toenhance the arms and hands representation of the operator
at remote environment. With the proposed method, the remote participants could understand the intended interactions
clearly though the operator cannot perform any physical manipulations.

Pico Projector

HMD &Leap motion

Full HD Cameras
Projected Hands
In Remote Place

3 DOF Head

DisplayFPV SuperimposedHands

Robot Base

Figure 3. Superimposed hands on user’s FPV, and the projected handson robot’s remote place.
Master Side (Local)

Slave Side (Remote)Figure 2. System Overview

SYSTEM IMPLEMENTATION

The developed system is divided into a Master-Slave systems
as shown in Figure 2. In the user side (master), a set of tracking tools are used to capture user’s head movement (OculusDK2) as well as hand movement and visuals (Leapmotion).
Head rotational motion is then mapped into the remote robot’s
mechanical head. Also head linear movement (translation) isused to control robot’s movement speed. The captured Hands
movement and visuals are used to provide visual feedback touser’s side, as well as to be projected in the robot’s side. Theuser observe his own hands motion over robot’s vision.
In the robot side (slave), a 3 Degrees of Freedom (DOF) headis used to physically map user’s head rotational motion at theremote place. FullHD stereo cameras and binaural microphones are used to enable bidirectional communication. Therobot provides user’s video and voice in the remote place via
a LCD display and a speaker mounted on the robot. This enables to see the user’s top body and arms during interactions.
The robot designed with fully wireless and mobile platformthat allows free motion in remote places.
To present user’s hands in the remote place, the captured images using the mounted camera on the HMD are first segmented to isolate the hands from the background. Then thosesegmented hands are superimposed on the visual stream fromthe robot side, so the user can have awareness of his handspresence. The position and size of the captured hands arepreserved in the FPV with his hands, so the pointing remains
natural. Those hands are sent to the robot side and projected using a pico projector (Model: Lumex Beampod) thatis mounted on the head module. Figure 3 shows the FPV superimposed hands on the remote robot’s vision, as well as thevirtual projected hands. There is a slight position mismatching in the picture, to fix this, a calibration step is required tobe applied to extract projection matrix and fix the mapping.
We have conducted initial usability tests and found that it became clear for the participants to understand where the user ispointing at or what the user is intended to do with his hands.

SUMMARY

We proposed a mutual telexistence embodied system that uses
virtual projected hands images in remote place. Using thesevirtual hands, both the user and remote participants can haveawareness of user’s hands motion and intended interactionsin the remote place.
ACKNOWLEDGMENTS

This work was supported by JSPS KAKENHI Grant#26240030
REFERENCES

1. Benthall, J., and Polhemus, T. The body as a medium ofexpression. Allen Lane, 1975.
2. Fernando, C. L., Furukawa, M., Kurogi, T., Kamuro, S.,
Sato, K., Minamizawa, K., and Tachi, S. Design of telesarv for transferring bodily consciousness in telexistence. InIntelligent Robots and Systems (IROS), 2012 IEEE/RSJ
International Conference on, IEEE (2012), 5112–5118.
3. Ishii, H., and Kobayashi, M. Clearboard: A seamlessmedium for shared drawing and conversation with eyecontact. In Proceedings of the SIGCHI conference on
Human factors in computing systems, ACM (1992),
525–532.
4. Kunz, A., Nescher, T., and Kuchler, M. Collaboard: anovel interactive electronic whiteboard for remotecollaboration with people on content. In Cyberworlds(CW), 2010 International Conference on, IEEE (2010),430–437.
5. Leithinger, D., Follmer, S., Olwal, A., and Ishii, H.
Physical telepresence: shape capture and display forembodied, computer-mediated remote collaboration. In
Proceedings of the 27th annual ACM symposium on Userinterface software and technology, ACM (2014),461–470.
6. Saraiji, M. Y., Fernando, C. L., Mizushina, Y.,
Kamiyama, Y., Minamizawa, K., and Tachi, S. Enforced
telexistence: teleoperating using photorealistic virtual
body and haptic feedback. In SIGGRAPH Asia 2014
Emerging Technologies, ACM (2014), 7.