第 18 回日本バーチャルリアリティ学会大会論文集(2013 年 9 月)

Study on Telexistence LXXV
Realtime self-superimposing in Virtual Environments
for Telexistence
テレイグジスタンスのためのバーチャル環境への自己身体の実時間重畳
MHD Yamen SARAIJI 1), Charith Lasantha FERNANDO 2),
古川 正紘 3), 南澤 孝太 4), 舘 暲 5)
MHD Yamen SARAIJI, Charith Lasantha FERNANDO,
Masahiro FURUKAWA, Kouta MINAMIZAWA and Susumu TACHI慶應義塾大学大学院メディアデザイン研究科(〒223-8626 横浜市港北区日吉 4-1-1, {yamen, charith, m.furukawa, kouta, tachi}@tachilab.org)
Abstract: On telexistence, visual consistency is important to deliver existing experience to the user. In thecurrent design, the utilization of humanoid robots provides the minimum required degrees of freedom forhuman bodies. However, it doesn‟t preserve the visual consistency feedback of user‟s body. Thus in this paper
we provide a new method to present operator self-body visuals and motion into robot‟s vision during
teleoperating. In this method, we utilize the estimated trajectory of user‟s hands and arms movement to mask
his local body via a see-through head mounted display and impose the result over robot‟s vision. In this paper
we describe the design and implementation of the technique.
Keywords: Telexistence, Virtual Reality, Self-superimposing

1. Introduction
During a virtual experience in an immersive environment, we
often check for our hands to confirm our existence in thatenvironment. This implies the importance of perceiving an
owned body at the expected location to maintain the immersive
experience, and as a way of verification of begin in that world.
On telexistence to the virtual environment as well as realenvironment, it is difficult to show the his/her own with keeping
appropriate position and orientation between the two first
person‟s view, masking, fusing and color collection in real time
due to some optical constraints, chrome key color errors or

Figure 1: Remained appearance in virtual world

depth measuring errors to cut out. So in this paper, we propose
essential method to cut out the own appearance and fuse it onto

The visual consistency can be categorized in to the two

the environment where the user exists as shown in Figure 1.

depending on the point of view; first person‟s view or not. On

2. Related workthetelexistence,visualconsistency

to theaudience‟s

Such issue can be considered coming from “visualperspective is also important. Thus the avatar robot, which

consistency”. In other words, the hands and arms which appear

represents the remote user but does not look the user, is requiredin from of you when you put them closer to your face, looks

an appearance as same as the user in terms of anonymous. Thisdifferent from your own one, although the appearance of themis the visual consistency on the second and third point of view.

in the virtual or in the remote place while telexistence is

On “TELEsarPHONE” [1], operator‟s body is captured andexpected to be your own one.

projectedon

his

remoteavatarvia

retro

reflective

technology [2] to provide mutual communication. However, theM ethod‟s flow can be described briefly in Figure 2, in thisuser in the avatar could not see his own hand in proper relative

diagram “Operator” represents the active user of the system, theposition between the remote environment and his/her self.

operator will generate movements which are transferred to slave

On the other hand, the first person‟s view requires the
expected trajectory of the hands, which is synchronized with therobot. “Kinematics Solver” element is responsible to convert
operator‟s movement into slave robot as joint angles values.

user‟s hand in real time as well as relative position between theuser‟s eyes and hands correctly. In other works, bodily
kinematics and visual consistency are key points to maintainduring the operation. In “TELESAR V” [3] , a 53 DOF
dexterous slave robot was designed to maintain the bodily andkinematics consistency between master and slave. In the regardof visual consistency, the aim is to provide seamless operating
visual experience to the user which to see his own body rather
than a different body – robot. Thus the problem is about how to
introduce a physical object, which is operator‟s body, into

Figure 2: Superimposing flow diagram

another environment such as in teleoperation environment or
The “Slave” represents the remote avatar of the operator

into a virtual environment.
Noyes and Sheridan [4] proposed a predictive graphic

which can be a physical robot or virtual environment. It outputs

displays where deployed to augment visual feedback andvisual images of the environment which is represented asprovide a superimposed virtual representation of the slave“Vision”. “Virtual M odel” is a simulated model of slave‟s bodyrobot. The slave robot is visible though some obstacles exist

which follows the trajectory of the user. By using this model,

between slave „s camera and hands. However, the method does

we generate a “M ask” image to be used to mask operator bodynot provide the user‟s appearance.

image from the other parts. The operator uses See-ThroughIn M irageTable [5] , physical objects including user‟s body

HM D “STHMD” which is mounted on his head, it has stereo

are captured via depth and color sensors of Kinect andcamera system, and outputs user‟s Point of View “PoV” images.

reproduced in remote environ ment on a table. This method

For masking, we define three basic operations: “(X)” masksrequires an additional sensor to acquire body outline of the user.

out parts of an image by using reference mask layer, “(-)”

In this paper, we focus on addressing visual consistency for

Inverts the reference mask, and “(+)” which combines two

the operator by proposing a new method to superimpose user‟s

images together into a single one. The resulted image isbody visuals on the slave‟s vision. The proposed self-projection

provided to the user in his HM D. Further details described intechniquethe following sections.

can

be

integrated

with

many

teleoperation

applications as well as in Augmented Reality applications byusing mounted camera(s) and body tracking techniques. We

4. System Implementation

describe how the system is organized, and implementation

4.1 Operator’s visual acquisition

details of this method.

To acquire operator‟s appearance in real-time, we utilized a
STHM D as shown in Figure 3, which captures stereo video

3. System Design

images from the operator‟s point of view. The HM D, which we

In this technique, we provide a presentation for user‟sdesigned, provides wide-angle vision of H×V (61◦×40◦), a HD

self-body superimposed over his visual feedback from the slave

LCD 1280x800px, and two USB cameras, Point Grey Research,robot. This presentation should override slave‟s robotic parts

Inc. Firefly (FFM V-03M 2M ), with almost identical field ofusing operator‟s arms and hands which is our focus.

view lens of the HM D (horizontal 62◦), max resolution of

We assume that the operator is using a HM D for visual

752x480px@60FPS were installed on the HM D with afeedback from the slave robot. We assume also the HMD has ahorizontal distance of 65mm. Camera resolution was cropped to

see-through part. This method works by obtaining images frommatch LCD resolution.

operator‟s point of view to capture his visual appearance, andIn Figure 4 a stereo output images from both cameras which

we calculate user‟s active movement and provide it toshow operator‟s hands. The virtual light path length between

kinematics solver to construct a virtual representation of his

the cameras and operator‟s eyes in this design is 8.5cm, it isbody. This virtual representation allows creating a mask image

important to have a conjugated focal plane: to have the same

to identify his body area in the acquired images. Also, we

optical length and center in order to get the similar point ofcorrect light and color information on the resulted masks.

view of the operator.

4.5 Lighting and color correction
In order to match lighting conditions of slave‟s environment,
an estimated lighting model is applied on PoV images. Because
we have a 3D model representing slave‟s body, we apply
lighting equations on this model.
An estimated light correction layer is generated by applyinga Fresnel light equation (1) at each pixel of the simulated
model: F  max( 0 , 1  ( N V )) 2

(1)

where “F” is the Fresnel value, “N” is Normal vector of the
Figure 3: See-Through Head Mounted Display (STHMD)

model‟s pixel, and “V” is the normalized view vector to the

4.2 Kinematics solver

pixel. Figure 7 shows the resulted layer. And in Figure 8, theThe Virtual M odel and the Slave representation in the remotecolor corrected image shows a consistent lighting appearance

environment need to be matched. Thus we utilize the same

between both environments.

kinematic solver [6] to control the slave robot “TELESAR V”
in our virtual environment‟s slave representation. The solver inthis system receives five tracking points from operator‟s body :both sides of shoulders, both sides of hands, and the head. It
map them into 23 DOF body joints. For the hands‟ fingers, theoperator uses data glove to track his fingers and mapping theminto 15 DOF per hand. Kinematic solver generates joint angle

Figure 4: See-Through HMD Cameras images (Left/Right)

data to transfer them to Virtual Telesar.
4.3 Masking
To isolate operator‟s hands and arms from the backgroundpictures, we generate a mask image by usin g the Virtual M odel.
The Virtual M odel is simulated by using Virtual Telesar
Platform [6] , this model receives the same joint angles whichwere generated by the kinematics solver for the slave robot.

Figure 5: Mask images generated by the Virtual ModelIn the simulated environment, the virtual camera is placedwith the same displacement between the eye ball and thephysical camera of the HMD. This ensures matching the
projected size and position between user and Virtual M odel.
In Figure 5, a generated black and white image representing
the mask of arms and hands. White areas shows the parts which
to remain, and black areas are the parts to erase.

Figure 6: Result after fusing the pictures with the mask

Technically speaking, the process of generating the mask wasdone in the GPU as a post-processing phase. The masks wererendered into a separated GPU Render Target. This allowed us
to achieve higher performance than processing it on the CPU.
4.4 Image fusion
After we have acquired the required information to identify
an estimated representation of user body via the mask image,

Figure 7: Estimated light correction layer

we apply masking operation over PoV images. An inverted
mask cuts the outer area of the slave‟s Vision. Figure 6 shows
the combined result of both masked images.
As a result of direct fusion of the images, the resulted super
imposed hands have flat look and hard ed ges with the remoteenvironment, which is a result of different lighting conditions
between the two environments.

Figure 8: Color correction and light matching4.6 Experimental Condition
The method was carried on real-time with frame rate

and avoiding sudden changes in the new calculated masked
color area of the hands.

exceeding 60FPS using CPU core i7 at 2.30GHz/8GB ofmemory and NVidia GeForce GT 650M . Image processing andfusion were done on the GPU.

5. Conclusion
In this paper we proposed self-superimposing system to

The laboratory room shown in Figure 4 was replicated inpreserve visual and appearance consistency for teleoperating

Virtual Telesar as shown in Figure 6 for prototyping and testing.

applications. In order to satisfy that, we presented a technique

Room matched the physical one in terms of scale and objects

which utilizes a see-through camera system, a kinematic solver

localization in the scene.

and simulated environment for mask generation.

User‟s active movement was captured by 14 cameras of
Natural Point OptiTrack V100:R2. HMD has 5 reflective

We will apply the proposed method to the telexistenceexperience to the real world as well.

markers and other 4 points have 3 markers each. The finger
movement is measured by 5DT Data Glove 5 Ultra which

Acknowledgement

provides only 5 DOF for each hand, this yielded the lack ofThis work was supported by JSPS Grant-in-Aid for Scientificmapping to 15 DOF fingers per hand.

Research A (23240017).

4.7 Discussion
It was possible to preserve the hands where we expect it toReferences

be. Also the color tone of the imposed images was improved[1] S. Tachi, N. Kawakami, H. Nii, K. Watanabe and K.

after applying light correction model to user‟s body images.

M inamizawa,

Light correction model can be changed depending on the

M astersituation. Also light placement is possible in the virtual

Retroreflective Projection Technology, SICE Journal ofenvironment to replicate the actual situation for the remote

Control, M easurement, and System Integration, Vol.1, No.5,

environment.

Slave

TELEsarPHONE:

M utualTelexistenceCommunication

Systembasedonpp.335-344 (2008.9)

The appropriate experimental distance for the hands in the[2] S. Tachi, Telexistence and retro-reflective projection

current conditions and design of the STHM D is approximately

technology (RPT), Proc. 5th Virtual Reality Int. Conf.

15 to 40 cm from operator‟s eyes. The camera position, which(IVRIC2003) pp. 69/1–69/9, 2003.

is not conjugated with the eye balls resulted the displacement[3] C.L. Fernando, M . Furukawa, T. Kurogi, S. Kamuro, K.

between mask and captured the user‟s appearance. The issue

Sato, K. M inamizawa, S. Tachi. , “Design of TELESAR V

can be improved using the conjugated optical system with somefor Transferring Bodily Consciousness in Telexistence”

external mirror.

IROS 2012, IEEE/RSJ

However, a noticeable shift of the masked area (especially at
the fingers) caused a slight outline of the local environment toInternational Conference onIntelligent Robots and Systems, 2012.
[4] M .V. Noyes and T.B. Sheridan. A novel predictor for

appear, which is mainly regarded as sensor tracking precision

telemanipulation throughproblem. Also hand position shift depending on the distanceConference M anual Control, M offett Field, CA, 1984.

from the cameras was noticeable. The reason of that shift is dueto the optical displacement of the see-through cameras.

a time delay. In Annual[5] Hrvoje Benko, Ricardo Jota, and Andrew

Wilson.

M irageTable: freehand interaction on a projected augmented

In this experiment, the binocular difference was set to 65mm

reality tabletop. In Proceedings of the SIGCHI Conference

which is cons idered as an average distance for humans,on Human Factors in Computing Systems (CHI '12),

although it can be changed via HMD slider. This value is199-208, 2012.

reflected into virtual environment camera horizontal placement
as well.
Regarding the performance, frame rate was limited to 60 FPS
for rendering the previous scene. A slight delay of tracking the
movement of the user was noticeable when the he performs
sudden moves. A solution for this issue can be performed byproviding a temporal tracking to operator‟s body appearance,[6] M .Y. Saraiji, C.L.Fernando, M . Furukawa, K. M inamizawa,S. Tachi, Virtual Telesar - designing and implementation ofa modular based immersive virtual telexistence platform,
SICE-SI, 2012.