This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere.
第 19 回日本バーチャルリアリティ学会大会論文集( 2014 年 9 月)

テレイグジスタンスの研究（第 7９報）	 
バーチャルボディを介した遠隔環境とのインタラクション	 
Study on Telexistence LXXIX
Remote interaction using humanoid virtual body
サライジ ムハマドヤメン 1)，フェルナンド チャリス 1)，古川 正紘 2) ，南澤 孝太 1) ，舘 暲 1)
MHD Yamen SARAIJI, Charith Lasantha FERNANDO, Furukawa MASAHIRO, Kouta MINAMIZAWA
and Susumu TACHI1) 慶應義塾大学大学院メディアデザイン研究科(〒223-8626 横浜市港北区日吉 4-1-1, {yamen, charith, kouta, tachi}@tachilab.org)
2) 大阪大学 大学大学院情報科学研究科(〒565-0871 大阪府吹田市山田丘 2-1, m.furukawa@ist.osaka-u.ac.jp)
Abstract: Telexistence systems require physical limbs for remote object manipulation. Having arms andhands synchronized with voluntary movements allows the user to feel robot’s body as his body through visual,and haptic sensation. In this method, we introduce a novel technique that provides virtual arms for existingtelexistence systems that does not have physical arms. These superimposed virtual arms follows the user’s
real-time arm movements and reacts to the dynamic lighting of real environment providing photorealistic
rendering adapting to remote place lighting. Thus, it allows the user to have an experience of embodied
enforcement towards the remote environment. Furthermore, these virtual arms can be extended to touch andfeel unreachable remote objects, and to grab a functional virtual copy of a physical instance where device
control is possible. This method does not only allow the user to experience a non-existing arm in telexistence,but also gives the ability to enforce remote environment in various ways.
Keywords: Telexistence, Augmented Reality.

1. Introduction
Experiencing hands and arms movements in a telexistence
robot system is an essential element for the user to feel thesense of existence in robot’s place. More importantly to allow
the user to use his limbs to interact with objects by touching andmanipulating them makes much stronger embodiment towards
the remote environement. Full-scale telexistence systems suchas TELESAR V [1] are equipped with mechanical arms, hands
and touch/tactile sensors, and such systems are designed toperform complicated tasks remotely. However, the complexity
and the cost of such systems can be decreased incredibly if thetarget applications do not involve the need of physical arms

Figure	 1	 System	 overview.	 	 

such as in investigation robots which need vision and head
movement mainly. An example of such portable systems is thephotorealistic virtual arms superimposed in the remote robot

6-DOF robot head: TORSO [2].

instead of physical arms as shown in Figure 1. Those virtual

To compensate the lack of arms and the haptic feedback, wearms provide a natural experience of feeling your own body

propose to adapt an alternative method that deploys

while interacting with the remote objects. However, virtual68arms cannot collide with physical objects in the remoteenvironment. A pseudo-haptic feedback method is developed
which uses a depth sensor to scan the remote place and convert
it into a 3D surface map in real-time, and the touch forces arecalculated using virtual hands fingers and the 3D geometry
surface map of the remote objects.
The visual quality of the hand plays an important role inorder to understand that virtual hand is your own.

Image baselighting (IBL) is an efficient method to estimate lights of realenvironment images, and to use it to illuminate virtual objects

Figure	 3	 System	 pipeline.	 	 

[3]. Here we propose a method using the captured images of theUser	 side	 and	 Robot	 side	 main	 functions	 

telexistence robot to apply it into the virtual arms in real-time.
In this paper, technique on how to integrate the virtual arms

Figure 3 shows an overview of the system pipeline. Most ofwith the robot, generating the pseudo-haptic forces into user’s

the tasks related to the arms such as generating IBL, animating

hands, the process of generating and applying IBL into virtual

the arms, 3D surface calculation, force estimation and haptic

arms was discussed. Also some user cases were discussed.

display are carried on the user side only. The robot side hasvery limited resources, and only provides main functionalities

2. System Design

such as head/base movement, and sending stereo video feed and2.1 Robot Design

depth map information over the network.

To realize this system, it is important to maintain the visual

2.2 Superimposed Handsconsistency between the user’s head and robot’s head motion.

Similar to head movement of the robot is designed to follow

The best-case scenario is to use a 6-DOF robot head such as

user’s head motion, arms movement and hand trajectoryTORSO. However in this paper, we developed a 3-DOF robot

follows the same movements as the user’s. Thus our inverse

head as shown in Figure 2 with higher dynamic response time

kinematics implementation takes into consideration theseto maintain user’s head motion when looking around.

points, and it generates the arm joints posture in real-time by

Wide-angle stereo camera’s are used in order to capture andtracking user’s hands, arms and head. Thus, joint space isdisplay a wide angle of view (110°) of the remote environment.

generated with kinematics and a 3D model representing theUser uses an Oculus Rift DK 1 to experience the telexistence

user’s upper body (arms and hands) is animated andwith virtual arms configuration

superimposed over the real robot at remote environment. The
robot head is also controlled using the same joint space of thevirtual representation to ensure the synchronized motion ofphysical robot and virtual body.

Figure	 2	 A	 3-DOF	 robot	 head	 with	 depth	 sensor	 A depth sensor camera is used to scan remote environment inreal-time and to construct a representative 3D surface in thelocal part to be used when interactions happen from user hands.
The depth sensor is mounted at the same joint of the camera

Figure	 4	 IBL	 processing	 to	 the	 skin.	 Notice	 how	 the	 skin	 part thus it shares the same point-of-vision.

has	 reddish	 and	 bluish	 color	 tone.	 

69To improve the visual appealing of the virtual arms, anFirst person view

Image Based Lighting (IBL) method was developed using thecaptured real-time image feed. Figure 4 shows the result ofapplying IBL to the arms, the ambient reflected color changes
dynamically when the contents of the remote view changes.
However, this estimation method only captures the colors fromthe front view and it lacks back contents.
2.3 3D surface generation and touch forces estimation
Due to the lack of physical arms in the robot side, it is notOld position
Motion Vector

possible to provide physical touch sensors at the interaction
points. So to overcome this limitation, an estimation method

Penetration

was developed using 3D surface representation of the remote
New position

Depth-map

environment and the collision between the virtual hands
trajectories.

Figure	 6	 Calculating	 touch	 force	 using	 depth	 map	 
Figure 6 shows how vertical forces are calculated and touch
is estimated using depth map and finger position. The pushingforces are simply calculated using Hooke’s law:

Fz = −kx(2)x is amount of penetration in the depth map.
Shearing forces are calculated using the tangential motion
Figure	 5	 Depth	 data	 processing	 and	 3D	 surface	 

vector of fingertips along the normal map. To display thesegeneration	 

forces on the fingers, gravity grabber haptic display [5] wasmounted on each finger, and driven using the calculated forces.

A depth sensor mounted on the robot side matching its
point-of-view is used to capture depth information of the3. Applications

remote environment and is sent into the user side as an image

Interacting with the remote environment is an essential thing

stream, forming a RGBD data stream. Next, a preconditioningin telexistence systems. In this system we achieve theis applied to the raw depth data resulting removing noise andinteraction using virtual arms in the remote environment. Figure

recovering missing depth information before generating the 3D

7 shows different scenarios in which the virtual arms can besurface of the remote place. Most of the gaps can be resolvedused. Further explanation about each scenario is presented inusing temporal recovery methods [4], and noise of the depth

this section.

map was resolved using bilateral filter in order to preserve the3.1 Using functional objects

hard edges while removing the high frequency noise. Figure 5

Controlling home appriances in the remote environment isshows how the depth map is processed to produce a 3D surface

possible if the remote robot has physical arms. However, withnormal map.

virtual arm we developed an method to grab a virtual copy ofNormal map is derived from the pre-conditioned depth map,

it’s fuctional remote controller. This way the physicaland each pixel represents a 3D normal vector of the surface.

properties are preserved in the remote environment, but remoteThose normal are calculated by deriving the depth map along X

participants can interact with the virtual copy as it is a physicaland Y space and calculating the cross product of both vectors:!!!!" !!!" !!"N x,y = ∂U × ∂V

remote. By linking the digital behavior of the objects (like lightswitchs, speaker volume,..) with virtual objects over the(1)internet, the remote interaction becomes possible with virtual

The combination of depth map and normal map will result aarms.

surface map relative to camera space. Using the above surface

Using virtual arms to operate and use those functional copies

map and virtual fingers position, it is possible to calculate touchgives a natural interaction with the environment by grapping

forces and shearing forces to be applied to the haptic display ofand operating it. As an example we show in Figure 7 (a) using a

the telexistence operator.

virtual remote control to turn on the TV.

70

Figure	 7	 Various	 scenarios	 using	 virtual	 arms.	 (a)	 Turning	 on	 TV	 using	 functional	 object	 of	 the	 remote	 control.	 
(b)	 Extending	 the	 arms	 to	 touch	 far	 objects.	 (c)	 Writing	 on	 remote	 whiteboard	 (digital	 screen)	 

5. Conclusion

3.2 Superhuman body manipulation
Virtual arms representation has advantage over rigid physicalIn this paper we presented a novel method for real-timearms, they can be extended, shortened depends on the situation

virtual arm substitution in armless telexistence robots. We

to reach a target which is not in reach. For example, the ability

showedto modify the length of the arm to expand the reaching spacesuperimposed virtual arms that follow operator’s body

heretheprocessofgenerating

photorealistic

allowing the user to touch far objects beyond the distance of his

movement in real-time. Depth-map based pseudo hapticarms. Or else, reaching far functional objects without the needestimation method was used to compensate the lack of force

to change the locomotion as in Figure 7 (b).

sensors in the robot side, allowing the operator to touch and feel

3.3 Collaborative work

the physical objects at the correct distance as if he is there.

Collaborative work between two places can be tedious

Using this method allows enforcing the remote environment tospecially when the spatial difference between the twobecome active regardless the lack of physical arms.

environments can vary. For example, when using a shareddigital workspace that remote participants can access as well as

Acknowledgement	 the robot operator in a same spatial dimension. Here an

This work was supported by JSPS Grant-in-Aid for Scientific
Research A (23240017).

example of using digital screen to act as a shared whiteboard
shown in Figure 7 (b) al l owi ng bot h par t i ci pant s t oReferences	 

i nt er act i vel y dr aw on i t . The oper at or per cei ves t he[1] Charith Lasantha Fernando, Masahiro Furukawa, Tadatoshi
Kurogi, Sho Kamuro, Katsunari Sato, Kouta Minamizawa,s cr een as he i s bei ng phys i cal l y next t o i t .

and Susumu Tachi. "Design of TELESAR V for transferring

4. Limitations

bodilyCurrently the interactions are mainly focused on the local

consciousness

intelexistence."

IROS

2012,

IEEE/RSJ International Conference on Intelligent Robots

side or the user operating the robot. Since the avatar robot doesand Systems, pp. 5112-5118, 2012.

not have physical manipulators, participants in the remote place[2] Kouichi Watanabe, Ichiro Kawabuchi, Naoki Kawakami,

will not be able to know whether the user driving the robot isTaro Maeda, and Susumu Tachi. "TORSO: Development ofcurrently touching or the current posture of the arms, thus it isa telexistence visual system using a 6-dof robot head."lacking mutual communication between both participants.

Advanced Robotics 22, no. 10, pp.1053-1073, 2008.

The limitation of manipulating the remote environment can[3] Paul Debevec. "Image-based lighting." IEEE Computerbe partially resolved using virtual functional objects that can be

Graphics and Applications 22.2 (2002): 26-34.

interacted with to do a specific function, such as the examples[4] Massimo

mentioned in the Applications section.

Camplani,andLuis

Salgado.

"Efficientspatio-temporal hole filling strategy for kinect depth maps."
In Proceedings of SPIE, vol. 8920. 2012.
[5] Kouta

Minamizawa,

Souichiro

Fukamachi,

HiroyukiKajimoto, Naoki Kawakami, and Susumu Tachi. "Gravity
grabber: wearable haptic display to present virtual mass
sensation."

InACM

SIGGRAPH

technologies, p. 8. ACM, 2007.

71

2007

emerging