䢴䣃䢴䢯䣏䢲䢷テレイグジスタンスの研究（第 77 報)―実環境内でのバーチャルな身体を用いた触覚伝送―
Study on Telexistence LXXVII-Haptic transfer system using virtual body in real world○MHD Yamen SARAIJI, Charith Lasantha FERNANDO, Masahiro FURUKAWA,Kouta MINAMIZAWA and Susumu TACHI
Keio University, {yamen, charith, m.furukawa, kouta, tachi}@tachilab.org
In this paper, we present a method to calculate touch forces in a remote place using low-cost depth camera andpresent those forces to the user in real-time. Using depth camera‟s captured data, a 3D virtual scene surfacerepresentation is constructed from the remote point of view and used to calculate pushing and stroking forces during
user fingers movement. The algorithm computes force vectors for each desired point using point‟s motion vector andcalculating god-points against the scene surface. By using such a method, it is possible to avoid using complex
articulated arms and hands in the remote side to measure haptic forces at fingers end points. We show here theprocess of capturing and constructing the environment, and the method of calculating the interaction forces using
depth data. Also we suggest using superimposed virtual hands for natural interaction with the remote environment.
Key Words: Augmented Reality, haptic

1. Introduction
In telexistence applications, the degree of presence that theoperator perceives in a remote environment reflects thequality of operation and the sense of presence in thatenvironment [1]. Telexistence robots aim to provide thefeeling of presence in a different location by mapping
human‟s senses and actions with a slave robot. TORSO [2] isa telexistence robot providing the operator dexterous head
movement and stereovision by mapping it with 6-DOF robot
head, but no hands are involved in this representation. InTELESAR V [3] the user is mapped into a full upper bodyslave representation with a 53 DOF robot allowing the user
to operate naturally in a different place. The fingertips of therobot are associated with physical force and tactile sensors.
In a previous study, visual and haptic sensory perception of ahuman works together to compensate each other‟s biased
sensory information misjudgment [4], and using cross-modal
perception from both of them, the judgment on object
properties (like shape and size) would be improved [5].
With regards to virtual environments, previous works [6][7]addressed the ability to access the virtual space and being
able to touch and gain haptic feedback of the virtual objects
by designing telexistence robots with mounted haptic sensors
and simulating them in the virtual environment. Also,
another work [8] was introduced for improving the sense ofpresence in a remote environment by superimposing user‟sbody into a virtual environment. It showed that it is possible
to include your own visual presentation of arms and hands byusing kinematics data and a body model matching operator'sdimensions. Providing the visual sense of user‟s arms andhands existence during operating in a different environment
gives better experience of natural interaction.
In order to provide haptic feedback for non-armedtelexistence robots, we present a depth-based haptic force
calculation and extraction from operator‟s first point of view
during interaction with the hand. This method uses amounted depth sensor aligned with stereo camera vision

mapping user‟s point of view in a remote place, andconstructs 3D scene surface using depth-map of the remoteplace. Using the constructed surface, touch force between
user‟s fingers and the remote environment are calculated andprovided to haptic displays mounted on user‟s fingers. Also
we adapt the visual superimposing of realistic virtual arms
matching operator‟s arms posture into remote environment
images.

2. Design Considerations
In this work, consumable depth sensor “PrimeSense
Carmine 1.08 short range” is used to capture and construct
physical environment‟s geometrical information. Depth
sensors such as Kinect and PrimeSense provide 11-bit
resolution of depth data, and 320x240 up to 640x480
depth-map generated at about 30Hz which is remarkable
quality for many real-time applications. Depth sensors
usually use Infra-Red (IR) light beams in specific patterns for
depth estimation and work in complete darkness.
However, several challenges remain when using raw depth
data from the sensor. Problems such as resulting noise fromthe sensor, and missing values or „holes‟ that no structured
light depth reading was possible as shown in Fig. 1. This canbe the result of certain materials in the environment doesn‟treflect IR light back to the sensor.
To solve such issues, incremental methods create spatial
representation of the environment using series of captured
depth-maps, such as KinectFusion [9] which uses a single
handheld depth camera. Such methods are effective forrobotic investigation in unknown environments, and for AR
based applications. However such systems are optimized toconstruct static scenes and non-movable objects such as
people passing by the camera which is a common case fortelexistence applications. For our purpose, we are onlyinterested in the current state of the scene, thus we calculate
the scene surface per-frame with no temporal information.

䣐䣱䢰䢢䢳䢶䢯䢴䢢䣒䣴䣱䣥䣧䣧䣦䣫䣰䣩䣵䢢䣱䣨䢢䣶䣪䣧䢢䢴䢲䢳䢶䢢䣌䣕䣏䣇䢢䣅䣱䣰䣨䣧䣴䣧䣰䣥䣧䢢䣱䣰䢢䣔䣱䣤䣱䣶䣫䣥䣵䢢䣣䣰䣦䢢䣏䣧䣥䣪䣣䣶䣴䣱䣰䣫䣥䣵䢮䢢䣖䣱䣻䣣䣯䣣䢮䢢䣌䣣䣲䣣䣰䢮䢢䣏䣣䣻䢢䢴䢷䢯䢴䢻䢮䢢䢴䢲䢳䢶䢴䣃䢴䢯䣏䢲䢷䢪䢳䢫

The viewpoint of the depth sensor matches right eye‟shorizontal offset. However because the design is not visual
conjugated between the depth sensor and the camera, asoftware offsetting and scaling step for the depth-map isrequired to correct the mismatched positions and the different
field of views.
Fig. 1 Left) Camera images. Right) Depth-map with gaps.

The method is aimed to be used in real-time, thus we avoid
using polygonal representation of the 3D scene due to thecomplexity, content size, and transmission. The calculations
are done on the local side, and only a surface map (ornormal-map) is generated from the depth-map. Using thisrepresentation it is possible to generated not only pushingforces, but also shearing forces along finger‟s surface.

3.2 Depth preprocessing and surface generation
As mentioned before, using raw depth data from the sensorwill produce several issues regarding resulted artifacts andlost information from the scene. Thus it is necessary toprocess and clean up the data before using it to generatescene surface. To do so, first we propose a filling strategy forthe missing depth values, then we apply smoothing filter on
the resulted depth-map to remove the presented noise while
preserving the hard edges of scene objects.
To recover the gaps of the depth-map, a statistical searchmodel is applied for each missing pixel. The model creates asliding window and picks the value highest frequent depth in
this window. Fig. 4 shows the results of applying this model.
However it couldn‟t fill the entire gaps due to the size of thegaps. To resolve this, temporal methods recover the missing
data from previous frames can be used [10].
The processed depth map is then segmented and sent into
the operator‟s side. Since depth-maps are 11-bits, image
based lossless compression algorithms will not be applied.
Thus it is compressed using ZIP algorithm before sending it.

Fig. 2 System data flow between local and remote sides.

The system is divided into two parts: Remote side inwhich the depth and video cameras are being at and local
side, the place that the user is being. Data flow between bothsides is shown in Fig. 2. In the following section, moredetailed explanation of each step is described.

3. Implementation
3.1 Robot head design
Head design should provide matching between camera
images with depth-map. In Fig. 3 a three degrees of freedom
head was designed with stereo cameras and a depth sensor.

Fig. 4 Recovering missing depth values.
Left) Original depth-map . Right) Recovered map.

Scene surface normal-map is derived from depth-mapvalues. Each pixel of the normal-map is a 3D vector
representing surface direction at that point. Normal vectors
are calculating using the cross product of the derivatives
along X and Y space of the depth values:⃗⃗⃗⃗⃗ × ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗
𝑁𝑥,𝑦 = ‖𝜕𝑈
𝜕𝑉‖
⃗⃗⃗⃗⃗
𝜕𝑈 = [𝜕𝑥, 𝐷𝑒𝑝𝑡ℎ, 𝑦]⃗⃗⃗⃗⃗𝜕𝑉 = [𝑥, 𝐷𝑒𝑝𝑡ℎ, 𝜕𝑦]⃗⃗⃗⃗⃗𝜕𝑈 and ⃗⃗⃗⃗⃗
𝜕𝑉 are derivative vectors for X and Y axis respectively.

Fig. 3 Remote head design

However, the noise from the depth sensor become obvious
on the normal-map, thus a smoothing filter is applied before
calculating the normal. Applying a Gaussian filter on thedepth values which works as a low pass filter in the image
space. However it doesn‟t take into consideration the hardedges of the objects in the image. Thus a bilateral filter isused [11] and applied on the depth-map.
Fig. 5 shows the generated normal-map before and after
applying the bilateral filter on the depth-map.

䣐䣱䢰䢢䢳䢶䢯䢴䢢䣒䣴䣱䣥䣧䣧䣦䣫䣰䣩䣵䢢䣱䣨䢢䣶䣪䣧䢢䢴䢲䢳䢶䢢䣌䣕䣏䣇䢢䣅䣱䣰䣨䣧䣴䣧䣰䣥䣧䢢䣱䣰䢢䣔䣱䣤䣱䣶䣫䣥䣵䢢䣣䣰䣦䢢䣏䣧䣥䣪䣣䣶䣴䣱䣰䣫䣥䣵䢮䢢䣖䣱䣻䣣䣯䣣䢮䢢䣌䣣䣲䣣䣰䢮䢢䣏䣣䣻䢢䢴䢷䢯䢴䢻䢮䢢䢴䢲䢳䢶䢴䣃䢴䢯䣏䢲䢷䢪䢴䢫Fig. 7 First point of view interaction using super imposed virtualarms. Left) Interaction with remote scene. Right) Normal-map andtouch vectors.

Fig. 5 Generated normal-map. Left) Original noise with nosmoothing. Right) Noise reduction using bilateral filter.

3.3 Calculating interaction forces
To calculate the touch forces with the environment, user‟sfinger position should be determined first. To do so, operator
body posture is captured via OptiTrack tracking points
mounted on the head, shoulders and hands. The kinematics ofarms trajectory is calculated using TELESAR V inverse
kinematics [3]. Also a 5DT data glove is used to provide
bending measurements of each finger. With these information
it is possible to determine finger‟s tips location with respectof user‟s head and direction in the 3D space. Also these
trajectories will be used in the next step for superimposing
virtual arms.

4. Conclusion
In this paper, we presented a real-time haptic interaction
system for non-armed telexistence robots. In this method, theuser perceive a superimposed virtual representation of hisarms, hands and fingers that represents his own body while
being operating in a different place. By using depth-map
images of the remote environment, scene surfaces arecalculated and normal-maps are generated which help todetermine when touching occurs with user‟s fingers. Finally,using normal-maps, force vectors of user‟s fingers arecalculated and passed into mounted haptic displays on user‟sfingers tips which provide the haptic feedback to the user.

Acknowledgement
This project is supported by JSPS Grant-in-Aid for
Scientific Research A (23240017).
References[1][2]Fig. 6 Touch point intersection with depth-map.
[3]

Fig. 6 shows the case of touch with the depth-map. Touchforces are separated into two components: pressure andshearing forces. Pressure is calculated using Hooke‟s law:
𝐹𝑧 = −𝑘𝑥
𝐹𝑧 is the resulted force along z axis (pressure axis).
𝑥 represents the amount of penetration in depth-map.

[4]
[5]
[6]

To calculate shearing forces, “motion vector” of finger tips
is used and projected on the depth-map when penetration
happens. This will provide an amount of friction which
represents the shearing value along (X, Y) axis.
To render the haptic on finger tips, Gravity Grabber isused [12] which can provide pressure force and one shearing
direction force along the finger tip.
3.4 Hands superimposing
The previously calculated and measured trajectories of thebody and fingers are provided into Virtual Telesar simulation
platform [6] that in turn apply those kinematics into a bodymodel and super impose it into the remote camera images.
Fig. 7 shows the result of superimposing the virtual arms into
the remote environment, also it shows the mapping of thenormal-map and the calculated touch forces between finger
tips and the normal-map.

[7]
[8]

[9]

[10]
[11]
[12]

Tachi, S. “Telexistence”. World Scientific Publishing Co., Inc.., 2009.
Watanabe, K., Kawabuchi, I., Kawakami, N., Maeda, T., & Tachi, S.
“TORSO: Development of a telexistence visual system using a 6-dof
robot head”. Advanced Robotics, 22(10), 1053-1073, 2008.
Fernando, C.L.; Furukawa, M.; Kurogi, T.; Kamuro, S.; Sato, K.;
Minamizawa, K.; Tachi, S., "Design of TELESAR V for transferring
bodily consciousness in telexistence," IROS, pp.5112-5118, 2012.
Wu, W. C., Basdogan, C., & Srinivasan, M. A. "Visual, haptic, andbimodal perception of size and stiffness in virtual environments".
ASME DYN SYST CONTROL DIV PUBL DSC., 67, 19-26, 1999.
Heller, M. A. "Visual and tactual texture perception: Intersensorycooperation". Perception & psychophysics, 31(4), 339-344, 1982.
Saraiji, M. Y., Fernando, C. L., Furukawa, M., Minamizawa, K., &Tachi, S. “Virtual Telesar-designing and implementation of a modular
based immersive virtual telexistence platform”. In System Integration(SII) IEEE/SICE International Symposium on (pp. 595-598). IEEE,
2012.
Saraiji, M. Y., Fernando, C. L., Furukawa, M., Minamizawa, K., &Tachi, S. “Haptic Layer Design and Integration for Virtual Telesar”.
On Robotics and Mechatronics (Robomec 2013), 2013.
Saraiji, M. Y., Fernando, C. L., Furukawa, M., Minarnizawa, K., &Tachi, S. "Real-time egocentric superimposition of operator's own
body on telexistence avatar in virtual environment". In Artificial
Reality and Telexistence (ICAT), 2013 23rd International Conferenceon (pp. 35-39). IEEE, 2013.
Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O.,
Shotton, J., & Fitzgibbon, A. "KinectFusion: Real-time dense surface
mapping and tracking". In Mixed and augmented reality (ISMAR),
10th IEEE international symposium on (pp. 127-136). IEEE, 2011.
Camplani, M., & Salgado, L. "Efficient spatio-temporal hole filling
strategy for kinect depth-map s". In Proceedings of SPIE (Vol. 8920),
2012.
Tomasi, C., & Manduchi, R. "Bilateral filtering for gray and colorimages". In Computer Vision, 1998. Sixth International Conferenceon (pp. 839-846). IEEE, 1998.
Minamizawa, K., Fukamachi, S., Kajimoto, H., Kawakami, N., Tachi,S,. Gravity grabber: wearable haptic display to present virtual mass
sensation, ACM SIGGRAPH 2007 emerging technologies, Article 8,
1 page, 2007.

䣐䣱䢰䢢䢳䢶䢯䢴䢢䣒䣴䣱䣥䣧䣧䣦䣫䣰䣩䣵䢢䣱䣨䢢䣶䣪䣧䢢䢴䢲䢳䢶䢢䣌䣕䣏䣇䢢䣅䣱䣰䣨䣧䣴䣧䣰䣥䣧䢢䣱䣰䢢䣔䣱䣤䣱䣶䣫䣥䣵䢢䣣䣰䣦䢢䣏䣧䣥䣪䣣䣶䣴䣱䣰䣫䣥䣵䢮䢢䣖䣱䣻䣣䣯䣣䢮䢢䣌䣣䣲䣣䣰䢮䢢䣏䣣䣻䢢䢴䢷䢯䢴䢻䢮䢢䢴䢲䢳䢶䢴䣃䢴䢯䣏䢲䢷䢪䢵䢫