BioMetal Glove
Masahiro Toyoura ∗Tatsuya Shono
University of Yamanashi

Xiaoyang Mao

Color markers on
BioMetal strings

Virtual objects

Figure 1: (Left) Thin and light BioMetal strings are stitched to the fingertips of a black glove. A user wears the black glove under a white
glove with color markers. (Right) Presenting contact sensation of virtual objects with BioMetal glove. The contact between the markers andthe virtual objects are detected with calibrated stereo cameras.

Abstract

1 Introduction

We propose a new haptic device for rendering contact sensation ofvirtual objects in camera-based Augmented Reality (AR) environments. Haptic feedback can help a user to intuitively sense virtual
objects. For vision-impaired users, it also means a transfer fromoptical information observed in the cameras to haptic information.
In our system, the contact between the virtual objects and the user’s
hand is detected with cameras. Therefore, when presenting the contact sensation, optical markers on the hand should not be occluded
from the cameras so as to avoid disturbing the estimation of 3D
position and posture of the hand. To fulfill such a requirement,
we used BioMetal, a promising and versatile light and thin material that shrinks when electric current is applied, which provides
the haptic feedback. Strings of BioMetal were stitched onto our
proposed BioMetal glove. Because BioMetal does not shrink instantly when energized, a major challenge is how to deal with thetime lag. We address this problem by setting buffering regions for
pre-heating the BioMetal strings.

We explain here the manipulation of virtual objects with the user’s
hand in Augmented Virtuality (AV) and Augmented Reality (AR)systems. AV is realized by capturing the position and posture of
the user’s hand by using special devices. The position of the virtual objects is changed according to the position and posture of theuser’s hand. On the contrary, AR is realized by displaying virtual
objects in “real world” images captured with cameras [Klein andMurray 2007]. In AR, the position and posture of the hand are captured with cameras, and the virtual objects are manipulated in thecoordinate space of the real world. In other words, a virtual handmanipulates virtual objects in a virtual world in the AV environment, while in the AR environment, a real hand in the real world
manipulates virtual objects in the real world. AR provides a senseof existence of virtual objects more effectively than AV.

CR Categories: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, Augmented, and Virtual Realities
Keywords: contact sensation, augmented reality, BioMetal∗ e-mail:mtoyoura@yamanashi.ac.jp

Conventional AR systems have been applied to annotate information to real objects. Mobile phones with cameras have been commonly used for this purpose. Previous AR studies have mainly addressed how to visually display the virtual objects.
In this paper, we propose a new method to present virtual objects
haptically in AR. Users can easily feel the depth of virtual objects
with a contact sensation from them. Depth information is difficult
to display only with visual feedback. Another benefit is that users
with impaired vision can feel the virtual objects in the AR environment with the information captured by cameras presented as haptic
feedback. Our method aims to recognize the posture and position ofhands with cameras and to present the virtual objects according to
the posture and position visually and haptically. Because the device
for presenting contact sensation should not occlude the hand fromthe cameras, it must be small and thin. Conventional contact sensation devices cannot be used in such a camera-based AR system.
R], a metal actuator, for
We propose the use of BioMetal [BioMetal°
making the contact sensation device. The BioMetal shrinks when
it is energized. We used the BioMetal to form a glove by winding
BioMetal strings on the fingertips. Power cables are put on the backof the hand. The glove is thin and wearable. We call it a BioMetal
Glove.

2

Related WorksTM

TM

CyberTouch[CyberTouch ] and CyberGrasp[CyberGrasp ] developed by Immersion Corporation provide contact sensation or reaction force from virtual objects. CyberTouch presents the contact
sensation with 3-cm-long vibrators on the fingertips. CyberGrasp
has an exoskeleton with controlled wires on it. It provides force
feedback through the wires. Both devices are so large that they
may disturb the estimation of hand posture from captured images
when used in a camera-based AR environment. In addition, they
do not have position sensors. Additional special-purpose devices
are required to detect a collision between the hand and the virtual
objects.
Sandor et al. [Sandor et al. 2007] installed visual feedback into a
haptic feedback system. Several visual markers are set on a haptic
feedback device and a head-mounted display (HMD). Calibration
between them is done with the markers. The users can interact onlywith the virtual objects that are located near the haptic device, andonly with a pen-style device. In addition, since the system includes
the haptic device, HMD, markers and position trackers, it is verylarge and cannot be moved.
Aoki et al. [Aoki et al. 2009] proposed a light contact sensation
system using small coils. The small coils pull wires on the fingertips to provide haptic feedback. The device is wearable and weighs
only 1.4 g, enabling moving users to interact with virtual objects
haptically. Minamizawa et al. [Minamizawa et al. 2008] also proposed a contact sensation system with small motors. Scheibe etal. [Scheibe et al. 2007] adopted Shape Memory Alloy (SMA) topresent contact sensation. The system was confirmed to be valid
for a driving simulator. These systems are achieved not only withcoils or SMAs, but also with additional position markers like LEDsor magnetic markers. Although the position markers are indeed
light, the receivers for them are not light and wearable. Moreover,although LEDs present the relative positions to the observing cameras, the change in position of cameras has not been considered. In
addition, none of those systems take into consideration the fact thatthe devices would visually occlude the hands. In fact, there was notrial to introduce contact sensation for a camera-based AR system.

feedback accordingly. Since the virtual object is manipulated by
the virtual hand rather than the real hand, the size of haptic devices
does not have to be considered. Actually, a larger size is desirable
in order to provide a guide for calibrating the position sensors to theposture sensors. Once the calibration is done, the position sensors
can no longer be moved. Moreover, it is difficult for the user to feelthe virtual objects as real ones since the contact is calculated basedon the positions of the virtual hand and the virtual objects.
In contrast, in the AR environment (Figure 2(b)), the 3D positionand posture of the hand are often calculated with images from cameras, because AR requires the position and posture of the hand relative to the cameras. Video-see-through HMDs or PDAs make itpossible to display the virtual objects on a real hand from the viewpoint of the user. Color glove [Wang and Popovic 2009] and Handy
AR [Lee and Höllerer 2007] are two typical examples of such systems. Both systems, however, do not provide any haptic feedback.
When introducing haptic feedback to such camera-based systems,
we need to be careful to avoid visually occluding the hand by the
gloves or devices for tracking the hand.

4 Virtual Object Manipulation with Stereo
CameraFigure 3 depicts how to achieve manipulation of virtual objects inan AR environment using the proposed BioMetal glove. BioMetal
is a kind of SMA. It tolerates repetitive shrinkage and stretch andworks quickly compared to other SMAs. It is also used as an artificial muscle of a robot.
The virtual objects are manipulated with colored fingertips. The positions of the fingertips are estimated using calibrated stereo cameras. We calibrate the stereo cameras with Zhang’s method [Zhang
2000]. The contact between the virtual objects and the hand in thereal world can be detected since the calibration makes it possible
to project the real coordinates on the virtual coordinates. The contact is presented both visually and haptically. Haptic feedback isachieved through the shrinking of the BioMetal strings.
Stereo cameras

3 Presenting Contact Sensation for AR
Before describing the proposed system, let us first clarify the requirement for presenting the contact sensation in AV and AR environments. As shown in Figure 2, the requirement for AR is different
from that for AV.

Virtualhand

User’s
hand

User’s hand

Virtual object(a) Augmented Virtuality

User’s hand

Virtual object(b) Augmented Reality

Figure 2: The difference between AV and AR. Requirements forcontact sensation devices are different in AV and AR environments.
In the AV environment, as shown in Figure 2(a), the 3D position andposture of the hand are often captured by special-purpose devices
such as data gloves or magnetic position sensors instead of cameras.
A virtual hand is displayed in a virtual environment. The contact is
detected by testing the collision of the virtual hand and the virtual
object in the virtual world, while presenting the visual and haptic

Color marker
BioMetal string
Glove
Power cables

Virtual object

Figure 3: Manipulation of virtual object with user’s hand incamera-based AR environment. Color markers indicate 3D positions of fingertips to stereo camera, and BioMetal strings present
contact sensation.
Each fingertip is colored with a marker in a unique color. Eachof the stereo cameras detects the colored regions in its own image.
HSV color system is adopted to detect the regions. The detected
regions are labeled, and the center of each colored region with the
maximum size indicates the 2D position of the corresponding fingertip. The 3D position of the fingertip is calculated with the calibrated stereo cameras. Although some previous systems used a single camera to estimate the position [Wang and Popovic 2009], we
adopted stereo cameras to achieve more accurate 3D positions. The

depth in particular can be much more accurately estimated when
the stereo cameras are used.

5

Design of BioMetal Glove

5.1 Physical Characteristics of Biometal

Shrinkage ratio (%)

BioMetal is a kind of light and noiseless SMA. One disadvantage
of BioMetal is that its response to energization is slow. A time lag
occurs with both shrinkage and stretching. We solve this problem
by providing a buffering region to the virtual objects and the details are described in Section 5.3. Figure 4 represents the thermal
responsiveness of the BioMetal.

0

70
100
Temperature (oC)longer when long strings are used, whereas the feedback is insufficient for the contact sensation when short ones are used. For eachstring, a pair of power cables is required to connect the string to the
TUSB-ADAPIO unit. The cables were positioned on the back of
the hand.

5.3 Buffering Region for Quick Response
As shown in Figure 6, to eliminate the time lag caused by theslow response of BioMetal to energization, we proposed covering
the virtual objects with a buffering region. When the fingertips of
the user enter the region, low energization is provided to pre-heatthe BioMetal strings. Pre-heated BioMetal strings can then shrink
quickly when fully energized at the moment the user touches thevirtual object. Since the ratio of shrinkage against energization depends on the air temperature, the pre-heating energization is tuned
by testing in an actual environment. When the air temperature is
20 ◦ C, we pre-heated the BioMetal strings to about 60 ◦ C with
100 mA current. A 3cm string shrunk within 0.5 second with thebuffering region, while it takes 0.8 second without the buffering
region.
Virtual object

Figure 4: Thermal responsiveness of BioMetal. Biometal drastically shrinks and strecthes in the vicinity of 70 ◦ C.
The BioMetal shrinks when heated. Energization at 150 mA initiates a heating to 70 ◦ C and a shrinkage of 4% of the natural length.
A spring style BioMetal string shrinks 50% with the same energization. Air cooling causes the BioMetal to stretch. Thinner BioMetal
string heats and cools more quickly than thicker types.
In our implementation, the energization was done using a TUSBADAPIO device by Turtle Corporation [TUSB-ADAPIO]. We wereable to control the energization from 0V to 2.5V in increments of
0.1V .

5.2

Installation of BioMetal Strings

The BioMetal strings were stitched onto a glove, as depicted in Figure 5. The terminals attached to the strings are fixed to the glove.
When the strings are shrunk, the contact sensation is presented on
the palmar region of the fingertips. Since the strings and terminals
are light and thin, they do not prohibit the hand from being capturedby the cameras.

GlovePower cable
Terminal
Biometal

Buffering region

Figure 6: Buffering region to compensate for slow response of
BioMetal strings.
A possible alternative is to compensate for the slow response by
setting the level of energization to be inversely proportional to the
distance between the fingertips and the virtual objects. This method
can avoid presenting false contact sensation when the fingertips arefar from the virtual objects. However, it is difficult to implement
due to the thermal characteristics of the BioMetal. The BioMetal
suddenly shrinks when it is heated above a threshold level. Therefore, we cannot control it as a smooth function of distance. As asolution, we adopted binary energization, which is controlled by
testing whether the fingertips have entered the buffering region.
Although the setting of the buffering region is effective for overcoming the slow shrinkage, it cannot solve the problem of slow
stretching. We need a method to cool the heated BioMetal instantly.
One possible way is to cover the BioMetal strings with refrigerant.
However, devices implemented in such a way tend to be large [Mascaro and Asada 2003]. We did not adopt such a solution so that we
could keep the system simple.

Figure 5: Design of BioMetal Glove.

6 Experimental Results
We limited the energization to 3V , and the temperature of the
BioMetal strings were controlled to be under 80 ◦ C. In addition,
since BioMetal has low thermal conductivity, users are hard to sense
its heating. To avoid burn injuries, we placed a piece of cloth between the BioMetal strings and the fingertips. Rubber sheets aresafer, but we chose cloth since rubber sheets would restrict the freemovement of the user’s fingers. Optical markers can be put on the
BioMetal strings directly, or on an additional glove covering the
BioMetal glove. This makes it possible to present the contact sensation without occluding the hand from cameras. The lengths of thestrings were set to 3cm empirically. Shrinkage and stretching take

We conducted a user study to investigate whether the BioMetal
glove is valid for presenting the sensation of virtual objects. For
the task, we asked subjects to find the virtual ball that had contact
sensation out of five virtual balls (Figure 1). Stereo cameras were
installed on the display monitor. The subjects had to try to touch
the virtual balls while watching the monitor. Four of the balls did
not present visual feedback even if the subjects’ fingers reached theballs. In the experiment, the BioMetal strings were attached to thefingertips of the forefinger and middle finger of the glove. Although
we could have attached strings to all five fingertips, we used only

two of them to keep the glove compact and light, since the main purpose of the current experiment was to test the validity of BioMetal
for presenting contact sensation.

device in camera-based AR systems is to avoid occluding the 3D
posture and position of the hand from the cameras. We achieved
this with our proposed BioMetal glove.

The ball with the contact sensation was randomly selected from thefive balls. The subjects were five university students. We analyzed
the number of correct selections by the subjects. We also asked thesubjects to answer the following questions on a scale of 1 to 5 – “5:agree very much,” “4: agree,” “3: neutral,” “2: disagree” and “1:disagree very much.”

In the future, we plan to improve the buffering region techniqueto alleviate the slow response of the BioMetal as. Currently, thebuffering region was set only to solve the slow shrinkage problem, and the problem of the slow stretching remains unsolved. This
makes it difficult for users to sense virtual objects since the contact
sensation was presented even when the users’ hand was detachedfrom the virtual objects. The importance of solving the slow stretch
problem was brought out through subject experiments. Another solution is to connect the two ends of the string to additional strings
to be energized when a fingertip is detached from the virtual object. The shrinking of the additional strings would pull the twoends of the string in the opposite directions and hence, result in thestretching of the string. By pre-heating the additional strings, we
can achieve quick stretching of the string. Predicting the contactfrom the posture and position of the user’s hand in a time sequence
may also help to solve the problem.

Q1: The glove is easy to wear.
Q2: The glove allows free movement of the fingers and hand.
Q3: I can feel the contact sensation with the balls.
Q4: I can easily recognize the ball with contact sensation.
Q5: I feel the balls exist in the real world.
Q6: I want to use the glove in the future.
We obtained an accuracy rate of 88% – 22 correct answers from 25
trials for the selection of the ball with the contact sensation. No user
gave two or more incorrect answers. The accuracy rate suggests thatthe BioMetal glove is effective for presenting contact sensation.
The results of the questionnaire are shown in Figure 7.
4

2
5
5

References

4
5

3
4

Q1

Q2

2

Q3

1

3
5

4

3

Q4

3

2

Q5

Another future goal is to refine the system from a hardware perspective. More strings covering all the fingertips and other areasof the hand would present more detailed contact sensation, whichis expected to be effective for sensing the shapes of virtual objects
and manipulating them in an AR environment.

AOKI , T., M ITAKE , H., K EOKI , D., H ASEGAWA , S., AND S ATO ,M. 2009. Wearable haptic device to present contact sensation
based on cutaneous sensation using thin wire. In Advances in
Computer Entertainment Technology, 115–122.
R
B IO M ETAL°. Toki Corporation.

4
Q6

C YBER G RASP
Figure 7: Results of questionnaire.
The answers of all subjects for Q1 were “agree very much” or“agree,” and four out of five subjects answered “agree very much”
or “agree” to Q2. Therefore, we can conclude that the BioMetal
glove was easy to wear, and the glove did not make it difficult tomove the hand.
All subjects agreed to Q3. This means that the BioMetal glove succeeded in presenting the contact sensation. On the other hand, thescore for Q4 was lower than that for Q3. The users have to focus
their attention in order to sense the contact sensation. The problem can be partially solved by having larger areas of the fingertips
covered by BioMetal strings. The slow stretching of the BioMetal
strings also contributes to the problem. Although we succeededin compensating for the slow shrinkage with the buffering region
around the virtual objects, we should also refine the system to respond quickly when the hand and the virtual objects are detached.
Most of the subjects answered “neutral” or “disagree” to Q5. Thismeans that our system still lacks immersiveness. A more refined
AR environment, such as a system using HMDs, should produce ahigher score.
The score for Q6 was high, which gives good prospects for thisresearch.

7 Conclusion and Future Works
We proposed a new method for presenting contact sensation in an
AR environment. A major requirement for the contact sensationTM

C YBERT OUCH

TM

. CyberGlove Systems.
. CyberGlove Systems.

K LEIN , G., AND M URRAY, D. 2007. Parallel tracking and mapping for small ar workspaces. In ISMAR.
L EE , T., AND H ÖLLERER , T. 2007. Initializing markerless tracking using a simple hand gesture. In ISMAR, 259–260.
M ASCARO , S. A., AND A SADA , H. H. 2003. Wet shape memory
alloy actuators for active vasculated robotic flesh. In IEEE ICRA,
vol. 1, 282–287.
M INAMIZAWA , K., K AMURO , S., K AWAKAMI , N., AND TACHI ,S. 2008. A palm-worn haptic display for bimanual operations in
virtual environments. In EuroHaptics, 458–463.
S ANDOR , C., U CHIYAMA , S., AND YAMAMOTO , H. 2007. Visuohaptic systems: Half-mirrors considered harmful. In Proceedings of the Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 292–297.
S CHEIBE , R., M OEHRING , M., AND F ROEHLICH , B. 2007. Tactile feedback at the finger tips for improved direct interaction in
immersive environments. In 3DUI.
TUSB-ADAPIO. Turtle Industory Corporation.
WANG , R. Y., AND P OPOVIC , J. 2009. Real-time hand-tracking
with a color glove. ACM Transaction on Graphics 28, 3.
Z HANG , Z. 2000. A flexible new technique for camera calibration.
IEEE TPAMI 22, 11, 1330–1334.