A Study for Vision Based Data Glove
Considering Hidden Fingertip with Self-Occlusion
Sanshiro Yamamoto∗
Dept. of Computer Science and Eng.
Nagoya Institute of Technology
Nagoya, Japan

Kenji Funahashi
Dept. of Computer Science and Eng.
Nagoya Institute of Technology
Nagoya, Japan
Email: kenji@nitech.ac.jp

Abstract—Data glove is widely used interface device whichmeasures hand posture (finger joint angles) and inputs it into
a computer in virtual reality field. But it dose not spreadthroughout home because of expensive interface. On the other
hand, researches to recognize a hand posture from photo/video
images are performed, which are a kind of hand posture measurement system and called Vision Based Data Glove (VBDG)
in recent years. In this paper, we propose a new VBDG system.
It estimates hand motion using detected fingertip positions withmonocular camera and inverse kinematics. However it cannot
estimate hand motion when a fingertip is undetectable with selfocclusion. So our system estimates hidden finger motion, thenestimates hand motion. Our experimental results show thatthe proposed method can estimate it with sufficient accuracy
in real-time. Since camera base system is inexpensive, it fits
personal use.
Keywords-Vision Based Data Glove, Self-Occlusion, Hidden
Fingertip, Inverse Kinematics, Personal UseI. I NTRODUCTION
Data glove is widely used interface device which measures
hand posture (finger joint angles) and inputs it into a
computer in virtual reality field. However, high-performancedata glove is expensive and low-priced data glove does
not have enough sensors to capture hand data correctly.
It dose not spread throughout home. On the other hand,researches to recognize a hand posture from photo/video
images are performed, which are a kind of hand posturemeasurement system and called Vision Based Data Glove(VBDG) in recent years. VBDG system can be classified
into two groups. One is use database of hand postures built
in advance, it estimates hand posture to match an input
image with database. Another detects fingertips from an
input image first, then it recognizes hand gesture and/or
estimates hand posture.
The hand posture estimation method to match an input
image with database [1,2] can be registered various handposture, so it can cope with self-occlusion. However aflexibility of human hand is so high and so many postures
can be targets to match with database. Therefore process∗ present affiliation is HAGIWARA ELECTRIC Co.,Ltd.

Yuji Iwahori
Dept. of Computer Science
Chubu University
Kasugai, Japan

speed and estimating accuracy are trade-offs. The method
to search efficiently is also proposed [3]. Although using
these methods, much data of hand posture is required, it has
wide individual difference probably. So it is difficult to build
database for each user at home.
On the other hand, the method detecting fingertips canbe further classified into two methods. These are a hand
gesture recognition method from detected fingertips [4,5]and a hand posture estimation method which solves Inverse
Kinematics (IK) with fingertip positions [6,7,8]. First one
recognizes a hand gesture with combination of detected
fingertip motion. But this does not estimate each finger joint
angle. It is not so useful for general application, i.e. to realize
many types of object operation. On second one, hand model
information (hand size, finger length, etc.) is required tosolve IK although, to create hand model is easier than tobuild database for each user at home. However it cannot
estimate hand posture when a fingertip is undetectable with
self-occlusion. Chen et al. proposed to choose suitable
images from several cameras and to estimate it [7]. It isdifficult to set several cameras at home.
In this paper, we propose a hand posture estimation
method after fingertip detection using computer vision technology, even when a camera cannot detect some fingertips
with self-occlusion. Our experiment system is based on ourpilot VBDG system without self-occlusion [9].
In proposed method, we assume that the finger motion
is continued for a while when a finger is hidden. Since
hand moves smoothly, we also assume that finger motion(time series variation of each finger joint angle) can be
expressed with polynomials. The finger posture in current
frame is estimated using the polynomials first. However thisestimation is not necessarily correct. So we also consider
the finger motion of the hidden fingertip. There we callthe finger of hidden fingertip as hidden finger. Both offingertip positions of estimated finger posture and previous
frame posture are judged whether these are hidden with
self-occlusion. Then hidden finger motion type is estimatedand hidden finger posture is estimated where the fingertip is
hidden with self-occlusion. Then our system estimates hand

Table I: Movable Range of Finger Angle
MP

CM

CM-AA

THUMB

0◦ – 80◦0◦ – 60◦

0◦ – 90◦

0 – 60◦

DIP

PIP

MP

MP-AA

INDEX

0◦ – 80◦0◦ – 100◦0◦ – 90◦−25◦ – 25◦

MIDDLE0◦ – 80◦0◦ – 100◦0◦ – 90◦−25◦ – 25◦

RING

0◦ – 80◦0◦ – 100◦0◦ – 90◦−25◦ – 25◦

LITTLE0◦ – 80◦0◦ – 100◦0◦ – 90◦−25◦ – 25◦

80
60
50
30
20
10
0
0

10 20 30 40 50 60 70 80 90 100
PIP joint

θ3

θ3′i

f ct (ti )
θ3

θ 3n

θ3

θ 2i
Figure 2: Derivation of a cubic polynomial showing timeseries change of MP joint.

PIP joint40

f c (θ 2i )f t (ti )

100
90
80
70
60
50
40
30
20
10
0

70

DIP joint

θ3i

IPIII. H IDDEN F INGER P OSTURE E STIMATION
A. Hidden Finger Posture Continuing Finger Motion
0

10

20

30

40 50 60
MP joint

70

80

90

(a)
(b)
Figure 1: Correlation of finger joints. The figure (a) shows
relation between PIP and DIP joint. The figure (b) shows
relation between MP and PIP joint.

posture solving IK from fingertip positions when a fingertip
is undetectable with self-occlusion.
II. P ILOT VBDG S YSTEMFirst of all, we would like to assume that fingertips are
already detected from an input 2D image with computer
vision technology. So our pilot VBDG system detect themusing simple and easy method as followed. Fingertips aredetected using color markers attached on the fingertips, andthe hand position as 6 degrees of freedom is measured
with an ARToolKit marker [10] attached on the palm.
Then Fingertip 3D positions are calculated considering hand
structural constrains. The hand posture (finger joint angles)
is estimated with IK.
A human hand has many constraints. For example, first,
the movement of each finger is limited with movable rangeof finger angle as shown in Table I. DIP is distal interphalangeal joint, PIP is proximal interphalangeal joint, IP isinterphalangeal joint, MP is metacarpophalangeal joint, CM
is carpometacarpal joint and AA is abduction and adduction.
Second, finger joint angles have correlation each other asshown in Figure 1. There are many constraints describedabove. If a finger moves following as these constraints,a fingertip moves on a defined curve. This pilot systemestimates fingertip positions using this curve.
It solves IK with Cyclic Coordinate Descent method [11],then gets finger joint angles. This CCD method calculates
them solving optimization problem which minimizes the distance between an end effector and the goal. Our experiment
system is built based on this pilot system.

Hand posture changes continuously. We assume that ahidden finger motion is continued for a while when a finger
is hidden. A hidden finger posture can be estimated fromthe hidden finger motion (transition of each finger joint
angle in past frames). Moreover, the hand moves smoothly.
Transition of each finger joint angle in past frames can beapproximated to a cubic polynomial. Therefore, it is possiblethat a finger motion in past frames has correlation as shownin Figure 1. The correlation in past frames is derived. And
the cubic polynomial showing time series change which
includes correlation is derived. When MP-AA joint data
without the correlation which becomes settled uniquely, or
the past frames data is not mutually related from otherfinger joint angles, a cubic polynomial is derived only fromtransition of the finger joint angle in past frames. A cubic
polynomial is derived using least-squares method.
I will show an example to illustrate derivation of a cubic
polynomial showing time series change of MP joint. Figure 2
shows the cubic polynomial derivation showing time serieschange of MP joint, where θ3i is MP joint angle, θ2i is
PIP joint angle. i is frame number (p ≤ i ≤ n), p isoldest frame to refer to past frames, now n is current
frame and ti is lapsed time to i frame when tp = 0.0.
Polynomial fθc3 (θ2i ) showing the correlation of θ3i and θ2i0
is derived from past frame data of each joint. Angle θ3iinpast frames which includes correlation is generated from thepolynomial fθc3 (θ2i ) and θ2i . Polynomial fθct3 (ti ) is derived
0
from θ3i
and ti . Then θ3n is obtained with giving tn to thepolynomial fθct3 (ti ) . When the finger motion is continued fora while after the fingertip hides, the hidden finger posture isestimated using the above method.
By the way, the same finger joint angle may be continued
by the finger motion stopping in past frames. In this case,the polynomial showing transition of finger joint angle inpast frames becomes a constant. Therefore, only fixed finger
joint angle can be estimated. Then, the finger joint angle
data in past frames to refer to is not updated when a finger
joint angle is the same as previous frame. However, if it was
separately updated about each finger joint, lack of data mightrLa
Figure 4: Phalanx. r is radius. La is axis of the cylinder.

Dfingertip
Figure 3: Updating the finger joint angle data in past frames.
The same finger joint angle with three of top middle. Leave
one of three with bottom middle.

Ls
La: shade spaceoccur when the correlation is derived. The finger joint angle
data in correlation is not updated only when both of finger
joint angles do not change. When data is not updated, time
between frames which is not updated is excepted as shownin Figure 3.
Estimated hidden finger posture has to continue finger
motion before hiding. However, since the hidden fingertip
is undetectable in fact, the finger motion is not necessarily
correct. Then, the hidden finger motion is estimated fromestimated hidden finger posture with the polynomial andprevious frame posture.
B. Shade Space of Self-occlusion
The hidden finger motion is estimated using current frame
fingertip being hidden with self-occlusion. A fingertip is
hidden with self-occlusion. In other words, a hidden fingertip
is behind other fingers from a camera viewpoint. There wecall the space behind other fingers as shade space. Collision
detection of shade space and fingertip position of previous
frame posture is performed. And collision detection of shadespace and estimated hidden fingertip position with the polynomial is performed. The hidden finger motion is estimatedfrom the results. However, shade space is complicatedshaped 3D space. Therefore computational complexity islarge when collision detection is performed as 3D space.
When the hidden fingertip is in shade space, a line
segment which connects the hidden fingertip and a camera
crosses the finger which makes shade space. We thought thata finger is cylinder and a finger joint is sphere joint. A finger
consists of phalanges. A phalanx can be expressed witha capsule as shown in Figure 4. Thus, collision detection
is possible using the distance calculation between a line
segment and axis of phalanx (Figure 5.).
When D ≤ r, Ls and LA cross and the fingertip is in
shade space. By performing the distance calculation to all
shade space, it is judged whether the hidden fingertip is in
shade space.

Figure 5: Collision detection. Ls is a line segment whichconnects the fingertip and the camera. D is the distance
between Ls and La .

C. Hidden Finger Posture Estimation Considering Finger
MotionThe hidden finger motion is judged as follows from theresults of collision detection:• Finger motion which completely continued the motion
before the fingertip hides.
• Finger motion slower than the motion speed before thefingertip hides.
• Finger motion different from motion before the fingertip hides.
• Finger motion is stopped.
The hidden finger posture is estimated from this judgment.
1) Two Fingertips are Inside: When two fingertip positions are in shade space, the hidden finger motion is moving
within shade space or is stopped.
First, we assume that the hidden finger is moving withinshade space. By the way the hidden finger posture isestimated based on transition of the finger joint angle whichis showed with the cubic polynomial. When the hidden
finger is in shade space over several frames, the hidden
finger moves along with the cubic polynomial. When thepolynomial is drawn like sine curve or cosine curve, thehidden finger motion is repetitive motion within shade space
as shown in Figure 6.
However, when the hidden finger is in shade space over
several frames, it is rare that the hidden finger moves
repeatedly as mentioned above. In such a case, other fingers
are moving. Since shade space is moved, the hidden finger
moves along with motion of shade space, without repetitive
motion. Or the fingertip comes out of shade space and is
detected. Thus, the hidden finger does not move repeatedly
in shade space. This is to say movement direction of thehidden finger is not reversed.

degreet

Figure 6: Repetitive finger motion.

When estimated finger posture with the polynomial is
the finger posture makes the movement direction reverse,
it is thought that the hidden finger posture is target finger
posture. It would be better to say that the hidden finger
motion stopped. In such a case, let previous frame posture
be hidden finger posture in current frame. When estimatedfinger posture with the polynomial is finger posture in which
the motion before the fingertip hides is continued, estimatedfinger posture with the polynomial is the hidden finger
posture in current frame.
2) Estimated Fingertip is Inside, Previous Frame Fingertip is Outside: When estimated hidden fingertip position with the polynomial is in shade space and fingertip
position of previous frame posture is out of shade space,
the hidden finger is changed into previous frame posturefrom the estimated finger posture with the polynomial. Thus,
estimated finger posture with the polynomial is the hidden
finger posture in current frame.
3) Estimated Fingertip is Outside, Previous Frame Fingertip is Inside: When estimated hidden fingertip position
with the polynomial is out of shade space and fingertip
position of previous frame posture is in shade space, thehidden finger motion is slower later than the motion speed
before the fingertip hides, is stopped or that movement
direction is reversed. Since it becomes repetitive motion
when movement direction is reversed, it is excepted. Whenthe finger motion speed is 0, the finger motion is stopped.
Thus, the hidden finger posture can be estimated with the
cubic polynomial.
The hidden finger posture estimation with the polynomial
is given current time tn to the polynomial. By changing time
given to the polynomial, the hidden finger posture of motion
to which speed becomes slow can be estimated. The time
given to the polynomial is from current time tn to previous
time tn−1 for every short time δ. Let a finger posture with thefingertip position which near most current time is judged to
be inside of shade space be hidden finger posture in current
frame.
4) Two Fingertips are Outside: When two fingertip positions are out of shade space, the movement direction of thehidden finger is reversed or the finger motion is changed todifferent motion.

First, we thought that case where the movement direction
is reversed. In this case, since fingertip position of previous
frame posture is outside shade space, it is permissible thatthe movement direction of the hidden finger is reversed. The
hidden finger posture in this case is estimated using the same
method as the case where speed becomes slow. The time
given to the polynomial is tn−1 − ∆t. Time ∆t is a timefrom tn−1 to tn . The hidden finger posture in the finger
motion to reversal direction of the finger motion is estimatedwith ∆t. Collision detection of shade space and the fingertip
position of estimated finger posture with ∆t is performed.
When the result is in shade space, the hidden finger moves
in the reversal direction. Let estimated finger posture with∆t be hidden finger posture in current frame.
When the fingertip position of estimated finger posturewith ∆t is outside shade space, the finger posture does notsatisfy a condition in which the fingertip hides with selfocclusion. Thus, It is thought that hidden finger motion
was changed. The hidden finger posture in this case isestimated based on image coordinates from fingertip positionof previous frame posture and shade space.
D. Hidden Finger Posture Estimation of Other Motions
When changed to motion from which the hidden finger
motion is different, it is difficult to estimate the hidden
finger posture from time series data. Therefore, the hidden
finger posture is estimated based on image coordinates fromthe hidden fingertip position of previous frame posture andshade space. In order to show shade space correctly using
image coordinates, we have to project form of phalanxof all detected fingers. However, computational complexity
will increase. For computational complexity reduction, shadespace in image coordinates is shown with a point P on
which the central point of phalanx which builds shade space
is projected. A phalanx which makes shade space where
the hidden fingertip exists is estimated based on P and
the fingertip position of previous frame posture. The hidden
finger posture is estimated based on estimated phalanx.
Point Q is the point which is projected the fingertip
position of previous frame posture on image coordinates.
Distance between P and Q on image coordinates is calculated. The hidden finger posture is estimated assumingthat the fingertip exists in shade space which phalanx of P
with shortest distance makes. Distance between a line whichconnected a camera and the central point of phalanx, and thefingertip position of of previous frame posture is calculated.
Let the point on the line which becomes shortest distance befingertip position in current frame. the hidden finger postureis estimated with solving IK from a fingertip position.
When the distance between the point on the line and thefingertip position of solution to IK is below a thresholdvalue, let the fingertip position of solution to IK be hidden finger posture in current frame. When the distance islarger than the threshold value, the fingertip does not reachTable II: Estimation Order
Thumb

Front side

OrderdetectIndex

Index → Middle → Ring → Little

detectLittleLittle → Ring → Middle → Index

undetect

Index

Index → Middle → Ring → Little → Thumbundetect

LittleLittle → Ring → Middle → Index → Thumb100

degree

90
80
70
60
50
40
30

Table III: Process Time
Algorithm

Time (msec)

Camera image fetch and wait
Noise reduction
Fingertip position calculation
Hand posture estimation
CG drawing

16.2
13.7
0.6
2.5
0.3

Total

33.3

20
10
0
0

5

10

15

20

25

30

35

frame

(a): PIP joint angle
100degree

90
80
70
60

the point on the line with hand structural constraint. The
hidden finger posture is estimated with phalanx with next
short distance on image coordinates. When the distance inall phalanges becomes more than the threshold value, wethought that the fingertip is out of a image. Thus, let previous
frame posture be finger posture in current frame.
E. Several Hidden Finger Posture Estimation
When there are some hidden fingers, a hidden fingertip
may exist in shade space of other hidden finger. A finger
posture must be estimated from fingers in front side. As
shown in Table II, an order of finger posture estimation isdetermined from direction of the hand and hand structural
constraints.
IV. E XPERIMENTAL R ESULT
A experimental system using the method mentioned above
is implemented with C++ language. The specification of PC
is the following; CPU: Dual-Core AMD Opteron Processor
1210 1.8GHz. The camera is BSW20K04 of BUFFALO
INC. (MAX fps: 30 fps). We evaluated process speed
and estimation accuracy of our experimental system. The
results of process time are shown in Table III. The average
process speed was the same 30 fps as the camera operation
maximum. The camera image fetch and wait time was 48%of them and the noise reduction of an image was 41%. The
speed is sufficient and it is useful as an interface.
To evaluate the estimation accuracy, it estimates the same
hand posture from two camera images. All of fingertips aretaken in one image, and the estimated value from this image
is assumed as true. Some of fingertips are hidden on theother image, then the estimated posture is compared withthe true one. Both of cameras are set at a distance of 1mfrom the hand, and the angle between the camera directions
is 40 degrees. The results of estimation accuracy and result
images are shown in Figure 7,8.

50
40
30
20
10
0
0

5

10

15

20

25

30

35

frame

(b): MP joint angle
Figure 7: The results of estimation accuracy. A hidden finger
is index finger. Index finger is hidden (11–30frames). A
dashed line is true value and a solid line is estimation value.

When all fingertips are detected from two viewpoints, themean errors are 6.77 degrees at PIP joint angle, and 5.60
degrees at MP joint angle. When hidden finger posture isestimated, the mean errors are 6.78 degrees at PIP joint
angle, and 18.10 degrees at MP joint angle. The estimationof our system is sufficient accuracy. However, the estimate
error of MP joint angle is slightly large. The estimated value
was more bent than the true value. Although it is not bent tomaximum of the movable range in fact, the system probably
estimated like that the finger motion was still continued andit bent to maximum of the range. We expect that suitable andflexible movable range considering other postures of visible
fingers works better.
V. C ONCLUSION
In this paper, we proposed a new Vision Based Data
Glove considering hidden fingertip with self-occlusion. It
can estimate hand posture (finger joint angles) on the assumption that fingertips are already detected using computer
vision technology, even when a camera cannot detect somefingertips with self-occlusion.
The experiment result shows that the system can estimate a finger posture with self-occlusion based on a fingerposture not only the finger motion but also other fingers
and relations with them. We will also apply the fingertip
detection method without any markers to our system. It is
expected that VBDG spreads through home as an inexpensive interface for VR system.

1.

R EFERENCES

2.

[1] Emi Tamaki, Takashi Miyata and Jun Rekimoto: A Robust andAccurate 3D Hand Posture Estimation Method for Interactive
Systems, Transactions of Information Processing Society ofJapan (in Japanese), Vol. 5, No. 2, pp. 229-239, 2010.

3.
[2] Robert Y. Wang and Jovan Popovic: Real-Time Hand-Tracking
with a Color Glove, ACM Transaction on Graphics (SIGGRAPH 2009), Vol. 28, No. 3, 2009.
[3] B. Stenger, A. Thayananthan, P. H. S. Torr, R.Cipolla: ModelBased Hand Tracking Using a Hi-erarchical Bayesian Filter,
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, No. 9, pp. 1372-1384, 2006.

4.

[4] M.K. Bhuyan, Debanga Raj Neog and Mithun Kumar Kar:
Hand Pose Recognition using Geometric Features, National
Conference on Communications (NCC), 2011.

5.

6.

[5] Kenji Oka, Yoichi Sato, and Hideki Koike: Real-time fingertip
tracking and gesture recognition, IEEE Computer Graphics andApplications, Vol. 22, No. 6, pp. 64-71, 2002.

7.

[6] Vitor F. Pamplona, Leandro A. F. Fernandes, João Prauchner,
Luciana P. Nedel e Manuel M. Oliveira: The Image-Based
Data Glove, Proceedings of X Symposium on Virtual Reality(SVR’2008), João Pessoa, 2008. Anais do SVR 2008, Porto
Alegre: SBC, 2008, (ISBN: 857669174-4). pp. 204-211.

8.

[7] Weiying Chen, Ryuji Fujiki, Daisaku Arita and Rin-ichiro
Taniguchi: Real-time 3D Hand Shape Estimation using Multiple Cameras, Proceedings of the 13th Japan-Korea Joint
Workshop on Frontiers of Computer Vision, pp. 15-20, 2007.

(a)

(b)

(c)

Figure 8: The result of estimation. The figure (a) is camera
images. The figure (b) is the result of estimation drawn inCG from the camera viewpoint. The figure (c) is the resultof estimation drawn in CG from another viewpoint. Index
finger is detectable at 1–4. Index finger is undetectable at
5–8. However index finger posture can be estimated at 5–8.

motion with sufficiently high process speed. Although ourexperiment system detects fingertips using some markers,
the estimation method is applicable to other types of VBDG
system which detect fingertips based on other techniques.
In the future, we would like to estimate hidden finger[8] Ryuji Fujiki, Daisaku Arita, Rin-ichiro Taniguchi: Real-time
3D hand shape estimation based on inverse kinematics andphysical constraints Proceedings of International Conference
on Image Analysis and Processing, pp. 850-858, 2005.
[9] Sanshiro Yamamoto and Kenji Funahashi: A Study for Image
Based Data Glove to Use at Personal Home, Proceedings of the
15th Annual Conference of the Virtual Reality Society Japan,
2A2-5 (DVD-ROM, in Japanese), 2010.
[10] Hirokazu and Mark Billinghurst: Marker Tracking and HMD
Calibration for a video-based Augmented Reality Conferencing
System, Proceedings of the 2nd International Workshop on
Augmented Reality (IWAR99), pp. 85-94, 1999.
[11] Chris Welman: Inverse kinematics and geometric constraints
for articulated figure manipulation, M.Sc Thesis, Simon Fraser
University, 1993.