J Multimodal User Interfaces
DOI 10.1007/s12193-014-0165-0

ORIGINAL PAPER

A novel set of features for continuous hand gesture recognition
M. K. Bhuyan · D. Ajay Kumar ·Karl F. MacDorman · Yuji Iwahori

Received: 8 October 2012 / Accepted: 29 May 2014
© OpenInterface Association 2014

Abstract Applications requiring the natural use of thehuman hand as a human–computer interface motivate researchon continuous hand gesture recognition. Gesture recognition depends on gesture segmentation to locate the startingand end points of meaningful gestures while ignoring unintentional movements. Unfortunately, gesture segmentation
remains a formidable challenge because of unconstrained
spatiotemporal variations in gestures and the coarticulationand movement epenthesis of successive gestures. Furthermore, errors in hand image segmentation cause the estimated hand motion trajectory to deviate from the actual one.
This research moves toward addressing these problems. Our
approach entails using gesture spotting to distinguish meaningful gestures from unintentional movements. To avoid theeffects of variations in a gesture’s motion chain code (MCC),
we propose instead to use a novel set of features: the (a) orientation and (b) length of an ellipse least-squares fitted tomotion-trajectory points and (c) the position of the hand. The
features are designed to support classification using conditional random fields. To evaluate the performance of the sysM. K. Bhuyan (B) · K. F. MacDorman
School of Informatics and Computing, Indiana University Purdue
University (IUPUI), 535 West Michigan St., Indianapolis,
IN 46202, USAe-mail: mkbhuyan@iupui.edu
K. F. MacDorman
e-mail: kmacdorm@indiana.edu
D. Ajay Kumar
Department of Electronics and Electrical Engineering,
IIT, Guwahati 781039, Indiae-mail: ajay.k@iitg.ernet.in
Y. Iwahori
Department of Computer Science, Chubu University,
1200 Matsumoto-cho, Kasugai 487-8501, Japane-mail: iwahori@cs.chubu.ac.jp

tem, 10 participants signed 10 gestures several times each,providing a total of 75 instances per gesture. To train the system, 50 instances of each gesture served as training data and25 as testing data. For isolated gestures, the recognition rateusing the MCC as a feature vector was only 69.6 % but rose to
96.0 % using the proposed features, a 26.1 % improvement.
For continuous gestures, the recognition rate for the proposedfeatures was 88.9 %. These results show the efficacy of the
proposed method.
Keywords Human–computer interaction (HCI) · Gesturerecognition · Motion chain code (MCC) · Conditionalrandom fields (CRF)

1 Introduction
Gestures provide an attractive, user-friendly alternative tousing an interface device like a keyboard, mouse, and joystick in human–computer interaction (HCI). Accordingly, thebasic aim of gesture recognition research is to build a systemthat can identify and interpret specific human gestures automatically and employ them to convey information (i.e., forcommunicative use as in sign-language) or to control devices(i.e., manipulative use as in controlling robots without any
physical contact).
One of the most important requirements for sign language
recognition is that natural gesturing be supported by therecognition engine so that a user can interact with the system without any restrictions. Since a sequence of gestures isgenerally mixed with coarticulation and unintentional movements, these non-gestural movements should be eliminatedfrom an input video before the identification of each gesture in the sequence. Movement epenthesis is a non-gestural
movement between gestures, and gesture coarticulation is123

J Multimodal User Interfaces

the modification of the beginning or end of a sign becauseof the sign that preceded or suceeded it, respectively [1,2].
In more natural settings, the gestures of interest are embedded in a continuous stream of motion, and their occurrence
must be detected. This is precisely the goal of gesture spotting, namely, to locate the starting point and the endpointof a gesture pattern and to classify the gesture as belonging
to one of the predetermined gesture classes. Once the gesture boundaries are known, a gesture in the sequence canbe conveniently extracted. However, the process of gestureboundary detection is not trivial for the following reasons:– Gesture boundaries vary from one signer to another.
– Gesture boundaries are sequence dependent; in particular, gesture segmentation is heavily influenced by higherlevel cognitive processes and by the context of each gesture.
– It is impossible to enumerate all gestures. Traditional
computer vision approaches characterize a gesture as aseries of poses. Clearly, given the virtually infinite number of poses that the human body can assume, a genericmodel-based approach to gesture segmentation is notviable.
Generally, a gesture starts and ends with the hand at a standstill. That is, a signer generally starts making a sign froma pause state and ends in a pause state even when gesturing continuously [3]. Based on this observation, we propose
to use the hand motion information to locate the boundary
points of each gesture in a continuous stream of gestures. A
boundary point is detected whenever the hand pauses during
gesturing. For gestures having global motion only or gestures having both global and local motions, the gesturing
hand traverses through space to form a gesture trajectory.
After a gesture trajectory is completed, the hand pauses for awhile and then moves with very high velocity to the starting
position of the next trajectory. Based on this, we propose todetect movement epenthesis—and distinguish it from a gesture stroke—by observing the motion of the hand between
pauses in the input hand motion video. Whereas a movement
epenthesis phase is simply a fast hand stroke, a gesture phasecan be divided into three motion stages: preparation, stroke,
and retraction. However, the scheme will result in the incorrect spotting of gestures under the conditions listed below:– It is not correct to assume that between gestures there
is always some non-gestural hand movement. For example, there is generally no extra movement if a gestureends in the same position or pose at which the next gesture begins. In this case, the two gestures are adjoined toeach other in the sequence with a common pause indi-

123

cating the end of the first gesture and the start of the next
gesture.
– For a sequence of static hand poses (fluent finger
spelling), there is generally no motion during the gesturing period while the hand may move in between two gesture poses because of movement epenthesis. That meansa pause itself in the sequence corresponds to a gesturesign as if the starting point and endpoint of the gesture
have merged together.
– For gestures involving global hand motion, there may besome pauses within a single gesture. When the hand traverses through space, it makes one or more hand strokes
to build up a complete gesture trajectory with pauses
between the strokes.
The proposed method is described in more details in thefollowing sections.

2 The state of the art
A threshold model based on a hidden Markov model (HMM)
was proposed to recognize and spot the gestures [4] in whichmotion chain code (MCC) is used as the feature. The system
was trained with a huge number of isolated samples. The gesture is spotted when it crosses the adaptive threshold and theboundary points are detected using the Viterbi back-tracking
algorithm. The major limitation of this method is that as thegesture vocabulary increases the number of states increases.
Most errors come from the failure of hand extraction that
distorts the hand trajectory and also the MCC. Our proposed
method focuses on solving this problem by replacing the
MCC with other trajectory-based features.
A gesture spotting technique was proposed using a finite
state machine (FSM) [5]. The features used are various
motion parameters like the velocity of the hand. The velocityof the hand is less at the beginning and end of the gesture,and the hand abruptly moves during movement epenthesis.
However, this method cannot be used as a generalized model
for all the gestures.
Gesture spotting with a threshold model based on conditional random fields (CRF) combines motion-based andlocation-based features [6]. The MCC of the hand trajectoryis used as a motion-based feature. As with other methods,
variations in the extracted MCC greatly affect the performance of the recognizer.
A framework that simultaneously spots and recognizes
gestures by using the density functions of the states for eachof the gesture classes was proposed in [7]. A 4D positionflow vector is used as the feature, which contains the pixel
position of the centroid of the hand and its two dimensional
velocity. This method is also not applicable to all kinds ofgestures.

J Multimodal User Interfaces

A sign language recognition system was developed using
three appearance-based features: PCA, kurtosis position, andthe MCC [8]. The hand is tracked and the trajectory is represented by MCC. These three features are given to three
separate HMM networks for recognition. Thus, the overall
recognition accuracy also depends on MCC.
Though the segmentation and coarticulation detection are
the main open research issues for continuous hand gesturerecognition, few vision-based approaches have been reported
prior to this work. The techniques developed so far for coarticulation detection have not always been successful for awide range of gesture vocabulary. Although powerful classifiers for labeling the sequential data exist, feature selection is key to obtaining a high recognition rate. In gesturerecognition, the features must be selected so as to enable theclassifier to unambiguously recognize and segment the continuous gestures. As explained earlier, most of the existing
approaches use the motion chain code as a feature. The MCC
is a rotation invariant feature for object recognition. But ingesture recognition, many variations occur in the extracted
MCC even when the same gesture is performed repeatedly.
This is because of ambiguity in hand image segmentationand trembling of the gesturing hand.
As an example, Fig. 1 shows frames of the segmented handregions of the gesture trajectory Eight. The area and the orientation of the segmented hand differs from frame to frame.
This causes the extracted trajectory to deviate from the actual
trajectory, resulting in drastic variations in the MCC, which
was encoded from the information of the motion trajectory.
This variation in the MCC causes difficulty in modeling a gesture. Figure 2 shows the extracted gesture trajectory varies
from the actual one. Hence, for a particular gesture, the MCC
will not be unique. To overcome this problem, we proposea much smoother chain code feature than the conventional
MCC. The new feature gives better results. Additionally, twoother features are proposed to cope with the problem of movement epenthesis.

3 Proposed system
This paper focuses on the recognition of continuous digits
gesticulated by the user. These gestures, shown in Fig. 3, aredynamic, consisting only of global hand motion. These gestures are performed continuously, so movement epenthesis
and coarticulation come into consideration as the gesturer
has to move his hand from the endpoint of one gesture to thestarting point of other gesture. Thus, the recognition systemshould be capable of detecting the meaningful gestures fromthe continuous stream of gesticulation. Due to coarticulation,
the appearance of a sign/gesture, especially at the beginningand end, can be significantly different under different sentence contexts, which makes the recognition of gestures inFig. 1 Various frames of the segmented hand of the gesture Eight

Fig. 2 Examples of trajectories of isolated gesturessentences hard. In other words, when gestures are produced
continuously, each gesture may be affected by the precedinggesture, and sometimes by the gesture that follows it. However, movement epenthesis occurs when the hand moves very
fast from the end position in a gesture to the starting positionin the following gesture. This implies that these movements
are deliberate and are essential for connecting two gestures ina sequence. A pictorial representation of movement epenthesis and the embedded gestures is shown in Fig. 4.
In our method, we use CRF to classify the sequential data(feature vector), because they can model the gestures by considering the dependencies in the sequential data. CRFs havethe ability to relax the independence assumption of hidden
Markov models [9,10]. However, an HMM is a stochastic
automaton that models the sequential data by generating theobservation while transiting from one state to another state[11]. Whereas, CRF is a non-generative finite state modelwith a single exponential for joint probability of the label
sequence given an observation sequence. Most of the existinggesture spotting approaches in the literature rely on dynamic
time warping (DTW), neural networks (NN), or HMM.
Once a particular gesture has been recognized, it needs tobe mapped to a corresponding action, for example, to controla robot or activate a window menu. A gesture-based HCI system also allows a person to communicate with a computer via

123

J Multimodal User Interfaces

Fig. 3 Examples of trajectories of isolated gestures

(a)

(b)
Fig. 5 Hand segmentation by the proposed method(c)Fig. 4 Problems in geture recognition include movement epenthesis,
indicated by dotted lines in (a) and (b), and the embedding of one gesturein another. In c Five is embedded in Eightsign language and, thus, enables those with hearing impairment to interact with a computer more easily. Our method isbasically intended for these kinds of HCI applications. The
proposed scheme for continuous hand gesture segmentationand subsequent recognition is described in the sections tofollow.
3.1 Hand segmentation and gesture trajectory estimation
Hand segmentation is typically a first step in tracking the
movement of the hand. In our method, the face region in theframe is removed by using a face detection algorithm. Skin
color-based segmentation is used to extract the skin regions

123of the hand from the video frame. Skin color is clustered in avery small region in a color space so that skin regions can easily be separated by selecting a proper threshold. In addition,color spaces are selected that are less susceptible to changes
in lightning. HSV and YCbCr are two color spaces that separate the chrominance ([H S] or [Cb Cr]) and luminance (Vor Y) components [12]. So, after removing the face region,
the RGB image is converted to HSV and YCbCr images, andsubsequently a threshold is applied to the chrominance components of both the HSV and YCbCr color spaces. Finally, a
logical AND operation is performed between them to obtainthe likely skin region.
After getting the skin color segmented regions, the largest
connected segment corresponds to the palm region of thehand. Figure 5 shows various intermediate results. After
determining the binary alpha plane corresponding to the palm
region of the hand, moments are used to find the center ofthe hand. The 0th and the 1st moments are defined as
M00 =


x

y

I (x, y), M10 =


x

y

x I (x, y),J Multimodal User Interfaces

M01 =


x

y I (x, y)(1)yFeatures

Subsequently, the centroid is calculated as
xc =

M10
M00

Table 1 Proposed features

and

yc =

M01
M00

(2)

CE

Chain code obtained by the angle of orientation of the ellipse

LE

Length of the major axis of ellipse (small/medium/large)

PH

Position of the hand (top/middle/bottom)

In the above equations, I (x, y) is the pixel value at theposition (x, y) in the image. Since the background pixels are
assigned 0, the centroid of the hand in a frame is also the centroid of the total frame. Therefore, in moment calculations,
we may either take the summation over all pixels in the frameor over only the hand pixels.
Finally, the gesture trajectory is formed by joining all the
calculated centroids in a sequential manner. However, thistrajectory may be noisy for the following reasons:– Points too close– Isolated points far from the correct trajectory due to achange in the hand shape– Unclosed endpoints– Hand trembling– Unintentional movements
Therefore, to reduce these effects, the final trajectory issmoothed out by considering the mean value of a specified
point and its two neighboring points, i.e.,(xˆt , ŷt ) = ((xt−1 + xt + xt+1 )/3, (yt−1 + yt + yt+1 )/3)
(3)
So, a dynamic hand gesture (DG) can be interpreted as aset of points in a spatiotemporal space:
DG = {(xˆ1 , yˆ1 ), (xˆ2 , yˆ2 ), . . . . . . , (xˆt , ŷt )}(4)

3.2 Extraction of proposed trajectory-based features
Given the sequence of centroid of segmented hand, three
features are extracted as shown in Table 1. The proposed
three dimensional feature vector is extracted to use the CRF
effectively, so that it can spot the gestures and recognize them.
As shown in Fig. 6, feature C E is extracted from the angleof the major axis of the ellipse which is fitted to the previousTr points P = {X t }t=T
−r of the motion trajectory, whereX t = (xt , yt ). The motivation behind this idea is to obtain achain code that preserves the shape of the motion trajectoryeven if the extracted trajectory is not smooth. The ellipse
is fitted using least squares fitting of conic sections. Let us
assume an ellipse with a set of parameters A = [a b c d e f ]Tas
E (A, X ) = ax 2 + bx y + cy 2 + d x + ey + f

Fig. 6 Extraction of features C E and L E

The algebraic distance E (A, X ) is the product of χt =[xt 2 xt yt yt 2 xt yt 1] and A.
E (A, X ) = [xt 2 xt yt yt 2 xt yt 1][a b c d e f ]T = χt A
The parameter set A is found using the least-squares technique that minimizes a distance metric  (A) between thedata points and the ellipse E.
 (A) =T


E( A, X t )2 = D A2

t=T −rwhere, D = χT −r χT −r +1 . . . χT is the design matrix. To
avoid the trivial solution A = 06 , different constraints areapplied to the parameter A. We chose the computationallyefficient algorithm given in [13] for ellipse fitting. The minimization of D A2 subject to the constraint 4ac − b2 = 1
gives only one solution, which is an ellipse. The counterclockwise angle of the major axis of that ellipse (θ M A ) isused to find our proposed chain code/feature C E , whereas aconventional MCC is calculated with the slope of the lines
joining to successive points.
In our proposed method, starting from the first trajectory
point (at the beginning of a gesture), an ellipse is fitted tothe set of points of the motion trajectory until r points are
enclosed. Then, the straight line joining the two extreme
points of the major axis of the ellipse is determined. Next, asshown in Fig. 6, the said ellipse is fitted to the next r points
starting from the second trajectory point, and the straight line
joining the two extreme points of the major axis of the ellipse
is determined. Finally, the angle (θ M A ) is calculated from123

J Multimodal User Interfaces

Fig. 7 Ellipse fitting to gesture trajectory Five for r = 6

these two lines, which is subsequently used for determining
the proposed chain code/feature. This process is performed
for all the points of the motion trajectory. In our experiment,
the value of r is set to 6. It is experimentally observed thatthe hand stops for at least six frames at gesture boundaries,
and the length of the major ellipse is very small for the endpoints. This consideration additionally helps us to detect theendpoints (gesture spotting). Figure 7 shows various instantsof ellipse fitting corresponding to gesture trajectory five.
As explained in Sect. 1, the conventional MCC fails to
describe the original track of the gesture due to segmentation
ambiguities. But the new feature C E preserves the shape ofthe gesture trajectory, because in this case the rate of change
in the orientation angle (θ M A ) is much lower. Hence, it
results in a smoother chain code. Table 2 shows the difference
between MCC and C E for the gesture One. The same gestureis performed three times and the parameters MCC and C E are
extracted. Clearly, the proposed feature C E nicely encodes
the shape of the gesture trajectory.
Next, the feature L E is obtained from the length of the
major axis of the ellipse L M A . As shown in Fig. 7, L M A
decreases at the endpoints of a gesture. It is a ternary-valued
feature [(S) or (M) or (L)], which is obtained by setting thethresholds Ta and Tb . It indicates whether the length of L M A
is small (S) or medium (M) or large (L). Hence, the temporal information of the hand trajectory can be known and the123

Table 2 Example showing the difference between MCC and C E forgesture trajectory One
MCC

CE676556566566066066666667
6055
3567007676660666665667620

6666666666666666666666666666
3446666666666666666666666

5665666766660666666672

5666666666666666666666

information helps in gesture spotting. This also helps in internal modeling of gestures because few gestures have intermediate stoppages in between the starting and the endpointsof the gestures, namely, motion trajectories of the gestures
three, two, four, seven, and nine. The feature L E is extracted
as follows:⎧⎨ S, L M A ≤ Ta
L E = M, Ta < L M A ≤ Tb⎩
L , L M A > Tb
Finally, the feature PH represents the position of the handin the frame. It is also a ternary-valued feature (T or M or
B). As shown in Fig. 8, each image frame is horizontally
divided into three regions, which are called as the top, middle, and bottom. Subsequently, PH in each frame is calculated by setting thresholds Tc and Td for the x coordinate of
the centroid of the hand. This feature is targeted to detect

J Multimodal User Interfaces

by using a log-linear exponential model:
⎛
p

1
⎝
exp
α j t j (wi−1 , wi , o, i)p(w|o) =z(o)j=1Fig. 8 Extraction of feature PH+qβk sk (wi , o, i)

k=1z(o) =



⎛

exp ⎝

w

+

p


α j t j (wi−1 , wi , o, i)

j=1
q


βk sk (wi , o, i)

k=1Fig. 9 An example showing the proposed set of features (C E , L E , PH )for the gestures Five, indicated by the red-colored trajectory, and Eight,
indicated by the blue-colored trajectory

the movement epenthesis. The boundaries of the movement
epenthesis stroke are generally either in the top or bottom
position of a video frame, whereas the middle region always
contains the body of the gesture. The feature PH is derived
as follows:⎧⎨ T, xt ≤ Tc
PH = M, Tc < xt ≤ Td⎩
B, xt > Td
The proposed features for the gestures ‘Eight’ and ‘Five’are shown in Fig. 9. It is evident that these three features may
be the same for both of the gestures until point p8 , but thefeature L E changes afterwards. Hence, the proposed features
can detect a gesture when it is embedded in another gesture.
3.3 Proposed gesture recognition scheme
Finally, as shown in Fig. 10, the extracted features C E , L E ,and PH are applied to the proposed classifier. In our experiment the train/test sequence o = {o1 , o3 , o3 , . . . , om } is associated with the label sequence w = {w1 , w3 , w3 , . . . , wm }.
The probability of the label sequence w given o is calculated

where, t j (wi−1 , wi , o, i) is a transition feature function of
the observation sequence and the labels at positions i andi − 1, sk (wi , o, i) is a state feature function of the label atposition i and the observation sequence, p and q are thetotal number of features for transition feature functions andstate feature functions, α j and βk are weights to the exponent
which are estimated from training data by using the maximum entropy criterion, and z(o) is a normalization factor toensure that w p(w|o) = 1.
The transition and state feature functions are calculatedusing the binary-valued predicates on the observation data.
An example of such a predicate in our experiment is givenby⎧⎨ 1 if o is at the position i defined by theg (o, i) =feature value (4, S, B)⎩
0 otherwise.
An example of the state and the transition feature functions
that takes on the value of the predicates are given byt j (wi−1 , wi , o, i)⎧⎨ g (o, i) if wi−1 = two=and wi = one⎩0otherwisesk (wi , o, i) =g (o, i)0

if wi = oneotherwise

These transition and state feature functions help the CRF
to model the data with arbitrary dependencies. An iterative algorithm called improved iterative scaling (IIS) is thenused to estimate the parameters θ = {α1 , α2 , . . . ; β1 , β2 , . . .}Tfrom the training data  = ot , w t t=1 that maximizes the
log-likelihood objective function, which is found as
L (θ ) =T
log pθ w t |ot

t=1

∝



p (o, w) log pθ (w|o)

o,w

123

J Multimodal User Interfaces
Fig. 10 Block diagram
showing proposed set offeatures C E , L E and PH

The conditional probability of the label sequence is computed
efficiently by using the matrices. Let there be (n + 1) labelsin the label sequence with a starting and stopping label w0 =start and wn+1 = stop. Then pθ (w|o) is calculated using a
set of n + 1 matrices. In this, {Mi (o) |i = 1, 2, . . . , n + 1}
where, Mi (o) is a |y × y| matrix with elements of the form
⎛

 

α j t j (wi−1 , wi , o, i)
Mi w , w|o = exp ⎝j+

βk sk (wi , o, i)k

The normalization factor z θ (o) is the product of these matricesz θ (o) = (M1 (o) M2 (o) . . . . . . Mn+1 (o))start, stop
And the conditional probability of the label sequence w is
given asn+1
Mi (wi−1 , wi |o)pθ (w|o) = i=1
n+1
M
(o)
i
i=1
start, stopIn our proposed method, CRF is trained with one label
for each gesture and an extra label is added to determine
the non-gestural movements. Apart from the isolated gestures, all possible non-gestural movements are trained. To
model the dependencies of non-gestural movements withgestures, a combination of gesture and non-gestural observation sequences are used in the training data.

4 System performance evaluation
The proposed system was tested in real-time on an Intel
Core I3-based personal computer. The input images are captured by a CCD camera at a resolution of 480 × 480 pixels.
The system is evaluated using both isolated and continuous
gestures. During the evaluation, we use a comparatively simple background as shown in Fig. 3. The background enhances
hand segmentation performance. Moreover, all the experiments are performed with nearly constant uniform illumina-

123

tion, and all the gestures under consideration are performed
with one hand only.
In the evaluation, we used gesture sequences corresponding to the numerals 0 to 9, as shown in Fig. 3. To obtainthe feature L E , the thresholds Ta and Tb were set to 60
and 180, respectively. To extract the feature PH , the captured image frame is divided into three equally spaced horizontal blocks. As the frame resolution is 480 × 480, thethresholds Tc and Td were set to 160 and 320, respectively. To ensure spatiotemporal variability, each gesture was
performed by 10 signers. Even if the same person tries
to perform the same sign twice, small variation occur inthe speed and position of the hands. In total, 50 isolated
samples per gesture were used to train the CRF. Subsequently, the same CRF was again trained with another 50
samples per gesture class with each feature sequence corresponding to the selected gesture appended with the possible movement epenthesis segments. This training method
creates an extra label in the CRF to handle movement
epenthesis as the CRF learns the dependencies of movement epenthesis for the gestures. In addition, by setting thewindow size to 3, the CRF is designed to use only three
past and future observations to predict the current state.
Motion trajectories extracted by our method are shown in
Fig. 11.

4.1 Isolated gesture recognition
The classification results corresponding to different gesture
sequences are shown in Table 3. The overall recognition rate
for our proposed method is 96.0 % as compared with 69.6 %when the MCC is used as a feature vector, a 26.1 % increase.
The improvement is mainly because of the smoother chain
code obtained by our proposed feature C H . However, a relatively low recognition rate is reported for the gesture One,
which might be due to the inability of CRFs to model the
long-range dependencies. This can be overcome by increasing the window size, but this generates many features, which
increases the computational complexity. Also, we note that
our recognizer misclassifies the gestures One and Seven insome critical conditions. The misclassification occurs when
the hand stops too long in the middle of the gesture Seven.

J Multimodal User Interfaces

evaluation of the proposed algorithm. Some of the test gesture
trajectories are shown in Fig. 12. Three parameters are used
to evaluate the performance of the proposed continuous gesture recognizer. They are labeled as insertion error, deletion
error, and substitution error. The insertion error occurs when
the spotter reports a nonexistent gesture. The deletion error
occurs when the spotter fails to detect a gesture. The substitution error occurs when the spotter falsely classifies a gesture.
The detection ratio is the ratio of correctly recognized gestures over the number of input gestures. Table 4 shows theperformance of our proposed system in terms of the above
mentioned parameters. The proposed system recognizes thecontinuous gestures with an accuracy of 88.9 %. As shownin Fig. 12, the recognized gestures are displayed in the formof blue-colored text appearing in the top right corner of theimage, where ? indicates the movement epenthesis present
in between the gestures. Additionally, as shown in Fig. 13, a
gesture-based numeric calculator is implemented to highlighta possible application of our proposed method.

5 Conclusions
Fig. 11 Motion trajectories extracted by the proposed method

4.2 Continuous gesture recognition
For continuous gesture recognition, 40 video clips of different continuous gestures that contains movement epenthesis are considered and 216 sample gestures are used to test
the system. We have only considered a few typical possible
movement epenthesis and the embedded gesture cases for the

One critical issue in continuous gesture recognition researchis to identify meaningful gestures in a continuous stream ofbody movements. This may be accomplished by spotting precisely when a gesture in the sequence starts and ends. Thisis the goal of gesture spotting. Gesture spotting is essential
for a recognition system to work continuously without needof human intervention. In particular, only if gesture spotting
is supported in a vision-based interface, it is possible for auser to interact with the recognition platform using natural
gestures without any restrictions. By spotting gestures in a

Table 3 Experimental results: isolated gesture recognition rates
Class label

No. of training samples per class: 50 No. of test samples per class: 25
No. of test pattern assigned to predefined class
Zero

Acc. rate %

One

Two

Three

Four

Five

Six

Seven

Eight

Nine

Zero

25

0

0

0

0

0

0

0

0

0

100

One

0

21

0

0

0

0

0

4

0

0

84

Two

0

0

25

0

0

0

0

0

0

0

100

Three

0

0

0

25

0

0

0

0

0

0

100

Four

0

0

0

0

23

2

0

0

0

0

92

Five

0

0

0

0

0

25

0

0

0

0

100

Six

0

0

0

0

0

0

25

0

0

0

100

Seven

0

4

0

0

0

0

0

21

0

0

84

Eight

0

0

0

0

0

0

0

0

25

0

100

Nine

0

0

0

0

0

0

0

0

0

25

100

Avg.

96.0

123

J Multimodal User Interfaces
Fig. 12 Continuous gestures:a Three-Two, b Four-Seven,c Eight-Six, d Two-Four-One

Table 4 Continuous gesturerecognition using the proposedfeaturesGestureNo. of gestures

InsertDelete

Zero

16

0

2

3

11

68.75

One

20

0

0

4

16

80.00

Correct

Detection (%)

Two

25

0

0

0

25

100.00

Three

26

0

0

0

26

100.00100.00

Four

21

0

0

0

21

Five

19

1

0

1

18

94.74

Six

20

0

3

5

12

60.00

Seven

25

1

1

3

21

84.00

Eight

29

0

0

1

28

96.55

Nine

25

2

1

1

23

92.00

Total

226

4

7

18

201

88.94

continuous video stream, we can automatically remove unintentional movements arising between gestures.
Again, coarticulation detection is one of the main challenges in continuous gesture recognition. It is difficult to distinguish between a valid gesture phase and the coarticulation
phase only from the motion information. That is why, few
vision-based approaches for estimating coarticulation havebeen reported in the literature to date. The methods proposedso far do not address the problems encountered in recognizing all types of continuous hand gestures in a vision-based
platform. Most of the existing algorithms use the glove environment and are successful only for some specific gesture

123

Substitutevocabularies. The techniques developed so far for coarticulation detection are generally not useful for a broad gesture
vocabulary. This is because, motion interpretation itself is
an ill-posed problem in the sense that a unique solution cannot be guaranteed. Moreover, the existing algorithms do notaddress the problems associated with the extraction of thesmoothed motion trajectory and the more consistent motion
chain code.
This work focuses primarily on solving two basic problems in continuous gesture recognition, namely, gesture spotting and movement epenthesis detection, by extracting asmoothed motion trajectory via a modified motion chain

J Multimodal User Interfaces

tracking of a moving hand from a cluttered dynamic background and the recognition of more fluent hand gestures in
the form of continuous sign language.

References

Fig. 13 A gesture-based calculator

code. Its main contribution is the extraction of three novel
trajectory-based features to tackle movement epenthesis incontinuous hand gesture recognition. The proposed system
for movement epenthesis detection and subsequent recognition of individual gestures in a continuous stream of gestures promises to perform well on different types of gesture sequences having different spatiotemporal characteristics and motion behavior.
The proposed gesture recognition scheme overcomes thefundamental problems associated with continuous hand gesture recognition with the help of novel trajectory-based features. The experiments demonstrate that the system is 96.0 %accurate for recognizing isolated gestures, and 88.9 % accurate for continuous gestures. The number of thresholds toobtain the features L E and PH are application dependent.
Evidently, CRFs can model the sequential data by considering the arbitrary dependencies, but fail to model the longrange dependencies. This is the only reason the recognition rate for gesture One is comparatively low. There is a
probabilistic model called hidden conditional random fields(HCRF), which can model the long-range dependencies ofthe observed data [14]. Extending the proposed system using
HCRF to recognize sign language is left for future work.
Future work also includes the accurate segmentation and

1. Segouat J, Braffort A (2010) Toward modeling sign language coarticulation. Lecture Notes in Computer Science (Gesture in Embodied Communication and Human–Computer Interaction), vol 5934,
pp 325–336
2. Yang R, Sarkar S (2006) Detecting coarticulation in sign language
using conditional random fields. In: 18th international conferenceon pattern recognition (ICPR), vol 2, pp 108–112
3. Eisenstein J, Davis R (2005) Gestural Cues for Sentence Segmentation. Technical report, MIT AI Memo, pp 1–14
4. Lee HK, Kim JH (1999) An HMM-based threshold model approachfor gesture recognition. IEEE Trans Pattern Anal Mach Intell
21(2):961–972
5. Bhuyan MK, Bora PK, Ghosh D (2011) An integrated approach to
the recognition of a wide class of continuous hand gestures. Int J
Pattern Recognit Artif Intell 25(2):227–252
6. Yang H-D, Sclaroff S, Lee SW (2009) Sign language spotting witha threshold model based on conditional random fields. IEEE Trans
Pattern Anal Mach Intell 31(7):1264–1277
7. Alon J, Athitsos V, Quan Y, Sclaroff S (2009) A unified frameworkfor gesture recognition and spatiotemporal gesture segmentation.
IEEE Trans Pattern Anal Mach Intell 31(9):1685–1699
8. Zaki MM, Shaheen IS (2011) Sign language recognition using a
combination of new vision-based features. Pattern Recognit Lett
32(4):572–577
9. Lafferty J, McCallum A, Pereira F (2001) Conditional random
fields: probabilistic models for segmenting and labeling sequence
data. In: Proceedings of 18th international conference on machinelearning, pp 282–289
10. Wallach HM (2004) Conditional random fields: an introduction.
University of Pennsylvania
11. Rabiner LR (1989) A tutorial on hidden markov models andselected applications in speech recognition. In: Proceedings of theIEEE, vol 77, no 2, pp 257–286
12. Chai D, Ngan KN (1999) Face segmentation using skin color map invideophone applications. IEEE Trans Circuits Syst. Video Technol.
9(4):551–564
13. Fitzgibbon A, Pilu M, Fisher RB (1999) Direct least square fittingof ellipses. IEEE Trans. Pattern Anal. Mach. Intell. 21(5):476–480
14. Quattoni A, Wang S, Morency LP, Collins M, Darrell T (2007)
Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach.
Intell. 29(10):1848–1852

123