# RGBD Video Based Human Hand Trajectory Tracking and Gesture Recognition System

Authors: Weihua Liu; Yangyu Fan; Zuhe Li; Zhong Zhang
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

The task of human hand trajectory tracking and gesture trajectory recognition based on synchronized color and depth video is considered. Toward this end, in the facet of hand tracking, a joint observation model with the hand cues of skin saliency, motion and depth is integrated into particle filter in order to move particles to local peak in the likelihood. The proposed hand tracking method, namely, salient skin, motion, and depth based particle filter (SSMD-PF), is capable of improving the tracking accuracy considerably, in the context of the signer performing the gesture toward the camera device and in front of moving, cluttered backgrounds. In the facet of gesture recognition, a shape-order context descriptor on the basis of shape context is introduced, which can describe the gesture in spatiotemporal domain. The efficient shape-order context descriptor can reveal the shape relationship and embed gesture sequence order information into descriptor. Moreover, the shape-order context leads to a robust score for gesture invariant. Our approach is complemented with experimental results on the settings of the challenging hand-signed digits datasets and American sign language dataset, which corroborate the performance of the novel techniques.


---

## Body

## 1. Introduction

In recent years, with the developing of the camera sensor, 3D tracking of human hands attracts considerable interest in the literature of gesture recognition, human grasping understanding, human-computer interfaces, and so forth. Due to the lack of depth information, the 2D hand tracking has to struggle with the light variations and cannot express the semantic integrity of gesture. Further, variability of hand appearance makes it challenging in detecting and tracking in 2D-dimension.The existing alternatives utilize various sensors, including colored gloved [1] and magnetic tracker [2], to detect hands accurately. Unfortunately, such methods require a costly hardware or complex execute time. In pace with the development of the depth sensor, more and more general tracking approaches are inclined to add depth information into algorithm. Given the fact that hand movements generally occur in front of human bodies in 3D space, the depth information can thus significantly distinguish hand part from complex background. Without utilizing the color information, many papers demonstrate that the single depth data suffices for hand tracking [3, 4], while such approaches are not robust enough and can easily lose tracking since they get rid of using color information. Zhang et al. [5] recently proposed an efficient and fast hand detector by just using the hand color and motion information. The experiment result shows that such detector outperforms the Mittal et al. [6] detector with multiply features and Karlinsky et al. [7] detector by using chain model in sign language video. Nevertheless, this sign language video is captured with pure background, which does not take the more complex real environment into consideration.Hand tracking can be thought of as a nonlinear and non-Gaussian problem because of the presence of background clutter, complex dynamics of hand motion, and varying illumination. Thus, particle filter is well designed and implemented in this field. In 2D methods, a hand is represented by its geometric features and appearance features such as contours, fingertips, and skin color. Color-based methods often use skin color to localize and track hands in single camera [8, 9]. Spruyt et al. [10] propose a complete tracking system that is capable of long-term, real-time hand tracking with unsupervised initialization. Shan et al. [11] proposed a general tracking approach by incorporating mean shift optimization into particle filter, which can improve the sampling efficiency by moving particles to local peaks in the likelihood. However, this method is also vulnerable to other hand skin-likelihood cues and may fail to track hand continuously. Also, it does not consider the scale variation of the tracking object. To overcome the scale problem, based on mean shift and particle filter, Wang et al. [12] proposed to combine Camshift based on mean shift and particle filter, to reduce the time complexity and introduce the adaptive scale adjustment factor on each particles to evaluate the optimize object size. For articulated hand tracking, Chang et al. [13] introduced an appearance-guided particles filter for high degree-of-freedom state space. They design a set of attractors which can affect the state parameter of target and thus achieve Bayesian optimal solutions. Another PF-improved approach was proposed by Morshidi and Tjahjadi [14]; they compute the gravitational force of each particle as its weight to attract nearby particles towards the peak of the particles distribution, thus improving the sampling efficiency.As for 3D hand tracking, Manders et al. [15] use commercially stereo camera to acquire depth information after calibrating camera pair and then depth joint probability model is established according to distance between face and camera and used as input to a Camshift tracker. Similarly, Van Den Bergh and Van Gool [16] use the time-of-flight camera to capture hand shape and location. Nonetheless, all of these color-depth information based methods are needed to be well camera pair calibrated, which are rather time consuming and of lower efficiency.In this paper, for the hand tracking part, we extend Zhang et al.’s work [5] by incorporating additional depth feature, salient hand skin feature, and motion feature into particle filter, yielding a robust, efficient detector with high tracking accuracy. To start the tracking system, an automatic tracking initializing method is proposed. Finally, the hand center is extracted by using Kernel density estimation on particles distribution.As for gesture recognition part, there are four main aspects, static human pose, static hand pose, activity human body gesture, and activity hand gesture. Since hand is one of the most dominant and visual sensitive body part to express the gesture meaning, we focus on studying the hand gesture recognition based on hand trajectory. Due to the variety of habits of users, hand gesture is often subtle and vulnerable to various changes, such as position, orientation, and distance of people performing gesture with respect to camera. The main trend methods for classifying gestures are related to indirect approach, such as dynamic programming (DP) methods; for example, an exemplar-based approach like DTW [17] and CTW [18] relies on aligning temporal trajectory between query and model sequence for similarity comparison. Similarly, Wobbrock et al. [19] propose a simple enough method, one dollar gesture recognizer to classify the gesture. In such approaches, trajectory locations are usually adopted as an important match feature for gesture classification, while they cannot be greatly invariant to scale and translation with respect to different sign position, orientation, and distance to camera. This restriction makes the traditional methods difficult to accommodate various ways of acting behaves. In Liu et al.’s paper [20], they consider the gesture as multi-small segmentations and build decision trees by evaluating those directions of segmentations. This method can greatly be invariant to scale and translation, but the classification accuracy still needs to be improved.Inspired by the shape matching work [21] that develops a local shape context descriptor on each of shape points, we extend this method to spatiotemporal domain by embedding the gesture sequence order to shape context descriptor and give a name as shape-order context. Given the gesture shape and sequence order, the dissimilarity of the two-gesture trajectory can be measured as sum of matching errors among corresponding shape points. Such local descriptor can also be invariant to gesture translation and scale and with high recognition accuracy.This paper is organized as follows. The framework of particle filter is described in Section2. In Section 3, we discuss the proposed hand tracking algorithm based on particle filter, detailing with dynamical model, and observation model and also refer to tracking initialization model, hand center localization model. Hand gesture recognition based upon proposed shape-order context feature is then presented in Section 4. In Section 5, we finally evaluate our approach and compare the results with the existing state-of-art hand tracking and hand gesture recognition algorithms. The flow diagram of our proposed work is shown in Figure 1, which consists of two main phases: gesture trajectory tracking phase and gesture recognition phase. In the output of tracking phase, gestures are represented as a set of hand points which can be visualized in gesture sequence representation model. Meanwhile, it is treated as the input of recognition phase as well for gesture classification.Figure 1
The flow diagram of RGBD based hand trajectory tracking and gesture recognition system.


## 2. Probabilistic Tracking

### 2.1. Overview Particle Filter

Particle filter is a nonparametric system offering probabilistic framework for dynamic state estimation. It defines finite current object statest conditioned on all observations z1:t up to time t by computing Bayes filter posterior st~Γ(st∣z1:t). The posterior Γ(st∣z1:t) is approximated by sampling N particles at time t from its distribution. All particles at time t-1 move independently to the time t by sampling from the state transition distribution Γ(st∣st-1,zt-1). Each i_th sample state st(i) associates with a weight wt(i), which depends on the probability of the observation state zt, given by wt(i)∝Γ(zt∣st(i)), called important factor. The probability of drawing each particle is given by its important weights. Hence, the fair particles st(i) will transit to the new particles s^t+1i after resampling.


### 2.2. Motion Model

In the pixel coordinate, we define the motion state, the position, and the velocity of the hand, at time instantt as dt=(pt,vt)T, pt=(xt,yt)T, and vt=(x^t,y^t)T, respectively. The motion state of bare hand can be established as a first order autoregressive dynamics model:
(1)dt=fdt-1+et;et~N0,1,
where et denotes white processing noise with zero-mean, for each state variable.


## 2.1. Overview Particle Filter

Particle filter is a nonparametric system offering probabilistic framework for dynamic state estimation. It defines finite current object statest conditioned on all observations z1:t up to time t by computing Bayes filter posterior st~Γ(st∣z1:t). The posterior Γ(st∣z1:t) is approximated by sampling N particles at time t from its distribution. All particles at time t-1 move independently to the time t by sampling from the state transition distribution Γ(st∣st-1,zt-1). Each i_th sample state st(i) associates with a weight wt(i), which depends on the probability of the observation state zt, given by wt(i)∝Γ(zt∣st(i)), called important factor. The probability of drawing each particle is given by its important weights. Hence, the fair particles st(i) will transit to the new particles s^t+1i after resampling.


## 2.2. Motion Model

In the pixel coordinate, we define the motion state, the position, and the velocity of the hand, at time instantt as dt=(pt,vt)T, pt=(xt,yt)T, and vt=(x^t,y^t)T, respectively. The motion state of bare hand can be established as a first order autoregressive dynamics model:
(1)dt=fdt-1+et;et~N0,1,
where et denotes white processing noise with zero-mean, for each state variable.


## 3. Observation Model

### 3.1. Salient Hand Skin Model

Since human skin is a relatively uniform and discriminable cue in color space, a statistical color model can be employed to compute the probability of every pixel being skin color. A skin color distribution and a nonskin color distribution are denoted asP(r,g,b∣skin) and P(r,g,b∣~skin), respectively, in RGB color space, which is quantized to 32×32×32 values. According to these two distributions, the probability of a pixel-level particle st(i), which corresponds to pixel in color vector [rgb], can be defined as skin particle by using Bayes rule:
(2)P(skin∣st(i))=Psti∣skinP(skin)P(r,g,b),
where P(r,g,b)=P(st(i)∣skin)P(skin)+P(st(i)∣~skin)P(~skin) and P(skin) always equals 0.5.A normalized skin probability of particlest(i) is defined as
(3)Γskinsti=P(skin∣st(i))max⁡⁡Pskin∣stii=1:K,
where K is particles number in each frame and we set K=2000 overall our experiments.To improve the effectiveness of the skin color detector, an optimization method based on skin saliency is proposed in order to enhance the particles. As for a local-level image region distributed by propagating particles, the contrast and the color of the tracking object are relatively unique and distinctive than the local background. Since the target hand can be concerned as the dominant object in the context of the local image, a particlest(i) can be viewed as a salient particle if its appearance patch with centered at pixel location of st(i) is relatively distinctive with respect to all other particle patches. Specifically, we represent each particles by cropping a patch (5×5 size) surrounding it. Let dcolor(·) and dposition(·) be, respectively, the Euclidean distance of the vectorized patches of colors (in RGB space) and pixel location between two particle patches centered at pt(st(i)), pt(st(j)), respectively, where pt(st(i)) and pt(st(j)) are the pixel locations of the particles st(i), st(j). Based on the hand skin model, let us define a dissimilarity measure between a pair of particles as
(4)dsti,stj=dcolorsti,stj1+c·dpositionsti,stj.
This implies that a single pixel-level particle is treated as a salient hand skin particle when the other particles similar to it are nearby, and it is less salient when the resembling particles are far away. The parameter c set as 0.2 in our experiments. Hence, the salient hand skin factor of particle st(i) is defined as
(5)ωisti=1-exp⁡-1K∑k=1Kdsti,stk,
where K most similar patches of the particles {st(k)}k=1:K of each frame are extracted according to dcolor(st(i),st(k)), and we set K=100 in the experiment. As mentioned above, a particle is identified as hand skin saliency when ωi is high and vice versa.


### 3.2. Temporal Motion Model

Motion information is another useful cue for hand detection in gesture video. Since the hand is performed as the most salient object in front of the camera, hand motion changes more frequently than other human parts and objects background, such as arm and face. Typically, people incline to use the frame difference method to detect hand motion, whereas it is arduous to discriminate the objects with skin-like color close to that of the signed hand region. To improve the robustness of our tracking system, we use velocity of optical flow [22] as one of the motion cues to represent the direction and the magnitude of moving hand.DenoteMu(pt) and Mv(pt) as the velocity of the x and y direction optical flow map at position pt, respectively. And we define a patch associate with the location of a particle as patch center to represent the motion cue of each particle st(i) on both directions of optical flow map:
(6)Musti=∑x∑yMupsti+rtWu_patch,
where Wu_patch is a priori fixed window patch through the definition of a 0-centered window with size of 50×60. Similarly, we can compute the motion cue of Mv(st(i)). Such patch size is empirically defined according to the face depth of performer and considered as the largest patch during tracking. The scale rt denotes the patch size control factor needed to be estimated according to the depth value of window center. We set rt=min⁡(hand_depth/face_depth,1) with the depth images values ranging from [0–2046] and value 2047 denotes an invalid depth pixel. It means that we control the maximum distance of hand to the camera not larger than the distance of face to the camera in our experiments. The similarity of the motion indicator between the current particle motion patches summation Mu,v(st(i)) and the pervious average motion velocity of particles M(u,v),t-1* is defined as
(7)Dis(st(i))=Mu,v(st(i))-M(u,v),t-1*
in which Mu,v(st(i))=Mu(st(i))2+Mv(st(i))2 and M(u,v),t-1*=Mu,t-1*2+Mv,t-1*2.The motion likelihood of the particlest(i) is given by
(8)ΓMsti=-12πσe-Dis(sti)2/2σ2,
where σ is a standard deviation and we empirically set it as constant 1.The motion cue of each pixel is normalized as(9)ΓmotionMsti=ΓMstimax⁡⁡ΓMstii=1:K.


### 3.3. Depth Model

The probabilityΓN(zt(i),zt-1*) explains the depth relationship between particle st(i) with depth zt(i) at current frame t and the minimum depth zt-1* across all particles at last frame t-1. Only if the depth variation involved in a small threshold will give a high probability of sampling that particle, we assume that the performed hand is closer to camera device than the connected forearm; hence, the depth variation of hand between two consecutive frames would not change too much and can be expressed by Gaussian distribution:
(10)ΓNzti,zt-1*=exp⁡⁡-κzti-zt-1*2,
in which κ is the speed controlling coefficient, which controls the speed of probability decreases with increasing depth difference. (κ=0.5 in the experiment).


### 3.4. Salient Skin-Motion-Depth (SSMD) Descriptor

For each particlest(i), the probability of observation Γ(zt∣st(i)) can be computed by combining the salient skin indicator and motion indicator with depth indicator to detect the most hand-like pixel. The combined indicator probability of a particle is defined as follows:
(11)Γzt∣sti=αΓskinstiωisti+βΓmotionMsti+γΓNzti,zt-1*.
To find the empirical constants α, β, and γ, the sum of squares of the position errors between particles and annotated hand center of each frame is minimized:
(12)min⁡⁡∑t=1M∑i=1Npt(st(i))-p^t.Since the particles positions are indirectly determined by the constantsα, β, γ, we consider that the optimal parameters values can lead to the minimization value of (12). The optimal α, β, and γ are chosen iteratively in training gesture video with step-size 0.5 under the constrain that α+β+γ=1. In this gesture training video, people sit in front of the camera and randomly move hand involved in the region between the performer’s face and camera. Thus, we set the empirical constant as 0.4, 0.3, and 0.3, respectively.The weightwt(i) of each particle st(i) is calculated from the combined observation model Γ(zt∣st(i)) based on (11). Thus, the desired posterior distribution Γ(st∣zt) can be represented by the set of weighted particles {st(i),πti}n=1N. We summarize the SSMD-PF algorithm as follows.SSMD-PF Algorithm for Hand Tracking. Given the particle set {st-1i,wt-1i}i=1N, zt-1*, Mu,v,t-1* at time t-1, perform the following steps.(1)
Resample a particle set{st′(i),N-1}i=1N from the particles {st-1(i),wt-1i}i=1N.(2)
Propagate each particlest′(i) by the dynamic model in (1) to give {sti,N-1}i=1N.(3)
Weight each particle by combining saliency skin, motion, and depth cues in (11): wt(i)∝Γ(zt∣st(i)) which are normalized so that ∑i=1Nwt(i)=1.(4)
Estimate the target state of depth and motion cueszt* and Mu,v,t* from {st(i),wti}i=1N.


### 3.5. Tracking Initialization

A major shortcoming of pervious tracking methods is the need for supervised initialization. Furthermore, manual reinitialization is needed if the tracking drifts or fails. To overcome this limitation, we also proposed an efficient and fast tracking initialization method which can automatically localize the hand, irrespective of their pose, scale, and rotation.In our hand tracking system, in order to accurately and explicitly express the hand, we assume that the performed hand is comparatively closer to the camera than the corresponding arm. Since the whole signer body can be considered as the most salient object in the foreground of each frame, the connected component of human body can be computed as follow:(13)Cmax⁡=argmax⁡{Ci}i=1:NCiCiZ,
where Ci denotes the i_th depth connected component from all sets {Ci}i=1:N and CiZ denotes the average depth of Ci. A depth connected component which represents the performer body can be considered as the one with larger connected component size and with smaller average depth, as shown in Figure 2. Before using (13), each Ci should be roughly divided into the background or foreground by comparing CiZ with a depth threshold Tr=2000. When CiZ>Tr, the corresponding connected component Ci is considered as background object and removed from {Ci}i=1:N; otherwise it will be considered as the foreground object and wait for computing in (13).Hand tracking initialization.
(a)
A depth frame with signer ready for signing gesture
(b)
The salient signer body with detected face in green rectangle and hand in red rectangleBefore performing a gesture, an actor is asked to posture the hand in the upper part of the actor body, as shown in Figure2(a). According to this starting state, we divide the extracted Cmax⁡ into upper and lower parts and select the upper part pixels by comparing each vertical location of pixel in Cmax⁡ with the threshold T. Finally, the pixel with minimum depth value in the upper part is selected as the initial hand position, as shown in Figure 2(b). In our experiments, we set T=v_face_center+0.72*(480-v_face_center), where v_face_center is the vertical positon of the face center and the actor face can be easily detected by using Viola-Jones method [23] in color image.


### 3.6. Hand Center Localization

To create the hand trajectory, for simplicity, hand center should be localized to represent the entire hand. Kernel density estimation (KDE) is a nonparametric density estimation method which can be used to estimate the probability density function of particles. Given the particles distribution of each frame, the KDE is(14)PKDEptstn=1Nh∑i=1NKptstn-ptstih,
where K(·) is the Gaussian kernel function and pt(st(i)), pt(st(n)) are the locations of particles st(i),st(n); h is bandwidth parameter to be set as h=1.06σN-1/5, where σ=1 is the standard deviation of particles and N is the number of particles. Then we choose the 2D position corresponding to the peak value of the KDE as the hand center, as shown in Figure 3.Figure 3
The Kernel density estimation of particles distribution in frames 90, 100, 130, and 180 of digital gesture “0.”


## 3.1. Salient Hand Skin Model

Since human skin is a relatively uniform and discriminable cue in color space, a statistical color model can be employed to compute the probability of every pixel being skin color. A skin color distribution and a nonskin color distribution are denoted asP(r,g,b∣skin) and P(r,g,b∣~skin), respectively, in RGB color space, which is quantized to 32×32×32 values. According to these two distributions, the probability of a pixel-level particle st(i), which corresponds to pixel in color vector [rgb], can be defined as skin particle by using Bayes rule:
(2)P(skin∣st(i))=Psti∣skinP(skin)P(r,g,b),
where P(r,g,b)=P(st(i)∣skin)P(skin)+P(st(i)∣~skin)P(~skin) and P(skin) always equals 0.5.A normalized skin probability of particlest(i) is defined as
(3)Γskinsti=P(skin∣st(i))max⁡⁡Pskin∣stii=1:K,
where K is particles number in each frame and we set K=2000 overall our experiments.To improve the effectiveness of the skin color detector, an optimization method based on skin saliency is proposed in order to enhance the particles. As for a local-level image region distributed by propagating particles, the contrast and the color of the tracking object are relatively unique and distinctive than the local background. Since the target hand can be concerned as the dominant object in the context of the local image, a particlest(i) can be viewed as a salient particle if its appearance patch with centered at pixel location of st(i) is relatively distinctive with respect to all other particle patches. Specifically, we represent each particles by cropping a patch (5×5 size) surrounding it. Let dcolor(·) and dposition(·) be, respectively, the Euclidean distance of the vectorized patches of colors (in RGB space) and pixel location between two particle patches centered at pt(st(i)), pt(st(j)), respectively, where pt(st(i)) and pt(st(j)) are the pixel locations of the particles st(i), st(j). Based on the hand skin model, let us define a dissimilarity measure between a pair of particles as
(4)dsti,stj=dcolorsti,stj1+c·dpositionsti,stj.
This implies that a single pixel-level particle is treated as a salient hand skin particle when the other particles similar to it are nearby, and it is less salient when the resembling particles are far away. The parameter c set as 0.2 in our experiments. Hence, the salient hand skin factor of particle st(i) is defined as
(5)ωisti=1-exp⁡-1K∑k=1Kdsti,stk,
where K most similar patches of the particles {st(k)}k=1:K of each frame are extracted according to dcolor(st(i),st(k)), and we set K=100 in the experiment. As mentioned above, a particle is identified as hand skin saliency when ωi is high and vice versa.


## 3.2. Temporal Motion Model

Motion information is another useful cue for hand detection in gesture video. Since the hand is performed as the most salient object in front of the camera, hand motion changes more frequently than other human parts and objects background, such as arm and face. Typically, people incline to use the frame difference method to detect hand motion, whereas it is arduous to discriminate the objects with skin-like color close to that of the signed hand region. To improve the robustness of our tracking system, we use velocity of optical flow [22] as one of the motion cues to represent the direction and the magnitude of moving hand.DenoteMu(pt) and Mv(pt) as the velocity of the x and y direction optical flow map at position pt, respectively. And we define a patch associate with the location of a particle as patch center to represent the motion cue of each particle st(i) on both directions of optical flow map:
(6)Musti=∑x∑yMupsti+rtWu_patch,
where Wu_patch is a priori fixed window patch through the definition of a 0-centered window with size of 50×60. Similarly, we can compute the motion cue of Mv(st(i)). Such patch size is empirically defined according to the face depth of performer and considered as the largest patch during tracking. The scale rt denotes the patch size control factor needed to be estimated according to the depth value of window center. We set rt=min⁡(hand_depth/face_depth,1) with the depth images values ranging from [0–2046] and value 2047 denotes an invalid depth pixel. It means that we control the maximum distance of hand to the camera not larger than the distance of face to the camera in our experiments. The similarity of the motion indicator between the current particle motion patches summation Mu,v(st(i)) and the pervious average motion velocity of particles M(u,v),t-1* is defined as
(7)Dis(st(i))=Mu,v(st(i))-M(u,v),t-1*
in which Mu,v(st(i))=Mu(st(i))2+Mv(st(i))2 and M(u,v),t-1*=Mu,t-1*2+Mv,t-1*2.The motion likelihood of the particlest(i) is given by
(8)ΓMsti=-12πσe-Dis(sti)2/2σ2,
where σ is a standard deviation and we empirically set it as constant 1.The motion cue of each pixel is normalized as(9)ΓmotionMsti=ΓMstimax⁡⁡ΓMstii=1:K.


## 3.3. Depth Model

The probabilityΓN(zt(i),zt-1*) explains the depth relationship between particle st(i) with depth zt(i) at current frame t and the minimum depth zt-1* across all particles at last frame t-1. Only if the depth variation involved in a small threshold will give a high probability of sampling that particle, we assume that the performed hand is closer to camera device than the connected forearm; hence, the depth variation of hand between two consecutive frames would not change too much and can be expressed by Gaussian distribution:
(10)ΓNzti,zt-1*=exp⁡⁡-κzti-zt-1*2,
in which κ is the speed controlling coefficient, which controls the speed of probability decreases with increasing depth difference. (κ=0.5 in the experiment).


## 3.4. Salient Skin-Motion-Depth (SSMD) Descriptor

For each particlest(i), the probability of observation Γ(zt∣st(i)) can be computed by combining the salient skin indicator and motion indicator with depth indicator to detect the most hand-like pixel. The combined indicator probability of a particle is defined as follows:
(11)Γzt∣sti=αΓskinstiωisti+βΓmotionMsti+γΓNzti,zt-1*.
To find the empirical constants α, β, and γ, the sum of squares of the position errors between particles and annotated hand center of each frame is minimized:
(12)min⁡⁡∑t=1M∑i=1Npt(st(i))-p^t.Since the particles positions are indirectly determined by the constantsα, β, γ, we consider that the optimal parameters values can lead to the minimization value of (12). The optimal α, β, and γ are chosen iteratively in training gesture video with step-size 0.5 under the constrain that α+β+γ=1. In this gesture training video, people sit in front of the camera and randomly move hand involved in the region between the performer’s face and camera. Thus, we set the empirical constant as 0.4, 0.3, and 0.3, respectively.The weightwt(i) of each particle st(i) is calculated from the combined observation model Γ(zt∣st(i)) based on (11). Thus, the desired posterior distribution Γ(st∣zt) can be represented by the set of weighted particles {st(i),πti}n=1N. We summarize the SSMD-PF algorithm as follows.SSMD-PF Algorithm for Hand Tracking. Given the particle set {st-1i,wt-1i}i=1N, zt-1*, Mu,v,t-1* at time t-1, perform the following steps.(1)
Resample a particle set{st′(i),N-1}i=1N from the particles {st-1(i),wt-1i}i=1N.(2)
Propagate each particlest′(i) by the dynamic model in (1) to give {sti,N-1}i=1N.(3)
Weight each particle by combining saliency skin, motion, and depth cues in (11): wt(i)∝Γ(zt∣st(i)) which are normalized so that ∑i=1Nwt(i)=1.(4)
Estimate the target state of depth and motion cueszt* and Mu,v,t* from {st(i),wti}i=1N.


## 3.5. Tracking Initialization

A major shortcoming of pervious tracking methods is the need for supervised initialization. Furthermore, manual reinitialization is needed if the tracking drifts or fails. To overcome this limitation, we also proposed an efficient and fast tracking initialization method which can automatically localize the hand, irrespective of their pose, scale, and rotation.In our hand tracking system, in order to accurately and explicitly express the hand, we assume that the performed hand is comparatively closer to the camera than the corresponding arm. Since the whole signer body can be considered as the most salient object in the foreground of each frame, the connected component of human body can be computed as follow:(13)Cmax⁡=argmax⁡{Ci}i=1:NCiCiZ,
where Ci denotes the i_th depth connected component from all sets {Ci}i=1:N and CiZ denotes the average depth of Ci. A depth connected component which represents the performer body can be considered as the one with larger connected component size and with smaller average depth, as shown in Figure 2. Before using (13), each Ci should be roughly divided into the background or foreground by comparing CiZ with a depth threshold Tr=2000. When CiZ>Tr, the corresponding connected component Ci is considered as background object and removed from {Ci}i=1:N; otherwise it will be considered as the foreground object and wait for computing in (13).Hand tracking initialization.
(a)
A depth frame with signer ready for signing gesture
(b)
The salient signer body with detected face in green rectangle and hand in red rectangleBefore performing a gesture, an actor is asked to posture the hand in the upper part of the actor body, as shown in Figure2(a). According to this starting state, we divide the extracted Cmax⁡ into upper and lower parts and select the upper part pixels by comparing each vertical location of pixel in Cmax⁡ with the threshold T. Finally, the pixel with minimum depth value in the upper part is selected as the initial hand position, as shown in Figure 2(b). In our experiments, we set T=v_face_center+0.72*(480-v_face_center), where v_face_center is the vertical positon of the face center and the actor face can be easily detected by using Viola-Jones method [23] in color image.


## 3.6. Hand Center Localization

To create the hand trajectory, for simplicity, hand center should be localized to represent the entire hand. Kernel density estimation (KDE) is a nonparametric density estimation method which can be used to estimate the probability density function of particles. Given the particles distribution of each frame, the KDE is(14)PKDEptstn=1Nh∑i=1NKptstn-ptstih,
where K(·) is the Gaussian kernel function and pt(st(i)), pt(st(n)) are the locations of particles st(i),st(n); h is bandwidth parameter to be set as h=1.06σN-1/5, where σ=1 is the standard deviation of particles and N is the number of particles. Then we choose the 2D position corresponding to the peak value of the KDE as the hand center, as shown in Figure 3.Figure 3
The Kernel density estimation of particles distribution in frames 90, 100, 130, and 180 of digital gesture “0.”


## 4. Hand Gesture Recognition

After hand tracking phase, each gesture can be represented by a set of hand locations frame by frame. The start and end frames of the gesture are manually annotated in this paper. The top line of Figure4 shows the representation of the original spatiotemporal gestures. In this section, we first describe how to preprocess the original gesture sequence. And then a naïve shape descriptor, which is called shape-order context, is introduced and we also talk about how to match two gestures using the proposed shape descriptor. At last, we compare the computational complexity between the proposed shape descriptor and the well-known shape descriptor: shape context.Hand digital gesture sequence normalization. (a) Original gesture sequence. (b) Gesture sequence with linear interpolation. (c) Gesture sequence with equidistant sampling.
(a)
(b)
(c)


### 4.1. Gesture Sequence Normalization

Typically, different gesture sequences have different lengths. However, for the same gesture sequences, the span of every two adjacent gesture points may be different due to the variation of the sign speed. For the peer to peer points matching, we need to normalize each trajectory sequence so that they share the same point numbers. Specifically, gesture trajectories are linearly interpolated between every two adjacent original hand locations; thenM locations are equidistantly extracted from all gesture points, so that each sequence has the same length, as shown in the middle and bottom row of Figure 4, respectively. In our method, this sequences normalization step is significantly important because the shape-order context descriptors matching requires that each gesture sample has the same length, and we need to match the corresponding shape-order context descriptors of pairwise points in both training and testing data.


### 4.2. Shape Context

Shape context [21] is a descriptor which expresses the object shape by considering the relationship of the set of vectors originating from a point to all other sample points on a shape. Obviously, the full set of vectors as shape descriptors contains much too details since it configures the entire shape relative to the reference points. This set of vectors is identified as a highly discriminative descriptor which can represent the shape distribution over relative positions. The shape context of pi is defined as a coarse histogram hi of the relative coordinates of the remaining n-1 points:
(15)hi(k)=#{q≠pi:(q-pi)∈bin(k)}.
The bins are uniform in log-polar space, making the descriptor more sensitive to positions of nearby sample points than to those of points farther away.


### 4.3. Our Approach: Shape-Order Context

A human hand-dominated gesture trajectory can be simplified as a set of hand location features in spatial-temporal domain. For the traditional shape context method, the object is represented only in spatial domain. As a normalized gesture sequence with lengthL, we improve the shape context method by computing the relationship of a trajectory point to all other trajectory points in spatial-temporal domain. Specifically, each k bin of the coarse histogram hi(k) of the relative log-polar coordinate is established by accumulating sequence order difference between gesture points pi and remaining gesture points q=∪jpj which is involved in the relative region:
(16)hi(k)=∑j∈J;i≠j(j-i)subjectto(q-pi)∈bin(k),q≠pi,
where J contains the subscripts of gesture points that belong to q. In this way, the temporal information of gesture trajectory is embedded into log-polar space bins, as shown in Figure 5. Therefore, the value of each bin is dominated not only by the distance and the angle relation in gesture spatial domain, but also by impact by the sequence order relation in gesture temporal domain. Since the descriptors of shape-order context contain rich spatial-temporal information for each point, they are inherently insensitive to small perturbations which are produced by different performers as shown in Figure 6.Shape-order context descriptor computation and matching. (a) and (b) Normalized trajectory of two gestures. (c) Diagram of log-polar histogram bins used in computing the shape-order contexts. (d), (e), and (f) Example of shape-order contexts for three reference samples. Note the visual similarity of (d) and (e) is much higher than (f).
(a)
(b)
(c)
(d)
(e)
(f)Figure 6
Gesture digital “6” under different performers.To build a robust gesture recognition system, translation and scale invariance are highly desirable. For our shape-order context approach, invariance under translation is intrinsically existing since all measurements are taken with respect to points on the gesture trajectory.To achieve scale invariance, we normalize all the gesture trajectory sets by means of calculating minimum enclosing circle (MEC) of those sets and then resize each set to the same circle’s radius. As for rotation invariance, since the shape-order contexts are extremely rich descriptors and inherently insensitive to small rotation and perturbations of the shape, we use the absolute image coordinates to compute the shape-order context descriptor for each point. Another reason for using the absolute image coordinates is that for two gesture trajectories even it achieves the same appearance after rotating; however, the complete rotation invariance impedes expressing the original meaning of the gesture. Hence, it should be emphasized that the completely rotation invariance of the gesture trajectory shape is not suitable for gesture recognition.


### 4.4. Final Classification

Due to the fact that each of the gesture sequence point is represented as histogram based on shape-order context, it is natural to use theχ2 test statistic to compute the histogram similarity of pairwise (pi,qi) in the two corresponding sequence positions. Let Cij=C(pi,qi) denote the cost of matching two corresponding points:
(17)Cij=12∑k=1Khik-hjk2hik+hjk,
where hi(k) and hj(k) denote the K-bin normalized histogram at pi and qi, respectively. Thus, the cost Cij of matching pairwise points not only includes the local appearance similarity but also contains local gesture order similarity, which is particularly useful when comparing the trajectory shapes. Finally, the similarity between two gestures is computed by accumulating each matching cost and using 1-NN nearest neighbor classification to determine which class the gesture belongs to.


### 4.5. Computational Complexity

The time complexity of the shape-order context matching algorithm can be measured as shown here. Letm be the number of gesture points after gesture normalization. Let r be the number of radial bins and let a be the number of angular bins. The time complexity of computing a gesture point histogram is n=r*a. The complexity of matching histogram in our shape-order context method is O(Pmn), where P is the number of gestures in the training dataset and mn is the feature vector size containing m points in each gesture trajectory. Our proposed gesture recognition method has the same computational complexity as the original shape-order context algorithm. However, it has relatively higher recognition rate as shown in the following section.


## 4.1. Gesture Sequence Normalization

Typically, different gesture sequences have different lengths. However, for the same gesture sequences, the span of every two adjacent gesture points may be different due to the variation of the sign speed. For the peer to peer points matching, we need to normalize each trajectory sequence so that they share the same point numbers. Specifically, gesture trajectories are linearly interpolated between every two adjacent original hand locations; thenM locations are equidistantly extracted from all gesture points, so that each sequence has the same length, as shown in the middle and bottom row of Figure 4, respectively. In our method, this sequences normalization step is significantly important because the shape-order context descriptors matching requires that each gesture sample has the same length, and we need to match the corresponding shape-order context descriptors of pairwise points in both training and testing data.


## 4.2. Shape Context

Shape context [21] is a descriptor which expresses the object shape by considering the relationship of the set of vectors originating from a point to all other sample points on a shape. Obviously, the full set of vectors as shape descriptors contains much too details since it configures the entire shape relative to the reference points. This set of vectors is identified as a highly discriminative descriptor which can represent the shape distribution over relative positions. The shape context of pi is defined as a coarse histogram hi of the relative coordinates of the remaining n-1 points:
(15)hi(k)=#{q≠pi:(q-pi)∈bin(k)}.
The bins are uniform in log-polar space, making the descriptor more sensitive to positions of nearby sample points than to those of points farther away.


## 4.3. Our Approach: Shape-Order Context

A human hand-dominated gesture trajectory can be simplified as a set of hand location features in spatial-temporal domain. For the traditional shape context method, the object is represented only in spatial domain. As a normalized gesture sequence with lengthL, we improve the shape context method by computing the relationship of a trajectory point to all other trajectory points in spatial-temporal domain. Specifically, each k bin of the coarse histogram hi(k) of the relative log-polar coordinate is established by accumulating sequence order difference between gesture points pi and remaining gesture points q=∪jpj which is involved in the relative region:
(16)hi(k)=∑j∈J;i≠j(j-i)subjectto(q-pi)∈bin(k),q≠pi,
where J contains the subscripts of gesture points that belong to q. In this way, the temporal information of gesture trajectory is embedded into log-polar space bins, as shown in Figure 5. Therefore, the value of each bin is dominated not only by the distance and the angle relation in gesture spatial domain, but also by impact by the sequence order relation in gesture temporal domain. Since the descriptors of shape-order context contain rich spatial-temporal information for each point, they are inherently insensitive to small perturbations which are produced by different performers as shown in Figure 6.Shape-order context descriptor computation and matching. (a) and (b) Normalized trajectory of two gestures. (c) Diagram of log-polar histogram bins used in computing the shape-order contexts. (d), (e), and (f) Example of shape-order contexts for three reference samples. Note the visual similarity of (d) and (e) is much higher than (f).
(a)
(b)
(c)
(d)
(e)
(f)Figure 6
Gesture digital “6” under different performers.To build a robust gesture recognition system, translation and scale invariance are highly desirable. For our shape-order context approach, invariance under translation is intrinsically existing since all measurements are taken with respect to points on the gesture trajectory.To achieve scale invariance, we normalize all the gesture trajectory sets by means of calculating minimum enclosing circle (MEC) of those sets and then resize each set to the same circle’s radius. As for rotation invariance, since the shape-order contexts are extremely rich descriptors and inherently insensitive to small rotation and perturbations of the shape, we use the absolute image coordinates to compute the shape-order context descriptor for each point. Another reason for using the absolute image coordinates is that for two gesture trajectories even it achieves the same appearance after rotating; however, the complete rotation invariance impedes expressing the original meaning of the gesture. Hence, it should be emphasized that the completely rotation invariance of the gesture trajectory shape is not suitable for gesture recognition.


## 4.4. Final Classification

Due to the fact that each of the gesture sequence point is represented as histogram based on shape-order context, it is natural to use theχ2 test statistic to compute the histogram similarity of pairwise (pi,qi) in the two corresponding sequence positions. Let Cij=C(pi,qi) denote the cost of matching two corresponding points:
(17)Cij=12∑k=1Khik-hjk2hik+hjk,
where hi(k) and hj(k) denote the K-bin normalized histogram at pi and qi, respectively. Thus, the cost Cij of matching pairwise points not only includes the local appearance similarity but also contains local gesture order similarity, which is particularly useful when comparing the trajectory shapes. Finally, the similarity between two gestures is computed by accumulating each matching cost and using 1-NN nearest neighbor classification to determine which class the gesture belongs to.


## 4.5. Computational Complexity

The time complexity of the shape-order context matching algorithm can be measured as shown here. Letm be the number of gesture points after gesture normalization. Let r be the number of radial bins and let a be the number of angular bins. The time complexity of computing a gesture point histogram is n=r*a. The complexity of matching histogram in our shape-order context method is O(Pmn), where P is the number of gestures in the training dataset and mn is the feature vector size containing m points in each gesture trajectory. Our proposed gesture recognition method has the same computational complexity as the original shape-order context algorithm. However, it has relatively higher recognition rate as shown in the following section.


## 5. Experiments and Results

In this section we describe the details of our proposed hand tracking and gesture recognition experimental results and analysis. All experiments were conducted on a 3.30 GHz based Windows PC with 8 GB of RAM.


### 5.1. Datasets

Hand-signed digit datasets (HSD) [24] is a commonly used benchmark hand gesture dataset for gesture recognition with 10 categories performed by 12 different people. In more detail these datasets have been organized as follow.Training Examples. 300 digit exemplars with 30 per class were stored in the database. A total number of 10 users wearing gloves have been employed to collect all training data. These video clips are captured by using a typical color camera at 30 Hz with an image size of 240 × 320. Each digit gesture video clip depicts a user gesturing ten digits in sequence, as shown in Figure 7.Figure 7
“0–9” digital gesture exemplar in training sets.Test Examples. We recapture 440 digit exemplars with 44 per class used as queries. 400 exemplars contain distracters, in the form of a human moving back and forth in the background and a skin-like object closing to the sign hand. The rest of video clips are captured in prune background without dramatic motion. These video clips are captured by using kinect camera at 30 Hz with an image size of 480 × 640. The corresponding depth and color frame are already well calibrated by using OpenNI platform.American sign language dataset (ASL) [25] is an ongoing work which contains video sequences of thousands of distinct ASL signs with prune background. We conducted our hand tracking experiments in a user independent manner using three of the sign language video datasets. The detail of these videos can be found in Table 1.Table 1
Frame length of the selected ASL video clip.
DS1
DS2
DS3
# of frame
1315
3300
3200


### 5.2. Hand Tracking Evaluation

To verify the effectiveness of the proposed observation model, we run the algorithm on some sequences by varying the ratio of(α,β,γ) in (11). The tracking on each frame is considered correct if
(18)area(B∩G)area(B∪G)>0.5,
where B is the detection hand area and G is the ground truth hand area which is acquired by manual annotation. For comparison, the tracking is implemented by using saliency skin model, temporal motion model, and depth model, respectively. The variation of parameters values is shown in Table 2. It is observed that, by using each single model on the HSD easy dataset, the tracker has a relatively convincing tracking accuracy as SSMD-PF method does. However, on the HSD hard dataset, the accuracy of trackers with (1,0,0) and (0,1,0) parameters are dropped rapidly due to the confusion caused by moving people in the background, whereas the trackers with depth model or proposed SSMD-PF model successfully handle the distraction and our proposed model with parameters (0.4,0.3,0.3) is much more robust and reliable than other three independent models.Table 2
Comparison accuracy by varying parameters values of (11).
(α, β, γ)
(1, 0, 0)
(0, 1, 0)
(0, 0, 1)
SSMD-PF
(0.4, 0.3, 0.3)
HSD easy data
95.0
96.6
98.5
100
HSD hard data
63.1
85.4
97.9
100We then assess the proposed SSMD-PF algorithm’s performance in comparison to the several conventional PF methods. Evaluation is done on the HSD datasets and the selected ASL video clips, respectively, as shown in Tables3, 4, and 5. In Table 3, our method achieves 100% accuracy on the HSD easy datasets, which is the same with the state-of-the-art methods in [11, 16]. It indicates that the pure and stable background can greatly improve the tracking accuracy. In Table 4, as expected, the tracking accuracy of SSMD-PF method achieves 100% on the HSD hard datasets, which is higher than [16, 26], which also uses the color and depth cue as tracking feature. The main reason of our method outperforming others is that of the use of the local minima hand depth cue as one of observation models in particle filter, which is strong enough to get rid of disturbing by other skin-like objects. The visualization of SSMD-PF based tracking algorithm in clean and complex background is shown in Figure 8. Figure 9 shows the tracking of digital gesture “3” with skin-like object close to that of the signed hand in the relatively same depth level. As we can see, the sign hand is not disturbed by such skin-like closing object because our system can discriminate it by taking advantage of the velocity as motion cue in observation model. Due to the lack of using the depth information, the other methods [11, 27] easily lose tracking, thus without showing tracking accuracy. In view of the fact that the ASL datasets only contain color information; we combine our salient skin and motion model (SSM-PF) to compare with other method in order to evaluate our color based on tracking method, as shown in Figure 10. In Table 5, we observe that our proposed method achieves better or comparable tracking accuracy with the Shan’s method [11] and has much better performance than Zhang’s [5], Perez’s [26], and Mittal’s method [6].Table 3
HSD easy datasets.
Methods
Accuracy (%)
SSMD-PF
100
SSM-PF
100
Van Den Bergh and Van Gool [16]
100
Shan et al. [11]
100
Pérez et al. [27]
95.2Table 4
HSD hard datasets.
Methods
Accuracy (%)
SSMD-PF
100
Doliotis et al. [26]
95.32
Van Den Bergh and Van Gool [16]
95.38
Shan et al. [11]
—
Pérez et al. [27]
—Table 5
ASL datasets.
Methods
Accuracy (%)
SSM-PF
91.7
Shan et al. [11]
90.3
Zhang et al. [5]
79.3
Mittal et al. [6]
40.2
Pérez et al. [27]
65.9Digital gesture sequence tracked by SSMD-PF in clean and complex background, respectively. (a) and (d) Original frame (frames 90, 100, 130, and 160 in the left; frames 19, 40, 85, and 101 in the right). (b) and (e) The corresponding depth frame. (c) and (f) The tracking result with particles distribution in blue color and the hand bounding box in red (gray) color.
(a)
(b)
(c)
(d)
(e)
(f)Figure 9
SSMD-PF based tracking on digital “3” with skin-like object close to that of signed hand.Figure 10
Tracking result of the consecutive sequence on the ASL dataset.To make an intuitive comparison, we also provide a visualization of gesture digital tracking from HSD hard dataset as shown in Figure11. Each 2D hand points in the pixel coordinate are transformed to 3D depth camera coordinate in advance based on compute camera intrinsic parameter [28]. Since the digital gestures are signed without dramatically depth variation, the gesture trajectory can be well fitted with ground truth.Visualization trajectory matching based on the HSD hard dataset.


### 5.3. Gesture Recognition Evaluation

For gesture recognition, we evaluate our proposed method on the HSD datasets. The shape-order context method is firstly evaluated by changing the sampling point number from each interpolated gesture sequence. As shown in Table6, with varying the sampling number, the proposed method remains stable and high classification accuracy over 98.4% with varying the sampling number from 20 to 80, and achieves 98.6% with 40 sampling numbers. It shows overall better performance than one dollar method [19] and Liu’s method [20] which also implement the sequence normalization with varying sequence length.Table 6
Classification accuracy versus sequence normalization length on 440 HSD datasets.
Sampling number
20
30
40
50
60
70
80
Classification rate (%)
Proposed method
98.4
98.4
98.6
98.4
98.4
98.4
98.4
Liu’s method [20]
94.6
96.8
95.5
93.2
93.2
91.6
89.8
One dollar [19]
88.2
89.1
89.1
90.2
91.6
90.2
91.4The results of the false positive rate (FPR) and the false negative rate (FNR) on the HSD datasets (40 sampling numbers, 440 testing gesture sequences, and 300 training gesture sequences) are illustrated in Table7. It shows that both the average FPR (with 0.0526) and FNR (with 0.0066) of proposed method are lower than Liu’s method and one dollar method. Moreover, both FPR and FNR of our method are not much fluctuating on each digital sign.Table 7
Comparison of the false positive rate and the false negative rate on each hand digital sign.
Digital gesture
0
1
2
3
4
5
6
7
8
9
Proposed method
FP (%)
0.052
0.074
0.052
0.0519
0.04
0.052
0.052
0.048
0.052
0.052
FN (%)
0
0
0
0
0.033
0
0
0.033
0
0
Liu’s method [20]
FP (%)
0.51
0.51
0.51
0
0
0.25
0.51
1.26
0
0
FN (%)
2.27
9.09
0
0
6.82
0
4.55
4.55
0
4.55
One dollar [19]
FP (%)
0.048
0.170
0.048
0.033
0.070
0.056
0.052
0.033
0.052
0.048
FN (%)
0.033
0.033
0.067
0.467
0
0.033
0
0.133
0
0.067We also compare our approach to other state-of-the-art methods with varying the training size on the HSD datasets. We use the testing set with 440 numbers of gesture sequences which are captured by using our proposed SSMD-PF method. The total training size contains 300 meaningful gesture sequences with 30 gesture sequences per class. There are 10 classes in total. To vary the training size, we increase the amount of the digital gesture examples in each class as[4,5,10,15,20,25,30]. It is worth noticing that each of the subsequent training set contains all pervious training set and gradually increases until covering all 300 training data. As shown in Figure 12, all the classification accuracy can be greatly impacted by increasing training data and the performance of our approach outperforms other method in overall training size, which again validates the effectiveness of the shape-order context method in temporal-spatial domain.Figure 12
The classification accuracy versus training size.Figures13(a) and 13(b) represent a considerable improvement of recognition accuracy with changing gesture scale and translation, respectively. To apply a certain amount of translation to the input trajectory sequences, we add a set of small increments in the pixel unit ([10,20,30,40,50,60,70,80,90]) to the x and y coordinates of the position of each gesture point. To apply a certain amount of scaling to the input gestures, we multiply the x and y coordinates of each gesture point by a set of small increments in the pixel unit ([1.1,1.2,1.3,1.4,1.5,1.6]). With gradually increasing scale and translating factors, the classification accuracy of dynamic time warping (DTW), Zhou’s method, and Jonathon’s method is rapidly dropped below 70%; however, the proposed method, Liu’s method, and one dollar method still remain a stable accuracy without change. Moreover, our method achieves better performance with 1.6% higher than Liu’s method. This advantage owes to the shape-order context descriptor which is invariant to several common transformations, which make it possible to recognize similarity gesture even with slight appearance variation.Classification accuracy versus gesture scale and translation. (a) Thex-axis is the scale factor, which expresses the increase of multiple of hand location in the x and y coordinates, applied to both the x and y dimensions of the test sequences. (b) The x-axis is the translation factor, which expresses the pixel translation increment of hand location, applied to both the x and y dimensions of the test sequences.
(a)
(b)


## 5.1. Datasets

Hand-signed digit datasets (HSD) [24] is a commonly used benchmark hand gesture dataset for gesture recognition with 10 categories performed by 12 different people. In more detail these datasets have been organized as follow.Training Examples. 300 digit exemplars with 30 per class were stored in the database. A total number of 10 users wearing gloves have been employed to collect all training data. These video clips are captured by using a typical color camera at 30 Hz with an image size of 240 × 320. Each digit gesture video clip depicts a user gesturing ten digits in sequence, as shown in Figure 7.Figure 7
“0–9” digital gesture exemplar in training sets.Test Examples. We recapture 440 digit exemplars with 44 per class used as queries. 400 exemplars contain distracters, in the form of a human moving back and forth in the background and a skin-like object closing to the sign hand. The rest of video clips are captured in prune background without dramatic motion. These video clips are captured by using kinect camera at 30 Hz with an image size of 480 × 640. The corresponding depth and color frame are already well calibrated by using OpenNI platform.American sign language dataset (ASL) [25] is an ongoing work which contains video sequences of thousands of distinct ASL signs with prune background. We conducted our hand tracking experiments in a user independent manner using three of the sign language video datasets. The detail of these videos can be found in Table 1.Table 1
Frame length of the selected ASL video clip.
DS1
DS2
DS3
# of frame
1315
3300
3200


## 5.2. Hand Tracking Evaluation

To verify the effectiveness of the proposed observation model, we run the algorithm on some sequences by varying the ratio of(α,β,γ) in (11). The tracking on each frame is considered correct if
(18)area(B∩G)area(B∪G)>0.5,
where B is the detection hand area and G is the ground truth hand area which is acquired by manual annotation. For comparison, the tracking is implemented by using saliency skin model, temporal motion model, and depth model, respectively. The variation of parameters values is shown in Table 2. It is observed that, by using each single model on the HSD easy dataset, the tracker has a relatively convincing tracking accuracy as SSMD-PF method does. However, on the HSD hard dataset, the accuracy of trackers with (1,0,0) and (0,1,0) parameters are dropped rapidly due to the confusion caused by moving people in the background, whereas the trackers with depth model or proposed SSMD-PF model successfully handle the distraction and our proposed model with parameters (0.4,0.3,0.3) is much more robust and reliable than other three independent models.Table 2
Comparison accuracy by varying parameters values of (11).
(α, β, γ)
(1, 0, 0)
(0, 1, 0)
(0, 0, 1)
SSMD-PF
(0.4, 0.3, 0.3)
HSD easy data
95.0
96.6
98.5
100
HSD hard data
63.1
85.4
97.9
100We then assess the proposed SSMD-PF algorithm’s performance in comparison to the several conventional PF methods. Evaluation is done on the HSD datasets and the selected ASL video clips, respectively, as shown in Tables3, 4, and 5. In Table 3, our method achieves 100% accuracy on the HSD easy datasets, which is the same with the state-of-the-art methods in [11, 16]. It indicates that the pure and stable background can greatly improve the tracking accuracy. In Table 4, as expected, the tracking accuracy of SSMD-PF method achieves 100% on the HSD hard datasets, which is higher than [16, 26], which also uses the color and depth cue as tracking feature. The main reason of our method outperforming others is that of the use of the local minima hand depth cue as one of observation models in particle filter, which is strong enough to get rid of disturbing by other skin-like objects. The visualization of SSMD-PF based tracking algorithm in clean and complex background is shown in Figure 8. Figure 9 shows the tracking of digital gesture “3” with skin-like object close to that of the signed hand in the relatively same depth level. As we can see, the sign hand is not disturbed by such skin-like closing object because our system can discriminate it by taking advantage of the velocity as motion cue in observation model. Due to the lack of using the depth information, the other methods [11, 27] easily lose tracking, thus without showing tracking accuracy. In view of the fact that the ASL datasets only contain color information; we combine our salient skin and motion model (SSM-PF) to compare with other method in order to evaluate our color based on tracking method, as shown in Figure 10. In Table 5, we observe that our proposed method achieves better or comparable tracking accuracy with the Shan’s method [11] and has much better performance than Zhang’s [5], Perez’s [26], and Mittal’s method [6].Table 3
HSD easy datasets.
Methods
Accuracy (%)
SSMD-PF
100
SSM-PF
100
Van Den Bergh and Van Gool [16]
100
Shan et al. [11]
100
Pérez et al. [27]
95.2Table 4
HSD hard datasets.
Methods
Accuracy (%)
SSMD-PF
100
Doliotis et al. [26]
95.32
Van Den Bergh and Van Gool [16]
95.38
Shan et al. [11]
—
Pérez et al. [27]
—Table 5
ASL datasets.
Methods
Accuracy (%)
SSM-PF
91.7
Shan et al. [11]
90.3
Zhang et al. [5]
79.3
Mittal et al. [6]
40.2
Pérez et al. [27]
65.9Digital gesture sequence tracked by SSMD-PF in clean and complex background, respectively. (a) and (d) Original frame (frames 90, 100, 130, and 160 in the left; frames 19, 40, 85, and 101 in the right). (b) and (e) The corresponding depth frame. (c) and (f) The tracking result with particles distribution in blue color and the hand bounding box in red (gray) color.
(a)
(b)
(c)
(d)
(e)
(f)Figure 9
SSMD-PF based tracking on digital “3” with skin-like object close to that of signed hand.Figure 10
Tracking result of the consecutive sequence on the ASL dataset.To make an intuitive comparison, we also provide a visualization of gesture digital tracking from HSD hard dataset as shown in Figure11. Each 2D hand points in the pixel coordinate are transformed to 3D depth camera coordinate in advance based on compute camera intrinsic parameter [28]. Since the digital gestures are signed without dramatically depth variation, the gesture trajectory can be well fitted with ground truth.Visualization trajectory matching based on the HSD hard dataset.


## 5.3. Gesture Recognition Evaluation

For gesture recognition, we evaluate our proposed method on the HSD datasets. The shape-order context method is firstly evaluated by changing the sampling point number from each interpolated gesture sequence. As shown in Table6, with varying the sampling number, the proposed method remains stable and high classification accuracy over 98.4% with varying the sampling number from 20 to 80, and achieves 98.6% with 40 sampling numbers. It shows overall better performance than one dollar method [19] and Liu’s method [20] which also implement the sequence normalization with varying sequence length.Table 6
Classification accuracy versus sequence normalization length on 440 HSD datasets.
Sampling number
20
30
40
50
60
70
80
Classification rate (%)
Proposed method
98.4
98.4
98.6
98.4
98.4
98.4
98.4
Liu’s method [20]
94.6
96.8
95.5
93.2
93.2
91.6
89.8
One dollar [19]
88.2
89.1
89.1
90.2
91.6
90.2
91.4The results of the false positive rate (FPR) and the false negative rate (FNR) on the HSD datasets (40 sampling numbers, 440 testing gesture sequences, and 300 training gesture sequences) are illustrated in Table7. It shows that both the average FPR (with 0.0526) and FNR (with 0.0066) of proposed method are lower than Liu’s method and one dollar method. Moreover, both FPR and FNR of our method are not much fluctuating on each digital sign.Table 7
Comparison of the false positive rate and the false negative rate on each hand digital sign.
Digital gesture
0
1
2
3
4
5
6
7
8
9
Proposed method
FP (%)
0.052
0.074
0.052
0.0519
0.04
0.052
0.052
0.048
0.052
0.052
FN (%)
0
0
0
0
0.033
0
0
0.033
0
0
Liu’s method [20]
FP (%)
0.51
0.51
0.51
0
0
0.25
0.51
1.26
0
0
FN (%)
2.27
9.09
0
0
6.82
0
4.55
4.55
0
4.55
One dollar [19]
FP (%)
0.048
0.170
0.048
0.033
0.070
0.056
0.052
0.033
0.052
0.048
FN (%)
0.033
0.033
0.067
0.467
0
0.033
0
0.133
0
0.067We also compare our approach to other state-of-the-art methods with varying the training size on the HSD datasets. We use the testing set with 440 numbers of gesture sequences which are captured by using our proposed SSMD-PF method. The total training size contains 300 meaningful gesture sequences with 30 gesture sequences per class. There are 10 classes in total. To vary the training size, we increase the amount of the digital gesture examples in each class as[4,5,10,15,20,25,30]. It is worth noticing that each of the subsequent training set contains all pervious training set and gradually increases until covering all 300 training data. As shown in Figure 12, all the classification accuracy can be greatly impacted by increasing training data and the performance of our approach outperforms other method in overall training size, which again validates the effectiveness of the shape-order context method in temporal-spatial domain.Figure 12
The classification accuracy versus training size.Figures13(a) and 13(b) represent a considerable improvement of recognition accuracy with changing gesture scale and translation, respectively. To apply a certain amount of translation to the input trajectory sequences, we add a set of small increments in the pixel unit ([10,20,30,40,50,60,70,80,90]) to the x and y coordinates of the position of each gesture point. To apply a certain amount of scaling to the input gestures, we multiply the x and y coordinates of each gesture point by a set of small increments in the pixel unit ([1.1,1.2,1.3,1.4,1.5,1.6]). With gradually increasing scale and translating factors, the classification accuracy of dynamic time warping (DTW), Zhou’s method, and Jonathon’s method is rapidly dropped below 70%; however, the proposed method, Liu’s method, and one dollar method still remain a stable accuracy without change. Moreover, our method achieves better performance with 1.6% higher than Liu’s method. This advantage owes to the shape-order context descriptor which is invariant to several common transformations, which make it possible to recognize similarity gesture even with slight appearance variation.Classification accuracy versus gesture scale and translation. (a) Thex-axis is the scale factor, which expresses the increase of multiple of hand location in the x and y coordinates, applied to both the x and y dimensions of the test sequences. (b) The x-axis is the translation factor, which expresses the pixel translation increment of hand location, applied to both the x and y dimensions of the test sequences.
(a)
(b)


## 6. Conclusion

We present a novel state-of-the-art hand tracking and gesture recognition system, respectively. By combining the enhanced skin, motion, and depth feature in particle filter model, the performing hand can be well localized and tracked in every frame. We also introduce a fast and simple tracking initializing method for fully automatically tracking. A shape-order context descriptor is then proposed for gesture sequence matching in temporal spatial domain. Such a rich descriptor can greatly improve the gesture recognition rate and be invariant to gesture to translate and scale. In the future work, we will explore more sophisticated features for more advanced tracking, such as the hand attaching on object and hand-arm parallel moving. Also, the hand appearance will be considered to embed into shape-order context descriptor for more robust gesture recognition.



---

*Source: 863732-2015-02-22.xml*