Depth data and fusion of feature descriptors for static gesture recognition

IET Image ProcessingVolume 14, Issue 5 p. 909-920 Research ArticleFree Access Depth data and fusion of feature descriptors for static gesture recognition Prachi Sharma, Corresponding Author Prachi Sharma prach.dee2015@iitr.ac.in Department of Electrical Engineering, IIT Roorkee, Uttarakhand, IndiaSearch for more papers by this authorRadhey Shyam Anand, Radhey Shyam Anand Department of Electrical Engineering, IIT Roorkee, Uttarakhand, IndiaSearch for more papers by this author Prachi Sharma, Corresponding Author Prachi Sharma prach.dee2015@iitr.ac.in Department of Electrical Engineering, IIT Roorkee, Uttarakhand, IndiaSearch for more papers by this authorRadhey Shyam Anand, Radhey Shyam Anand Department of Electrical Engineering, IIT Roorkee, Uttarakhand, IndiaSearch for more papers by this author First published: 03 March 2020 https://doi.org/10.1049/iet-ipr.2019.0230Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this study, the authors propose a novel methodology for static gesture recognition in a complex background using only depth map from Microsoft's Kinect camera. Four different types of features are extracted and analysed on two public static gesture datasets. The features extracted from the segmented hand are geometrical, local binary patterns, number of fingers (Num) raised in a gesture and distance of hand palm centre from the fingertips and the valley between the fingers. The hand region is first segmented from the image using depth data followed by the forearm removal. Four multi-class support vector machine (SVM) kernels are also compared and used for recognition of gestures with extracted feature vector as an input. The experimental results achieved recognition accuracy of 99 and on two public complex static gesture datasets using Gaussian SVM kernel function as a classifier. The proposed approach is found to be comparable and even outperforms some of the state-of-the-art techniques in terms of high recognition accuracies, even after using a single cue for hand segmentation and extraction of features in the complex background which results in non-dependency on too many cues and much hardware. 1 Introduction Hand gesture recognition is a challenging yet crucial research problem. It is the major application of human–computer interaction where a human can not only interact with the computer using text, touch and speech but also with gestures. Significant work has been reported on gesture recognition since the 1970s utilising two approaches – glove-based [1] and vision-based. In the former method, the user performs the gesture by wearing a glove embedded with some sensors to record the changes in hand position based on which, the performed gesture is recognised. In a vision-based method, the user is free from any hardware attached to his body as the performed gesture is recognised with the help of computer vision in which the hand features are extracted from the bare hand. That is why; this method is preferred over the glove-based method due to the ease of performing a gesture and reduction in the hardware used. To capture the hand gesture, researchers have used many sensors or cameras to measure different types of data from a scene like RGB data, depth data, and skeleton data (position of body joints). The work done by Dahmani and Larabi [2], Haria et al. [3] and Fagiani et al. [4] involves segmentation of the hand using only a simple RGB camera followed by features extraction followed by classification. The segmentation of hand is not reliable using only an RGB camera due to the following reasons: the presence of more skin colour objects present in the image other than the hand, too much variation in the skin colour of the signers and complex background. The challenges involved in a Gesture Recognition System, such as occlusion of hands with each other or with the face and illumination changes, are still unsolved. The introduction of the cost-effective Microsoft Kinect camera in 2010 provides the RGB and depth map of an image or video frame and skeleton points of a human body, which can be used for the segmentation, tracking, and recognition of gestures. Thus, this device has become quite popular in the researcher's community. The Kinect camera, in conjunction with suitable feature extraction, feature selection and classification techniques, makes the gesture recognition more reliable and robust. He et al. [5] detected 3D parameters of a human body using only depth data from the Kinect sensor. The Santos et al. [6] involved segmentation and feature extraction using a depth map with Dynamic Time Warping (DTW) and Hidden Markov Models (HMMs) as a hybrid classifier. Discrete Cosine Transform was used as a feature vector and Artificial Neural Network as a classifier followed by hand segmentation using depth thresholding by Joudaki et al. [7]. The work done by the above-mentioned researchers have used only a single cue from the Kinect camera for segmentation of hand and extraction of features but the datasets used include simple gestures performed in a uniform background. Thus, to work on complex datasets, many authors combined two or more cues from the Kinect camera for more reliable and accurate gesture recognition. Mahnud et al. [8] used both RGB and depth map for hand segmentation and extraction of features. Plouffe and Cretu [9] utilised both the depth map and skeleton information for hand segmentation and DTW for classification. Yang [10], Wang et al. [11] and Lee et al. [12] used depth and skeleton cues for hand segmentation with different classification techniques, namely, Hierarchical Conditional Random Fields, template matching and support vector machine (SVM) respectively. The work, done in [13, 14], also used SVM for classification of gestures. The work was done by Almeida et al. [1] used all the three attributes of the Kinect camera: intensity data, depth map, and skeletal points for segmentation and feature extraction with Radial Basis Function (RBF) kernel of SVM as the classification technique. Sometimes, authors combine the data from the Kinect sensor with other sensors [15] to enable fast and better recognition. One such sensor is the Leap Motion sensor, launched in November 2010 by an American company, to measure the 3D position of hands with the help of two infrared cameras. Kumar et al. [16] combined the data from the Kinect sensor with the Leap Motion sensor for segmentation of hand and extraction of features. They used the combination of HMMs and Bidirectional Long Short-Term Memory Neural Network for classification. Combining multiple cues from the Kinect sensor or the fusion of data from two or more sensors enhances the recognition accuracy, but strikingly increases the dependency of the gesture recognition system on more modalities and hardware. This issue is addressed in this paper by recognising the static gestures using the depth map as the only cue for hand segmentation and extraction features. This paper deals with a gesture recognition system, which can work even with the following challenges: varying illumination, occlusion of hand with the face, varying skin tones, complex background and varying orientations of hand with respect to the camera. A novel algorithm has been developed that utilises only the depth values for hand region segmentation from a complex background followed by the removal of the region below the wrist (forearm region) and retaining only the hand palm and fingers in the image. The features extracted from the segmented hand are geometrical features, number of fingers in the performed gesture, local binary patterns (LBPs) and distance of fingertips and finger valleys from the centre of the hand palm. The extracted feature set is then used as an input to a multi-class SVM for classification of gestures. Two public complex static gesture datasets are used for training and testing. Experimental results indicate to achieve improved accuracy over the state-of-the-art approaches for both the datasets. The paper is divided into the following sections: Section 1 introduces the gesture recognition system and reports the literature. Section 2 briefs the proposed methodology and Section 3 explains the segmentation of hand region and forearm removal. Feature extraction is discussed in Section 4 and classification techniques in Section 5. Section 6 covers the experiments performed and their results with the conclusion in Section 7. 2 Proposed methodology The proposed methodology of static gesture recognition consists of three parts as shown in Fig. 1. These three parts deal with the segmentation of hand, extraction of features and classification of gestures. The hand region is segmented from the image utilising the depth map obtained from the Kinect camera. The segmented hand region is further segmented into the two regions: region above the wrist consisting of hand palm and fingers and the region below the wrist which is the forearm. The forearm is then removed retaining only the hand palm and fingers as forearm does not play any role in distinguishing different gestures. From the segmented hand, the following features are extracted: hand geometry, number of fingers present in the gesture, LBPs and the distance of fingertips and finger valleys from the centroid of the hand. In this paper, the distance features and the number of fingers are combined to form a single feature vector, named, Dist+Num. Lastly, the extracted feature vectors are given as an input to a multi-class SVM for classification of gestures. Each part of the proposed methodology is discussed in detail in Sections 3 to 5. Fig. 1Open in figure viewerPowerPoint Schematic diagram of proposed methodology 3 Segmentation of hand A hand region can be divided into two regions: the region above the wrist consisting of hand palm and fingers and the region below the wrist, i.e. the forearm. The region above the wrist plays an important role in signifying the differences between different gestures through orientation and position of fingers. Thus, it is necessary to separate the region above the wrist from the forearm region after the segmentation of the hand region from the complex background. The two steps, involved in segmenting only hand palm and fingers from an image, are discussed below. 3.1 Hand region segmentation Depth map provided by the Kinect camera is the preferred cue [8, 17] to segment the closest object in an image as it aids in addressing the following challenges present in gesture recognition-cluttered background, light source with varying illumination levels, varying skin tones of the user's hands performing the gesture and occlusion of hands with each other and with the face. In this paper, the segmentation of hand region is carried out by applying only a depth threshold unlike in [18, 19] where multiple cues and multiple sensors are used for segmentation respectively. As the hand is always the closest object to the camera, first, the smallest depth value is located from the depth image. This value corresponds to the point on the hand region which is closest to the camera plane. Then, an empirical value of depth threshold, say A, is added to to give the value of segmentation threshold. Thus, the segmentation threshold is given as where A is estimated based on different possible hand shape orientations. After multiple measurements and testing on the validation set, the value of A is chosen such that defines a range of depth values that allows a signer to perform a gesture in between. Thus, the hand pixels can be represented as the pixels having depth value and . Empirically, the value of A equals to for Dataset 1 and for Dataset 2 is found to be suitable for experimental results. The obtained hand pixels, sometimes, have unwanted pixels due to the presence of noise at the same depth level. To remove those pixels, each pixel in the image is checked for eight-connected components such that if any two pixels share an edge or vertex, they are said to be in an eight-connected object in the image. Out of all the connected components, the one having the largest number of white pixels is selected as the hand region. It is then followed by median filtering which is an effective method of preserving the edges and reducing the noise. 3.2 Forearm removal To distinguish between different gestures, the information about the position of fingers is crucial and can be obtained from the region above the wrist consisting of fingers and hand palm. Thus, to reduce the computation time while extracting the features and classifying a gesture, it is necessary to remove the unwanted forearm region from the image. The forearm removal algorithm developed in this section is based on the fact that usually when a user performs a gesture, the hand does not tilt completely in either horizontal left or right direction and not in a straight vertical position, where fingers point in the downward direction. The datasets used in this paper have the same scenario and so the forearm removal algorithm presented in this section will follow the same hand orientation conditions. To locate the forearm region in the image, it is necessary to locate the wrist line first. Once the wrist line is located, it is easy to locate the forearm region also, as it is the region below the wrist line. A circle is fitted on the hand palm region [18], whose intersection with the hand boundary will detect the wrist line. The circle can intersect anywhere with the hand boundary but those points will be selected which will properly define the wrist line. For this reason, the hand has to be positioned in a particular orientation irrespective of the original hand position in such a way that the hand is always in a straight vertical position with fingers pointing upwards. Now, we can say that the wrist line will always be present in the region below the centre of the hand palm. So, first, the hand is rotated in the desired orientation and then, the circle is fitted on the hand palm. Axis of hand and its inclination are calculated in order to rotate it in the desired orientation. To find the hand's axis, at least two points are needed on the hand such that when joined, the line should pass approximately through the middle of the hand as shown in Fig. 2c. The first point is the centroid of the hand that can be calculated by filtering the hand image by a two-dimensional (2D) Gaussian smoothing kernel with a large sigma or standard deviation value as given by Dominio et al. [18] and is shown in Fig. 2b. After filtering, the point in the image having the 100% of the density, i.e. maximum value of grey level intensity in the image is then selected as the centroid of the hand. The number of maximum density points can be more than one lying on the forearm region or near the hand palm region also. If these points are two or more than two, then the axis of the hand can be easily detected, but if the count of these points is 1, then, one more point has to be found out on the hand region. For this, the image is scanned from left to right and bottom to top and the first white pixel detected is selected as the second point as marked in Fig. 2c. The two points are then joined together to obtain the axis of hand. Fingers are spread in a larger area due to which the scanning is not performed top to bottom because the second point will then lie on any of the fingertips resulting in an improper hand's axis when joined with the first point. Fig. 2Open in figure viewerPowerPoint Depiction of hand rotation (a) Segmented hand region, (b) Gaussian blurring, (c) Maximum density pixel as the centroid of the hand with one more point through the scanning of the image from bottom to top and left to right, (d) Rotated hand Now the inclination of the hand's axis is calculated based on the concept that is equal to the slope of a line. The amount of rotation R to be carried out, in order to get the desired hand orientation, depends on the sign of as shown in Fig. 3. Thus, R will be positive, when is negative and negative when is positive. This will decide the direction of rotation: clockwise when R is positive and anti-clockwise when R is negative. Fig. 3Open in figure viewerPowerPoint Rotation of hand (a) When is negative, (b) When is positive Circle fitting on the hand palm utilises the centroid of the hand as the starting point. The centroid of the rotated hand is again calculated by the same concept of Gaussian blurring and selecting that point in the image having of the maximum density thereby reducing the chances of getting the maximum density point on the forearm region. Sometimes, there are two or more maximum density points near the hand palm region. To select the best centroid among all, scanning of the image is done from top to bottom and left to right to detect the first white pixel in the image. This white pixel lies on one of the fingertips of the hand and is denoted as . The distance of from the maximum density points is calculated and compared with each other. The maximum density point having the minimum distance from is selected as the centroid of the hand C because less the distance, more the centroid is on the central area of the hand palm. With the centroid of the hand C as the centre, a circle is fitted on the hand palm region with an initial radius of 1 pixel. The hand region is now divided into four quadrants with C as the centre. Due to a fixed position of the hand after rotation, the wrist line will always lie in the third and the fourth quadrants (Fig. 4b). Thus, the radius of the circle is increased until it intersects the hand boundary in both these quadrants. To form the wrist line, two intersecting points are required – one in each of the two quadrants but many times, the count is more as shown by markers in Fig. 4d. As the boundary line of the wrist is more near to the vertical axis passing through C compared to the boundary of the hand palm, that point is selected in each of the quadrant, which is near to that vertical axis. This will help in reducing the chances of selecting the intersecting points lying more on the hand palm region. With the help of the calculated two extreme points of the wrist, a wrist line is formed and thus, the region below that line, i.e. forearm is discarded retaining only the hand palm and fingers in the image as shown in Figs. 4e and f. Fig. 4Open in figure viewerPowerPoint Circle fitting on hand palm (a) Circle fitting on the hand palm,(b) Red markers show theintersection of the circle with the hand boundary,(c) Circle fitting with radiusincreased, (d) Red markers show theintersection of the circle in the third and fourth quadrants,(e) Line joining the pointsobtained after the intersection of circle with the wrist region,(f) Removal of region belowthe wrist (forearm) 4 Extraction of features The features extracted from the hand segmented from the complex background are as follows. 4.1 Geometrical features These features describe the geometry of the hand. The hand's geometry changes with the gesture and this fact is utilised to distinguish between two or more gestures. The geometrical features, which are used in this paper, are – area, perimeter, minor and major axis lengths, eccentricity, convex area and solidity. Let an image of size , where M is the number of rows from 0 to and N is the number of columns from 0 to with as the intensity of a pixel located at row r and column c in an image. The area A of any region R, hand in our case, in a binary image, can be defined as (1)To find the boundary length or perimeter of any region, Herbert Freeman [20] used the concept of Code Chain, which simply provides the steps necessary to move from one pixel to another on the boundary of that region with any random starting point. The distance travelled from the centre of one pixel to another is given by a code value based on the directions: right, left, up, down, diagonally up and right, up and left, down and right and down and left. Let cc be a Chain Code sequence , where m is the number of codes in a sequence and n represents the code value. The optimal value of perimeter can be found according to Vossepoel and Smeulders [21] and is given by (2)where E is the number of even codes present in the chain code sequence cc and Co is the corner count which is the number of times the cc value changes. To calculate the eccentricity, and major & minor axis lengths, an ellipse is fitted on the region of interest, i.e. hand region with C as the centre. The fitting of the ellipse on the region is such that the region and the ellipse have the same second central moment which is a covariance matrix in the 2D case. With the help of eigenvalues, the major axis length a and minor axis length b of the ellipse can be determined. Now, with the help of a and b, the eccentricity E of a region is calculated as: (3)The convex hull is the smallest convex polygon enveloping the region such that the boundary pixels of the region lie on or inside that polygon. Solidity is calculated using the convex hull properties and is defined as the proportion of pixels in the convex hull present inside the region of interest. The numerical values of all the geometrical features used are presented in Fig. 5 for three different types of gestures. Fig. 5Open in figure viewerPowerPoint Representation of geometrical features for three different gestures 4.2 Distance features and number of fingers ([Distance, Num]) This feature vector is a combination of two different features – number of fingers raised in a gesture (say, Num) and the distance of C from the fingertips and the valley between the fingers. To extract both these features, fingertips and the valley between the fingers have to be detected. The hand region is first divided into 12 equal sectors [18] with C as the centre and then the distance of hand contour pixels is found out from C as shown in Fig. 6. The sectors, in which the fingers are raised, will have a maximum and minimum distance point representing a fingertip and a finger valley, respectively. Thus, a maximum and a minimum distance point are selected from each sector making a total of pixel points. This feature set of size is the distance feature set for each gesture. If is the sector with q range from 1 to 12 and be the hand contour pixels, the distance of from the centre C is given by (4)The fingertips and valley between the fingers are then calculated as (5) (6) Fig. 6Open in figure viewerPowerPoint Distance features and count of fingers. Left to right. Top first row: Centroid of hand and division of boundary image into 12 equal sectors. Top second row: Fingertips (red marker) & valley between the fingers (blue marker) and distances of both the markers from the centroid Top third row: Approximation of hand palm by formation of polygon, in blue colour, joining the valley markers and retained fingers The number of fingers raised in a gesture is calculated using the finger valley points , i.e. the points, where the palm region ends and the region of fingers starts. These points are joined forming a polygon that represents the hand palm and is removed from the hand region retaining only the fingers in the image as shown in Fig. 6. To count the number of fingers correctly, the fingers should be separated from each other properly. However, after removal of the hand palm, sometimes it happens that a finger is still connected to another finger from the bottom resulting in the wrong count of fingers. To avoid this, the polygon formed is first dilated and then removed reducing the error in finding the number of fingers. This feature is combined with the distance feature set making a single feature vector of size [] for each gesture and is named as [Distance, Num]. This feature set is the modified version of the distance features calculated in [18]. The two modifications are: usage of whole hand including palm and fingers when calculating the distance of edge pixels from the centre of the hand palm, keeping both the maximum (fingertip) and minimum (finger valley) distance points from centre in the feature set, and concatenating the value of the count of fingers raised in a gesture with the distance features as [Distance, Num] [Distance, Number of Fingers]. 4.3 Local binary patterns These features help to determine the texture features [17] and can also be used to determine the shape and appearance of hand gestures. For extracting these features, the image is first divided into a dense grid of equal cells, whose size is based on the application at hand. If the application requires features of minute details present in the image, the small sized cells will do the task, otherwise, the size of the cells is increased. To get the shape information of the hand present in a binary image, a cell size of is used, which will encode the shape information better than the other sizes as tested experimentally and is shown in Fig. 7. Each pixel in a cell is compared with its eight neighbouring pixels counter-clockwise or anti-counter-clockwise. Based on the comparison, the eight neighbouring pixels are allotted the binary numbers (0 or 1) forming an eight digit binary number that can be converted to a decimal number. A histogram is formed for each cell individually indicating the frequency of 0s and 1s. These histograms are then combined with each other to form a single histogram for the whole grid of cells. Fig. 7Open in figure viewerPowerPoint Local binary patterns 4.4 Combination of feature vectors All the feature vectors discussed above are given individually and also in combination with each other as an input to the classification stage so as to be able to select the best combination of features out of all in terms of recognition accuracy. The combination of feature vectors is made by simply concatenating them, for example, a combination of all the feature vectors is given as – [Geometrical, LBP, Distance, Num]. Inputting these features directly to the classifiers will result in insignificant feature weights of different scales. Thus, the standardisation of these feature vectors is done before giving as an input for classification. The feature vectors are standardised in such a way to maintain unit standard deviation and zero mean. 5 Gesture classification In this paper, multi-class SVM classifiers are used to classify the gestures. Significant work is reported on gesture recognition using SVM as a classifier. Kumar et al. [13] optimised an RBF SVM kernel and used it in conjunction with HMMs for classification of isolated Indian Signs achieving a recognition accuracy of 83.77%. A multi-class -SVM, where C is the regularisation parameter, is used for classification of ten static gestures of American sign language (ASL) in a work done by Zhi et al. [14] and after comparison with adaptive naive Bayes classifier and k-nearest neighbour (k-NN) classifier, -SVM is found to be superior to others with a recognition accuracy of 98.25%. 5.1 SVM classifier In an SVM classifier, a hyperplane is searched, that can separate all the data points of one class from another. The width of the slab, which is called Class Margin and is parallel to the hyperplane with no interior data points, should be maximum to get the best hyperplane. Thus, SVM finds the optimal separating hyperplane by maximising the class margin or minimising where w is normal to the hyperplane and is the Euclidean norm of w. If are the n input vectors present in the training dataset and , the corresponding target values, then for a linearly separable binary problem, the objective function to be minimised is given in equation (7) which is also called as Primal Problem [22]. (7)where b is bias and C is the regularisation parameter and is the coefficient of slack variables representing the distance by which the training examples violate the margin. The Lagrangian formulation of the above Primal Problem can be represented by the following equation: (8)where represents Lagrange multiplier. C is, sometimes, called a Box Constraint as it maintains the permissible values of Lagrange multipliers in a bounded region or a ‘box’ as shown in the inequality given in (8). If the problem to be solved is non-linear in nature, the dot product in (8) is replaced by (9)for generalisation for non-linear kernels which is known as ‘Kernel-Trick’. Here, is the kernel function, is the feature map and is the feature vector such that with target space be the feature space [22]. This ‘Kernel-Trick’ shows that rather than mapping each data vector into higher dimensional space, computing dot product in (9) is sufficient to maximise the margin. For classification of gestures, four SVM kernels have been used in this work, namely, Linear, Quadratic, Cubic and Gaussian as defined in (10)–(12): Linear Kernel: (10)Polynomial Kernel: (11)where M is the order of the polynomial which is 2 for Quadratic kernel and 3 for Cubic kernel. Gaussian or RBF Kernel: (12)where, and is the standard deviation that determines the width of a Gaussian distribution. 5.2 Bayesian optimisation Several internal parameters of a classifier function, that can strongly affect the performance of a cla