Static gesture segmentation technique based on improved Sobel operator

The Journal of EngineeringVolume 2019, Issue 22 p. 8339-8342 Jiangsu Annual Conference on Automation (JACA 2019)Open Access Static gesture segmentation technique based on improved Sobel operator Xinzhi Wang, Xinzhi Wang School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorYifan Fang, Yifan Fang School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorChangdi Li, Changdi Li School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorShenjian Gong, Shenjian Gong School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorLei Yu, Corresponding Author Lei Yu yu_lei@suda.edu.cn School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of China Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China State Key Lab of CAD&CG, Hangzhou, Zhejiang University, People's Republic of ChinaSearch for more papers by this authorShumin Fei, Shumin Fei School of Automation, Southeast University, Nanjing, People's Republic of ChinaSearch for more papers by this author Xinzhi Wang, Xinzhi Wang School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorYifan Fang, Yifan Fang School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorChangdi Li, Changdi Li School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorShenjian Gong, Shenjian Gong School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of ChinaSearch for more papers by this authorLei Yu, Corresponding Author Lei Yu yu_lei@suda.edu.cn School of Mechanical and Electric Engineering, Soochow University, Suzhou, People's Republic of China Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China State Key Lab of CAD&CG, Hangzhou, Zhejiang University, People's Republic of ChinaSearch for more papers by this authorShumin Fei, Shumin Fei School of Automation, Southeast University, Nanjing, People's Republic of ChinaSearch for more papers by this author First published: 25 November 2019 https://doi.org/10.1049/joe.2019.1075Citations: 7AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The gesture recognition technology based on computer vision can provide a more natural way of human–computer interaction, and become a research hotspot in the field of gesture recognition. This study proposes an improved Sobel operator which is used for static hand contour extraction. The method is based on the noise existing in the picture, and a bilateral filtering algorithm is employed to denoise and preserve the edges to a large extent. Based on the colour space, the elliptical skin colour model is used to extract the skin colour domain, and the hand is separated according to the block domain feature. Finally, in order to solve the oblique indirection problem of the traditional Sobel operator, two oblique direction templates are added and the weights are reassigned. Then the improved Sobel operator is used to extract the gesture edges, and compared with the classical extraction edge operator Laplace and Canny extraction effects. The experimental results show that the proposed algorithm has high efficiency and accuracy in static gesture segmentation in a simple environment. 1 Introduction Vision-based gesture tracking and recognition technology [1, 2] is an important way in modern human–computer interaction. Users can use simple gestures to control or interact with the device, allowing the computer to understand human behaviour. Its core technologies are gesture segmentation, gesture analysis, and gesture recognition. As tracking and identification improve efficiency and accuracy, the technology brings more industrial value and economic value. At present, there are still many research hotspots and difficulties in gesture recognition and segmentation. The main factors affecting vision-based gesture tracking and recognition are: gesture distortion, illumination changes, skin colour changes, tracking accuracy, recognition accuracy etc. Static hand image edge detection technology is a hot research issue in recent years. In the research of gesture segmentation, many researchers have proposed a variety of methods in recent years. Including thermal infrared capture method [3], which is simple but the external conditions are limited; template matching method [4] which is difficult to compare with the database, and the amount of calculation is large; auxiliary colour marking method [4] which is simple but the external conditions are limited. There are also geometric feature-based segmentation methods [5]. Due to the variety of gestures, the complexity of the algorithm is still high. After extracting the hand image, the edge is extracted using the classic image edge extraction operator, the most commonly used Roberts operator, Laplacian of Gaussian (LoG) operator, Prewitt operator, Krisch operator etc. [6]. These algorithms are relatively simple, but there are problems such as insufficient edge positioning accuracy and false edges. The improved Sobel algorithm extracts edges based on the sum of grey-scale weighted differences, and can extract more accurate edge direction information and has certain noise reduction capability. In view of the inaccurate detection of existing gestures and the problem of gesture segmentation deviation, this paper adopts the method of gesture detection and segmentation based on elliptical skin colour model [7]. The proposed method firstly projects the skin colour in the YCbCr colour space into the CbCr space, and detects the skin colour region by using the skin colour projection to approximate the elliptical property of the space. Moreover, the block processing of the region separates the hand according to the feature [5], and then chooses the improved Sobel calculation. 2 Sobel operator The recognition of the target object first obtains its outline, and the line-type features in the image are important clues for visual perception. Line features are generally represented by the outline, texture, or edge line of the target object, while edge detection is the key information for acquiring the target object or image outline. The classical Sobel operator is a commonly used gradient amplitude detection operator in edge detection. The operator first uses a 3 × 3 convolution template to perform weighted averaging or neighbourhood averaging on the detected image, and then detects the edge of the image by performing the first-order differential calculation. The famous Sobel edge operator was proposed by Irwin Sobel and is widely used in image edge detection. Set as an image function , the gradient at the point is a vector [8, 9], defined as (1) where is used to represent the gradient, the value of which can be calculated as follows: (2) Traditional Sobel operators contain convolution templates in both horizontal and vertical directions, as shown in Fig. 1. Fig. 1 a is a horizontal gradient direction template, which is mainly used to detect the vertical direction edge of the image; Fig. 1 b is a vertical gradient direction template, which is mainly employed to detect the horizontal edge of the image. Fig. 1Open in figure viewerPowerPoint Traditional Sobel operator template (a) Horizontal direction, (b) Vertical direction 2.1 Principle of the traditional Canny algorithm The basic idea of the Sobel algorithm is that the edge of the image is located where the brightness of the image changes relatively significantly, so that the pixel point whose pixel grey value in the neighbourhood exceeds the set threshold can be regarded as the edge point. The specific steps of the Sobel algorithm are as follows: (i) The two direction templates are traversed from left to right and top to bottom, and the center point of the template corresponds to the corresponding pixel in the image. (ii) Discrete the weight of each position in the template and its corresponding image pixel value. (iii) Replace the image pixel value corresponding to the template center point with the maximum value of the two template convolutions, denoted as p max. (iv) Take an appropriate threshold T, and if p max ≥ T, determine that the pixel is the edge of the image. The advantage of the traditional Sobel operator is that the principle is simple, the calculation is small, and the running speed is fast. However, since the algorithm uses only two horizontal and vertical templates, it is insensitive to edges in other directions. Therefore, for images with more complex textures and more oblique edges, the extracted image contours are not ideal, and the anti-noise ability is also low. 2.2 Improved filtering and Sobel algorithm The traditional Sobel algorithm only uses two horizontal and vertical templates, which are insensitive to edges in other directions. For contours with more complex textures and more oblique edges, the contour extraction is not ideal, and the anti-noise ability is also low. The use of this has brought limitations. Conventional median filtering, mean filtering and Gaussian filtering etc., all image edges are smoothed while smoothing the image, resulting in a reduction in image edge gradient. Therefore, it is better to use the edge-preserving filtering method to process images. This study uses bilateral filtering. (i) Increase of direction template Fig. 2Open in figure viewerPowerPoint Improved four direction template The image mainly includes four edge directions, namely horizontal, vertical, 45° and 135° directions. The outline of the actual image is complex and mostly dominated by oblique edges. Traditional Sobel operators cannot accurately extract image edges. In this paper, the Sobel operator is improved [10, 11], and two templates in the 45° and 135° directions are added, and the weights of the original two templates are re-adjusted, and the weight of the diagonal direction of the new template is increased. The improved algorithm direction template is shown in Fig. 2. (ii) Improvement of filtering algorithm Even with static gesture recognition, there is still a noise problem. At this time, if Gaussian filtering is used, median filtering can reduce image noise to a certain extent, but at the same time it will weaken the edge of the image, which has a great influence on the edge of the extracted image, which may cause misrecognition. To this end, this paper proposes to first use the bilateral filtering to reduce the image processing, and then the subsequent processing of the image. The basic idea of bilateral filtering is to simultaneously consider the spatial domain information and the range information of the pixel to be filtered. It is a joint filtering method that considers both aspects of information at the same time. First, according to the pixel value, the neighbourhood to be used for filtering is divided or classified, and then the weight of the category to which the point belongs is relatively high, and then the neighbourhood weighted sum is obtained to obtain the final result. It is assumed that the gradation value of the image I at the coordinate point is , the gradation value of the image obtained by the filtering binary image (BI) at the coordinate p is , and the bilateral filtering formula are as follows: (3) (4) In the above formula, is the neighbourhood pixels of the central pixel p, the set of neighbourhood pixels is S, is a normalisation factor, is a spatial proximity factor, is a grey similarity factor. Then, the expression is represented by the following formulas: (5) (6) where is the distance standard deviation based on the Gaussian function, is the grey standard deviation based on the Gaussian function. The two radial ranges that control the spatial domain filtering kernel function and the grey-scale filtering kernel function are all selectable parameters, which directly determine the performance of the bilateral filters. The range of degrees is adjusted to adptive the weight of the pixel. 3 Static gesture segmentation 3.1 Elliptical skin model Extracting the hand should first model the skin tone to identify the skin colour area. Commonly used detection methods include the threshold method and the Gaussian model method. The threshold method is simple to operate, but the detection results are not ideal. The Gaussian model has high precision, but the modelling is difficult and the application is complicated. Therefore, this paper considers the elliptical skin colour model [12-14] based on YCrCb colour space to obtain efficient and simple detection results. YcrCb is composed of Y (luminance component), Cr (blue chrominance component), Cb (red chrominance component), and each component is linearly converted from the RGB colour space model. The conversion formula is shown as (7) The 137 image statistics are selected in the Heinrich-Hertz-Institute (HHI) image library, and the ellipse model is employed to describe the skin colour distribution in the transformed region. The elliptic formula group is expressed as follows: (8) In the formula, the ellipse center , , the long axis a = 25.39, the short axis b = 14.03, and the counterclockwise rotation with respect to the x -axis , , . 3.2 Block domain segmentation algorithm Threshold segmentation is a region-based image segmentation technique. The principle is to divide image pixel points into several categories. It is especially useful for images where the target and background occupy different grey scale ranges. After acquiring the skin colour image, this paper uses the threshold segmentation method to separate the hand. The skin colour detection is divided into three major fields: the left hand, the right hand, and the face. Based on the area and perimeter of the block domain, the screening threshold is selected, and the dynamic upper and lower area thresholds are taken between 1/k 1 and 1/k2 times of the largest area, and the upper and lower perimeter thresholds are taken between 1/k 3 and 1/k 2 times of the maximum circumference. The hand separation block is then acquired by traversing the image, and finally the image contour is extracted using the improved Sobel algorithm. 4 Sobel algorithm experiments Opencv3 is selected as the core of image processing. Python 3.6 is used as the processing language. Firstly, the elliptical skin colour model detection algorithm based on YCrCb space is used to screen out the skin colour domain in the image, and then the hand segmentation skin colour domain is obtained through the hand features. Finally, the edge is extracted based on the improved Sobel operator's skin colour domain, and compared with the traditional Sobel operator extraction edge, the experimental results show that the improved Sobel operator extracts the gesture edge more accurately. The experimental results are shown in Figs. 3-8. Fig. 3 shows the original image and the Gaussian noise image. The subsequent processing is based on the latter. Fig. 4 shows the Gaussian filtered image and the bilateral filtered processed image. The subsequent processing is based on the latter. Fig. 5 is the extraction of the skin colour domain. Fig. 6 is the hand segmentation map and its mask, and Fig. 7 is the hand edge detection map of the improved Sobel operator and the traditional Sobel operator. Fig. 8 is a diagram showing the effect of the classic image edge extraction operator Laplace [15] and Cannyextracting the hand edge [16, 17]. Fig. 3Open in figure viewerPowerPoint Original image and added noise map Fig. 4Open in figure viewerPowerPoint Gaussian filter and bilateral filter Fig. 5Open in figure viewerPowerPoint Elliptical skin colour detection and mask Fig. 6Open in figure viewerPowerPoint Gesture segmentation and mask Fig. 7Open in figure viewerPowerPoint Traditional Sobel operators and improved operator extracting edges Fig. 8Open in figure viewerPowerPoint Laplace and Canny operator extracting edges As shown in Fig. 4, the Gaussian filter is weighted averaged in a certain neighbourhood, and only the spatial weight is considered, and the image edge is blurred. The bilateral filtering considers both the spatial and the value domain information, and can maintain a clear outline to a greater extent, and the edge detection accuracy is higher. In Fig. 5, the elliptical skin colour model based on the big data training can extract the skin colour domain more accurately, and the black eyeballs are all eliminated. This image was not removed because the hair was close to the skin. This error is acceptable and can be processed later. Fig. 6 is a result of gesture segmentation. By adaptive block domain segmentation, a hand contour with a connected domain area and a connected length in an appropriate range is extracted. The final result of the experiment is shown in Fig. 7. The algorithm of the improved filtering and direction template obtained in this paper has clear hand edges and can reflect most gesture information. However, the traditional Sobel operator template is less difficult to fine tune, which makes it difficult to remove small edges and large veins inside the palm [18]. Fig. 8 is a diagram of the edge extraction effect of the classical extraction edge operator Laplace and Canny extraction. The extraction effect is not satisfactory, and the extraction edge is not smooth enough and the small edge details are difficult to remove. From the comparison effect, the improved Sobel operator proposed in this paper overcomes the defects of detecting edge roughness and discontinuity, and is a suitable edge extraction algorithm for static gesture segmentation. Experiments show that bilateral filtering can preserve the edge of the image while denoising, and the effect is better than Gaussian filtering for gesture extraction. In this paper, the traditional Sobel operator template is improved, the 45° and 135° direction templates are added, and the weights of the four direction templates are redistributed, and the high contrast edges are filtered to the maximum extent. The gradient image is refined to distinguish the inside of the palm. The small edge and the large vein overcome the defects of the traditional Sobel operator algorithm for detecting edge roughness and discontinuity, and achieve better edge extraction. The bilateral filter and improved Sobel operators have higher accuracy in the extraction of static gesture contours, while at the same time it can be able to distinguish the contours with obvious changes. 5 Conclusion In order to solve the oblique indirection problem of the traditional Sobel algorithm, an improved Sobel algorithm is proposed and applied in static gesture segmentation. The method firstly filters the static image bilaterally, denoising and retains the edge to the greatest extent; then uses the elliptical skin colour model to filter out the skin colour domain, and separates the hand according to the skin colour domain feature; finally, the Sobel algorithm 45° and 135° two direction templates are added. Experiments show that it has higher accuracy in simple static gesture segmentation and has a good application prospect. 6 Acknowledgments The work is supported by the National Natural Science Foundation of China (Grant no. 61873176); Natural Science Foundation of Jiangsu Province, China (Grant no. BK20181433); The Open Laboratory Foundation of Key Laboratory of Industrial Internet of Things & Networked Control, Ministry of Education(Grant no. 2018FF05); The Open Project Program of the State Key Lab of CAD & CG, Zhejiang University(Grant no. A1918); Postgraduate Research & Practice Innovation Program of Jiangsu Province(Grant no. SJCX18_0844). The authors would like to thank the referees for their constructive comments. 7 References 1Belgacem S., Chatelain C., Paquet T.: 'Gesture sequence recognition with one shot learned CRF/HMM hybrid model', Image Vis. Comput., 2017, 61, pp. 12 – 21 2Cui Y.T., Weng J.J.: 'View-based hand segmentation and hand-sequence recognition with complex backgrounds'. Proc. of the IEEE Int. Conf. on Pattern Recognition, Vienna, Austria,, 1996, pp. 617 – 621 3Oka K., Sato Y., Koike H.: 'Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems'. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, Washington DC, USA, 2002, pp. 429 – 434 4Attila L., Tamas S.: 'User adaptive hand gesture recognition system with interactive training', Image Vis. Comput., 2005, 10, (2), pp. 1102 – 1114 5Padam Priyal B.: 'A robust static hand gesture recognition system using geometry based normalizations and Krawtchouk moments', Pattern Recognit., 2013, 46, (8), pp. 2202 – 2219 6Lee T., Park Energy efficient Canny edge detector for advanced mobile vision applications', IEEE Trans. Circuits Syst. Video Technol., 2018, 4, (28), pp. 1037 – 1046 7Zhanwu X.U., Miaoliang Z.H.U.: 'Color-based skin detection: survey and evaluation'. Proc. of the 12th Int. Multimedia Modeling Conf. (MMM2006), Beijing, China,, 2006, pp. 143 – 152 8Health A., Sarkar S., Teta1 S.: 'Comparison of edge detectors: a methodology and initial study', Comput. Vis. Image Underst., 1998, 69, (1), pp. 38 – 54 9Vincent O.R., Folorunso O.: 'A descriptive algorithm for Sobel image edge detection'. Proc. of Informing Science & IT Education Conf., Macon,, 2009, pp. 97 – 107 10Jin-Yu Z., Chen Y., Huang X.X.: 'Edge detection of images based on improved Sobel operator and genetic algorithms'. Int. Conf. on Image Analysis and Signal Processing, Linhai, 2009 11Gao W.S., Zhang X.G., Yang L. et al.: 'An improved Sobel edge detection'. IEEE Int. Conf. on Computer Science and Information Technology (ICCSIT'10), Chengdu, China, 2010 12Chaid N.: 'Face segmentation using skin-color map in videophone applications', IEEE Trans. Circuits Syst. Video Technol., 1999, 9, (4), pp. 551 – 564 13Duan-sheng C., Zheng-kai L.: 'A survey of skin color detection', Chin. J. Comput., 2006, 29, (2), pp. 194 – 207 14Kai L., Xiao-jian L., Jin-xiang Z.: 'Hand signal recognition based on skin color and edge outline examination', J. North China Univ. Technol., 2006, 18, (3), pp. 12 – 15 15van Vliet L.J., Young I.T., Beckers G.L.: 'A nonlinear Laplace operator as edge detector in noisy images', Comput. Vis., Graph.Image Process., 1989, 45, (2), pp. 167 – 195 16Song R., Zhang Z., Liu H.: 'Edge connection based Canny edge detection algorithm', Pattern Recognit. Image Anal., 2017, 27, (4), pp. 740 – 747 17Lee J., Tang H., Park J.: 'Energy efficient Canny edge detector for advanced mobile vision applications', IEEE Trans. Circuits Syst. Video Technol., 2018, 28, (4), pp. 1037 – 1046 18Lee H.K., Kim J.H.: 'An HMM-based threshold model approach for gesture recognition', IEEE Trans. Pattern Anal. Mach. Intell., 1999, 21, (10), pp. 961 – 973 Citing Literature Volume2019, Issue22November 2019Pages 8339-8342 FiguresReferencesRelatedInformation