Towards pen‐holding hand pose recognition: A new benchmark and a coarse‐to‐fine PHHP recognition network

IET BiometricsVolume 11, Issue 6 p. 581-587 ORIGINAL RESEARCHOpen Access Towards pen-holding hand pose recognition: A new benchmark and a coarse-to-fine PHHP recognition network Pingping Wu, Pingping Wu orcid.org/0000-0002-8234-8566 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorLunke Fei, Corresponding Author Lunke Fei [email protected] School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Correspondence Lunke Fei, School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China. Email: [email protected]Search for more papers by this authorShuyi Li, Shuyi Li Department of Computer and Information Science, University of Macau, Taipa, Macau, ChinaSearch for more papers by this authorShuping Zhao, Shuping Zhao School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorXiaozhao Fang, Xiaozhao Fang School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorShaohua Teng, Shaohua Teng School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this author Pingping Wu, Pingping Wu orcid.org/0000-0002-8234-8566 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorLunke Fei, Corresponding Author Lunke Fei [email protected] School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Correspondence Lunke Fei, School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China. Email: [email protected]Search for more papers by this authorShuyi Li, Shuyi Li Department of Computer and Information Science, University of Macau, Taipa, Macau, ChinaSearch for more papers by this authorShuping Zhao, Shuping Zhao School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorXiaozhao Fang, Xiaozhao Fang School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorShaohua Teng, Shaohua Teng School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this author First published: 17 May 2022 https://doi.org/10.1049/bme2.12079Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Hand pose recognition has been one of the most fundamental tasks in computer vision and pattern recognition, and substantial effort has been devoted to this field. However, owing to lack of public large-scale benchmark dataset, there is little literature to specially study pen-holding hand pose (PHHP) recognition. As an attempt to fill this gap, in this paper, a PHHP image dataset, consisting of 18,000 PHHP samples is established. To the best of the authors' knowledge, this is the largest vision-based PHHP dataset ever collected. Furthermore, the authors design a coarse-to-fine PHHP recognition network consisting of a coarse multi-feature learning network and a fine pen-grasping-specific feature learning network, where the coarse learning network aims to extensively exploit the multiple discriminative features by sharing a hand-shape-based spatial attention information, and the fine learning network further learns the pen-grasping-specific features by embedding a couple of convolutional block attention modules into three convolution blocks models. Experimental results show that the authors' proposed method can achieve a very competitive PHHP recognition performance when compared with the baseline recognition models. 1 INTRODUCTION Hand pose recognition, which aims to understand the activities of human hands, is a longstanding and interesting problem with a variety of practical applications such as virtual reality and robotics. A number of hand pose recognition methods have been proposed over the past decades, including the monocular RGB [1, 2], video-based, depth-based and 3D recognition methods [3-5]. While these hand pose recognition methods have achieved promising performance on the special tasks, 2D hand pose recognition remains an essential component and still plays an important role in a sub-module of the 3D recognition. As a sub-branch of 2D hand pose recognition, pen-holding hand pose (PHHP) recognition has attracted increasing research attention in the communities of hand pose recognition in recent years because it has many meaningful practical applications, such as pen-holding hand posture correction [6, 9] and user-dependent personal authentication [7, 8]. For example, in the early years, Cheng et al. [7] proposed a video-based PHHP recognition method to identify the writer. Yuuki et al. [8, 9] proposed a biometric authentication method by using pen holding style images. In addition, Bi et al. [10] designed a CNN-based method by smartwatch signals for PHHP recognition. Matulic et al. [11] also proposed a CNN-based method based on Electromyogram data to detect PHHP. While these PHHP recognition methods have achieved encouraging performance in controlled environments, they mainly use the electrical or movement signals captured by using special engineer devices, making them less effective for open environments. Hence, how to perform PHHP recognition based on 2D images, which can be easily captured by using normal imaging devices, is an interesting and challenging problem. In this paper, to fill the gap of lacking vision-based method and the related benchmark image dataset, we first establish a large-scale PHHP image dataset consisting of 18,000 PHHP samples, which to the best of our knowledge is the largest in scale. Moreover, we design a coarse-to-fine network for PHHP recognition to solve two main problems. Specifically, we first use a sharing attention-based coarse feature learning network to jointly learn the multiple features and then engineer three convolution blocks to further extract pen-grasping-specific features for PHHP recognition. Experimental results on the newly established dataset clearly show that our proposed method can achieve the best PHHP recognition accuracy in comparison with the base recognition models. The main contribution of this paper can be summarised as follows: We design a coarse-to-fine network to first learn multiple features of PHHP images by sharing an attention module and then further learn the pen-grasping-specific features by embedding convolutional block attention modules (CBAM) into three convolution blocks so that more discriminative pen-grasping-specific information can be exploited in the learnt features. We establish a new PHHP image dataset that contains 18,000 samples with multiple capture views and also more complex backgrounds. To the best of our knowledge, the PHHP image dataset is the largest 2D pen-grasping image dataset ever collected. We conduct several comparative experiments to evaluate our proposed method on the newly established dataset, and the experimental results show that our proposed method outperforms the state-of-the-art PHHP recognition methods. The rest of this paper is organised as follows. Section 2 reviews two related studies. Section 3 introduces the newly established PHHP benchmark dataset. Section 4 elaborates our proposed PHHP recognition method. Section 5 presents the experimental results, and finally section 6 concludes this paper. 2 RELATED WORK In this section, we briefly review two related studies including the hand segmentation and the methods of specific pen-grasping-feature learning. Hand segmentation Hand segmentation is the process to solve the stick hand pose recognition problem; as we can see in Figure 1c, PHHP image background is also with illumination complexity and plenty of other things such as books, papers, strings, laptop, pattern on the table, and so on. It is difficult to learn the hand information directly from the deep learning network with the changing illumination in different views and complex backgrounds. To address this, there are several methods for the background filtering recently. For example, Zhou et al. [12] utilised skin colour segmentation method and canny edge detection algorithm for hand contour image extraction. And Zhang et al. [13] propose a hand segmentation based on saliency and skin colour detection consisting of three hand detection methods. However, although skin colour segmentation and edge detection algorithm have achieved significant effect in some cases, it does not work in PHHP recognition since the most postures are on the wood table whose colour is similar to the human skin colour. In recent years, the attention mechanism has been successfully applied in target detection tasks such as hand gesture recognition [14-16]. Inspired by this, we proposed the sharing attention mechanism for the PHHP network to double force the hand regions. Specifically, after first hand segmentation, we not only simply combine the segmentation map with the feature map but also put adaptive feature mechanism to re-learn the hand regions and refine the coarse multi-feature jointly learning network. FIGURE 1Open in figure viewerPowerPoint (a) Nine different hand postures of pen-holding hand pose (PHHP) images, such as standard grip, close grip, fold grip, tuck grip, squeeze grip, hook grip, wrap grip, mount grip, and tripod grip. (b) Two different views for capturing PHHP images. (c) Some typical PHHP samples captured under complex backgrounds Subtle and specific pen-grasping-feature learning Different from the common hand pose recognition, PHHP recognition has nine different but similar hand postures for recognition. As shown in Figure 1a, there are slight differences between the standard grip contour and the other gesture contours since the distinction of pen-holding hand poses is the mix and match of three fingers and a pencil. In other words, the standard grip and the other grips are shown as the same as three fingers grabbing the pen point and the pen staying in the V-shaped webbed area between the thumb and index. So, they are small and subtle features to be distinguished from an image. In general, due to the development of deep learning network, there are some normal methods to tackle this problem in recent years such as Full Connected Network [18], Faster Regions with CNN features (RCNN) [20], Mask RCNN [21], you only look once (YOLO) [22], U-net [23] and so on. While these deep learning networks have excellent effect in target detection tasks, it is worthless and less effective in PHHP since the deeper convolutional network would be overfitting. In addition, Ju et al. [24] proposed a method to mix up different feature maps generated through the whole step of a deep 32-layer convolutional network and three down-sampling processes to learn tiny features effectively. By contrast, we design a fine pen-specific feature learning network with only six layers but can extract and learn small features more efficiently in a limited time, which means that it can be embedded into electronic devices used commonly such as smart phones. 3 A NEW BENCHMARK PEN-HOLDING HAND POSE IMAGE DATASET There have been extensive methods proposed for hand pose recognition in recent years [1-4]. However, there are still limited studies that focus on PHHP recognition, mainly due to few samples, low image quality and single background of the existing hand pose image datasets. To address this, we have an aim to establish a new large-scale PHHP image dataset. Specifically, our dataset was collected from 100 volunteers, including 82 males and 18 females, with the ages from 16 to 75 years old. Each volunteer was asked to provide nine-view pen-holding hand poses as shown in Figure 1a, and each pose were captured for 20 images. Moreover, to better show the PHHP, we captured 10 hand images under the overshot view and another 10 images under the side shot view, as shown in Figure 1b. All samples were captured by using an ordinary 2D high-definition camera with 1080 × 1080 $1080\times 1080$ pixels. Therefore, our PHHP dataset consists of 18,000 samples of 100 subjects with nine different pen-holding hand poses. Figure 1c shows some typical samples selected from the newly established PHHP dataset, and more than 90% of PHHP samples were captured under complex backgrounds. Table 1 tabulates the basic information of our newly established PHHP dataset comparing with the other three hand pose image datasets. Specifically, Pen-holding Style (PHS) [8] is a private database, which consists of only 300 images captured in a single scene, including the same metallic grey background, the same capture view and the same black pen. In addition, these images are mainly used for biometric feature extraction of personal authentication. Silhoutte-based pen holding style (SBPHS) [17] database is another private pen-holding style database, which contains 4305 images captured by the single side hands under same background. In addition, OneHand 10K [19] database consists of 11,703 hand pose images captured under different scenes and multiple views, which are mainly used for hand pose estimation but not PHHP recognition task. TABLE 1. Our newly established pen-holding hand pose (PHHP) dataset and the other three representative hand pose image datasets Dataset Year Samples Labels Subjects Scenes View PHS [8] 2010 300 30 30 1 1 SBPHS [17] 2013 4305 22 22 1 1 OneHand [18] 2019 11,703 - - Multiple Multi-angle PHHP dataset 2022 18,000 9 100 Multiple Multi-angle Abbreviations: PHS, Pen-holding Style; SBPHS; Silhoutte-based pen holding style. Different from these databases, our newly established dataset consists of 18,000 PHHP images collected from 100 volunteers, and each of them was asked to provide more than 10 samples with nine different pen-holding hand gestures under two different views. To the best of our knowledge, it is the largest PHHP image database ever collected, and thus it can serve as a benchmark for video-based PHHP recognition and other related hand pose approaches. 4 THE PROPOSED COARSE-TO-FINE PHHP RECOGNITION NETWORK In this section, we first elaborate the proposed coarse-to-fine PHHP recognition network. Then, we detail the architectures of coarse multi-feature jointly learning network and the fine pen-grasping-specific learning network. Overview of the coarse-to-fine PHHP recognition network Figure 2 shows the overall framework of our proposed coarse-to-fine PHHP recognition network, which mainly consists of a coarse multi-feature jointly learning network and a fine pen-grasping-specific learning network. Specifically, we design a two-stage learning network, which first exploits the coarse multi-features from PHHP images and then further learns fine pen-grasping-specific features. To filter the complex background of PHHP images, we use a simple DNN embedded with a spatial attention module as the backbone of the coarse feature learning network, which can simultaneously focus on exploring hand pose-related information and meanwhile keep the model small. Moreover, we engineer a convolutional module (consists of two convolutional layers and one max pooling layer) with an attention module to further learn the fine pen-grasping-specific features, in which the convolutional module aims to learn the PHHP-specific features, and the attention module is to highlight the minor differences of pen-holding hand poses among different feature maps. By doing these, more fine hand-pen features can be learnt from PHHP images. In the following, we detail the network architectures and implementations of the two sub-networks. FIGURE 2Open in figure viewerPowerPoint An overview of the proposed pen-holding hand pose (PHHP) recognition network, which mainly consists of a coarse multi-feature jointly learning network and a fine pen-grasping-specific learning network Coarse multi-feature jointly learning network The coarse multi-feature jointly learning network aims to extensively exploit the discriminative features from PHHP images. It is recognised that a PHHP image in practical application usually contains complex background, where the hand-pen position as well as hand region are the crucial information for recognition. Due to this, our coarse multi-feature learning network first extracts the important gradient information (3 channels) of PHHP images to obtain gradient maps by segmentation and edge detection algorithm. Then, we simultaneously learn the discriminative features from both the original and gradient map based on two DNN networks. Particularly, we employ a spatial attention mechanism to further learn the spatial-specific information (only one channel) from the gradient-based feature maps (8 channels) and share it with double DNN feature maps to obtain a combined multiple feature representation (32 channels). Let tensors G ∈ R c × w × h $G\in {R}^{c\times w\times h}$ be the DNN-based feature maps, where w $w$ and h $h$ represents the width and height of the input images and c $c$ denotes the channel number. In this paper, we empirically set c $c$ to 8 for edge feature map and 24 for original image feature map, because the original image usually contains more information than the edge feature maps. Specifically, G 1 ∈ R 8 × 224 × 224 ${G}_{1}\in {R}^{8\times 224\times 224}$ refers to the gradient feature maps, and G 2 ∈ R 24 × 224 × 224 ${G}_{2}\in {R}^{24\times 224\times 224}$ represents the original feature maps. Then, we specially learn the adaptive spatial attention information by performing global maximum pooling and average pooling on G 1 ${G}_{1}$ by calculating it with two 3 × 3 $3\times 3$ convolution blocks and further use a sigmoid function to update the attention map. Finally, we share the spatial attention map with two DNN feature maps and concatenate them to obtain the combined multi-feature representation as follows. Multi_feature_map = Spatial_attention ⊗ G 1 , G 2 , $\text{Multi\_feature\_map}=\text{Spatial\_attention}\otimes \left\{{G}_{1},{G}_{2}\right\},$ (1)where M u l t i _ f e a t u r e _ m a p $Multi\_feature\_map$ is the combined multi-feature representation and ⊗ $\otimes $ denotes the dot product operation. Fine pen-grasping-specific feature learning network As shown in Figure 2, the fine pen-grasping-specific feature learning network consists of three convolution block models combining with three CBAM [25]. Specifically, to learn the more feature but keep the small size of model, each convolution blocks module comprises two layers of 3 × 3 $3\times 3$ convolution kernels to replace the receptive field of 5 × 5 $5\times 5$ convolution kernel and adopts a max pooling layer to reduce the size of maps and enlarge the number of channels. Moreover, we embed an attention module in each convolution block module, which arranges and adjusts the channel-wise and spatial-wise information of feature maps of each convolution layer to learn the pen-grasping-specific features. In general, a CBAM consists of a channel attention module [26] followed with a spatial attention module, which pools the multi-feature maps on the channel dimension to get a channel array, and further multiply the channel array with the original feature to obtain the fine-tuned feature map. Lastly, two full connection classification layers are used for PHHP feature recognition. 5 EXPERIMENT In this section, we evaluate the proposed coarse-to-fine PHHP recognition network on our newly established PHHP image dataset. Dataset settings Since our study focusses on the PHHP recognition under complex backgrounds, our newly established PHHP dataset is used to evaluate our proposed method. In this experiment, we first select the first half part of the images from each subject to form a smaller sample set (refers to half-PHHP dataset) to evaluate the effectiveness of the proposed method on small samples. Then, we use the whole dataset to conduct the experiments. For each dataset, we use 80% of the samples as the training set, 10% as validation and 10% as the test set. Since the left hand and right hand are symmetrical, all samples are mirror flipped so that the mirror images can be considered as left-hand PHHP images, as shown in Figure 3. In addition, each image is flipped horizontally and upside down to test the generalisation ability of our network, and the brightness of each image is randomly changed from 50% to 150% to characterise the illumination changes. Lastly, we cropped all images as well as the flipped samples into the sizes of 224 × 224 $224\times 224$ pixels. FIGURE 3Open in figure viewerPowerPoint Original image and the mirror image Recognition results To better evaluate the proposed method, we compare the proposed method with five baseline representative deep learning methods such as Visual Geometry Group (VGG16, VGG19) [27], ResNet34 [28], ResNet101, Mobile-NetV2 [29] and Mobile-NetV3 [30] in terms of recognition accuracy rate. For a fair comparison, all experiments are conducted under the same PyTorch framework and similar Adam optimiser with the cross entropy optimisation function. In addition, all methods were performed under the same platform with Giga Texel Shader eXtreme 3060 graphics card (including 3584 Compute Unified Device Architecture) cores, AMD 3600 CPU, 32 GB RAM and 500 GB hard disk drive. For the proposed method, we initialise the parameters with the Xavier and the learning rate and batch size of our method are empirically set to 1E-6 and 32, respectively. For our proposed method, we train the proposed model on the PHHP dataset due to the fact that our proposed method is specially designed for the PHHP recognition. For the other compared methods, we first pre-train them with ImageNet dataset and then fine-tune the parameters to achieve the best results of them for comparison. Table 2 tabulates the recognition accuracy rates of different methods on the half-PHHP and PHHP datasets. TABLE 2. The recognition accuracy rates of our proposed method and the representative feature learning methods Methods PHHP validation acc. PHHPTest acc. Half-PHHP validation acc. Half-PHHPTest acc. GFLOPs Model size VGG16 0.861 0.850 0.792 0.784 15.5 268M VGG19 0.942 0.930 0.784 0.764 19.6 288M ResNet34 0.957 0.941 0.921 0.919 3.8 83.2 M ResNet101 0.961 0.932 0.922 0.905 7.8 166M Mobile NetV2 0.936 0.921 0.924 0.917 0.31 8.8 M Mobile NetV3-large 0.946 0.937 0.930 0.923 0.23 16M Ours 0.960 0.953 0.956 0.946 0.64 13M Abbreviations: GFLOPs, giga floating-point operations per second; PHHP, pen-holding hand pose; VGG, Visual Geometry Group. From Table 2, we can see that our proposed method can achieve obvious better recognition accuracy rates than the other large-scale networks such as VGG and ResNet on both the half-PHHP and PHHP datasets. This is because our proposed network first uses a spatial attention mechanism to exploit the discriminative gradient information and then employs a couple of CBAM to specially learn pen-grasping-specific features, such that a better accuracy rate can be obtained. In particularl, the size of our proposed network is only 1/5 or even 1/20 times smaller than the VGG and ResNet. In addition, the proposed method also outperforms the two compared mobile networks, demonstrating the promising effectiveness and efficiency of the proposed network. Moreover, our proposed method has a low computational complexity such as a lower Giga Floating-point Operations Per Second (GFLOPs), such that the proposed method can be efficiently executed with limited RAM. Therefore, our proposed method can be easily embedded into mobile phones or even microcontrollers for efficient PHHP recognition. In addition, it is noted that the Mobile-Net obtains lower GFLOPs than ours. This is because our proposed method consists of many attention-based modules, which is more computation expensive. Considering the significant improvement of PHHP recognition, our proposed method makes a good trade-off when both the accuracy and model size are the main concerns. Ablation experimental results In this subsection, we evaluate the effectiveness of different components of the proposed method such as the edge feature map extraction, shared attention module and the CBAM. Table 3 tabulates the recognition results of our proposed method with embedding different components, where 'X' means the corresponding component is removed from the proposal network and 'O' represents the component is employed in the network. TABLE 3. The recognition accuracy as well as the loss of the proposed network with embedding different components Method (With or without) Edge information CBAM Shared attention Validation acc. Test acc. (1) Plain net X X X 0.897 0.888 (2) With edge O X X 0.930 0.915 (3) Without SA O O X 0.933 0.921 (4) Without CBAM O X O 0.951 0.942 (5) Coarse-to-fine network O O O 0.959 0.946 Abbreviations: CBAM, convolutional block attention modules; SA, Shared Attention. From Table 3, we can easily observe that the proposed method can achieve better recognition accuracy than that with removing different components. For example, both the edge information extraction and the spatial attention module can significantly improve the recognition accuracy of the proposed method over the plain net, which removed all components from our model. This is because these can effectively learn the rich discriminative edge information from PHHP images, such that a better accuracy can be obtained. In addition, the CBAM can also further improve the overall recognition performance of the proposed network. This is because the CBAM can specially focus on pen-grasping-specific feature learning and extraction. 6 CONCLUSION In this paper, we present an efficient end-to-end PHHP recognition network. We first design a coarse multi-feature learning network by hand segmentation to jointly learn multiple discriminative features from PHHP images and then engineer a fine pen-grasping-specific feature learning network by embedding a couple of adaptive methods into three convolution block models. Moreover, we newly established a large-scale PHHP image dataset to evaluate our proposed network, and the experimental results show that the proposed network achieves better PHHP recognition performance than the baseline networks. For future work, it seems to be an interesting direction to exploit multi-angle features of PHHP to further improve PHHP recognition and estimation performance. In addition, how to apply our established PHHP dataset to other 2D hand pose estimations such as 21 keypoints annotation seems to be another interesting future work. ACKNOWLEDGEMENT This work was supported in part by the National Natural Science Foundation of China under Grants 62176066, 62106052 and 61972102, in part by the Natural Science Foundation of Guangdong Province under Grant 2019A1515011811, and in part by the Guangzhou Science and technology plan project under Grant 202002030110. CONFLICT OF INTEREST The authors declare there are no conflicts of interest. Open Research DATA AVAILABILITY None. REFERENCES 1Shin, J., et al.: Deep learning-based hand pose estimation from 2d image. In: 2020 3rd IEEE International Conference on Knowledge Innovation and Invention (ICKII), pp. 108– 110. IEEE (2020) 2Wang, J., Liu, T., Wang, X.: Human hand gesture recognition with convolutional neural networks for k-12 double-teachers instruction mode classroom. Infrared Phys. Technol. 111103464 (2020). https://doi.org/10.1016/j.infrared.2020.103464 3Oberweger, M., Wohlhart, P., Lepetit, V.: Generalized feedback loop for joint hand-object pose estimation. IEEE Trans. Pattern Anal. Mach. Intell. 42(8), 1898– 1912 (2019). https://doi.org/10.1109/tpami.2019.2907951 4Liu, S., et al.: Semi-supervised 3d hand- object poses estimation with interactions in time. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14687– 14697. (2021) 5Huang, L., et al.: Hot-net: non- autoregressive transformer for 3d hand-object pose estimation. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 3136– 3145. (2020) 6Murata, A., Gotoh, K.: Effects of pen holding posture on handwriting motion. Adv. Soc. Organ. Factors. 12, 469 (2014) 7Cheng, H.-Y., et al.: Video-based signature verification and pen-grasping posture analysis for user-dependent identification authentication. IET Comput. Vis. 6(5), 388– 396 (2012). https://doi.org/10.1049/iet-cvi.2010.0136 8Hashimoto, Y., Muramatsu, D., Ogata, H.: Biometric person authentication method using features extracted from pen-holding style. In: Mobile Multimedia/Image Processing, Security, and Applications 2010, vol. 7708, pp. 149– 156. SPIE (2010) 9Schwellnus, H., et al.: Effect of pencil grasp on the speed and legibility of handwriting in children. Am. J. Occup. Ther. 66(6), 718– 726 (2012). https://doi.org/10.5014/ajot.2012.004515 10Bi, H., Zhang, J., Chen, Y.: Smartge: identifying pen-holding gesture with smartwatch. IEEE Access. 8, 28– 28830 (2020). https://doi.org/10.1109/access.2020.2967770 11Matulic, F., et al.: Eliciting pen-holding postures for general input with suitability for emg armband detection. In: Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces, pp. 89– 100 (2019) 12Zhou, Y., Jiang, G., Lin, Y.: A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recogn. 49, 102– 114 (2016). https://doi.org/10.1016/j.patcog.2015.07.014 13Zhang, Q., et al.: Segmentation of hand posture against complex backgrounds based on saliency and skin colour detection. IAENG Int. J. Comput. Sci. 45(3), 435– 444 (2018) 14Hou, J., et al.: Spatial-temporal attention res-tcn for skeleton-based dynamic hand gesture recognition. In: Proceedings of the European Conference on Computer Vision. ECCV Workshops (2018) 15Dhingra, N., Kunz, A.: Res3atn-deep 3d residual attention network for hand gesture recognition in videos. International Conference on 3D Vision (3DV), pp. 491– 501. IEEE (2019) 16Li, Y., et al.: Deep attention network for joint hand gesture localization and recognition using static rgb-d images. Inf Sci. 441, 66– 78. (2018) 17Muramatsu, D., Yagi, Y.: Silhouette-based online signature verification using pen tip trajectory and pen holding style. In: 2013 International Conference on Biometrics (ICB), pp. 1– 8. IEEE (2013) 18Wang, Y., Peng, C., Liu, Y.: Mask-pose cascaded cnn for 2d hand pose estimation from single color image. IEEE Trans. Circ. Syst. Video Technol. 29(11), 3258– 3268 (2018). https://doi.org/10.1109/tcsvt.2018.2879980 19Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431– 3440. (2015) 20Ren, S., et al.: Faster r-cnn: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28, 91– 99 (2015) 21He, K., et al.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961– 2969. (2017) 22Redmon, J., et al.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779– 788. (2016) 23Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234– 241. Springer (2015) 24Ju, M., et al.: A simple and efficient network for small target detection. IEEE Access. 7, 85771– 85781 (2019). https://doi.org/10.1109/access.2019.2924960 25Woo, S., et al.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision, pp. 3– 19. ECCV (2018) 26Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132– 7141. (2018) 27Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556 (2014) 28He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770– 778. (2016) 29Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications (2017). arXiv preprint arXiv:1704.04861 30Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314– 1324 (2019) Citing Literature Volume11, Issue6Special Issue: Recent Advances in Representation Learning for Robust Biometric Recognition SystemsNovember 2022Pages 581-587 FiguresReferencesRelatedInformation