Sewing gesture image detection method based on improved SSD model

Electronics LettersVolume 57, Issue 8 p. 321-323 Image and vision processing and display technologyOpen Access Sewing gesture image detection method based on improved SSD model Wenjie Wang, Wenjie Wang orcid.org/0000-0001-6426-0416 School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this authorMengling He, Mengling He School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this authorXiaohua Wang, Corresponding Author Xiaohua Wang wangxiaohua@xpu.edu.cn School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. China Email: wangxiaohua@xpu.edu.cnSearch for more papers by this authorWeiming Yao, Weiming Yao School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this author Wenjie Wang, Wenjie Wang orcid.org/0000-0001-6426-0416 School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this authorMengling He, Mengling He School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this authorXiaohua Wang, Corresponding Author Xiaohua Wang wangxiaohua@xpu.edu.cn School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. China Email: wangxiaohua@xpu.edu.cnSearch for more papers by this authorWeiming Yao, Weiming Yao School of Electronics and Information, Xi'an Polytechnic University, Xi'an, P. R. ChinaSearch for more papers by this author First published: 15 March 2021 https://doi.org/10.1049/ell2.12149Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this letter, the authors present a novel sewing gesture image detection method based on an improved single shot MultiBox detector (SSD) model. The deeper Resnet50 residual network replaces the VGG16 basic network of the original SSD model to improve the feature extraction ability. High and low level features are fused based on a feature pyramid network (FPN) for enhanced small-target detection performance. The model is trained via transfer learning to resolve the small sample shortage problem. The proposed model shows an average precision of 88.69% on a sewing gesture data set constructed by the authors. The proposed model outperforms the Faster R-CNN, YOLO, and SSD networks in terms of accuracy with acceptable operating speed on the same data set and fully satisfies the real-time requirements for sewing gesture detection. Introduction Intelligent equipment has gradually begun to replace human labourers throughout the garment industry as producers continue seeking new ways to minimize costs [1]. It is difficult to achieve desired garment-making effects with a single robot; man-machine collaborative work is more effective. The precise and rapid recognition and understanding of sewing gestures by a collaborative robot is exactly the key to realize the cooperative interaction between human and robot [2-4]. Existing target-detection methods have generally low recognition rates for small gestures in complex backgrounds, which is the case in the context of sewing. Fu et al. [5] proposed a deconvolutional single shot detector (DSSD) which uses a feature pyramid network (FPN), where the SSD backbone network VGG is replaced with ResNet; this effectively improves the shallow feature representation ability and ultimately improves the target detection accuracy. However, the relatively large depth of the model structure drives down its detection speed. Chen et al. [6] amplified and extracted small target feature regions from SSD networks, extracted semantic information from multiple deep feature layers, and improved the medium-scale feature representation of mesoscale objects. However, their method also has low detection speed. This paper proposes a method for sewing gesture recognition based on an improved single shot MultiBox detector (SSD) model. A network structure based on FPN is used to fuse the high bottom layer to further improve the detection accuracy. The proposed SSD deep network model consists of 59 convolution layers and five maximum pool layers. The main trunk of the model is a deep network based on Resnet50 superimposed with five convolution modules. The FPN, the main network structure, is divided into three parts: top-down, bottom-up, and horizontal connection. FPN fusion enhances the detection effects for small target gestures [7, 8]. The network structure of the proposed model is shown in Figure 1. Fig 1Open in figure viewerPowerPoint Improved SSD model network structure diagram To sum up, the major contributions of the proposed method are summarized as follows: We use a residual network to replace the original SSD backbone network, and adopt a feature pyramid to enhance the model's feature representation ability. The low recognition rate and poor detection effect for small target gestures, common issues in traditional algorithms, are resolved by the proposed method. The gradient instability problem is resolved without any increase in model complexity. The proposed method is fast enough for real-time detection. The rest of this paper is organized as follows. Section 2 describes the sewing gesture detection method in detail. Section 3 presents the experimental results we obtained to validate the proposed method. Section 4 provides a brief summary and concluding remarks. Methodology The SSD is a single detection depth neural network which integrates the YOLO regression concept and Faster R-CNN priori frame mechanism. It effectively extracts global features at a certain position, but does not sufficiently utilize underlying features. The SSD is mathematically defined as follows: θ l = f l ( θ l − 1 ) = f l ( f l − 1 ( … f 1 ( X ) ) ) (1) Detections = T ( τ l ( θ l ) , … , τ l − n ( θ l − n ) ) , l > n > 0 (2)where θl denotes the feature graph in layerl, fl is a nonlinear mapping of feature diagrams in layer (l − 1) that are converted to tier l, and f(X) is the first nonlinear block that transmits the input sample image X to the first layer feature graph; τl( · ) converts the feature graph of layer l into a specific scale of detection results, then combines all n intermediate results to complete the final detection. The original SSD network structure was improved in this study to enhance its detection performance for small (sewing) target gestures. SSD improved model Deep network layers respond better than shallower layers to gradient instability issues (e.g. disappearance and explosion). The VGG16-based SSD model increases the depth of the network rather than its layers, which degrades the network performance. The ResNet allows for residual connection, transforms the characteristic interlayer multiplicative transfer into additive transfer, and resolves gradient instability problems. The "shortcut" connection does not increase the parameters or complexity of the model. These qualities altogether improve the SSD detection performance [9 ]. Sewing workers perform various tasks requiring various gestures; the gestures encompass areas of different scales. In a small gesture target area, the original SSD method does not use sufficient underlying features and missed or false detections may occur. The low level feature map contains rich local texture information, which can detect some small target gestures. The position information expression of the gesture is relatively rough, however, and the high level feature map may have strong global semantic expression ability. This model is better-suited to large target detection. High-score and strong semantic feature information can be obtained via FPN-based top-down high and low feature fusion, which strengthens the detection ability for small targets. After the improved SSD deep network model, a shallow middle layer feature map detects the small and medium size target on the larger receptive field. The deep feature map's receptive field acts on the whole image. All image feature information is covered in each pixel, which is mainly conducive to large target sewing gesture detection. The network leads the feature map from the residual network Conv4_3 layer, the Conv5_3 layer, and the three convolution layers with one average pooling layer that is added later. In the sewing gesture detection process, differently scaled feature maps make different predictions for different scale targets of input samples. Each feature graph produces a priori frames of multiple scales. A variety of scale prediction frames are then generated according to different border ratios to judge sewing gestures of different scales. The FPN network structure amplifies the length and width of the deeper feature map to twice its original size by using different levels of feature maps in the convolution neural network. The maps fall in order from deep to shallow. High and low layer features are fused accordingly. The convolution operation in the fusion module minimizes the number of feature maps without changing their size, which can be used to integrate information of different dimensions. Three of the 3 × 3Conv-batch normalization (BN) layers structures buffer to prevent gradients from affecting the trunk network too severely and to ensure the stability of the network. Model training Rich data samples lead to better model training results than rough samples. Sewing gesture data was collected in this study from an operating garment factory. The data set includes four gesture categories: inner package seam, curling, cutting cloth, and frilling. In the workshop production environment, there is a deviation between the colour of the sample image collected and the colour of the real sample. The gray world algorithm was applied to eliminate the effects of light on colourability. The lighting environment changes substantially over the course of the 24-h production process. The lighting changes at different times were simulated here by image brightness transformation, which resolved any issues with the lack of all-day lighting environment (as the samples were collected at the same time) and thus improved the detection effects of the model. Too high or too low a brightness value will affect the accuracy of the manual label bounding box, so the range of select brightness changes was controlled here between 0.6 to 1.5 times. The original image was randomly rotated and mirrored to enhance the robustness of the model at different imaging angles. Fuzziness due to the focusing and movement of the camera focus will also affect the detection results, so the model's detection ability was further enhanced by randomly blurring the images. The amplified dataset has a total of 5800 images with a 6:2:2 ratio to training, test, and validation sets. The size of the amplified data is shown in Table 1. Table 1. Image quantity table generated by data enhancement Categories Origin Colour Brightness Rotation Cladding sewing 900 290 290 1500 Crimp sewing 840 280 280 1400 Fabric cutting 960 320 320 1600 Knots-making sewing 780 260 260 1300 Deep learning performs well when there is a large number of data samples. The sewing gesture data set used in this study is relatively small, however; the trained model can only depict local parts and the information reliability is relatively low. The model is thus prone to overfitting and to falling into local optimal solutions. The feature extraction ability learned by the network was applied on the ImageNet of large-scale data sets to mitigate any issues with the small sample data and fine-tune it to further improve the detection effect of the proposed model. Data training and testing were carried out at a deep learning workstation configured with i7-6800k CPU, 12Gb GPU-GTX-1080Ti, and 128 Gb of running memory. The Keras framework was operated in an Ubuntu16.04 system to write the code and call the CUDA, Cudnn, and OPENCV libraries. To generalization ability and robustness of the proposed algorithm were validated on the NUS-II data set Pascal VOC common data set with a complex background. The average accuracies (mAP values) of the three algorithms were compared as shown in Table 2. Table 2. Detection accuracy comparison mAP/% Method VOC NUS-II YOLO 0.755 0.842 SSD 0.785 0.883 Improved SSD 0.793 0.903 All three algorithms show high average accuracy (mAP) on the NUS-II data set. The loss function was used to further evaluate the performance of the models after optimization in the training network. Location and target category regression were conducted followed by reverse propagation to constantly update the model to continually minimize loss. The loss function can be calculated as follows: L ( z , c , l , g ) = ( L conf ( z , c ) + α L loc ( z , l , g ) ) ( L conf ( z , c ) + α L loc ( z , l , g ) ) N N (3)where L(z, c, l, g) denotes position loss, Lloc(z,l, g) is the loss function of the function real box and the prediction box, Lconf(z,c) is the loss determination on multi-category confidence c, N is the number of a priori boxes that match the object box, zis the matching value of the default box with different categories of real object boxes, αis the weight factor of the relationship between the confidence loss and position loss (the default value is generally 1), cand l are the confidence levels of the object category and the location information of the target box, respectively, and gis the location information of the label box. The dynamic process of training was observed by drawing a loss curve, as shown in Figure 2. Fig 2Open in figure viewerPowerPoint Loss value change curve The loss value of the model decreases rapidly over the curve and the model fits quickly in the early iterations. The loss of the model decelerates after about 4000 iterative training times and converges at approximately 8000 iterations. Experiments There are light changes in the workshop over the course of the day, as mentioned above; the model has certain application significance in regards to its robustness to light changes. The illumination condition was used as a control variable to detect the proposed model's real-time effects in morning, noon, and night environments. The results are shown in Figure 3. Fig 3Open in figure viewerPowerPoint Detection effect under different lighting conditions. Note: (The first column is morning light, the second column is evening light, and the third column is light environment). (a) Inner overlock seam; (b) crimp seam; (c) cut fabric; (d) pleated seam Table 3 shows a comparison between the enhanced data set with no light changes and the simulated 24-h light conditions. Table 3. Experimental results under different light conditions Dataset Morning Noon Light A-F1 score 0.834 0.732 0.763 B-F1 score 0.842 0.825 0.838 In Table 3, "A" marks the enhanced data set without light changes and "B" marks the data set with simulated 24-h light conditions. The harmonic average F1 values (accuracy and recall rates) increase significantly when the lighting is relatively bright and when the environment is relatively dark. An increase in brightness would otherwise create disadvantages in the data set that are suppressed by the proposed method; the proposed model performs well in the complex illumination environment. The detection effect of the trained model is better than that of the model with no brightness transformation operations. The two data sets were again compared experimentally to test the validity of their processing methods as shown in Table 4. Table 4. Comparison of ablation experiment results Method Original Corrected mAP/% 77.23 85.44 As shown in Table 4, the model trained by the enhanced data set is 8.21% more accurate than that without any enhanced data treatment. The proposed data enhancement appears to markedly impact model performance. Next, several different algorithms were compared on an actual sewing gesture dataset. The results are given in Table 5 and Figure 4. Table 5. Recognition results of five algorithms AP (average precision) Method a b c d mAP/% Detection speed/s−1 FasterR-CNN 88.2 84.6 86.9 83.6 85.8 13 YOLO 85.2 85.0 77.9 85.5 83.4 47 SSD 87.3 83.6 84.2 87.5 85.7 48 RetinaNet 87.4 86.0 87.3 87.9 87.2 51 improved SSD 87.2 89.5 88.6 89.5 88.7 52 Fig 4Open in figure viewerPowerPoint Experimental detection effects of different algorithms. Note: (The first column is F-CNN, the second column is YOLO, the third column is SSD, and the fourth column is the improved SSD). (a) Inner overlock seam; (b) crimp seam; (c) cut fabric; (d) pleated seam The proposed method shows higher accuracy than the Faster R-CNN, YOLO, or SSD network on the data set used in this study while satisfying the real-time requirements for sewing gesture detection. The residual shortcut connection prevents any gradient instability and does not increase the parameters or complexity of the model. The average real-time detection speed on each image is 52 frames per second, which satisfies the requirements for sewing gesture detection in a man-machine cooperation context. Conclusion A novel sewing gesture recognition model was established in this study based on an improved SSD deep convolutional neural network. The proposed model does not have the low recognition rate or poor detection effects of traditional algorithms in complex environments. The residual network replaces the original SSD backbone network and a feature pyramid enhances the model's feature representation capability. It was tested on a data set constructed by the authors, which was expanded by traditional image enhancement to pre-process sewing gesture images in terms of colour, brightness, rotation, and blurring. Experiments showed that the average precision of the improved model is 88.69%. The residual shortcut connection prevents gradient instability without increasing the complexity of the model. Its average real-time detection speed is 52 frames per second, which satisfies the real-time requirements for sewing gesture detection in man–machine cooperation scenarios. Acknowledgments This work partially supported by the National Natural Science Foundation of China (Program No. 51905405), the Natural Science basic Research Plan in Shaanxi Province of China (Program No. 2019JQ-855), the Natural Science Special Project of Education Department in Shaanxi Province of China (Program No. 19JK0375) References 1Zeng, Y.R., Zeng, Y.: Choi Multifactor-influenced energy consumption forecasting using al enhanced back-propagation neural network. Energy 127, 381– 396 (2017) 2Kim, M., Sul, I.H., Kim, S.: Development of a sewing machine controller for seam pucker reduction using online measurement feedback system. J. Eng. Fibers Fabr. 12(2), 67– 72 (2017) 3Berberian, B.: Man–machine teaming: A problem of agency. IFAC-PapersOnLine 51(34), 118– 123 (2019) 4Mabrok, M.A., et al.: Human models in human-in-the-loop control systems. J. Intell. Fuzzy Syst. 38(3), 2611– 2622 (2019) 5Fu, C.Y., et al.: DSSD: deconvolutional single shot detector. arXiv:1701.06659, (2017) 6Li, X., et al.: Trident SSD: A trident single-shot Multibox object detector with deconvolution. J. Phys.: Conf. Ser. 1631(1), 012182 (2020) 7Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image. arXiv: 1409.1556, (2014) 8Szegedy, C., et al.: Going deeper with convolutions. In: IEEE Conf. on Computer Vision and Pattern Recognition, CVPR 2015, pp. 1– 9. IEEE, Piscataway, NJ (2015) 9Redmon, J., et al.: You only look once: Unified real-time object detection. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 6517- 6525. IEEE, Piscataway, NJ (2017) 10Xu J., Liu H., Wu D., et al.: Generating universal adversarial perturbation with ResNet. Information Sciences 537, 302– 312 (2020) Citing Literature Volume57, Issue8April 2021Pages 321-323 FiguresReferencesRelatedInformation