###### Abstract  
In this study, we consider fully automated action recognition based on deep learning in the industrial environment. In contrast to most existing methods, which rely on professional knowledge to construct complex hand-crafted features, or only use basic deep-learning methods, such as convolutional neural networks (CNNs), to extract information from images in the production process, we exploit a novel and effective method, which integrates multiple deep-learning networks including CNNs, spatial transformer networks (STNs), and graph convolutional networks (GCNs) to process video data in industrial workflows. The proposed method extracts both spatial and temporal information from video data. The spatial information is extracted by estimating the human pose of each frame, and the skeleton image of the human body in each frame is obtained. Furthermore, multi-frame skeleton images are processed by GCN to obtain temporal information, meaning the action recognition results are predicted automatically. By training on a large human action dataset, Kinetics, we apply the proposed method to the real-world industrial environment and achieve superior performance compared with the existing methods. deep learning; action recognition; convolutional neural network; spatial transformer network; graph convolutional network; industrial workflows +
Footnote †: journal: Medical  
To find a way to extract the spatial and temporal information, action recognition has been greatly studied. Accurate action recognition is a highly difficult task [5] in industrial workflows principally for the following challenges: (1) Intra-class and inter-class differences. For the same action, the performance of different people may vary greatly. (2) Complexity of the environment, including occlusion, multiple shooting angles, illumination, low resolution and dynamic backgrounds. (3) Complexity in the temporal dimension. Human actions are often composed of multiple poses, and different starting points will affect the accuracy of action recognition enormously. Motivated by the aforementioned challenges, we propose a multi-deep-learning model integrated spatial transformer graph convolutional network for action recognition that combines convolutional neural networks (CNNs), spatial transformer networks (STNs) and graph convolutional networks (GCNs) to recognize manual operations in industrial workflows. The spatial information in video is extracted by the CNN, the temporal information between the frames, and the spatio-temporal information is established through a GCN. To solve the problem that the human body presents different shapes under different camera angles, we use a STN to align the human body to enhance the recognition effect. At the same time, considering the difference in the workload of the upper and lower body of a human in industrial operations, we have added an attention mechanism to the model to adjust the weight of keypoints in action recognition, and improve the accurate identification of manual operations in industrial environments. To validate the effectiveness of our method, we perform thorough experiments and achieve superior performance under both accuracy and false recognition rate metrics. The main contributions of the proposed method are as follows:  
* We propose a novel method that combines multiple neural networks to recognize manual operations in an industrial environment, which is a rare attempt in the field. * The proposed method leverages a spatial transformation network to reduce the recognition errors caused by the diversity of human poses in the real working environment. * A graph convolutional neural network is constructed to extract the spatial and temporal information of the skeleton image at the same time, which, combined with the classifier, can accurately recognize human action. * An attention mechanism. Considering the unique characteristics of the real-world production environment, different weights are applied to more than a dozen keypoints of the human body to improve the accuracy of recognition.