## 3 The Proposed Method
### Spatial Transformer Networks  
After human detection and pose estimation, the input video data is converted into a sequence of skeleton images. In an industrial production environment, people may appear anywhere in the field of view, presenting different poses and sizes, which causes some interference in the recognition of manual operations. To make our method more robust to the environment and reduce the impact of the diversity of human poses on the recognition results, we employ an STN to adaptively adjust each frame, and the adjustments include translation, rotation, and zoom.  
According to [PERSON] et al. [37], any affine transformation process can be described using six parameters. Therefore, the STN achieves adaptive correction of human poses in industrial environments by regressing these six parameters. The structure of the STN is shown in the spatial transformer part of Figure 1. Differently from manually correcting the pose of the human body in each frame or directly using the CNN to extract features without any pre-processing, the STN consists of three parts: the localization net, grid generator, and sampler.  
The localization net is essentially a regression network, the input of which is the skeleton image, and the output is the six parameters of the affine transformation. The structure of the localization net is usually a fully connected network or a convolutional network followed by a regression layer to train the parameters, which is denoted as \(\theta\) in Figure 1. The grid generator (i.e., \(T_{\theta}(\mathrm{G})\) in Figure 1) transforms the skeleton image by taking advantage of the parameter \(\theta\), and calculates the coordinates in the original skeleton image corresponding to each position in the target image. The generation process can be expressed as follows:  
\[\left(\begin{array}{c}x_{i}^{s}\\ y_{i}^{s}\end{array}\right)=\mathcal{T}_{\theta}\left(G_{i}\right)=\theta \left(\begin{array}{c}x_{i}^{t}\\ y_{i}^{t}\\ 1\end{array}\right)=\left[\begin{array}{ccc}\theta_{11}&\theta_{12}&\theta_{13 }\\ \theta_{21}&\theta_{22}&\theta_{23}\end{array}\right]\left(\begin{array}{c}x_ {i}^{t}\\ y_{i}^{t}\\ 1\end{array}\right) \tag{1}\]  
where \(\left(x_{i}^{t},y_{i}^{t}\right)\) is the coordinates of a position on the target image and \(\left(x_{i}^{s},y_{i}^{s}\right)\) is the coordinates of the corresponding position on the input skeleton image. The _sampler_ samples the skeleton image according to the coordinate information in \(T_{\theta}(\mathrm{G})\), and copies the pixels to the target image. However, \(\left(x_{i}^{s},y_{i}^{s}\right)\) always falls in the middle of several pixel points of skeleton image, so in this study we use bilinear interpolation to calculate the gray value corresponding to this point. Most importantly, according to the research in [PERSON] et al. [37], the gradient can be transmitted during the STN transformation process, so end-to-end training can be continuously performed in the network to correct the parameters.  
In summary, the training result of STN is six parameters, and the skeleton images can adaptively perform affine transformation after STN. In the case where the input data has a large spatial difference, this network can be added to the existing convolutional network to improve the accuracy of classification.