# 3D-STARNET: Spatial-Temporal Attention Residual Network for Robust Action Recognition
## 3 Proposed Methods
### Skeleton Extraction and 3D Heat Map Generation  
As the basic work of this study, the effect of skeleton key point extraction directly affects the accuracy of action recognition. This study first uses the mmpose [26] open source framework of openmmlab to perform top-down skeleton key point extraction, which is used as the basic condition for downstream tasks.  
Since the basic action recognition framework used by the model in this article is posec3d [27], the data input format of its backbone network is a 3D heat map instead of a 2D skeleton sequence. This is mainly because a 3D heat map (the 3D heat map is the 2D heat map stacked in the time dimension so that it can be input into the 3D CNN model) has more time dimension information than a 2D skeleton sequence, which can better describe the temporal transformation relationship between actions. It has good robustness when dealing with occlusion, lighting changes, and complex backgrounds, can better handle noise and uncertainty in posture estimation, and reduces the dependence on precise coordinates of bones. At the same time, 3D heat maps can provide the model with more contextual information, which helps the model generalize between different scenes and different actions.  
In this framework, we first use the pose estimator mmpose [26] to extract 2D human skeleton key points from action video clips, and then a Gaussian distribution heat map is generated for each skeleton key point to form a \(\text{K}\times\text{H}\times\text{W}\) heat map, where K represents the number of skeleton points, and H and W are the height and width of the frame, respectively. These 2D heat maps are then stacked along the time dimension to form a 3D heat map volume of shape \(\text{K}\times\text{T}\times\text{H}\times\text{W}\). Assuming that the position information of each bone point is represented by a triple \((x_{k},y_{k},c_{k})\), the k joint points are mapped to a graph through Gaussian transformation to form a three-dimensional bone point heat map J:  
\[J_{kij} =e^{-\frac{(i-x_{k})^{2}+(j-y_{k})^{2}}{2\omega^{2}}}*c_{k}, \tag{1}\]  
Among them, \(x_{k}\) and \(y_{k}\) represent the position information of the k-th skeleton point, respectively, and \(c_{k}\) represents the confidence score of the skeleton point; this value is  
Figure 1: Visualization of 3D-STARNET input data processing and overall network architecture.  
obtained during the key point extraction stage. Its value is usually between 0 and 1, representing the accuracy of detecting the skeleton point. It affects the size of the variance. The larger the variance, the wider the Gaussian expansion range, which will be more advantageous in tasks with strong globality such as action recognition. And \((i,j)\) represents the point on the Gaussian graph, where \(\sigma\) represents the degree of diffusion around the key point, ensuring that the generated Gaussian kernel can cover the area around the key point, which will be more advantageous in tasks with strong globality such as action recognition. Figure 2 intuitively shows the transformation from an RGB image to skeleton points and then to a heat map.