# 3D-STARNET: Spatial-Temporal Attention Residual Network for Robust Action Recognition
## 3 Proposed Methods
### Spatiotemporal Attention Module  
The attention mechanism is a cognitive enhancement process that enables the network to selectively learn interesting feature areas while ignoring other less relevant features. The input data of this paper are a continuous action video frame. The ordinary network assigns the same weight to all frames and pixel positions within the frame, which brings great confusion to the action recognition task. To this end, this paper proposes a spatiotemporal attention mechanism STA, as shown in Figure 3. Our proposed spatiotemporal attention mechanism applies attention mechanisms in both temporal and spatial dimensions to enhance the model's ability to identify key frames and key locations in specific frames among consecutive video frames. F denotes the input continuous video frames, where each frame is represented as a feature map \(\text{f}_{\text{i,j}}\) in \(\text{R}^{\text{h}\times\text{w}}\), with \(\text{i}=1,2,\cdots,1\) indicating the time index, and \(\text{j}=1,2,\cdots,c\) indicating the channel index. Here, \(\text{i}\) is the total number of video frames and \(\text{c}\) is the number of channels. Through the Spatiotemporal Attention (STA) module, F can generate corresponding weights \(\text{W}=\left\{\text{w}_{\text{i,j}}\right\}\) in both the temporal and spatial dimensions. We define the attention function \(\text{T}\) as a mechanism to learn these weights W from F, which in turn allows us to define the output feature \(\text{O}=\left\{\text{o}_{\text{i,j}}\right\}\) in \(\text{R}^{\text{h}\times\text{w}}\) after F has passed through the STA module. It should also be noted that T is composed of two parts, \(\text{T}_{\text{t}}\) and \(\text{T}_{\text{s}}\), where \(\text{T}_{\text{t}}\) and \(\text{T}_{\text{s}}\) represent the temporal attention function and the spatial attention function, respectively. Figure 4 shows the entire spatiotemporal attention mechanism. The construction of the entire attention mechanism follows the principles of making the architecture simple and effective enough and making the network robust. Figure 2: Data acquisition process. By introducing the spatiotemporal attention mechanism we proposed, the network can recognize actions in the video by focusing on only a small part of the frames in the video. This is difficult to do with previous 3D CNN models. Previous studies mostly focused on spatial and channel dimensions, while giving indifferent weights to video frames, which made it difficult for the network to have stable recognition performance and greatly increased the processing cost of the network. The temporal attention mechanism in the STA module can effectively enhance the model's ability to identify key frames, thereby more effectively extracting decisive information about video actions. Specifically, the input feature \(\text{F}_{\text{t}}\) learns the weight of the time dimension through the attention function \(\text{T}_{\text{t}}\). In practical applications, since the input frame sequence is too short, it is impossible to extract the variance information of the frame sequence. To address this problem, we use a group convolution to expand the time channel to twice the original size to retain richer time information. Here, the attention function \(\text{T}_{\text{t}}\) can be defined as a composite function that generates temporal weights for each frame:  
\[\text{T}_{\text{t}}=\text{S}_{\text{t}}\circ\varphi_{\text{t}}\circ\varepsilon_ {\text{t}} \tag{2}\]  
where \(\text{S}_{\text{t}}\) and \(\varepsilon_{\text{t}}\) represent the compression and expansion operations of the time channel, respectively, \(\varphi_{\text{t}}\) represents the Relu activation function, and \(\circ\) represents the composite operation of multiple functions; that is, the output of one function becomes the input of the next function. The upper part of Figure 4 shows the detailed network framework of the temporal attention module.