# Action Recognition Network Based on Local Spatiotemporal Features and Global Temporal Excitation
## 3 Method
### Local Spatiotemporal Extraction Module (LSTE)  
The LSTE module obtained the local motion features through a motion branch, which is fused with the spatial features extracted through a spatial branch, as shown in Figure 3. In the spatial branch, a frame \(I_{i}\) randomly selected from each video segment was fed to a \(7\times 7\) convolutional layer to obtain the spatial features \(X_{s}\in R^{N\times 64\times\frac{W}{2}\times\frac{W}{2}}\). \(N\) represents the batch size, \(H\) represents the feature map's height, \(W\) represents the feature map's width. In the motion branch, the local motion features around a frame \(I_{i}\) was extracted. Firstly, we counted the pixel-wise differences of neighboring video frames \([I_{i-2},I_{i-1},I_{i},I_{i+1},I_{i+2}]\) and stacked them along channel dimension \(D=[D_{1},D_{2},D_{3},D_{4}]\), \(D\in R^{N\times 12\times H\times W}\). To reduce the computational cost, we performed an average pooling on \(D\) to halve the spatial size, and we obtained difference features \(X_{D}\) by a \(7\times 7\) 2D convolutional layer, which is represented in Equation (1):  
\[X_{D}=CNN(Avg\_pool(D(I_{i}))),X_{D}\in R^{N\times 64\times\frac{W}{2}\times \frac{W}{2}} \tag{1}\]  
Then, we strengthened the difference features \(X_{D}\) by stressing the effect of the feature channels sensitive to difference information, which can be divided into the following three operations. In the first operation, we scaled the spatial dimension to \(1\times 1\) by using a global average pooling on \(X_{D}\) to obtain the global information \(F\), which is represented in Equation (2):  
\[F=\frac{1}{H\times W}\sum_{i=1}^{H}\sum_{j=1}^{W}X_{D}[:,:,i,j],F\in R^{N\times 6 4\times 1\times 1} \tag{2}\]  
In the second operation, we performed two \(1\times 1\) convolution operations on \(F\) and obtained the importance weight \(s\) corresponding to each channel through a Sigmoid activation function, which is represented in Equation (3):  
\[s=\sigma(W_{2}*\delta(W_{1}*F)) \tag{3}\]  
where \(W_{1}\) is the first \(1\times 1\) convolution, which was used to fully capture the interdependencies between each channel and reduce channel dimension by a ratio \(r\) (\(r=16\)). \(\delta\) is the ReLU activation function. \(W_{2}\) is the second \(1\times 1\) convolution, which was used to recover the number of feature channels. \(\sigma\) is Sigmoid activation function. In the third operation, the local motion features \(\overline{X}_{D}\) were obtained by performing a channel-wise multiplication between the difference features \(X_{D}\) and the importance weight \(s\), which is shown in Equation (4):  
\[\overline{X}_{D}=s\odot X_{D},\overline{X}_{D}\in R^{N\times 64\times\frac{W}{2} \times\frac{W}{2}} \tag{4}\]  
Figure 3: The framework of the LSTE module.  
Finally, we up-sampled \(\overline{X}_{D}\) to match the spatial features \(X_{s}\) and fused them, which is shown in Equation (5):  
\[X_{L}=X_{s}+Upsample(\overline{X}_{D}),X_{L}\in R^{N\times 64\times\frac{H}{2} \times\frac{N}{2}} \tag{5}\]  
where \(X_{L}\) is the output feature of the LSTE module. Through the above operations, we obtained the local spatiotemporal features of each video segment.