# 3D-STARNET: Spatial-Temporal Attention Residual Network for Robust Action Recognition
## 3 Proposed Methods
### MS-Residual Structure  
In order to make the model competitive in the efficient dissemination of information, we improved the previous Resnet [28] residual block into a phased residual block, and divided it into start block, middle block, and end block, without changing the complexity of the model. Here, we only changed the order of the different units to promote the dissemination of data in the network. At the same time, since a shortcut exists in the main information dissemination path, we have also conducted research on its improvement. The following is a detailed introduction to the improvement. The residual structure of all layers of the traditional resnet network architecture adopts the same unit structure, as shown in Figure 5. The propagation of information in the residual block can be expressed as follows:  
\[X_{i+1}= \begin{cases}X_{i}+F(X_{i},W_{i}),size(F(X_{i},W_{i}))=size(X_{i})\\ \lambda_{i}X_{i}+F(X_{i},W_{i}),size(F(X_{i},W_{i}))\
eq size(X_{i})\end{cases}  \tag{4}\]  
where \(X_{i}\) and \(X_{i+1}\) represent the input and output information of the i-th residual block, respectively, and the function F represents a learnable residual mapping function, which represents the downsampling operation performed when the sizes of \(F(X_{i},W_{i})\) and \(X_{i}\) are different. As can be seen from Figure 5, the outputs of the main branch and shortcut are added and then processed using RELU which will cause the negative elements in the information  
Figure 5: The original residual structure. to be zero, thereby causing information loss. Pre-activation [29] has improved this by moving the BN and RELU at the end of the network to the front of the network to obtain better performance. However, this improvement does not achieve a normalization of the complete signal, but is only applied to the branch signal, which reduces the information control through the network and affects the dynamic flow of the gradient. As the number of network layers increases, the overall information lacks normalization and nonlinearity. To address the above issues, we replace the original residual structure with a staged residual structure [30], as shown in Figure 6. The same number and type of unit blocks as the original residual structure, but the order of the unit blocks is changed. Specifically, in the start stage, the last Relu operation is removed to prevent the loss of signal caused by the zeroing of negative numbers. In the middle stage, we adopted the residual structure in pre-activation. In the end stage, we said that BN and Relu are applied to the complete signal to replace the last BN and Relu of each residual block in each stage, so as to effectively control the complete signal. When the gradient of the signal becomes very large after passing through the residual block in the initial stage, it will also be subjected to nonlinear normalization processing using BN and Relu in the final stage to improve the nonlinear expression ability of the model and reduce the overall signal loss. In the residual structure, when the dimension of the input \(X_{i}\) is inconsistent with the mainstream output dimension, a shortcut projection method is used to make the second condition in Formula (4) meet. The original ResNet's shortcut projection is implemented by a convolution with a kernel of \(1\times 1\times 1\) and a stride of 2, which is used to satisfy the condition of adding operations with the mainstream branch. For an input dimension of \(\text{T}\times\text{H}\times\text{W}\), after the above convolution operation, the output dimension will be changed to \(\frac{\text{T}}{2}\times\frac{\text{H}}{2}\times\frac{\text{W}}{\text{T}}\). Therefore, this convolution operation will cause about 87.5% of the information to be lost, and the remaining 12.5% of the information will be meaningless, which is equivalent to adding noise to the mainstream output and will have a negative impact on network performance.