# HandFormer: A Dynamic Hand Gesture Recognition Method Based on Attention Mechanism
## 3 Method
### Spatial Attention Former Block (SAFB)  
To learn the spatial feature information of each frame of the gesture, we propose the Spatial Attention Former Block (SAFB) that combines convolutional operations and Transformer structures based on the spatial attention mechanism, as shown in Figure 2b, which is built based on the MetaFormer block. The SAFB replaces the token mixer in the MetaFormer block with the Spatial Attention Convolution Block (SACB) based on the spatial attention mechanism, as shown in Figure 3. The SACB uses convolutional neural networks and spatial attention mechanisms to learn the spatial features of the input image. The attention mechanism learns to select spatial information useful for gesture recognition using different weights. Firstly, the input image is fed into two different branches. One branch uses convolutional operations to extract the spatial features of the gesture. Specifically, it first applies a depthwise convolution (DWC) with kernel size k to the input feature map channel by channel, and then applies another depthwise convolution (DWC) with kernel size k to extract gesture spatial features with different receptive fields. The feature maps obtained by the two convolutional operations are then fused and input into a pointwise convolution (PWC) to map the feature maps to different dimensions [26]. This branch can be represented as follows:  
\[F_{cb}=\sigma(\phi(\varphi_{1}^{3,3}(F_{in})\oplus\varphi_{1}^{5,5}(F_{in}))). \tag{4}\]  
The above equation represents the spatial feature extraction branch in the SACB, where \(F_{in}\) represents the input feature map, \(\varphi_{s}^{k,k}(\cdot)\) represents the depthwise convolution with kernel size \(k\) and stride \(s\), \(\phi(\cdot)\) represents the pointwise convolution, \(\oplus\) represents the feature fusion operation, and \(\sigma(\cdot)\) represents the activation function. The other branch generates the spatial attention weight matrix through the spatial attention mechanism and performs matrix multiplication with the spatial feature maps learned by the first branch to select different spatial feature information. It first applies a 32-channel convolution with a kernel size of 3 and an activation function, followed by a 64-channel convolution with a kernel size of 3 and another activation function to gradually increase the dimensionality of the feature map and enhance the non-linear expression ability of the features. Finally, it applies a 1-channel convolution to generate the attention weight matrix for the corresponding spatial feature map. This branch can be represented as  
\[W_{SA}=f_{1}^{1,1,1}(\sigma(f_{1}^{3,3,64}(\sigma(f_{1}^{3,3,32}(F_{in}))))). \tag{5}\]  
The above equation represents the spatial attention weight extraction branch in the SACB, where \(F_{in}\) represents the input feature map, \(f_{s}^{k,k,c}(\cdot)\) represents the convolution operation with kernel size \(k\), output channels \(c\), and stride \(s\), \(\sigma(\cdot)\) is the activation function, and \(W_{SA}\) represents the obtained attention weight matrix. After learning the spatial feature map \(F_{cb}\) and the spatial attention weight matrix \(W_{SA}\) through the two branches, the Spatial Attention Convolution Block (SACB) applies the spatial attention weight matrix to the spatial feature map by performing a matrix multiplication operation between the two matrices. This operation can be represented as  
\[F_{SA}=W_{SA}\otimes F_{cb}. \tag{6}\]  
The above equation represents the multiplication of the obtained spatial attention weight matrix and the spatial feature map to obtain the spatial attention feature map, where \(\otimes\) represents the matrix dot product operation, and \(F_{SA}\) represents the obtained spatial attention feature map. Finally, by replacing the token mixer in MetaFormer with the Spatial Attention Convolution Block (SACB), we obtain the Spatial Attention Former Block (SAFB), which can be represented as  
\[F_{out}=F_{in}+SACB(Norm(F_{in})). \tag{7}\]  
Figure 3: Spatial Attention Convolution Block (SACB).