# Content-Adaptive and Attention-Based Network for Hand Gesture Recognition
## 2 Related Work  
### Convolution-Based Methods  
Action is composed of two components: appearance information carried by a single frame and motion information between neighboring frames. Capturing motion information between neighboring frames from sequence data is a key step in gesture recognition systems. In reference [14], [PERSON] et al. proposed a two-stream approach that combined predictions from a single-color image frame and a stack of externally computed multiple optical flow frames to predict action classes. Two-stream networks are only modeled in a short time and cannot effectively model the temporal structure over a long range. To address this problem, [PERSON] et al. presented a TSN [15] that combined a sparse temporal sampling strategy with video-level supervision to efficiently learn the long-range temporal structure of the video. The TSN's main idea is that the entire video can be evenly divided into several segments, each of which is processed separately. The video-level prediction will then be formed from a consensus among the segments. Subsequently, many studies [16; 17; 25; 26] have employed this idea and designed different deep models for action recognition. However, we observed that this algorithm does not work well for scene-independent action recognition, such as hand gestures. In addition, although optical flow can describe the motion effectively, computational consumption is expensive.  
Additionally, 3D convolution and recurrent convolution are two effective methods for modeling temporal structure. In [16], [PERSON] et al. proposed an approach for action classification using 3D CNNs for spatial-temporal feature learning. Compared to 2D CNNs, 3D CNNs are excellent for learning spatial-temporal features. However, the exponential increase in the number of parameters to be optimized for 3D CNNs-based models leads to high computational consumption and the need for more data to train the model. RNNs are neural networks composed of recurrent units [19; 20]. The input of the current moment recurrent unit consists of the output of the previous moment and the input of the current moment, which can also be understood as the output obtained from the network's current moment calculation being jointly determined by the previous sequence. For long  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Layer Type**} & \multicolumn{2}{c}{**Minimum Sequential**} & \multirow{2}{*}{**Maximum Path Length**} & \multirow{2}{*}{**Complexity Per Layer**} \\  & & & & \\  & & **Operations** & & \\ \hline CNNs & Convolutional & \(O(1)\) & \(O(\log_{k}(n))\) & \(O(k\cdot n\cdot d^{2})\) \\ RNNs & Recurrent & \(O(n)\) & \(O(n)\) & \(O(n\cdot d^{2})\) \\ Transformer & Self-Attention & \(O(1)\) & \(O(1)\) & \(O(n^{2}\cdot d)\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of the time complexity of each layer of RNNs, CNNs, and transformer. \(n\) indicates the length of the sequence, \(d\) indicates the size of the hidden layer, and \(k\) indicates the kernel size of the convolution.  
videos, long distances can cause information to be lost during delivery, and this sequential modeling approach also limits the parallel computing of the system. To address all these issues, we proposed a transformer-based network for hand gesture recognition. It is a fully self-attentional architecture without any convolutional and recurrent convolutional layers. In contrast to using the segmental consensus-based approach [25; 26], our proposed method classifies actions by focusing on the temporal structure of the entire video sequence.