# Content-Adaptive and Attention-Based Network for Hand Gesture Recognition
## 1 Introduction
Therefore, it is attractive to solve machine learning tasks using deep networks based on a self-attention mechanism. Recently, some research [22; 23] has attempted to use transformers in the field of computer vision, which traditionally relies on CNNs. [PERSON] et al. proposed a vision transformer (ViT) network [24] for image recognition tasks. This is a pure transformer-based network that outperformed the convolutional-based approach. Inspired by ViT and the fact that self-attention enjoys parallel processing, we introduced a transformer-based network for a hand gesture recognition task. This is an entirely self-attentional architecture that predicts gestures by focusing on the whole video sequence information. More specifically, we use a transformer encoder-derived architecture for hand gesture recognition tasks, resulting in a high-accuracy and low-latency model for real-time applications. The architecture of our approach consists of three submodules: a content-based adaptive sampler, a temporal attention-based encoder, and a classification multi-layer perceptron (MLP) header. After obtaining the sequence of tokens mapped from the video data, each token is given a position embedding information gained through learning or fixed absolute encoding. In the same manner as with ViT, we prefix the sequence of the features with a special classification token (CLS). We use the final state of the features related to this CLS token as the final representation of the input video after propagating the sequence through the transformer encoder. Finally, an MLP head processes the CLS token to obtain the final class prediction. Furthermore, we present a dynamic sampling strategy based on content adaptation to efficiently sample the video sequence. We use a sliding window to perform gesture detection on the input video content. When the sampler detects a gesture, it starts sampling evenly according to the time strategy until it reaches the set maximum number of frames. Another role of this sampler is to act as a switch for the encoder. When a gesture is detected in the video sequence, the encoder is activated and starts receiving the input from the sampler. Since most of the time, no gesture appears in the video sequence, there is no need for the feature extractor and encoder to process these frames, which would significantly increase the consumption of the system. The following are the main contributions of our work:  
1. We suggest a transformer-based network for hand gesture recognition tasks, which is a fully self-attentional architecture. The framework classifies gestures by focusing on the entire video sequence without any convolution or recurrent layers. It effectively manages the high number of spatial-temporal tokens that may appear in the video. 2. We propose an adaptive sampling strategy based on video content to reduce the sampling of gesture-free frames and improve the performance of the model. We use a sliding window to continuously detect the input frame sequence. When a gesture is detected, continuous sampling from the current frame is conducted, with no additional gesture detection. The rest of this paper will consist of the following. The related studies and background techniques in gesture recognition are discussed in Section 2. Section 3 describes our proposed approach in detail. The experiments and experimental results are shown in Section 4. Section 5 contains the conclusion and suggestions for further studies.