# Content-Adaptive and Attention-Based Network for Hand Gesture Recognition
## 2 Related Work
### Attention-Based Methods  
Since 2017, the transformer has received substantial attention due to its performance beyond RNNs in many NLP tasks. The transformer was first introduced for solving machine translation tasks [21]. Since then, transformer-based sequence modeling approaches have gradually become the state-of-the-art solution for other NLP tasks [27; 28]. Traditional self-attention-based models usually rely on RNNs for input representations [29; 30]. In contrast, the transformer model does not have any recurrent or convolutional layers and is a model entirely based on attention mechanisms [21; 31]. The transformer consists of two submodules: the encoder and the decoder. Both encoder and decoder are stacked with self-attention-based modules. The embedding representations of the input and target sequences are fed into the encoder and decoder, respectively, after adding positional encoding. The encoder of the transformer is composed of \(N\) identical layers stacked on top of each other (in reference [21], \(N\) is set to 6). Each layer consists of two sublayers: a self-attention sublayer and a feed-forward network sublayer. Similar to the design of the ResNet module, both sublayers are connected using a residual connection layer, followed by layer normalization. Specifically, when computing the encoder's self-attention, the queries, keys and values are taken from the output of the previous encoder layer. Like the encoder, the decoder of the transformer is stacked with \(N\) identical layers, with each sublayer also using residual connections and layer normalizations. Unlike the encoder, the decoder adds a sublayer called the encoder-decoder attention layer between the two sublayers of the self-attention layer and the feed-forward network. The encoder-decoder attention sublayer is still based on the self-attention mechanism, so it has the same structure as the self-attention sublayer. As with the input of the encoder self-attention, the queries, keys, and values in the self-attention decoder are also taken from the output of the previous decoder layer. In the new encoder-decoder attention sublayer, the input keys and values are taken from the output of the transformer encoder layer, while the queries are from the output of the previous decoder layer. Although the original transformer was proposed for solving sequence-to-sequence tasks, it is now widely used in other deep learning areas, such as speech processing, reinforcement learning, and computer vision [31; 32; 33; 34]. Inspired by the advanced performance achieved by transformers in the field of NLP, [PERSON] et al. [24] proposed a vision transformer network (ViT) for image classification. For an input image, ViT first splits it into \(N\) fixed-size patches and then uses the linear embeddings sequence of these patches as the input to the transformer encoder. In ViT, image patches are treated in the same manner as words (tokens) are processed in an NLP application. To our knowledge, ViT was the first approach to apply transformer to computer vision tasks. After this, transformers started to be employed in a wide range of other computer vision tasks. For example, DETR [35] for object detection and VisTR [36] for video instance segmentation have yielded competitive results using transformer-based approaches. Since the transformer has a component based on the self-attention mechanism, it enables the network to capture contextual information from the whole sequence data, which leads to the success of the transformer. However, the computation of self-attention grows quadratically with the length of the sequence data, resulting in the standard transformer-based model being unable to effectively process long sequence data. To address this challenge, [PERSON] et al. proposed an improved transformer architecture, called longformer [37]. The longformer has an improved self-attention mechanism, and the operations of this attention mechanism are scaled linearly with the sequence length, enabling it to handle longer sequence data.