# Content-Adaptive and Attention-Based Network for Hand Gesture Recognition
## 3 Proposed Method
### Attention-Based Encoder  
The architecture of the transformer encoder is shown in Figure 3. As shown in Figure 3a, the encoder consists of a stack of \(N\) identical layers. Following the suggestions in reference [21], in our work we set \(N\) to 6. Each layer consists of two submodules: a multi-head self-attention module, and a position-wise fully feed-forward neural network. Each submodule is connected using a residual connection (denoted as \"Add\" in Figure 3), followed by layer normalization. The multi-head self-attention module uses a self-attention mechanism to perform a new representation of the input sequence; the feed-forward network uses a fully connected feed-forward neural network to further transform the input vector sequence. Self-Attention. Mapping a query and a group of key-value pairs to an output is the main role of the attention function. In other words, a given query can interact with the keys to guide the biased selection of values. If a key is closer to a given query, then more attention weight is assigned to the value of the key. The self-attention mechanism treats the representation of each position in the sequence as query and the representation of all positions as key and value. The self-attention model computes a weighted sum of the values of each position by calculating the degree of match between the current position and all positions, which is the attention weight in the attention mechanism. Mathematically, given  
Figure 3: The architecture of transformer encoder: (**a**) encoder architecture; (**b**) multi-head attention; (**c**) scaled dot-product attention. a query \(q\) and \(m\) key-value pairs \((k_{1},v_{1}),\ldots,(k_{m},v_{m})\), the attention function \(f\) is instantiated as a weighted sum of the values, which can be defined as following Equation (1) [21]:  
\[f(q,(k_{1},v_{1}),\ldots,(k_{m},v_{m}))=\sum_{i=1}^{m}\alpha(q,k_{i})v_{i} \tag{1}\]  
where query \(q\), key \(k\), value \(v\), and output are all vectors. The attention weight of query \(q\) and key \(k_{i}\) is obtained by mapping the two vectors into scalars using the attention-scoring function \(a\) and then by SoftMax operation:  
\[\alpha(q,k_{i})=\frac{\exp(a(q,k_{i}))}{\sum_{j=1}^{m}\exp\bigl{(}a\bigl{(}q,k _{j}\bigr{)}\bigr{)}} \tag{2}\]  
where \(a\) indicates the attention-scoring function. From Equation (2), it is clear that choosing different attention-scoring function \(a\) leads to different attentional behaviors. Additive attention and scaled dot-product attention are the two most commonly used attention-scoring functions. The scaled dot-product attention of queries \(q\) and keys \(k\) can be expressed as follows:  
\[a(\mathbf{q},\mathbf{k})=\mathbf{q}^{\top}\mathbf{k}/\sqrt{d} \tag{3}\]  
where \(d\) denotes the length of the query \(q\) and the key \(k\). Since the time complexity of the self-attention operation of the original transformer model is \(O(n^{2})\) (\(n\) is the length of the input sequence), this limits its ability to process long sequences. To address this limitation, we use an efficient self-attention variant called longformer [37] as our attention module. Longformer combines local and global attention, and its temporal complexity is linearly related to the length of the sequence, which allows long-range dependence problems to be better solved. Multi-Head Attention. Another important technique used in the transformer is the multi-head attention mechanism. To capture the various ranges of dependencies within a sequence, we can transform queries, keys, and values using \(h\) sets of linear projections learning independently. As shown in Figure 2(b), these \(h\) groups of transformed queries, keys, and values are then fed in parallel to the scaled dot-product attention. Finally, the outputs of \(h\) scaled dot-product attention are concatenated together and transformed by another linear projection that can be learned to produce the final output.