# SDViT: Stacking of Distilled Vision Transformers for Hand Gesture Recognition
## 3 Stacking of Distilled Vision Transformers (SDViT)
### Vision Transformer
#### 3.1.2 Transformer Encoder  
The transformer encoder plays a crucial role in the transformer model. It comprises several layers, denoted as \(L\), each of which consists of an MSA layer and a position-wise feedforward layer (MLP), as depicted in Figure 3. Layer normalization (LN) is employed before the commencement of each block, and residual connections are implemented following the conclusion of each block. LN is applied to stabilize the model representations. Following the MSA layer, another layer normalization step is performed, and the resulting sequence is passed through the MLP layer. Equations (2) and (3) represent the flow of information within the transformer encoder block. Overall, the transformer encoder leverages self-attention mechanisms and MLPs to effectively grasp global interdependencies and efficiently handle representation processing. \[\mathbf{z}_{\ell}^{\prime}=\text{MSA}(\text{LN}(\mathbf{z}_{\ell-1}))+\mathbf{ z}_{\ell-1},\quad\ell=1\cdots L \tag{2}\]  
\[\mathbf{z}_{\ell}=\text{MLP}\big{(}\text{LN}(\mathbf{z}_{\ell}^{\prime})\big{)} +\mathbf{z}_{\ell}^{\prime},\quad\ell=1\cdots L \tag{3}\]  
* Multi-head Self-Attention  
The MSA layer in the ViT model captures contextual relationships among patch embeddings through self-attention. It utilizes numerous self-attention heads to assess the significance of patch embeddings by considering their interactions. Within the MSA layer, there are query, key, and value projections; dot product attention; and linear projection layers. It takes the patch embeddings as its input, carries out linear projections to produce query, key, and value matrices, and computes attention scores. These attention scores are employed to weigh the values, resulting in a weighted summation. The outcomes of this self-attention process are aggregated along the dimension of heads and subsequently subjected to a linear projection to return them to their original dimension. This process enables the MSA layer to capture patch relationships and produce the final output. Figure 3: The design of the transformer encoder. * Multi-Layer Perceptron  
Within the ViT encoder, there is an MLP block responsible for modifying the representations acquired from the MSA layer. This block is composed of two linear transformations separated by a Gaussian error linear unit (GELU) activation function. The inclusion of the MLP introduces non-linear characteristics, enabling the model to capture intricate connections. The result of the MLP block is obtained through a series of operations, which involve multiplying the input by a weight matrix, adding a bias, applying the GELU activation function, and then repeating the process with another weight matrix and bias. This sequence of operations is iterated multiple times within the ViT encoder to learn progressively more abstract features from the input image. The transformer encoder extracts features using self-attention in the MSA layer to capture global interdependencies among patch embeddings. Attention scores computed through query, key, and value projections facilitate the weighing of values, capturing contextual relationships. The resulting representations pass through the MLP block, introducing non-linear characteristics to progressively learn more abstract features. This feature extraction process enables the ViT model to capture intricate patterns and relationships within images, making it well suited for a variety of computer vision tasks. In the encoder block, the first token \(z_{L}^{0}\) is then selected and normalized to generate the image representation \(\mathbf{y}\). \[\mathbf{y}=\text{LN}\Big{(}\mathbf{z}_{L}^{0}\Big{)}. \tag{4}\]  
This representation is subjected to pooling in order to derive a solitary feature vector. Afterwards, it is flattened, normalized through batch normalization, and passed through a fully-connected layer with SoftMax activation for the ultimate classification. While training, the rectified Adam optimizer effectively adjusts the model's parameters.