# SDViT: Stacking of Distilled Vision Transformers for Hand Gesture Recognition
## 3 Stacking of Distilled Vision Transformers (SDViT)
### Stacking
plateau involve monitoring accuracy, a factor of 0.2, patience of 2, verbose of 1, minimum delta of 0.0001, minimum learning rate of 0.000001, and mode set to max. The adaptive learning rate proves beneficial by adjusting the learning rate when a specified metric, such as accuracy, shows no improvement for a set number of epochs. These callback methods, when integrated into the training process, enhance the model's efficiency, stability, and generalization performance. Two experiments were conducted on the pretrained ViT. The first experiment focused on comparing the ViT configured with a patch size of 32 \(\times\) 32, referred to as ViT B32, with fine-tuning of all available layers using weights from ImageNet21k and ImageNet2012 datasets. The results can be observed from Table 2, while the best results were achieved when using weights from ImageNet21k + ImageNet2012, with testing accuracies of 99.97% for the ASL dataset, 99.01% for the ASL with digits dataset, and 99.85% for the NUS hand gesture dataset. On average, across all three datasets, the accuracy reached 99.61%. The second experiment involved fine-tuning the ViT B32 with ImageNet21k + ImageNet2012 weights by unfreezing different numbers of layers, ranging from 3 to 19 layers. The results can be observed from Table 3, while the best results were obtained when fine-tuning 12 layers, achieving testing accuracies of 99.98% on the ASL dataset, 99.40% on the ASL with digits dataset, and 99.85% on the NUS hand gesture dataset. On average, across all three datasets, the accuracy reached 99.74%. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Weights**} & \multicolumn{4}{c}{**Datasets**} & \multirow{2}{*}{**Average**} \\ \cline{2-2} \cline{4-5}  & **ASL** & **ASL with Digits** & & **NUS Hand Gesture** \\ \hline ImageNet21k & 99.94\% & 98.53\% & 99.15\% & 99.21\% \\ \hline ImageNet21k + ImageNet2012 & 99.97\% & 99.01\% & 99.85\% & 99.61\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results of weights experiment of ViT on the three benchmark datasets. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Fine Tune Layers**} & \multicolumn{4}{c}{**Datasets**} & \multirow{2}{*}{**Average**} \\ \cline{2-2} \cline{4-5}  & **ASL** & **ASL with Digits** & & **NUS Hand Gesture** \\ \hline
3 & 99.93\% & 97.46\% & 98.20\% & 98.53\% \\ \hline
4 & 99.95\% & 98.57\% & 99.15\% & 99.22\% \\ \hline
5 & 99.97\% & 98.69\% & 99.05\% & 99.24\% \\ \hline
6 & 99.98\% & 98.77\% &



To perform knowledge distillation on the pretrained vision transformer (ViT), a series of experiments were carried out to identify the best hyperparameter setup. The teacher and student models used in the knowledge distillation process have the same configurations as the pretrained ViT experiments but with different fine-tuning layers. The student model is a scaled-down version of the ViT, while the teacher model is a larger ViT. The model was then configured using the rectified Adam optimizer, utilizing a learning rate of 0.0001. Accuracy was used as the evaluation metric. The student loss function was categorical cross-entropy with label smoothing of 0.2, and the distillation loss function was Kullback-Leibler divergence, with an alpha value of 0.1 and a temperature of 10. Additionally, early stopping and reduce learning rate on a plateau were implemented during the training phase. The first experiment for knowledge distillation focused on comparing teacher and student models with different fine-tuning layers, as more fine-tuning layers result in a more complex model. Various combinations of teacher-student models were tested, including models with more, fewer, or the same number of fine-tuning layers. The best results were obtained when the teacher model had 12 fine-tuning layers and the student model had 11 fine-tuning layers. With this setup, the model achieved accuracy scores of 99.99% on the ASL dataset, 99.36% on the ASL with digits dataset, and 99.90% on the NUS hand gesture dataset. On average, across all three datasets, the accuracy reached 99.75%.