# SDViT: Stacking of Distilled Vision Transformers for Hand Gesture Recognition
## 3 Stacking of Distilled Vision Transformers (SDViT)
### Knowledge Distillation  
Knowledge distillation is a powerful method for compressing models, allowing the transfer of expertise from a large, intricate model to a smaller one throughout training. This method is commonly known as the teacher-student model, with the larger model assuming the role of the teacher and the smaller model functioning as the student. The motivation behind knowledge distillation stems from the fact that large models possess a significant amount of knowledge capacity due to their numerous layers and parameters. However, this knowledge is not fully utilized when deploying these large models to devices that have limited memory and computational resources. To overcome this limitation, a model compression technique was initially proposed by [38] and later generalized by [39] as knowledge distillation. The fundamental concept of knowledge distillation is depicted in Figure 4, which illustrates the network architecture used in the proposed SDViT approach. Both the teacher and student models utilized in the knowledge distillation process are based on transfer learning from pretrained vision transformers, with varying fine-tuning layers. During training, the teacher model transfers its knowledge to the student model by guiding its learning process. This knowledge transfer is achieved by instructing the student model to replicate not only the final predictions made by the teacher model but also the internal representations and patterns learned by the teacher. Through this distillation of knowledge from the teacher model, the student model can capture and leverage the valuable insights and decision-making capabilities of the larger model while maintaining a compact size. This enables the student model to achieve comparable performance to the teacher model, making it a highly efficient solution for real-world applications. In this paper, response-based knowledge distillation is employed. The teacher model's output probabilities or logits are utilized as the knowledge, which provide valuable information about the confidence or certainty of the model's predictions. By incorporating this response-based knowledge into the training process of the student model, it can learn to mimic the teacher's decision-making process and improve its own predictive capabilities. In addition, we adopt the widely used offline distillation approach. Offline distillation involves employing a pretrained teacher model to steer and inform the training of the student model through knowledge transfer. This method offers ease of implementation and is commonly employed in deep learning applications. During the knowledge distillation process, knowledge transfer from the teacher model to the student model takes place through the minimization of a loss function. The objective in this process is to match the class probability distribution generated by the teacher model, which is derived by applying a SoftMax function to the teacher model's logits. However, the teacher model's probability distribution often assigns a high probability to the correct class while assigning close to zero probabilities to other classes. This limited distribution fails at offering extra information beyond the actual labels present in the dataset. To overcome this limitation, SoftMax temperature [39] is employed to calculate the probability \(p_{i}\) of class \(i\) directly from the logits \(z\) using the equation:  
\[p_{i}=\frac{\exp\left(\frac{Z_{i}}{T}\right)}{\sum_{j}\exp\left(\frac{Z_{j}}{T }\right)} \tag{5}\]  
This equation represents the calculation of class probabilities, where the temperature parameter \(T\) plays a crucial role. A temperature of one yields the standard SoftMax function, while increasing the value of \(T\) results in a softer probability distribution. This softer distribution provides more information about which classes the teacher model considers similar to the predicted class.