## 4 Experiments
### Second-Stage: Relational Knowledge Distillation-Based Pig Face Recognition
#### 4.2.2 Implementation Result  
To validate the effectiveness of our proposed method that integrated the knowledge distillation technique with the lightweight student model, we conducted experiments comparing the performance of the student model, both with and without the distillation from the teacher model. The teacher model used was a Vision Transformer (ViT-base), and the student model was ShuffleNet-V2. The results of these experiments are summarized in Table 2.  
The ViT-base teacher model (T) achieved high performance based on all metrics, with an Accuracy of 0.972, Precision of 0.978, Recall of 0.970, and an F1 Score of 0.974. These metrics set a benchmark for the student model to aspire to through the distillation process. The FLOPs and parameters for the teacher model are denoted as XX, indicating their substantial computational requirements. The ShuffleNet-V2 student model, when trained independently without knowledge distillation, reached an Accuracy of 0.941, Precision of 0.943, Recall of 0.936, and F1 Score of 0.944. These results, while commendable for a lightweight model, suggested room for improvement in mirroring the teacher model's performance. Applying Relational Knowledge Distillation (RKD) to the ShuffleNet-V2 student model yielded a marked improvement in performance. The distilled student model (RKD) achieved an Accuracy of 0.968, Precision of 0.972, and Recall of 0.970, with an F1 Score of 0.973. Notably, the distilled student model attained this enhanced performance while maintaining the same level of computational efficiency as the undistilled model, with 0.15 billion FLOPs and 1.26 million parameters, demonstrating the efficacy of the RKD technique in transferring the teacher's knowledge to the student model effectively. These results showcase the potential of knowledge distillation as a method to enhance the performance of lightweight models, enabling them to perform complex tasks with a high degree of accuracy while operating within the constraints of limited computational resources. Figure 10 presents training and Accuracy curves that depict a compelling narrative of the efficacy of our distillation approach in pig face recognition. Notably, the student model, ShuffleNet-V2 with Relational Knowledge Distillation (RKD-S), was trained without leveraging pre-trained weights from public datasets, which is often a common practice to accelerate the initial learning phase. This deliberate choice is reflected in the initial lower accuracy rates compared to the non-distilled ShuffleNet-V2, which benefits transiently from such pre-trained weights. The left graph exhibits the training loss for both models. The RKD-S shows a steep decline from a high initial loss, indicating an intense early learning phase, and quickly stabilizes, demonstrating rapid convergence. This behavior underscores the distilled model's swift adaptation to the intricacies of the task at hand, mimicking the high-level feature interrelations learned from the teacher model, ViT-base (T). On the right, the Accuracy curves tell a story of transformation and rapid improvement. After the  
initial epochs, the RKD-S model's accuracy surges ahead of the non-distilled ShuffleNet-V2, showcasing the distilled model's rapid assimilation of the teacher's knowledge. The student model's performance not only overtakes its non-distilled counterpart, but also exhibits a clear trend toward the teacher model's high-accuracy benchmark. The accuracy of the ViT model depicted in the graph represents the zenith of our training efforts with the ViT model, serving as the ideal teacher model's weights for the distillation process. Figure 11's confusion matrices reveal the performance differences between the Vision Transformer (ViT), ShuffleNet-V2, and ShuffleNet-V2 with Relational Knowledge Distillation (RKD). ViT shows high classification accuracy, consistent with its advanced design. ShuffleNet-V2 displays some misclassifications, as expected from a lightweight model.