# A Robust and Efficient Method for Effective Facial Keypoint Detection
## 4 Experimental Results
### Main Results
#### 4.2.3 Distillation Combined with Pruning Results  
To show how effective it is to combine distillation with pruning in face keypoint detection, we carried out multiple experiments and compared the detection results of various model combinations. These combinations included regression + heatmap, regression + heatmap + distillation, and regression + heatmap + distillation + pruning.  
As shown in Table 4, a teacher model was developed and evaluated using the regression + heatmap approach, achieving an MSE LOSS of 0.0457. In parallel, a student model was created and assessed with the same regression + heatmap approach, resulting in an MSE LOSS of 0.0462.  
To enhance the student model's accuracy, we incorporated a distillation step into the regression + heatmap method, transferring knowledge from the teacher model to the student model. This process lowered the MSE LOSS of the student model to 0.0434. The process of knowledge distillation enabled the lightweight student model to better learn the deep features from the teacher model, thereby improving its detection accuracy. Building on this, we further incorporated a pruning step, pruning 50% of the student model to remove redundant parameters. The experimental results showed that the MSE LOSS of the pruned student model was 0.0445. Although there is a slight increase, compared to the unpruned student model, the pruned model significantly reduced computation and model size while maintaining high precision, thus improving the model efficiency. The improvement is attributed to the combination of distillation and pruning, which allows the network to be trained and optimized with minimal accuracy loss. Distillation helps the lightweight models learn more powerful representations, while pruning optimizes the model structure, making it more efficient and lightweight.  
To assess the generalization capability of various algorithms, including regression + heatmap on ResNet-50, regression + heatmap on MobileNet, regression + heatmap + distillation on MobileNet, and regression + heatmap + distillation + pruning (50%) on MobileNet, we conducted a three-fold cross-validation. This method involved pairwise  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Backbone** & **Algorithm** & **MSE** & \(t\)**-Test/\(p\)-Value** & **Levene-Test/\(p\)-Value** \\ \hline \multirow{2}{*}{MobileNet} & \multirow{2}{*}{regression + heatmap} & \multirow{2}{*}{0.0556} & T: 0.52, 0.97, 0.44 & L: 0.73, 1.23, 0.06 \\  & & & P: 0.59, 0.32, 0.65 & P: 0.39, 0.26, 0.80 \\ \multirow{2}{*}{MobileNet} & \multirow{2}{*}{regression + heatmap} & \multirow{2}{*}{0.0462} & T: 0.64, 0.93, 0.29 & L: 0.03, 1.87, 1.28 \\  & & & P: 0.52, 0.34, 0.76 & P: 0.84, 0.34, 0.25 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of the performance between regression alone and regression combined with heatmap.  
combinations of three different test sets, thus yielding three validation values and corresponding \(p\)-values for each set of experiments. Performance metrics were collected for each fold to ensure a thorough evaluation. We applied the \(t\)-values to examine the mean differences in performance metrics across the folds and employed the Levene test to check for homogeneity of variances. The large \(p\)-values from both the \(t\)-test and Levene test indicated that the mean performance differences were not statistically significant and the variances were homogeneous.