# A Robust and Efficient Method for Effective Facial Keypoint Detection
## 3 Methods
### Model Compression
#### 3.5.1 Knowledge Transfer  
Following the above step, the teacher CNN (Re) is rendered static, and all the following optimization efforts are exclusively focused on the small CNN Mo. Given its inherently reduced parameter space, Mo inherently possesses a more constrained capacity for keypoint detection, translating to a comparatively diminished accuracy. To address this performance  
Figure 2: Thispicture shows the three stages of the model pruning network architecture combined with distillation: (**a**) in the model initialization stage, the input image is processed through the three convolutional layers of the initial network structure (large, medium, and small purple blocks) to extract features and output the prediction results (thin green bars); (**b**) in the model distillation stage, the input image is processed through the three convolutional layers after distillation (large, medium, and small blue blocks), the prediction results are output, and the outputs of the initial network and the distillation network are connected by a red dotted line to calculate the distillation loss; (**c**) in the model pruning stage combined with distillation, the input image is processed through the three convolutional layers pruned layer by layer (large, medium, and small yellow blocks), the prediction results are output, and the outputs of the distillation network and the pruned network are connected by a red dotted line to calculate the pruning loss and guide the pruning process.  
disparity, we introduce a mechanism aimed at transferring the keypoint detection to the smaller model. This approach endeavors to bridge the accuracy gap, empowering Mo to achieve a heightened level of precision that more closely aligns with that of Re.  
Research has underscored the notion that the similarity of feature maps between a small and a large CNN, given identical input imagery, correlates strongly with the similarity of their predictions, thereby implying comparable capabilities. Leveraging this insight, we devise a transfer methodology aimed at aligning the feature maps of both networks at a designated layer. Specifically, the teacher model's backbone produces a feature map \(\mathbf{M}_{r}\in\mathbb{R}^{B\times C_{Lr}\times H_{Lr}\times W_{Lr}}\), whereas the student model's backbone yields \(\mathbf{M}_{m}\in\mathbb{R}^{B\times C_{Lm}\times H_{Lm}\times W_{Lm}}\). The disparity in these dimensions poses a challenge for direct knowledge transfer.  
To overcome this obstacle, we employed average pooling to homogenize the dimensionality of both \(\mathbf{M}_{r}\) and \(\mathbf{M}_{m}\), yielding reduced representations \(\mathbf{M}_{r}^{\prime}\in\mathbb{R}^{B\times C_{Lr}}\) and \(\mathbf{M}_{m}^{\prime}\in\mathbb{R}^{B\times C_{Lm}}\), respectively. Furthermore, we utilized \(1\times 1\) convolutional layers to ensure that the vectors \(\mathbf{M}_{r}\in\mathbb{R}^{B\times C_{La}}\) and \(\mathbf{A}_{m}\in\mathbb{R}^{B\times C_{Lm}}\), derived from these reduced feature maps, share a common vector space dimensionality.  
Finally, we adopted mean squared error (MSE) as the loss function, denoted as loss\({}_{3}\), to quantify the discrepancy between corresponding elements in vectors \(\mathbf{a}_{r}\) and \(\mathbf{a}_{m}\) belonging to \(\mathbf{A}_{r}\) and \(\mathbf{A}_{m}\). This loss function computes the squared differences, facilitating the optimization process aimed at minimizing the divergence between the feature representations of the teacher and student models.  
\[\textit{loss}_{3}=\min\frac{1}{B\times C_{Lm}}\sum_{i=1}^{B}\sum_{j=1}^{C_{Lm} }(a_{r_{ij}}-a_{m_{ij}})^{2} \tag{6}\]