# A Robust and Efficient Method for Effective Facial Keypoint Detection
## 3 Methods
### Model Compression
#### 3.5.2 Model Cutting  
The overarching goal of this endeavor is to enhance the network's efficiency by diminishing its parameter count, subsequently leading to a reduction in computational complexity. In the context of knowledge distillation, it is imperative that the parameter disparity between the teacher and student CNNs remains within a manageable threshold. This is due to the inherent learning constraints of the smaller CNN, where an excessively wide gap can hinder the effective transfer of knowledge. Consequently, the size of the student CNN cannot be overly diminutive, presenting a paradoxical challenge: maximizing efficiency while ensuring that the model does not sacrifice too much in terms of accuracy. To reconcile this dilemma, we introduce a model pruning or \"cutting\" strategy, tailored to strike an optimal balance between accuracy preservation and efficiency gains. This approach aims to meticulously identify and eliminate redundant or less impactful parameters, enabling the small CNN to achieve a highly efficient footprint while still retaining sufficient representational power to deliver acceptable levels of prediction accuracy. We take the layer \(t\) in the network architecture as an example. The process of parameter reduction can be conceptualized as the elimination of insignificant filters within the tensor \(\mathbf{Z}_{t}\in\mathbb{R}^{C_{t}\times C_{t-1}\times s\times s}\), essentially diminishing the count of filters in this layer. To quantify the significance of each filter, a logical approach involves assessing its impact on the subsequent feature maps, where less influential filters exhibit reduced effects. Specifically, upon hypothetically removing a filter, the dimensionality of \(\mathbf{Z}_{t}\) would adjust to \(\mathbb{R}^{C_{t-1}\times C_{t-1}\times s\times s}\), with corresponding alterations in the feature maps \(\mathbf{M}_{t}\in\mathbb{R}^{C_{t-1}\times H\times W}\) and the subsequent convolutional kernel \(\mathbf{Z}_{t+1}\in\mathbb{R}^{C_{t+1}\times C_{t-1}\times s\times s}\), leading to an altered feature map \(\mathbf{M}_{t+1}\in\mathbb{R}^{C_{t+1}\times H\times W}\). To objectively evaluate the importance of each filter, we leverage a subset of images and employ the adjacent, unmodified feature map \(\mathbf{M}_{t+1}\) as a reference. By sequentially removing each filter and computing the resultant reconstruction error for the reference map, we can infer the significance of each filter. This methodology enables us to identify and prune those filters that contribute minimally to the overall representation, thereby refining the model's efficiency without unduly compromising its accuracy. \[\mathit{Score}^{Ci}=\frac{\sum_{l=1}^{L}\left\|\mathbf{M}_{t+1}^{Ci}(s_{l})-\mathbf{ M}_{t+1}(s_{l})\right\|_{2}^{2}}{L} \tag{7}\]  
Within the sampled subset \(\{s_{l}|l=1,\ldots,L,L\ll N\}\), where \(L\) is significantly smaller than the number of training images \(N\), \(\mathbf{M}_{t+1}^{i}(s_{l})\) denotes the transformed representation subsequent to the removal of the \(i\)-th filter from the set of filters at layer \(t\). The significance metric is derived by assessing the impact of eliminating the \(i\)-th filter from \(\mathbf{Z}_{t}\). By arranging these significance metrics in descending order, we can systematically prune filters with lower scores, guided by a predetermined pruning ratio \(\alpha\). Specifically, we discard \(\left\lceil\mathbf{Z}_{t}\times\alpha\right\rceil\) filters while retaining \(\left\lceil\mathbf{Z}_{t}\times(1-\alpha)\right\rceil\) filters, thereby generating a refined set of filters for the \(t\)-th layer, denoted as \(\mathbf{Z}_{t}\in\mathbb{R}^{\left\lceil\mathbf{C}_{t}\times(1-\alpha) \right\rceil\times\mathbf{C}_{t-1}\times 5\times 5}\). This process ensures that the new filter configuration at each layer is optimized for both efficiency and performance considerations.