# Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
## 2 Related Works
### Knowledge Distillation  
Knowledge distillation (KD), as introduced by [PERSON] et al. [2], constitutes a pivotal concept in which the understanding of a larger, more complex model (teacher) is imparted to a smaller counterpart (student). The process involves utilizing the teacher's soft probabilities (logits) as \"soft targets\" during student training, steering its learning trajectory. This strategy showcases that despite the student's compactness and computational efficiency, it can achieve comparable or even superior performance to the teacher. Attention transfer (AT), presented by [PERSON] and [PERSON] [3], introduces attention transfer within knowledge distillation. The integration of attention maps directs the focus toward significant regions in the input data. By orchestrating the student's learning to mimic the attention patterns exhibited by the teacher, the student's generalization and overall performance can be enhanced. Variational information distillation (VID), an innovation by [PERSON] et al. [4], injects the realm of variational distributions into knowledge distillation. The student model assimilates knowledge from the teacher while accounting for the uncertainty inherent in the teacher's predictions. By considering the variance in the teacher's logits, the student captures both the central tendency and the confidence level associated with the teacher's assessments. \"FitNets: Hints for Thin Deep Nets\", proposed by [PERSON] et al. [5], used hints or intermediate representations from a deeper teacher network to guide the training of a shallower student network. The FitNet architecture is designed to facilitate the distillation process, allowing the student network to learn not only from the final output of the teacher network but also from intermediate representations. This approach helps transfer knowledge more effectively, particularly in scenarios where the teacher network is significantly deeper than the student network. Contrastive representation distillation (CRD), proposed by [PERSON] et al. [6], brings contrastive learning into the fold of knowledge distillation. The alignment of teacher and student models employs contrastive loss, leveraging positive pairs from the same class and negative pairs from distinct classes. This fosters the student's ability to encapsulate distinguishing features while upholding intra-class coherence and inter-class divergence. Similarity-preserving knowledge distillation, as outlined by [PERSON] et al. [7], introduces a paradigm where the student's learning is guided by preserving the pairwise similarities inherent in the teacher's features. This meticulous preservation of relative feature similarities empowers the student to capture the nuanced intricacies present in the teacher's representations. Self-residual representation learning (SRRL), brought forth by [PERSON] et al. [8], advances knowledge distillation with a focus on self-residual representations. The teacher's self-residuals serve as a conduit for amplifying the knowledge transfer process. The student's training entails learning to harness the teacher's self-residuals, leading to an elevation in its generalization capabilities. \"Feature Normalized Knowledge Distillation for Image Classification\" (FN), proposed by [PERSON] et al. [9], presents an innovative approach to address label noise in the context of knowledge distillation (KD). The authors systematically analyze the impact of one-hot label encoding noise on the L2-norm of penultimate layer features, noting its potential distortion. They propose a novel feature normalized knowledge distillation method, introducing sample-specific correction factors to replace the conventional temperature parameter in KD. Semantic conditioned knowledge distillation (SemCKD), introduced by [PERSON] et al. [10], injects the concept of semantic conditioning into knowledge distillation. The student's learning journey is molded to emphasize specific semantic facets present in the teacher's predictions.