# Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
## 4 Experiments
### Pruning
A remarkable trend emerges where weight pruning consistently achieves levels of accuracy that are comparable across different combinations of teacher and student models, or even surpass the accuracy of the teacher models at the breaking point. For instance, consider the transition from ResNet 101 to ResNet 50. The student model attains an impressive 58.92% accuracy after weight pruning, outperforming both teacher and channel pruning accuracies, which stand at 54.34% and 56.78%, respectively. The findings of our analysis of channel pruning yield a notable observation. Across the range of student models analyzed, a consistent trend is observed: with the exception of the ResNet-50 architecture, pruned student models are less accurate than their corresponding teacher models. This model emphasizes the complexity of channel pruning and the importance of a delicate approach to balance reducing model complexity while maintaining predictive accuracy. Further exploration of the factors contributing to this divergence can provide valuable insights to refine channel pruning strategies and optimize the performance of different architectures. The distinct characteristics exhibited by weight pruning and channel pruning highlight the importance of careful and thoughtful evaluation when choosing between these two pruning methods. In the case of the ResNet 101 to ResNet 50 transition, after weight pruning, the student model's parameter count is only 0.5 \(\times\) 10\({}^{7}\), while after channel pruning, it amounts to 2.1 \(\times\) 10\({}^{7}\). Weight pruning shines as a method capable of optimizing both accuracy and efficiency. Its potential to surpass teacher accuracy while achieving substantial parameter reduction positions it as an attractive choice for applications where resource efficiency is paramount. On the other hand, channel pruning excels at producing lightweight models by drastically reducing parameter counts. While accuracy is sometimes compromised, this method is well-suited for environments with strict computational constraints. The choice between weight pruning and channel pruning should be guided by a thorough understanding of the desired balance between model accuracy, parameter reduction, and available resources. To illustrate the comparative effectiveness of these pruning techniques, we provide detailed results in Tables 1 and 2, comparing accuracy and parameter count at different pruning breakpoints for CIFAR-10 and CIFAR-100 across various models. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline
**Teacher,** & **Res101,** & **Res152,** & **Res101,** & **Dens169,** & **Dens161,** & **Dens161,** & **Effic bi1,** & **Effic bi3,** & **Effic bi3,** & **Effic bi5,** \\
**Student** & **Res50** & **Res34** & **Res18** & **Mobile** & **Dens 169** & **Dens 201** & **Effic bi0** & **Effic bi2** & **Mobile3** & **Effic bi5,** \\ \hline \multicolumn{11}{c}{**Weight Pruning [1]**} \\ \hline
**ACC(t)** & 54.07 & 52.2 & 54.07 & 57.2 & 58.2 & 59.2 & 58.1 \(\pm\) 0.5 & 58.1 & 59.5 \\
**ACC(t)** distillation & 59.12 & 51.3 & 55.6 & 52.1 & 56.45 & 54.78 & 57.1 & 55.87 & 52.07 & 59.4 \\ \hline
**P1(b)**before pruning \(\times\) 10\({}^{7}\) & 4.45 & 6.02 & 4.45 & 1.42 & 2.68 & 2.68 & 0.78 & 1.22 & 1.22 & 3.04 \\
**P2(s)**before pruning \(\times\) 10\({}^{7}\) & 2.56 & 2.13 & 1.17 & 0.42 & 1.42 & 2 & 0.56 & 0.91 & 0.42 & 1.93 \\ \hline
**ACC(t)**backpoint** & 54.34 & 47.48 & 53.15 & 57.95 & 57.71 & 58.94 & 60.17 & 60 & 60 & 60.06 \\
**ACC(s)**backpoint** & 58.92 & 53.47 & 55.25 & 55.1 & 57.61 & 55.23 & 58.63 & 55.91 & 54.8 & 59.8 \\ \hline
**P(b)**breakpoint \(\times\) 10\({}^{7}\) & 0.91 & 1.1 & 0.91 & 0.35 & 0.5 & 0.52 & 0.39 & 0.6 & 0.55 & 1.42 \\
**P(b)**backpoint \(\times\) 10\({}^{7}\) & 0.5 & 0.37 & 0.32 & 0.18 & 0.29 & 0.41 & 0.27 & 0.42 & 0.15 & 0.8 \\ \hline \end{tabular}
\end{table} 
Table 1: Accuracy and parameter count at breakpoint for CIFAR-100. Weight pruning often results in a notably reduced parameter count compared to channel pruning, with differences sometimes reaching as high as a factor of four.