# Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
## 4 Experiments
### Comparing the Results  
Table 3 illustrates a comprehensive comparison of state-of-the-art compression methods applied to a diverse set of teacher models, including ResNet-101, ResNet-152, DenseNet-169, DenseNet-161, EfficientNet-b1, EfficientNet-b3, and EfficientNet-b5. Our method, labeled \"channel pruning\" and \"weight pruning\" [1] exhibits compelling results, outperforming seven benchmark techniques encompassing two pruning methods ([2,10]), two knowledge distillation approaches ([11,13]), and two hybrid compression methods ([17,21]). We performed evaluations on our own proprietary models and datasets due to the lack of previous results for these specific configurations. This approach ensures that performance evaluation is tailored to our distinct context and emphasizes the compatibility and effectiveness of each compression method. The superiority of our method across a range of teacher models underlines its efficiency in achieving significant compression results.  
Quantitative evaluation, measured in terms of parameter reduction (\(\times 10^{7}\)), consistently demonstrated the superiority of our method over most teacher models. Notably, our approach achieved the best results in the ResNet and DenseNet architectures and demonstrated its effectiveness in compressing these models. In the case of EfficientNet-b1, our method closely competed with the performance of the [17] approach, indicating comparable efficiency. These results highlight the robustness and efficiency of our proposed compression method and emphasize its potential for model reduction while maintaining  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Teacher** & **Res101** & **Res152** & **Dens169** & **Dens161** & **Effici b1** & **Effici b3** & **Effici b5** \\ \hline
**(Channel pruning) \(\times\)\(10^{7}\)** & 0.8 & 1.8 & 0.35 & 1.1 & 0.45 & 0.27 & **1.6** \\ \hline (Weight pruning) [1]\(\times\)\(10^{7}\) & **0.32** & **0.37** & **0.18** & **0.29** & 0.27 & **0.15** & **0.8** \\ \hline \([2]\times 10^{7}\) & 1.87 & 2.42 & 0.41 & 1.95 & 0.52 & 0.43 & **1.92** \\ \hline \([10]\times 10^{7}\) & 1.12 & 2.1 & 0.36 & 1.76 & 0.51 & 0.38 & **1.81** \\ \hline \([11]\times 10^{7}\) & 2.23 & 1.2 & 0.68 & 1.52 & 0.58 & 1 & **2.4** \\ \hline \([13]\times 10^{7}\) & 3.5 & 2.8 & 0.71 & 1.43 & 0.65 & 1.1 & **2.51** \\ \hline \([17]\times 10^{7}\) & 0.81 & 1.15 & 0.37 & 1.09 & **0.25** & 0.3 & **1.54** \\ \hline \([21]\times 10^{7}\) & **0.74** & **0.83** & **0.32** & **0.85** & **0.43** & **0.21** & **1.67** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison number of parameters on CIFAR-100.  
or even improving performance, especially in our specific models and datasets where no previous results were available. The pivotal factor in this success was the strategic use of knowledge distillation (KD) before pruning. This approach not only provided more flexibility in compression options but also demonstrated that, when it comes to weight versus channel pruning, weight pruning in conjunction with KD yielded superior compression.  
In summary, our experiments showcase a novel and highly effective compression strategy--distilling models to lower architectures, followed by weight pruning. This approach not only outperforms prior methods but also highlights how using knowledge distillation (KD) before pruning is a powerful way to achieve the best compression results.