# Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression
## 4 Experiments
### Impact of Pruning Methods on Efficiency
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline
**Teacher, Student** & **Res101, Res52, Res50** & **Res152, Res34** & **Res101, Res 18** & **Dens169, Mobile** & **Dens161, Dense 169** & **Dens161, Dense 201** & **Effici b1, Effici b3, Effici b2** & **Effici b3, Effici b5, Effici b4** \\ \hline \multicolumn{11}{c}{**Weight Pruning 11**} \\ \hline
**ACC(t)** & 81.1 & 83.3 & 81.7 & 83.15 & 83.07 & 83.07 & 81.1 & 80.5 & 80.5 & 83.4 \\
**ACC(s)\({}_{\text{distillation}}\)** & 81.7 & 80.1 & 80.56 \(\pm\) 0.75 & 78.2 & 82.76 & 82.33 & 80.2 & 81.14 & 80.2 & 83.3 \\ \hline \multicolumn{11}{c}{**P(b)before pruning \(\times 10^{7}\)**} & 4.45 & 6.02 & 4.45 & 1.42 & 2.68 & 2.68 & 0.78 & 1.22 & 1.22 & 3.04 \\ \multicolumn{11}{c}{**P(s)before pruning \(\times 10^{7}\)**} & 2.56 & 2.13 & 1.17 & 0.42 & 1.42 & 2 & 0.56 & 0.91 & 0.42 & 1.93 \\ \hline \multicolumn{11}{c}{**ACC(b)\({}_{\text{basepoint}}\)**} & 82.7 & 83.15 & 83.32 & 83.53 & 82.86 & 83.53 & 81.62 & 80.52 & 80.12 & 83.1 \\ \multicolumn{11}{c}{**ACC(s)\({}_{\text{basepoint}}\)**} & 83.51 & 80.71 & 80.78 & 82.93 & 83.02 & 83.02 & 80.74 & 81.15 & 80.13 & 83.78 \\ \hline \multicolumn{11}{c}{**P(b) breakpoint \(\times 10^{7}\)**} & 0.91 & 1.1 & 0.91 & 0.25 & 0.38 & 0.58 & 0.37 & 0.47 & 0.22 & 1.4 \\ \multicolumn{11}{c}{**P(s) breakpoint \(\times 10^{7}\)**} & 0.6 & 0.51 & 0.34 & 0.16 & 0.2 & 0.41 & 0.27 & 0.27 & 0.18 & 0.8 \\ \hline \multicolumn{11}{c}{**Channel Pruning**} \\ \hline
**ACC(t)** & 81.1 & 83.3 & 81.7 & 83.15 & 83.07 & 83.07 & 81.1 & 80.5 & 80.5 & 83.4 \\
**ACC(s)\({}_{\text{distillation}}\)** & 81.7 & 80.1 & 80.65 \(\pm\) 0.75 & 78.2 & 82.76 & 82.33 & 80.2 & 81.14 & 80.2 & 83.3 \\ \hline \multicolumn{11}{c}{**P(b)before pruning \(\times 10^{7}\)**} & 4.45 & 6.02 & 4.45 & 1.42 & 2.68 & 2.68 & 0.78 & 1.22 & 1.22 & 3.04 \\ \multicolumn{11}{c}{**P(s)before pruning \(\times 10^{7}\)**} & 2.56 & 2.13 & 1.17 & 0.42 & 1.42 & 2 & 0.56 & 0.91 & 0.42 & 1.93 \\ \hline \multicolumn{11}{c}{**ACC(b)\({}_{\text{basepoint}}\)**} & 80.2 & 80.1 & 80.2 & 79.12 & 82.45 & 82.45 & 80.1 & 80.65 & 80.5 & 83.4 \\ \multicolumn{11}{c}{**ACC(s)\({}_{\text{basepoint}}\)**} & 81.7 & 80 & 79.56 & 80 & 81.3 & 82.33 & 80.3 & 79.1 & 80.2 & 83.3 \\ \hline \multicolumn{11}{c}{**P(b) breakpoint \(\times 10^{7}\)**} & 2.1 & 3 & 2.1 & 0.73 & 1.5 & 1.5 & 0.67 & 1 & 1 & 2.5 \\ \multicolumn{11}{c}{**P(b)\({}_{\text{basepoint}}\)**} & 2 & 1 & 0.6 & 0.33 & 0.75 & 1.58 & 0.43 & 0.75 & 0.27 & 1.6 \\ \hline \end{tabular}  
* P: number of parameters P(t): parameter count in the teacher; P(s): parameter count in the student. \({}^{2}\); P(s)\({}_{\text{bbefore}}\)/P(s)\({}_{\text{base}}\): number of parameters before pruning/number of parameters after pruning at the break point for the student. \({}^{3}\). Mobile: MobileNet.  
\end{table} 
Table 2: Accuracy and parameter count at breakpoint for CIFAR-10. emerge from the chart provide valuable insights into the effectiveness of each pruning method. Our investigation sheds light on how each method affects the efficiency of the models and offers valuable insights into their relative advantages and limitations. From the generated charts for both the CIFAR-10 and CIFAR-100 datasets, it is evident that the impact of pruning methods on efficiency varies across different student models. Weight pruning consistently results in improved efficiency compared to the other methods, with reductions in the number of parameters leading to better performance. This aligns with the expectation that weight pruning targets redundant parameters, leading to more efficient representations while maintaining accuracy. In the realm of channel pruning, it is evident that the efficiency of the pruned models surpasses that of the distilled student models, albeit by a relatively modest degree. However, a more pronounced distinction emerges when comparing channel pruning to weight pruning. In this context, the efficiency achieved through weight pruning consistently outperforms that of channel pruning.