# Robust CNN Compression Framework for Security-Sensitive Embedded Systems
## 4 Experiments
### The Effect of Knowledge Distillation
#### 4.1.2 Filter Pruning  
The filter pruning [11; 12; 13; 14; 15] generates the sparse patterns more adequate for GPU acceleration compared to the element-wise pruning [9; 10]. However, the sparsity that the filter pruning can achieve is often lower than that of element-wise pruning [39]. Therefore, we set the smaller compression rates of \(\times\)1.5, \(\times\)2, and \(\times\)2.5 than those of the element-wise pruning. As with element-wise pruning, we set the same sparsity for each convolution layer. We compared our method (APD) with the adversarial pruning (AP) to show the effectiveness of the knowledge distillation on the filter pruning. The results on MNIST and CIFAR10 are summarized at Tables 3 and 4, respectively.  
With LeNet, APD improved both original accuracy and adversarial accuracy in the entire compression rates. For instance, in the largest compression rate of \(\times\)2.5, APD improves the original accuracy by 0.36% and the adversarial accuracy by 1.44%. The improvement  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Network (Dataset)** & **Comp Rate** & **Method** & **Org Accuracy (\%)** & **Adv Accuracy (\%)** \\ \hline \multirow{6}{*}{LeNet (MNIST)} & \multirow{2}{*}{\(\times\)2} & AP & 97.82 & 92.83 \\ \cline{3-5}  & & APD & **98.83** & **95.11** \\ \cline{2-5}  & & AP & 90.94 & 72.71 \\ \cline{2-5}  & & APD & **98.55** & **94.51** \\ \cline{2-5}  & & AP & 94.45 & 71.63 \\ \cline{2-5}  & & APD & **98.48** & **94.25** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of element-wise pruning results of APD (ours) and AP on MNIST.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Network (Dataset)** & **Comp Rate** & **Method** & **Org Acc (\%)** & **PGD5** & **PGD10** \\ \hline \multirow{6}{*}{VGG16 (CIFAR10)} & \multirow{2}{*}{\(\times\)3} & AP & 81.67 & 49.20 & 40.91 \\ \cline{3-5}  & & APD & **82.44** & **50.63** & **42.24** \\ \cline{2-5}  & & AP & 80.72 & 48.70 & 40.87 \\ \cline{2-5}  & & APD & **81.77** & **50.30** & **42.31** \\ \cline{2-5}  & & AP & 79.69 & 48.77 & 40.97 \\ \cline{2-5}  & & APD & **80.57** & **49.93** & **42.06** \\ \hline \multirow{6}{*}{ResNet18 (CIFAR10)} & \multirow{2}{*}{\(\times\)3} & AP & 85.13 & 51.27 & 42.45 \\ \cline{2-5}  & & APD & **87.56** & **54.65** & **45.55** \\ \cline{1-1} \cline{2-5}  & & AP & 84.67 & 51.79 & 42.64 \\ \cline{1-1} \cline{2-5}  & & APD & **86.87** & **54.40** & **45.31** \\ \cline{1-1} \cline{2-5}  & & AP & 84.65 & 51.25 & 42.55 \\ \cline{1-1} \cline{2-5}  & & APD & **86.73** & **54.23** & **45.61** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of element-wise pruning results of APD (ours) and AP on CIFAR10.  
on the original accuracy tends to be smaller than the improvement on the adversarial accuracy since the original accuracy is already closed to that of the baseline network. APD also showed an improvement in both accuracy measures on the CIFAR10 dataset. With VGG16, APD improved the original accuracy significantly in high compression rate. For instance, in the compression rate of \(\times\)2.5, the original accuracy is improved by 5.23%. The adversarial accuracy against both PGD5 and PGD10 attacks is also improved by APD. In the compression rate of \(\times\)2.5, the adversarial accuracy increases by 2.09% against PGD5 attacks and 0.6% against PGD10 attacks. With ResNet18, APD also showed a consistent improvement on both original accuracy and adversarial accuracy in the entire compression rates. For instance, in the largest compression rate of \(\times\)2.5, APD improves the original accuracy by about 2% and adversarial accuracy by about 1% against both PGD5 and PGD10. Those results imply that the knowledge distillation in our method improves the adversarial robustness of the filter pruning solution.