# Robust CNN Compression Framework for Security-Sensitive Embedded Systems
## 4 Experiments
### The Effect of Knowledge Distillation
#### 4.1.1 Element-Wise Pruning  
Generally, the element-wise pruning ([PERSON] et al., 2014; [PERSON] et al., 2015) can achieve higher sparsity with only a few accuracy drops compared to the filter pruning ([PERSON] et al., 2014; [PERSON] et al., 2015; [PERSON] et al., 2015; [PERSON] et al., 2015). Therefore, we tested the element-wise pruning on the relatively high compression rates (\(\times\)2, \(\times\)3, \(\times\)4) compared to the filter pruning ([PERSON] et al., 2015). As in [PERSON] et al. (2019), we applied the same sparsity for every convolution layer in the target neural network. For instance, if the compression rate of a given network is determined to \(\times\)2, we set the fraction of zero weights in every layer of this network equal to 0.5. With this pruning scheme, we compared the element-wise pruning result of our method (7) with adversarial pruning (6). Both methods were optimized with proximal gradient descent. With this comparison, we demonstrated how much improvement was achieved by the knowledge distillation of our method. The results on MNIST and CIFAR10 are summarized at Tables 1 and 2, respectively.  
A popular small network LeNet ([PERSON] et al., 2017) is enough to achieve a high accuracy on the MNIST dataset. Our baseline LeNet, trained by the original training process achieves the original accuracy of 99.34% and the adversarial accuracy of 0%. With LeNet, our method (APD) showed a large improvement in both original accuracy and adversarial accuracy over the adversarial pruning (AP). In the compression rate of \(\times\)2, APD improved the original accuracy by 1.01 and the adversarial accuracy by 2.28% over AP. In the relatively high compression rate of \(\times\)3 and \(\times\)4, APD achieved a larger improvement in both original accuracy and adversarial accuracy. In particular, the amount of improvement in the adversarial accuracy achieved by APD in the compression rate of \(\times\)3 and \(\times\)4 was over than 20%. Compared to the baseline performance, APD achieved the compression rate of \(\times\)4 with the adversarial accuracy of 94.25% while reducing the original accuracy by about 1%.  
We also applied APD and AP to the two CNNs, VGG16 ([PERSON] et al., 2017) and ResNet18 ([PERSON] et al., 2017) with the CIFAR10 dataset. Achieving high adversarial robustness on the CIFAR10 dataset is more challenging since it requires a higher architectural capacity of the CNN compared to the MNIST dataset. Our baseline VGG16 achieved the original accuracy of 92.99% and the adversarial accuracy of 0%. Despite the difficulty, APD showed an improvement with VGG16 in the entire compression rates. For instance, in the compression rate of \(\times\)4, APD improved the original accuracy by 0.88% and the adversarial accuracy against both PGD5 and PGD10 by more than 1% over AP. Though ResNet18 consists of fewer parameters than VGG16 (11 M vs. 138 M), the generalization performance of Resnet18 for the CIFAR10 dataset is higher than that of VGG16. The baseline ResNet18 showed the original accuracy of 94.40% and the adversarial accuracy of 0.03%. With ResNet18, APD improved the original accuracy and adversarial accuracy against both PGD5 and PGD10 by more than 2% over AP in the entire compression rates. Based on those results, we can conclude that consistently providing the SoftMax output of the baseline CNN with the knowledge distillation improves the adversarial robustness of the element-wise pruning solution.