# Robust CNN Compression Framework for Security-Sensitive Embedded Systems
## 4 Experiments
### Comparison with the State-of-the-Art Methods  
To show the relative benefit of our method (denoted as APD) compared to other state-of-the-art methods, we also compared APD to Defensive Distillation [22] (denoted as DD), Filter Pruning via Geometric Median [15] (denoted as FPGM), and [PERSON] et al. [19]. The results are summarized at Table 5.  
Figure 1: The original accuracy and the adversarial accuracy of AP and APD (ours) with respect to the epoch of the element-wise pruning procedure for **(a)** LeNet, **(b)** VGG16, and **(c)** ResNet18. The left of each row is the result in the compression rate of \(\times 2\), the middle of each row is the result in the compression rate of \(\times 3\), and the right side of each row is the result in compression rate of \(\times 4\). The blue line means the original accuracy and the red line indicates the adversarial accuracy. The solid line is the result of APD and the dashed line is the result of AP. DD is a well-known defense strategy that generates a robust model by using knowledge distillation. It trains a teacher model with a high temperature value in a modified SoftMax output and then applies knowledge distillation to a student model whose architecture is the same as that of the teacher model. We compared the original accuracy and the adversarial accuracy of APD and DD with LeNet in the compression rate of \(\times 2\). For DD, we set the temperature \(t\) as 40 and the number of epochs as 100. In comparison, APD showed about 6% higher original accuracy and 10% higher adversarial accuracy than DD. FPGM is a SOTA filter pruning method that effectively prunes the redundant filters by measuring the Geometric Median [40] of each filter. To show that the pruning method only is not enough to generate sparse but robust solutions, we compared our pruned VGG16 with the compression rate of \(\times 1.5\) to FPGM's pruned VGG16 with the compression rate of \(\times 1.3\). APD showed 26.45% higher adversarial accuracy and 12.12% lower original accuracy compared to FPGM. The mean value of the original and the adversarial accuracy of APD is 61.82 and that of FPGM is 54.65. This result demonstrates that the model generated by the pruning method alone is vulnerable to adversarial attack. [PERSON] et al. is a SOTA robust pruning method. To solve the adversarial pruning (6) problem using alternative direction method of multipliers (ADMM), the method introduced two additional tensors for auxiliary parameters and Lagrangian multipliers. The size of those two tensors is exactly the same as the size of the weight parameters and therefore, it requires two times more memory than the memory required to store the weight parameters during the optimization procedure. On the other hand, APD solves our optimization problem (7) with the proximal gradient descent, which does not require any auxiliary tensor. We compared the result of APD and [PERSON] et al with LeNet and ResNet18. VGG16 was excluded in this comparison since the exact values of the original accuracy and the adversarial accuracy with VGG16 are not available in the original paper of [PERSON] et al. We set the compression rates to \(\times 2\), \(\times 4\), and \(\times 8\) for LeNet, and \(\times 2\) for ResNet18. With LeNet, APD slightly improved both original accuracy and adversarial accuracy over [PERSON] et al. in entire compression rates. With ResNet18, APD improved the original accuracy by 0.26% and the adversarial accuracy by 0.03% compared to [PERSON] et al. The adversarial robustness of APD appears to be similar to that of [PERSON] et al. ; however, APD requires far less memory that Ye et al. and therefore will be more suitable for generating robust models in memory-constrained environments as we discuss in the next section.