# Robust CNN Compression Framework for Security-Sensitive Embedded Systems
## 3 Methods
### Optimization  
Most of the adversarial pruning approaches use the alternative direction method of multiplier (ADMM) method to solve the resulting optimization problem, for example, [PERSON] et al. [19] and [PERSON] et al. [20]. However, by construction, the ADMM requires using two additional tensors to the learning weights during optimization, which can be preventive on a resource-constrained environment with limited memory. Here, we suggest another algorithm based on the proximal gradient method to solve our proposed optimization problem (7) which does not require such auxiliary tensors. For simplicity, we denote the linear combination of two cross-entropy loss in (7) by \(\mathcal{L}_{\text{APD}}\):  
\[\mathcal{L}_{\text{APD}}(w)=\frac{1}{n}\sum_{i=1}^{n}(1-\alpha) \mathcal{L}(f(x_{i}+\delta;w),y_{i})+\alpha t^{2}\mathcal{L}(f^{t}(x_{i}+ \delta;w),f^{t}(x_{i}+\delta;w^{*})). \tag{8}\]  
Here, APD stands for adversarial pruning with distillation. Then we can rewrite (7) as  
\[\min_{w\in\mathbb{R}^{p}}\mathcal{L}_{\text{APD}}(w)+\lambda\|w \|_{0}. \tag{9}\]  
By applying a second order Taylor approximation on \(w_{k}\) and Hessian approximation with \(\
abla^{2}\mathcal{L}_{apd}(w_{k})\approx\frac{1}{\eta_{k}}I_{p\times p}\) for a \(\eta_{k}>0\) to (9), we obtain the following formulation:  
\[\mathcal{L}_{\text{APD}}(w)\approx\mathcal{L}_{\text{APD}}(w_{k})+\
abla \mathcal{L}_{\text{APD}}(w_{k})^{\top}(w-w_{k})+\frac{1}{2\eta_{k}}\|w-w_{k} \|^{2}.\]  
Here, \(I_{p\times p}\) indicates the identity matrix where the shape is \(p\times p\). Based on this successive approximation result, the weight update can be formulated as follows:  
\[w_{k+1}=\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\mathcal{L}_{\text{APD}} (w_{k})+\
abla\mathcal{L}_{\text{APD}}(w_{k})^{\top}(w-w_{k})+\frac{1}{2\eta_{ k}}\|w-w_{k}\|^{2}+\lambda\|w\|_{0}.\]  
By removing the redundant parts of the above weight update equation, we can obtain  
\[w_{k+1}=\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{2 \eta_{k}}\Big{(}2\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k})^{\top}w+\|w\|^ {2}-2w^{\top}w_{k}\Big{)}+\lambda\|w\|_{0}.\]  
We can rewrite the above equation as follows:  
\[w_{k+1}=\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{2 \eta_{k}}\Big{(}\|w\|^{2}-2w^{\top}(w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD }}(w_{k}))\Big{)}+\lambda\|w\|_{0}.\]By adding a constant \(\|w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k})\|^{2}\), we can obtain  
\[w_{k+1}=\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{2\eta_{k}}\big{(} \|w\|^{2}-2w^{\top}(w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k}))+\|w_{ k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k})\|^{2}\big{)}+\lambda\|w\|_{0}.\]  
Then, we can get the following equation:  
\[w_{k+1}=\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{2\eta_{k}}\|w-( w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k}))\|^{2}+\lambda\|w\|_{0}.\]  
This is exactly the form of proximal operator which is described as  
\[w_{k+1}=\text{prox}_{\eta_{k}\lambda\|w\|_{0}}(w_{k}-\eta_{k}\
abla\mathcal{L }_{\text{APD}}(w_{k})).\]  
For each element, proximal operator with \(\ell_{0}\) regularization term can be computed as  
\[(w_{k+1})_{i}= \begin{cases}(w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k} ))_{i},&|(w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k}))_{i}|>\sqrt{ \lambda}\\ 0,&|(w_{k}-\eta_{k}\
abla\mathcal{L}_{\text{APD}}(w_{k}))_{i}|\leq\sqrt{ \lambda}.\end{cases} \]  
It is simply the thresholding operation which sets the updated weight parameter smaller than \(\sqrt{\lambda}\) to zero. Let us note that by controlling the value of \(\lambda\), we can explicitly manipulate the sparsity of network. The entire process of our method is described at Algorithm 1.