# Robust CNN Compression Framework for Security-Sensitive Embedded Systems
## 3 Methods
### Adversarial Pruning with Distillation  
Adversarial training is a type of robust optimization procedure which can be stated by the following min-max problem:  
\[w^{*}_{den}\in\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{ i=1}^{n}\max_{\delta\in\Delta}\mathcal{L}(f(x_{i}+\delta;w),y_{i}). \tag{4}\]  
To solve the inner maximization problem of (4), we consider the projected gradient descent (PGD) attack method [6] with an \(\ell_{\infty}\)-norm feasible set. For a given data pair \((x,y)\), the PGD attack is defined as follows:  
\[x^{t+1}=\Pi_{\mathcal{B}(x,\epsilon)}(x^{t}+\alpha\cdot\operatorname*{sgn}( \
abla_{x^{t}}\mathcal{L}(f(x^{t}+\delta;w),y))). \tag{5}\]  
Here, \(\Pi_{\mathcal{B}(x,\epsilon)}\) is a projection operation to the \(\ell_{\infty}\)-norm ball around \(x\) defined as \(\mathcal{B}(x,\epsilon):=\{x+\delta:\|\delta\|_{\infty}\leq\epsilon\}\). Let us note that uniformly distributed random noise is added to \(x\) in the initial stage of the PGD attack to prevent the label leaking problem [23]. The solution of (4) which we denote as \(w^{*}_{den}\) is generally non-sparse since there is no sparse constraint on this optimization problem. By adding a sparse regularization term to (4), we can obtain the objective of adversarial pruning,  
\[w^{*}_{spa}\in\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{ i=1}^{n}\max_{\delta\in\Delta}\mathcal{L}(f(x_{i}+\delta;w),y_{i})+\lambda\|w\|_{0}, \tag{6}\]  
where \(\lambda>0\) is a hyperparameter to control the sparsity of \(w\).  
Generally, the solution of (1), denoted by \(w^{*}\), is used as initial weights for solving (6). Here, our question is how we effectively preserve the accuracy of \(w^{*}\) on original inputs during adversarial pruning procedure. The accuracy on the original inputs is largely dropped during the adversarial pruning procedure since the one-hot encoded label \(y_{i}\) in (6) does not contain any information about the decision boundary of \(w^{*}\).  
To consistently provide the information of \(w^{*}\) during pruning, we combine the knowledge distillation idea with adversarial pruning. In our method, the pretrained network works as a teacher and provides SoftMax output \(f^{t}(:w^{*})\) on original input during adversarial pruning procedure. The proposed objective is formulated as follows:  
\[\min_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{i=1}^{n}(1-\alpha)\mathcal{L}(f(x_{ i}+\delta;w),y_{i})+\alpha t^{2}\mathcal{L}(f^{t}(x_{i}+\delta;w),f^{t}(x_{i};w^{* }))+\lambda\|w\|_{0}. \tag{7}\]Here, \(\delta\) is the solution of (3) and \(t\) is a distillation hyperparameter [21]. The \(t^{2}\) is multiplied in front of the second term to prevent the shirking of gradient problem [21]. The second term in (7) is distillation loss which indicates the cross-entropy between SoftMax output of the currently pruned model \(f(\cdot;w)\) and the teacher model \(f(\cdot;w^{*})\). The overall formulation of (7) can be interpreted as the linear combination of the adversarial pruning loss (6) and the distillation loss. By solving (7), we can obtain a sparse but robust solution that approximates the decision boundary of \(w^{*}\). Our framework can be extended for filter pruning by replacing the third regularizer term with the number of non-zero filters as follows:  
\[\min_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{i=1}^{n}(1-\alpha) \mathcal{L}(f(x_{i}+\delta;w),y_{i})+\alpha t^{2}\mathcal{L}(f^{t}(x_{i}+ \delta;w),f^{t}(x_{i};w^{*}))\] \[+\lambda\sum_{g=1}^{G}\mathbb{1}\{\|w_{g}\|_{2}\
eq 0\}.\]  
Here, \(G\) is the number of filters and \(w_{g}\) is the weight vector of \(g\)th filter.