# Improving the Robustness of Model Compression by On-Manifold Adversarial Training
## 4 Methodology
### Model Compression with On-Manifold Adversarial Training  
Our proposal method is to combine the sparse coding technique for model compression (discussed in Section 3.1) and the PGD adversarial training (Section 3.3) using on-manifold adversarial examples (Section 4.1). Our purpose is to train compressed DNN models robust to natural noise: in particular, we are interested in a high-compression regime to produce models suitable for small computation devices.  
To run our method, denoted by MC-AT(on), we prepare an on-manifold dataset \(\{(x_{i},y_{i})\}_{i=1}^{m+n}\). First, we generate \(n\) on-manifold adversarial examples by manipulating the latent vector \(z\). Then, we select on-manifold adversarial examples by checking Definition 1. Finally, we combine selected \(m\) on-manifold adversarial examples and \(n\) original training examples.  
Using this data, we can formulate MC-AT(on) as the following \(\ell_{0}\)-norm based sparse coding problem:  
\[\text{MC-AT(on):}\quad\min_{w\in\mathbb{R}^{p}}\frac{1}{m+n}\sum_{i=1}^{m+n} \ell(f(x_{i};w),y_{i})+\lambda\|w\|_{0}, \tag{7}\]  
Figure 2: (**a**) Simulated latent samples; (**b**) On-manifold adversarial examples in the latent space. The distribution of simulated latent vectors (**a**) and the distribution of generated on-manifold adversarial examples with the simulation dataset (**b**). Each color represents a different class. The same number of examples (17,500 examples) are sampled from each of the four classes. We observe that on-manifold adversarial examples sit near the true decision boundary.  
where \(\lambda>0\) is a hyperparameter controlling the sparsity of the model. Algorithm 1 describes the overall process of MC-AT(on) algorithm based on the proximal gradient descent algorithm [20].  
```
target classifier \(f\) with weight vector \(w\), latent classifier \(f_{z}\), VAE consists of encoder \(\mathcal{E}\) and decoder \(\mathcal{D}\), train dataset \(\{(x_{i},y_{i})\}_{i=1}^{n}\) where \(x_{i}\in\mathbb{R}^{d}\), \(\eta\) for \(\ell_{\infty}\) norm bound of perturbation ; Pre-train target classifier \(f\): \(\min_{w}\ \ \ \mathbb{E}_{(x,y)}[\ell(f(x;w),y)]\) Train VAE: \(\min_{w_{\mathcal{L}},w_{\mathcal{D}}}\ \ \mathbb{E}_{z}[\|\mathcal{E}_{w_{ \mathcal{D}}}(\mathcal{D}_{w_{\mathcal{D}}}(z))-z\|_{2}^{2}]\) Train latent classifier \(f_{z}\): \(\min_{w_{z}}\ \ \mathbb{E}_{(x,y)}[\ell(f(\mathcal{E}(x);w_{z}),y)]\) Initialize on-manifold adversarial dataset: \(M_{\text{adv-on}}\leftarrow\emptyset\) for all\((x_{i},y_{i})\in\{(x_{i},y_{i})\}_{i=1}^{n}\)do  Compute a latent vector: \(z_{i}\leftarrow\mathcal{E}(x_{i})\)  Compute a perturbation: \(\tilde{\zeta}_{i}\leftarrow\arg\max_{\tilde{\zeta}_{i}}\ell(f(\mathcal{D}(z_ {i}+\zeta_{i});w),y_{i})\ \ s.t.\ \ \|\zeta_{i}\|_{\infty}\leq\eta\)  Add perturbation to the latent vector: \(z_{i}^{j}\gets z_{i}+\zeta_{i}\)  Generate an on-manifold adversarial example: \(x_{i}^{j}\leftarrow\mathcal{D}(z^{\prime})\) if\(f(x_{i}^{j})\
eq y_{i}\) and \(f_{z}(z_{i}^{j})=y_{i}\)then  insert \((x_{i}^{j},y_{i})\) to \(M_{\text{adv-on}}\) endif endfor \(M_{\text{adv-on}}\gets M_{\text{adv-on}}\cup\{(x_{i},y_{i})\}_{i=1}^{n}\)  Apply MC-AT(on) with (7)
```  
**Algorithm 1** Model compression by the on-manifold adversarial training algorithm