# Improving the Robustness of Model Compression by On-Manifold Adversarial Training
## 3 Background
### Model Compression Based on Sparse Coding  
Suppose that \(f(\cdot;w):\mathbb{R}^{d}\rightarrow\mathbb{R}^{K}\) is a classifier parametrized by the learning weight \(w\in\mathbb{R}^{p}\), whose outcome is the scores indicating the likelihood of a given input to be each of the \(K\) classes. Given a dataset \(\{(x_{i},y_{i})\}_{i=1}^{n}\) of \(n\) input points \(x_{i}\in\mathbb{R}^{d}\) and its labels \(y_{i}\in\{1,2,\ldots,K\}\), we solve the following optimization problem in sparse coding [46; 47],  
\[w^{*}\in\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{i=1}^{n }\ell(f(x_{i};w),y_{i})+\Psi(w),\]  
where \(\ell(f(x_{i};w),y_{i})=-\log(f(x_{i};w)y_{i})\) is the cross-entropy between the classifier's outcome and the labels with \(K\) classes, and \(\Psi:\mathbb{R}^{p}\rightarrow\mathbb{R}\) is a (usually) convex function used for inducing certain structure in \(w\): a popular choice is \(\Psi(w)=\lambda\|w\|_{1}\) with \(\lambda>0\), which induces element-wise sparsity in \(w^{*}\)[1].  
In model compression, the \(\ell_{1}\)-norm based sparse coding has a practical issue in that it is difficult to determine the value of the hyperparameter \(\lambda\), which corresponds to a specific compression rate. As a result, one would need to solve the above optimization problem for several values of \(\lambda\)'s until he or she finds the one for the desired compression rate. Therefore, \(\ell_{0}\)-norm based sparse coding has been popular recently using \(\Psi(w)=\lambda^{\prime}\|w\|_{0}\) where \(\|w\|_{0}\) is the count of nonzero elements in \(w\): in this case, the value of \(\lambda^{\prime}\) can be computed during optimization to achieve a specific rate of model compression [20].  
Although we can set the values of unimportant weights to zero using sparse coding, this usually does not entail a reduction in memory usage and computation automatically. Since sparse tensor operations are not fast on GPUs [2], it will be better for the resulting sparse weights to have certain patterns in the locations of zero values so that we can create dense weight tensors of reduced sizes. This is the perspective taken in the filter-wise pruning method [4], where sparsity is induced by the unit of convolution filters. We can write the filter-wise pruning as a general form of group-wise sparse coding based on the \(\ell_{0}\)-norm, where we can define groups of optimization variables that will be selected or removed altogether by the mechanism:  
\[w^{*}\in\operatorname*{arg\,min}_{w\in\mathbb{R}^{p}}\frac{1}{n}\sum_{i=1}^{n }\ell(f(x_{i};w),y_{i})+\lambda^{\prime\prime}\sum_{g=1}^{G}\mathbb{I}_{\|w_ {g}\|_{2}\
eq 0}. \tag{1}\]  
Here, \(\lambda^{\prime\prime}>0\) is a hyperparameter controlling the sparsity of the model, \(\mathbb{I}_{[e]}\) represents the indicator function whose value is \(1\) if the statement \(e\) is true and \(0\) otherwise, and \(w_{g}\) denotes the group subvector corresponding to the group indexed by \(g\). This problem can be solved efficiently using algorithms, such as the alternating direction method for multipliers (ADMM) [17; 18] and proximal gradient methods [20].