# Improving the Robustness of Model Compression by On-Manifold Adversarial Training
## 5 Experiments
### Robustness of Compressed Models on Off-Manifold Adversarial Test Data  
Figure 4 shows our evaluation of the five robust model compression methods against the adv-off test data. Since the goal of our study is to present a model compression method that can enhance the robustness against noises that may exist on the data manifold, which embraces observed data well, we briefly describe the adv-off test results. Overall, MC-TRADES showed improved robustness against adv-off on average from 7.82% to 13.06% at high compression rates (70% or more) compared to other methods.  
From the results, MC-TRADES and MC-AT(dual), which are model compression methods with adversarial training that consider adv-off, showed robustness against noise that left the manifold, that is, adv-off. Therefore, it can be said that MC-TRADES and MC-AT(dual) methods are more valid for model compression for robustness against adv-off when compared with other methods.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Sparsity**} & \multirow{2}{*}{**Original Accuracy**} & \multirow{2}{*}{**Method**} & \multicolumn{4}{c}{**Test Accuracy (\%)**} \\ \cline{5-8}  & & & & **Original** & **Adv-On** &
\begin{tabular}{c} **Combination** \\ **(Original \& Adv-On)** \\ \end{tabular} & \multirow{2}{*}{**Adv-Off**} \\ \hline \multirow{4}{*}{Simulation} & \multirow{4}{*}{95\%} & \multirow{4}{*}{97.20\%} & MC(regular) & 96.93 & 56.50 & 76.72 & 16.32 \\  & & & MC-AT(on) & **96.98** & **88.00** & **92.49** & 17.34 \\  & & & MC-AT(off) & 92.69 & 79.50 & 86.09 & 23.45 \\  & & & MC- & & & \\  & & & AT(dual) & 95.30 & 80.00 & 87.65 & 25.45 \\  & & & MC-TRADES & 95.96 & 79.50 & 87.73 & **46.68** \\ \hline \multirow{4}{*}{MNIST} & \multirow{4}{*}{95\%} & \multirow{4}{*}{99.07\%} & MC(regular) & **98.49** & 57.98 & 78.24 & 79.67 \\  & & & MC-AT(on) & 97.31 & **72.90** & **85.11** & **93.51** \\  & & & MC-AT(off) & 11.35 & 2.70 & 7.02 & 14.16 \\  & & & MC- & & & \\  & & & AT(dual) & 94.74 & 57.42 & 76.08 & 93.47 \\  & & & MC-TRADES & 97.68 & 70.47 & 84.07 & 91.19 \\ \hline \multirow{4}{*}{FMNIST} & \multirow{4}{*}{90\%} & \multirow{4}{*}{90.63\%} & MC(regular) & **86.62** & 2.49 & 44.55 & 25.39 \\  & & & MC-AT(on) & 86.51 & **95.61** & **91.06** & 17.29 \\  & & & MC-AT(off) & 40.65 & 26.66 & 33.65 & 67.50 \\  & & & MC- & & & \\  & & & MC- & & & \\  & & & AT(dual) & 82.93 & 91.02 & 86.98 & 37.31 \\  & & & MC-TRADES & 83.14 & 36.80 & 59.97 & **67.88** \\ \hline \multirow{4}{*}{CIFAR-10} & \multirow{4}{*}{70\%} & \multirow{4}{*}{84.28\%} & MC(regular) & **78.41** & 16.53 & 47.47 & 51.04 \\  & & & MC-AT(on) & 69.57 & **49.54** & **59.56** & 45.21 \\ \cline{1-1}  & & & MC-AT(off) & 49.74 & 38.48 & 44.11 & 45.58 \\ \cline{1-1}  & & & MC- & & & \\  & & & AT(dual) & 71.32 & 43.79 & 57.55 & 64.95 \\ \cline{1-1}  & & & MC-TRADES & 75.99 & 22.86 & 49.43 & **68.12** \\ \hline \multirow{4}{*}{HAR} & \multirow{4}{*}{95\%} & \multirow{4}{*}{90.97\%} & MC(regular) & 87.78 & 0.00 & 43.89 & 27.09 \\  & & & MC-AT(on) & **89.75** & **40.52** & **65.14** & 27.30 \\ \cline{1-1}  & & & MC-AT(off) & 63.22 & 0.00 & 31.61 & 41.16 \\ \cline{1-1}  & & & MC- & & & \\ \cline{1-1}  & & & AT(dual) & 73.33 & 40.52 & 56.92 & 52.62 \\ \cline{1-1}  & & & MC-TRADES & 87.51 & 0.00 & 43.76 & **65.53** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy results on five datasets at a sparsity ratio where the original test accuracy is maintained above 90% of the pre-compressed models. The original accuracy denotes the accuracy result of MC(regular) without model compression on the original testset. In all five datasets, MC-AT(on) showed the highest combination test accuracy results.