# A Mask-Based Adversarial Defense Scheme
## 4 Results
### On Mask Rates and Grid Sizes  
The effectiveness of MAD also depends on two critical hyperparameters. The mask rate is a constant which is used as the probability for a grid to be masked at the training phase as well as test phase. As mentioned before, a larger mask rate potentially removes more perturbation (which means better defense) at the cost of less usable information for the classifier, which may lead to worse accuracy on clean images.  
We carry out an experiment on a selection of mask rates (\(1/3,1/2,3/4,4/5\), sample images illustrated in Figure 4) with grid size fixed at \(8\times 8\) for the training phase and \(4\times 4\) for the test phase, on a VGG16 model with image data from CIFAR-10. The experimental results are shown in Table 1. It can be seen from the results that a higher masking percentage makes a better defense but has lower accuracy on benign inputs. Considering that in a real-life scenario, a DNN may receive a mixed set of benign and adversarial inputs, say \(1/2\) benign samples and \(1/2\) adversarial samples, for the mask rate of \(3/4\) and adversarial samples generated by FGSM with \(\epsilon=15\), the real-life defense score can then be calculated as the weighted average \(\frac{1}{2}\times 82.63\%+\frac{1}{2}\times 82.65\%=82.64\%\) as a fair guess in this hypothetical scenario (data as from the top two rows at column \(3/4\) of Table 1). Therefore, we believe that \(3/4\) could be a reasonable mask rate, which yields acceptable accuracy for both benign and adversarial inputs, and we use it for all experiments conducted in rest of the paper.  
The grid size is another hyperparameter to be determined before the main experiment. Intuitively, a larger grid size allows better continuity of information on average that is preserved in a masked image (examples illustrated in Figure 5).  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multicolumn{2}{c}{**Masked**} & \multicolumn{5}{c}{**Classification Accuracy**} \\ \hline \multirow{2}{*}{**Training**} & \multirow{2}{*}{**Testing**} & \multirow{2}{*}{**Benign**} & **BIM Linf** & **PGD Linf** & **CW L2** & **DeepFool** \\  & & & \(\epsilon\) = 0.01 & \(\epsilon\) = 0.01 & \(\epsilon\) = 1.0 & **Linf** \\ \hline \(\times\) & \(\times\) & 84.18\% & 38.29\% & 40.29\% & 63.11\% & 50.70\% \\ \(\checkmark\) & \(\times\) & **84.65\%** & 3.16\% & 6.13\% & 15.89\% & 35.72\% \\ \(\times\) & \(\checkmark\) & 49.39\% & 67.98\% & 69.37\% & 70.06\% & 67.98\% \\ \(\checkmark\) & \(\checkmark\) & 82.65\% & **81.80\%** & **84.78\%** & **91.28\%** & **85.23\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Defense effect of using masked and nonmasked images during training and testing.  
Table 3 presents the preliminary experimental results on a VGG model with CIFAR-10 dataset for a selection of grid size values in the masking operation at the time of training and testing, respectively. Given images in CIFAR-10 are of size \(32\times 32\), we try a number of combinations for values that divide 32. If we focus on the three columns where \(8\times 8\) grids are used for training, the results seem to suggest that using a larger grid in the test phase produces a better classification accuracy on benign images, and a smaller grid produces a better defense. We choose \(8\times 8\) (training) and \(4\times 4\) (testing) for the VGG16 model and CIFAR-10 dataset, and adopt this same setting for ResNet and SVHN, which also use \(32\times 32\) images. For the LeNet model and MNIST dataset (\(28\times 28\) images), we use \(7\times 7\) grids for both training and testing.  
Figure 4: Images from CIFAR-10 with different masking rates applied. From top to bottom, \(1/3\), \(1/2\), and \(3/4\) rates are respectively applied on the same images.  
Figure 5: The \(3/4\) masked images from CIFAR-10 with different grid sizes: from top to bottom, \(2\times 2\), \(4\times 4\) and \(8\times 8\) grids are used for masking the same images.