# Partial Retraining Substitute Model for Query-Limited Black-Box Attacks
## 4 Experimental Evaluation
### Impact of Image Size (Vggface2)  
The purpose of this section is to compare attack results when the target model is trained by small images (MNIST, Section 4.2) and bigger images (VGGFace2, Section 4.3). Each sample in the VGGFace2 dataset is higher-resolution than in MNIST and images are in color, meaning that black-box attacks on VGGFace2 dataset are more complicated. There are 10 classes (the same number as for MNIST) and, for each test image, the target classes are the other 9 classes. The network architecture of the target model is ResNet50 [35], and we apply the recent loss functions, SphereFace [37] and Ring Loss [38] to improve the classification accuracy of the target model. The ratio of the training data is 50% (approximately 270 images of each face) and they are flipped horizontally for data augmentation. Using the ResNet50 network architecture for the target model, the classification accuracy is approximately 99% after training. We construct two substitute models whose network architectures are respectively VGG16 [1] and Inception [36]. The maximum upper limit of the number of queries is 10,000.  
Table 3 shows that the proposed method achieves better performance on large-sized image classification models. The attack success ratios are much higher and the number of queries is lower in this query-limited environment. The main reasons are the partial retraining of the substitute models and pre-training them without queries as well. Large-size image classification requires large-scale networks as target models, which also makes the attack methods more complicated. In other words, substitute models require more training data to emulate either the entire target model or the partial decision boundary.  
Figures 5 and 6 show query efficiency when the substitute models are VGG16 and InceptionV3, respectively. Similar to the results for MNIST, the success ratio of the proposed method is always higher than that of the previous method when the upper bound is the same. The success ratio of the proposed method rapidly increases when the upper bound approaches 5000 in both Figures 5 and 6. In contrast, the points of rapid increase for the previous method are near 10,000, the maximum upper bound. In both graphs, the proposed method decreases the number of queries by more than 50% compared with the previous method to achieve the same attack success ratio. For example, the proposed method  
Figure 4: Attack success ratio according to the upper bound of the number of queries (Sub3, MNIST).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Network** & **Attack** & \multicolumn{2}{c}{**Previous Method**} & \multicolumn{2}{c}{**Proposed Method**} \\ \cline{3-5}
**Architecture** & **Algorithm** & **\% Succ.** & **\# Query** & **\% Succ.** & **\# Query** \\ \hline VGG16 & I-FGSM & 29.7\% & 6128 & 82.4\% & 5015 \\ VGG16 & C\&W & 25.1\% & 7351 & 79.7\% & 4942 \\ InceptionV3 & I-FGSM & 40.1\% & 5986 & 84.1\% & 3914 \\ InceptionV3 & C\&W & 38.9\% & 7010 & 82.4\% & 4598 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Black-box attack results (VGGFace2, query limit = 10,000).  
requires less than 5000 queries to reach a 30% attack success ratio while the previous method needs more than 10,000 queries, as shown in Figure 5.