Question: Will the proposed approach achieve similar performance on other datasets e.g. CIFAR10？
Evidence: We again repeat the main untargeted attack experiment, this time using CIFAR-10 as the dataset,
with the target and surrogate networks sourced from https://github.com/akamaster/
pytorch_resnet_cifar10. We use ResNet-110 as the target, and the much simpler ResNet-20 as its single surrogate. The ℓ2 norm bound is again set to ν =√0.001D, which for CIFAR-10≈ 1.75. Fig. 9 demonstrates the results. All methods completely solve the problem well within the 10K query limit: we focus the plot on the first 500 queries. This problem is easier than the main
ImageNet version studied in this work, but GFCS nonetheless retains its relative dominance.
Answer: We have added the results of the requested experiments to the submission pdf (in the main paper or in the Appendix) as follows:
Lower norm bound (of 5) (Section A.5). Adversarially trained network (Section A.7). CIFAR10 (Section A.6).
Note also that, as also discussed with reviewer KEn1, the loss-gradient-only ablation figures have been added to Table 1.
Originally, we did not include any comparison against surrogate-free methods such as Square Attack since one of our competitors [2] (ODS) ran relevant experiments in their own paper, and claimed dramatic improvement over Square Attack in the L2-bound case (see Table 5 in that reference). Given that Square Attack does not use a surrogate, we did not find this result surprising. Since Square Attack was outright beaten by ODS-RGF in [2], and since we compare ourselves against ODS-RGF under the same experimental setup, we considered the Square Attack comparison to be redundant in our case. Having said that, we feel that this is nevertheless a worthwhile comparison, and have performed this experiment. We have revised the pdf of the submission to include SquareAttack in the main comparison of Fig.2 (a,b,c) and also in the comparison on targeted attack of Fig.4.
We are happy to cite the Bayesian optimisation approaches from a conceptual perspective, as part of the related work (along with latent-space methods we have already included such as AutoZOOM). However, we don’t see that these represent a competitive comparison. For one thing, both of the above papers present results only for the L_inf norm, and these results are not directly comparable to ours. Beyond this, the very low query counts achieved by [2] on the 0.05 l_inf bound on ImageNet come at the expense of far higher failure rates than the sort we are discussing here (ranging from about 20% to about 35%): The variant of the method that was eventually published (as “Simple and Efficient Hard Label Black-box Adversarial Attacks in Low Query Budget Regimes”) stays within the hard-label/decision-attack threat model, only comparing against other such methods. The performance of [1] is not comparable to ours or any of our chosen competitors: under the standard l_inf bound of 0.05 on ImageNet, their success rate is 60%, with a median query count of 1247. Note that the “state-of-the-art black-box methods” against which they compare are ZOO, AutoZOOM, and GenAttack. Among l_inf-bounded surrogate/prior-leveraging approaches, TREMBA represents an empirically superior modern approach to either of these methods, in both the untargeted and targeted settings.