SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9: Standard and robust accuracy against Transfer-PGD  $\ell_{\infty}$  threat model on ImageNet, under different classifier architectures and attack budgets.
Architecture	Natural Acc	Attack Budget	Accuracy (%)
Standard	Robust
ResNet-50	78.52	ε = 4/255	72.85	70.51
ε = 16/255	69.34	62.70
ResNet-152	80.66	ε = 4/255	73.83	71.29
ε = 16/255	72.85	66.60
WRN-50-2	80.47	ε = 4/255	74.61	70.51
ε = 16/255	71.68	62.10
Based on the results from Table 7 to 9, we can summarize that ScoreOpt achieves better performance than previous defenses consistently and purify adversarial examples on transfer-based attacks successfully, across various datasets.
Code is available at https://github.com/zzzhangboya/ScoreOpt.git.
C.1 Computing resources
All of our experiments are conducted using GPUs. Specifically, the diffusion models and base classifiers are trained in parallel on eight GPUs. Each test-time robustness evaluation is performed on a single GPU. The GPUs used in our experiments are NVIDIA Titan RTX with 24GB of memory.
C.2 Dataset descriptions
We utilize three datasets in our experiments: CIFAR-10, CIFAR-100, and ImageNet. CIFAR-10 and CIFAR-100 datasets contain 50,000 training images and 10,000 test images, with 10 and 100 classes respectively. All CIFAR images have a resolution of  $32 \times 32$  pixels and three color channels (RGB). On the other hand, ImageNet consists of a validation set with 50,000 examples, featuring 1,000 classes and images with a resolution of  $256 \times 256$  pixels and three color channels.
C.3 Attack details
We conduct evaluations against the BPDA+EOT attack using 50 BPDA iteration steps and 15 EOT attack samples. Regarding the white-box PGD+EOT attack discussed in Section 4.2.2, we employ 20 PGD steps and 20 replicates for EOT attacks, consistent with the setup used in Lee and Kim [34].
For the Transfer-PGD attack, we adopt 40 PGD update iterations to generate adversarial examples, following previous studies [69, 63]. The step size  $\alpha$  is set to  $\epsilon /4$ .
C.4 Training details
Diffusion models We utilize pre-trained class-unconditional diffusion models obtained from Karras et al. [29] for CIFAR-10 and from Dhariwal and Nichol [9] for ImageNet. For the CIFAR-100 dataset, we train our own class-unconditional EDM using the ddpm++ model architecture. The training configuration and network architecture for CIFAR-100 align with the settings used in Karras et al. [29] for CIFAR-10.
**Classifiers** Our classifiers are trained alone on the training set of each dataset. For CIFAR-10, WRN-28-10 and WRN-70-16 classifiers both achieve  $95.19\%$  natural accuracy on the whole test set. For CIFAR-100, the WRN-28-10 model achieves  $81.45\%$  natural accuracy and WRN-70-16 achieves  $81.28\%$  natural accuracy.
C.5 Implementation details of our methods
We evaluate our ScoreOpt method using the Adam optimizer in all experiments. During our experiments, we observed that both the Diff loss and the SR loss can yield satisfactory results when the number of optimization steps is relatively small. However, in comparison to SR optimization, Diff optimization offers faster inference speed and requires less memory. Unless otherwise stated, all experiments in the main text are conducted using the WRN-28-10 classifier architecture. Please refer to Table 10 for the optimization hyper-parameters we used to obtain the final evaluation results in Section 4 and Appendix B.
Table 10: Hyper-parameters choices of our optimization process for experimental results in the main text and appendix.
Attack Type	Perturbation Budget	Method	LR	Step	Noise Level
Dataset: CIFAR10
BPDA+EOT	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	0.1	5	[0.40,0.60]
BPDA+EOT	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	0.1	5	[0.50,0.50]
PGD+EOT	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	0.1	20	[0.25,0.25]
PGD+EOT	\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-N	0.1	20	[0.25,0.25]
Transfer-PGD	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	0.01	20	[0.15,0.35]
Transfer-PGD	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	0.1	5	[0.25,0.25]
Transfer-PGD	\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-O	0.01	20	[0.15,0.35]
Transfer-PGD	\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-N	0.1	5	[0.25,0.25]
Dataset: CIFAR100
BPDA+EOT	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	0.1	3	[0.15,0.35]
BPDA+EOT	\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	0.1	3	[0.25,0.25]
D Ablation Studies and Additional Results
D.1 Different classifier architectures
To showcase the effectiveness and reliability of incorporating our method into any off-the-shelf classifiers, we conduct additional evaluations using the WideResNet-70-16 classifier architecture. All of these experiments are conducted using the same optimization hyper-parameters as those employed for the WRN-28-10 architecture. Table 11 demonstrates that our method with WRN-70-16 consistently achieves high robust accuracy against various attacks on both the CIFAR-10 and CIFAR-100 datasets.
Table 11: Evaluation results obtained by WRN-70-16 classifier architecture.
Attack Type	Perturbation Budget	Method	Acc (%)
Standard	Robust
Dataset: CIFAR10
BPDA+EOT	l∞(ε = 8/255)	ScoreOpt-N	93.75	91.60
Transfer-PGD	l∞(ε = 8/255)	ScoreOpt-N	92.47	92.32
Transfer-PGD	l2(ε = 0.5)	ScoreOpt-N	92.47	92.29
Dataset: CIFAR100
BPDA+EOT	l∞(ε = 8/255)	ScoreOpt-O	76.56	65.82
BPDA+EOT	l∞(ε = 8/255)	ScoreOpt-N	69.53	66.99
D.2 Number of EOT attack replicates
We present additional results for our ScoreOpt defense by varying the number of EOT attack replicates. All experiments are conducted using the same settings as described in Section 4, except the EOT number. Figure 4 illustrates the robustness performance against PGD+EOT attacks with different numbers of EOT replicates, while Figure 5 showcases the results against BPDA+EOT attacks. The evaluations are performed using the WRN-28-10 architecture on the CIFAR-10 dataset. Interestingly, our findings indicate that the number of EOT replicates does not significantly impact the standard and robust accuracy of our method.
(a)  $\ell_{\infty}(\epsilon = 8 / 255)$
(b)  $\ell_2(\epsilon = 0.5)$
Figure 4: Robustness performance against PGD+EOT attack under the  $\ell_{\infty}(\epsilon = 8 / 255)$  and  $\ell_2(\epsilon = 0.5)$  threat models, respectively. We evaluate our methods with WRN-28-10 on CIFAR-10.
Figure 5: Robustness performance against BPDA+EOT attack under the  $\ell_{\infty}(\epsilon = 8 / 255)$  threat model. We evaluate our methods with WRN-28-10 on CIFAR-10.
D.3 Hyper-parameter choices of the optimization process
To further assess the effectiveness of our methods, we conduct additional experiments presented in Table 12. In these experiments, we use different optimization hyper-parameters compared to the ones used in Table 10. It is worth noting that varying the optimization hyper-parameters typically does not have a significant impact on the final robustness results. This observation suggests that our defense mechanism remains effective and consistent across different choices of optimization hyper-parameters, further reinforcing its reliability.
Table 12: Evaluation results with different hyper-parameters. The * symbol indicates that the hyperparameter is identical to the corresponding experiment in the main text.
Perturbation Budget	Method	Hyper-parameter	Acc (%)
LR	Step	Noise Level	Standard	Robust
Attack: BPDA+EOT
\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	*	*	[0.30,0.50]	93.16	79.30
\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	*	*	[0.30,0.60]	92.38	81.05
\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	*	*	[0.40,0.40]	94.53	92.19
\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-N	*	*	[0.60,0.60]	91.60	89.26
Attack: Transfer-PGD
\( \ell_{\infty}(\epsilon=8/255) \)	ScoreOpt-O	0.1	10	*	91.42	88.56
\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-O	*	*	[0.20,0.40]	91.55	90.21
\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-N	*	10	*	91.64	91.17
\( \ell_{2}(\epsilon=0.5) \)	ScoreOpt-N	*	10	[0.30,0.30]	91.24	90.94
D.4 More results on the whole ImageNet test set
The experimental results presented in Table 9 are evaluated on a fixed subset of 512 randomly sampled images. To provide a more comprehensive assessment, we include additional results on the complete 50,000 test set of ImageNet in Table 13.
Table 13: Standard and robust accuracy against Transfer-PGD  $\ell_{\infty}$  threat model on Imagenet, under different classifier architectures and attack budgets.
Architecture	Natural Accuracy	Attack Budget	Accuracy (%)
Standard	Robust
ResNet-50	76.15	ε = 4/255	70.07	66.02
ε = 16/255	66.93	60.45
WRN-50-2	78.48	ε = 4/255	72.38	68.22
ε = 16/255	69.27	61.73
E Discussions and Limitations
The experiments conducted demonstrate the significant improvements achieved by our method in terms of both robustness performance and inference speed compared to previous methods. Our approach can be seamlessly integrated into various off-the-shelf classifiers. However, it is important to acknowledge a few potential drawbacks of our method.
Firstly, the computational memory cost remains a challenge that needs to be addressed. Although our method requires only a small number of optimization steps, the inclusion of the U-Net Jacobian term in the loss function introduces additional computational overhead. This memory constraint limits the evaluation of our method on stronger adversarial benchmarks like RobustBench.
Secondly, while we have experimentally demonstrated that our test-time optimization process converges to a certain local optimum within a small number of steps, we have not provided a theoretical convergence analysis.
Our future work will focus on addressing these limitations and further exploring the theoretical aspects of our method. By addressing these issues, we can enhance the practicality and theoretical understanding of our approach, paving the way for more robust and efficient adversarial defense mechanisms.