SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: Top-1 classification accuracy on ImageNet versus the parameter  $a$  of Hampel loss function defined in Kim & Scott (2012) under different settings. We use  $2 \times a$  and  $3 \times a$  as parameters  $b$  and  $c$ . The best results are highlighted in bold font and the second best are highlighted in underline. We choose  $a = 0.2$  in rest of the experiments.
Hampel Loss Parameter	0.1	0.2	0.4	0.6	0.8	1
Clean Data	71.63	72.94	71.84	71.23	70.87	70.41
FGSM	56.42	55.92	55.83	55.66	54.97	53.68
PGD	45.18	44.23	43.89	43.62	43.01	42.34
SPSA	52.96	52.48	52.13	51.46	50.92	50.23
Imagenet-C	44.76	45.61	46.04	46.13	45.82	45.31
Table 9: Top-1 classification accuracy on ImageNet versus the parameter  $\beta$  of SPKDE defined in Eq. (6) under different settings.  $\beta = \frac{1}{1 - \varepsilon} > 1$ , where  $\varepsilon$  is the percentage of anomalous samples. A larger  $\beta$  indicates a more robust model. The best results are highlighted in bold font and the second best are highlighted in underline. We choose  $\beta = 1.4$  in rest of the experiments.
β	1.05	1.2	1.4	1.6	1.8	2
Clean Data	74.25	73.56	73.22	73.01	72.86	72.64
FGSM	53.69	55.08	56.03	55.37	54.21	53.86
PGD	42.31	43.68	44.51	44.32	44.17	43.71
SPSA	51.29	52.02	52.64	52.84	52.16	51.39
Imagenet-C	44.68	45.49	44.76	44.21	43.96	43.33
Table 10: Top-1 classification accuracy on ImageNet versus the number of iterations of the KIRWLS algorithm in Eq. (25) employed in Transformer-RKDE. Since the increased number of iterations does not lead to significant improvements of performance while the computational cost is much higher, we use the single-step iteration of the KIRWLS algorithm in Transformer-RKDE.
Iteration #	Huber Loss	Hampel Loss
1	2	3	5	1	2	3	5
Clean Data	72.83	72.91	72.95	72.98	72.94	72.99	73.01	73.02
FGSM	55.83	55.89	55.92	55.94	55.92	55.96	55.97	55.99
PGD	44.15	44.17	44.17	44.18	44.23	44.26	44.28	44.31
SPSA	52.42	52.44	52.45	52.45	52.48	52.53	52.55	52.56
Imagenet-C	45.58	45.61	45.62	45.62	45.61	45.66	45.68	45.71
Table 11: Computation time (measured by seconds per iteration) of baseline methods, Transformer-SPKDE, Transformer-MoM and Transformer-RKDE with different number of KIRWLS iterations. Transformer-SPKDE requires longer time since it directly obtains the optimal set of weights via the QP solver.
Iterations of KIRWLS	DeiT	RVT	SPKDE	MoM-KDE
1	2	3	5
Time (s/it)	0.43	0.51	0.68	0.84	0.35	0.41	1.45	0.37
(b)
Figure 5: Contour plots of density estimation of the 2-dimensional query vector embedding in an attention layer of the transformer when using (b) KDE (Eq. (4)) and (c) RKDE after one iteration of Eq. (25) with Huber loss (Eq. (24)), (d) KDE with median-of-means principle (Eq. (10)), where (a) is the true density function. We draw 1000 samples (gray circles) from a multivariate normal density and 100 outliers (red cross) from a gamma distribution as the contaminating density. RKDE and KDE with the median-of-means principle can be less affected by contaminated samples when computing self-attention as nonparametric regression.