SlowGuess/NAACL_2022_OCR huggingface dataset:
For each NLP dataset, pre-trained base (12-layer, 768-hidden dimension, 110M parameters) Transformer encoders were fine-tuned during training. Table A.1 gives the training hyperparameters: learning rate (lr), batch size (bs) and the number of training epochs. In all training regimes an Adam optimizer was used. With respect to hardware, NVIDIA Volta GPU cores were used for training all models.
Dataset	Model	lr	bs	epochs
IMDB	BERT	1e-5	8	2
Twitter	ELECTRA	1e-5	8	2
AG News	BERT	1e-5	8	2
DBpedia	ELECTRA	1e-5	8	2
L-Bus	BERT	1e-6	16	5
Table A.1: Training Hyperparameters
Figure A.1 presents the impact of adversarial attacks of different perturbation sizes,  $N$  on each NLP dataset. All classification datasets' models underwent saliency ranked,  $N$ -word substitution attacks described in Equation 8, whilst the regression dataset, L-Bus, was subject to a  $N$ -word concatenation attack as in Equation 9. For the classification tasks the impact of the adversarial attacks was measured using fooling rate, whilst for the L-Bus dataset task, the average output score from the system is given. Figure A.2 gives the encoder embedding space PCA residue plots for all the datasets not included in the main text.
Table A.2 compares the impact on error sizes (using  $l_{2}$  and  $l_{\infty}$  norms) and the residue plot metric,  $N_{\sigma}$  for the original text space discrete attacks and an artificial input embedding space continuous attack. The purpose of this table is to present the results for the datasets not included in the main text in Table 6.
$^{6}$ https://huggingface.co/transformers/pretrained_models.html
(a)IMDB
(d) DBpedia
Figure A.1: Adversarial Attack Impact against  $N$ -word attack.
(b) Twitter
(c) AG News
(e) L-Bus
Dataset	Attack	Nσ	l2error	l∞error
IMDB	Discrete	0.181	73.5±22.5	4.02±0.95
Continuous	0.111	6.73±3.98	0.09±0.03
Twitter	Discrete	1.201	50.2±19.2	3.26±0.86
Continuous	0.676	5.35±3.95	0.08±0.03
AG News	Discrete	0.642	67.9±28.1	3.35±0.95
Continuous	0.393	5.41±4.10	0.09±0.04
DBpedia	Discrete	1.355	57.4±18.6	3.29±0.88
Continuous	0.991	6.57±3.23	0.09±0.04
L-Bus	Discrete	0.201	94.6±30.1	5.91±1.12
Continuous	0.135	8.22±3.54	0.07±0.04
Table A.2: Comparison of token level discrete attack and input embedding layer continuous PGD attack.
(a)IMDB
(b) Twitter
(c) AG News
(e) L-Bus
(f) CIFAR-100
Figure A.2: Encoder Embedding space residue plot using PCA decomposition.