SlowGuess/NAACL_2022_OCR huggingface dataset:
Yelp, and 4 for IMDB. The learning rate is set to  $2e - 5$  and is adjusted using linear scheduling.
Adversarial training For adversarial training, the number of training epochs is set to 3 with an additional first epoch of finetuning on clean examples. The adversarial examples are generated on the fly in each batch during training. For the Geometry Attack in adversarial training, at most  $\min \{K,0.4\cdot \mathrm{len}(X_i)\}$  words can be perturbed in an example. The upper limit of the candidate set size is set to 50.
Back Translation We use pretrained translation models opus-mt-en-roa and opus-mt-roa-en from Helsinki-NLP to generate one translation for each example.
G Hard Positive Examples from Geometry Attack for Contrastive Loss
In Table 9, we show hard positive examples generated by our Geometry Attack for contrastive loss from the AG's News dataset.
Original	Zurich employees plead guilty in probe new york (reuters) - two senior insurance underwriters at zurich american insurance co pleaded guilty on tuesday to mis-demeanors related to bid-rigging in the insurance market.
Adversarial	Zurich employees plead guilty in probe new york (reuters) - two senior insurance agents at zurich ameri-can insurance co testified guilty on tuesday to violations related to bid-rigging in the insurance market.
Original	Black watch troops move into position the first units of a black watch battlegroup are due to arrive today in their new positions south of baghdad as tony blair indicated that more british troops may replace them in the american - controlled zone before the end of the year.
Adversarial	Black watch troops move into place the first units of a black watch operation are due to arrive today in their new positions south of baghdad as tony blair indicated that more british troops may replace them in the american - controlled zone before the end of the year.
Table 9: Hard positive examples generated by Geometry Attack for contrastive loss. Blue words in the original examples are replaced by red words in the adversarial examples.
H Additional t-SNE plots
We give t-SNE plots of the vector representations of clean examples and adversarial examples from Yelp, IMDB and DBpedia in fig. 4, fig. 5 and fig. 6, respectively.
(b) ADCL + FTC
(c) NP + ADV
(d) ADCL + ADV
Figure 4: t-SNE plot of the vector representations of clean examples and adversarial examples from the Yelp dataset.
(d) ADCL + ADV
Figure 5: t-SNE plot of the vector representations of clean examples and adversarial examples from the IMDB dataset.
Figure 6: t-SNE plot of the vector representations of clean examples and adversarial examples from the DBpedia dataset.