SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset	Vectorizer	0%	10%	20%	30%	40%	50%	60%	70%	80%	90%	100%
AG News	Whitespace	92.0%	91.6%	91.0%	90.2%	88.6%	86.8%	83.8%	79.3%	72.3%	63.5%	49.1%
SentencePiece	91.8%	91.0%	89.9%	88.5%	86.2%	82.8%	78.7%	74.0%	67.9%	60.9%	51.5%
BPE	91.6%	90.5%	89.1%	87.6%	84.8%	80.6%	76.1%	70.2%	63.8%	56.2%	46.3%
fastText	92.8%	92.1%	91.6%	90.7%	89.1%	87.8%	85.1%	82.5%	77.2%	70.5%	59.7%
RETVec-raw	91.0%	90.3%	89.6%	88.3%	87.3%	85.5%	83.7%	81.8%	78.2%	74.7%	68.5%
RETVec	92.6%	92.1%	91.6%	91.1%	90.5%	89.9%	88.7%	87.3%	86.0%	84.2%	81.2%
Yelp P.	Whitespace	93.2%	92.5%	91.6%	90.5%	88.6%	86.7%	84.1%	80.4%	75.8%	69.7%	60.9%
SentencePiece	93.1%	91.5%	89.7%	87.7%	85.1%	81.9%	78.7%	75.1%	71.2%	67.4%	62.6%
BPE	93.0%	91.7%	90.2%	88.5%	86.2%	83.5%	80.5%	77.2%	73.1%	69.0%	64.2%
fastText	94.1%	93.5%	92.8%	92.0%	90.7%	89.3%	87.9%	85.8%	83.1%	80.2%	75.6%
RETVec-raw	92.4%	91.7%	90.9%	89.8%	88.6%	87.2%	85.7%	83.7%	81.2%	78.6%	74.5%
RETVec	92.7%	92.2%	91.6%	90.9%	90.0%	89.2%	88.0%	86.8%	85.1%	83.5%	80.9%
Multilingual Amazon P.	Whitespace	92.7%	92.1%	91.6%	91.1%	90.2%	89.1%	87.9%	86.2%	83.7%	80.9%	75.0%
SentencePiece	89.6%	88.5%	87.5%	86.2%	84.5%	82.6%	80.7%	78.7%	76.1%	73.9%	70.7%
BPE	88.9%	87.7%	86.6%	85.4%	83.6%	81.8%	80.1%	77.9%	75.3%	73.4%	70.3%
fastText	86.2%	85.5%	84.8%	84.0%	82.9%	81.0%	80.1%	77.9%	75.9%	73.9%	70.2%
RETVec-raw	92.3%	91.6%	90.8%	90.0%	89.0%	87.7%	86.3%	84.8%	82.7%	80.6%	77.0%
RETVec	92.9%	92.5%	92.0%	91.7%	91.3%	90.6%	90.2%	89.5%	88.6%	87.7%	86.1%
MASSIVE	Whitespace	70.0%	58.9%	58.8%	58.4%	55.8%	51.7%	49.2%	43.9%	37.1%	34.8%	17.3%
SentencePiece	69.0%	58.5%	58.4%	57.8%	55.8%	52.7%	50.7%	46.7%	41.8%	40.9%	29.6%
BPE	65.5%	54.6%	54.4%	54.1%	52.0%	48.6%	46.9%	42.7%	38.4%	37.0%	26.4%
fastText	16.7%	15.5%	16.1%	15.2%	14.8%	14.3%	13.9%	13.7%	13.1%	13.0%	12.5%
RETVec-raw	69.6%	59.7%	59.7%	59.2%	57.4%	54.4%	52.7%	49.0%	44.8%	43.6%	32.5%
RETVec	73.2%	65.7%	65.7%	65.5%	63.9%	61.8%	60.4%	57.5%	54.1%	52.9%	43.5%
Table 17: Random mixed typo resilience results (0% to 100% word typo rate) for each classification dataset and vectorizer. Following the methodology described in Section 6, test accuracy on each dataset is reported and results are averaged across the three model architectures we benchmarked in 5.
F Typo Resilience Evaluation
Detailed results for random mixed typo resilience across every dataset and vectorizer can be found in Table 17.
G Adversarial Resilience Evaluation
We report adversarial attack resilience results for all vectorizers, classification models, and adversarial attack algorithms we benchmarked in Table 18. The TextAttack [21] framework was used to conduct all three types of adversarial attacks.
Model	Vectorizer	TextBugger	Pruthi	DeepWordBug
Original Acc	Acc under Atk	Atk Success %	Original Acc	Acc under Atk	Atk Success %	Original Acc	Acc under Atk	Atk Success %
LSTM	Whitespace	90.6%	9.9%	89.1%	90.6%	84.2%	7.1%	90.6%	9.9%	89.1%
SentencePiece	90.3%	0.8%	99.1%	90.3%	68.1%	24.6%	90.3%	0.8%	99.1%
BPE	88.1%	3.3%	96.3%	88.1%	73.3%	16.8%	88.1%	3.3%	96.3%
fastText	92.7%	14.4%	84.5%	92.7%	83.3%	10.1%	92.7%	14.4%	84.5%
RETVec-raw	90.8%	22.3%	75.4%	90.8%	74.8%	17.6%	90.8%	22.3%	75.4%
RETVec	91.8%	23.7%	74.2%	91.8%	80.9%	11.9%	91.8%	23.7%	74.2%
CNN	Whitespace	90.9%	17.6%	80.6%	90.9%	83.8%	7.8%	90.9%	9.0%	90.1%
SentencePiece	90.3%	2.9%	96.8%	90.3%	72.2%	20.0%	90.3%	3.5%	96.1%
BPE	89.3%	31.8%	64.4%	89.3%	54.1%	39.4%	89.3%	43.5%	51.3%
fastText	91.9%	17.3%	81.2%	91.9%	74.9%	18.5%	91.9%	15.4%	83.2%
RETVec-raw	86.9%	30.1%	65.4%	86.9%	59.6%	31.4%	86.9%	37.7%	56.6%
RETVec	91.4%	34.3%	62.5%	91.4%	77.4%	15.3%	91.4%	45.0%	50.8%
BERT	Whitespace	89.4%	9.8%	89.0%	89.4%	83.6%	6.5%	89.4%	3.3%	96.3%
SentencePiece	89.8%	2.8%	96.9%	89.8%	70.8%	21.2%	89.8%	3.7%	95.9%
BPE	90.8%	8.2%	91.0%	90.8%	78.2%	13.9%	90.8%	1.2%	98.7%
fastText	92.6%	22.9%	75.3%	92.6%	80.5%	13.1%	92.6%	18.1%	80.5%
RETVec-raw	93.0%	30.8%	66.9%	93.0%	82.2%	11.6%	93.0%	38.9%	58.2%
RETVec	93.7%	30.1%	67.9%	93.7%	84.6%	9.7%	93.7%	40.5%	56.8%
Table 18: Detailed adversarial resilience results on AG News. Results are reported on the same randomly selected 1000 examples from the AG News test split, following the methodology described in Section 7.
H Pre-training and Fine-tuning BERT
Table 19 details the hyperparameter settings used for pre-training and fine-tuning BERT-Base models. Table 20 shows detailed results on the GLUE benchmark, including the models' average performance and standard deviation for each GLUE task.
Hyperparameter	Pre-training	Fine-tuning
Training steps	100k steps	20 epochs
Batch size	64	32
Sequence length	512	512
Adam ε	1e-8	1e-8
Adam β1	0.9	0.9
Adam β2	0.999	0.999
Weight decay	0.01	0.01
Max learning rate	5e-5	2e-5
End learning rate	0	0
Warmup steps	10000	First 5% of steps
Decay function	Linear	None
Table 19: Pre-training and fine-tuning hyperparameters for BERT-Base models described in Section 8.
Vectorizer	MNLI	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B	GLUE Avg
SentencePiece	80.6 (0.1)	88.4 (0.3)	90.1 (0.0)	66.1 (1.0)	90.8 (0.3)	85.4 (0.15)	50.3 (0.4)	82.0 (0.5)	79.2 (0.3)
RETVec-raw	82.0 (0.5)	89.5 (0.1)	90.4 (0.1)	64.5 (0.9)	91.5 (0.1)	86.3 (0.9)	47.9 (1.1)	79.1 (0.2)	78.9 (0.4)
RETVec	80.9 (0.4)	88.9 (0.3)	90.4 (0.1)	65.0 (0.7)	90.7 (0.2)	86.9 (0.4)	47.2 (0.7)	79.6 (0.2)	78.7 (0.3)
I fastText Word Dataset
Table 21 contains statistics on word length computed on the fastText word dataset using words from all 157 available languages.
Table 20: Detailed results on GLUE Benchmark for pre-trained BERT-Base models using RETVec compared to SentencePiece. Each model is trained three times with different seeds, and the average and standard deviation is reported here. Bold indicates best results, underline indicates second best.
Avg	Median	Std	p90	p95	p99	p99.9
8.4	7.9	4.6	13.0	15.0	20.8	36.1
Table 21: Word length statistics computed on all fastText words from 157 languages. p90 denotes the 90th percentile, p95 denotes the 95th percentile, and so on.
J Embedding Visualization
We use the TensorBoard Embedding Projector to visualize RETVec embeddings for words, as shown in Figure J.
Figure 7: RETVec embedding visualization (using the TensorBoard Embedding Locator) of the most common 10000 English words and a typo-laden version of each word. The word 'discover' is selected as an example, and the 5 nearest neighbors and their cosine distances are shown.
discovers	0.078
discovery	0.145
discovered	0.177
discooves	0.238
discoveries	0.278