SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Delving deeper into RETVec's typo resilience capabilities, as reported in Figure 5, character deletion is the hardest form of typo for RETVec to deal with. Without the pre-trained model, RETVec-raw also struggles with character insertion and RETVec's model also greatly helps nullify the impact of character swapping.
7 Evaluation: Adversarial Attack Resilience
Setup We use the TextAttack framework [21] to evaluate the resilience of RETVec and other vectorizers against character-level adversarial attacks, specifically: TextBugger [18], DeepWordBug [9], and Pruthi [23]. For each model and vectorizer, the attacks are carried out on the same 1000 examples drawn randomly from the AG News test dataset.
Results Table 4 demonstrates that RETVec models are significantly more resilient to adversarial text attacks than other embedding schemes. In particular, the average attack success rate against RETVec-raw and RETVec models are  $45.6\%$  and  $44.8\%$  respectively, which is significantly lower than for SentencePiece  $(71.3\%)$  and BPE  $(67.8\%)$ . The best performing, non-RETVec, vectorizer is fastText  $(56.3\%)$  attack success rate), which makes sense because fastText handles OOV tokens by averaging the vector representations of the token's n-grams. Detailed results broken down by adversarial attack algorithm are reported in Appendix G.
8 Evaluation: Pre-training BERT
In this section, we evaluate RETVec's performance when used to pre-train transformers. We evaluate both RETVec's competitiveness on the GLUE benchmark [29] and typo resilience.
Setup We use BERT-Base [7] for all experiments. Given that RETVec outputs word-level embeddings and is not restricted by a vocabulary size, we cannot directly use the standard BERT masked-language modeling (MLM) task and predict token ids using a softmax layer. Instead, we use an approximate MLM task for RETVec-based models by only masking and predicting input tokens that are in the top 100k words. Other than this, we follow the standard MLM procedure by selecting  $15\%$  of tokens at random and replacing a selected token with (1) the [MASK] token with  $80\%$  chance (2) a random token with  $10\%$  chance (3) the same token unchanged with  $10\%$  chance. For the baseline, we use SentencePiece with a vocabulary size of 32k and the standard MLM task described in [7].
For each model, we pre-train for 1M steps with batch size 64 on the English C4 dataset [32]. We use Adam with a max learning rate of 5e-5,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and 0.01 L2 weight decay. We warmup for 10000 steps and linearly decay the learning rate. We fine-tune all models for 20 epochs on GLUE using 3 different random seeds and report the best result for each dataset. We pre-train and fine-tune each model using 8 NVidia V100s. Detailed pre-training and fine-tuning hyperparameters can be found in Appendix H.
To benchmark typo resilience of RETVec-based pre-trained BERT models, we fine-tune on AG News to produce comparable results to Section 6, and follow the same evaluation methodology against random and common human typos.
Vectorizer	MNLI	QNLI	QQP	RTE	SST-2	MRPC	CoLA	STS-B	Avg
SentencePiece	80.6	88.7	90.1	67.2	91.1	85.8	50.7	82.5	79.6
RETVec-raw	82.5	89.6	90.5	65.3	91.5	87.2	49.1	79.3	79.4
RETVec	81.4	89.2	90.4	65.7	90.8	87.3	47.9	79.8	79.1
Table 5: Results on GLUE dev sets. We report matched accuracy for MNLI, Matthews correlation for CoLA, Pearson correlation for STS-B, F1 score for QQP, and accuracy for all other tasks. Bold indicates best results, underline indicates second best.
Figure 6: Comparison of typo resilience against mixed random typos (left) and common human typos (right) when fine-tuning a pre-trained BERT-Base model on AG News dataset for RETVec and SentencePiece.
Results Overall, as reported in Table 5, BERT pre-trained with RETVec and RETVec-raw using MLM on 100k words achieves competitive results compared to SentencePiece. Additionally, using RETVec reduces the pre-training time of BERT by around  $10\%$  in our experiment, when training for an equivalent number of steps. Surprisingly, RETVec-raw is more competitive than RETVec, which we attribute to the fact that BERT-Base has more than enough capacity to learn a competitive representation without RETVec's word embedding model. However, this increased competitiveness for RETVec-raw comes at the cost of decreased typo resilience, as reported in Figure 6, although the difference in typo resilience between pre-trained BERT models using RETVec-raw versus RETVec is only significant when the percentage of injected typos exceeds  $50\%$  of the words. As such, it seems that using RETVec-raw when pre-training large models is the best choice. How to optimally do so is left for future work.
9 Evaluation: In the Wild
As a final evaluation, to ensure RETVec will work in practice against adversarial content, we trained a version of a transformer model we use as part of our spam email filtering system at Google. As with previous experiments, we kept everything constant besides swapping SentencePiece with RETVec. Benchmarking on our internal production evaluation dataset, we found that RETVec improved the recall at  $0.80\%$  false positive rate by  $\sim 0.51\%$ , helping detect  $38\%$  of the spam emails that would have been missed by the baseline model while reducing model serving latency by  $30\%$ . This result gives us confidence that RETVec is competitive for real-world classification tasks, especially in the face of adversarial content.
10 Ablation Study
In this section, we report how key hyperparameters variations affect RETVec's performance. An unforeseen challenge we discovered while developing RETVec models is that a lower pre-training loss didn't directly translate into meaningful performance on our benchmarks. To overcome this
challenge, we had to include an extra validation step that trains and evaluates models on the Multilingual Amazon Reviews dataset [19] using the methodology described in Section 5 to measure downstream improvement. We choose the Amazon dataset due to its multilingual nature and difficulty. Detailed results on classification performance by model type, pre-training loss, and model size for the ablation studies can be found in Appendix C, along with additional experiments on less impactful hyperparameters.
Architecture	Loss	Accuracy
MLP	0.0248	92.89%
BERT + MLP	0.0129	92.54%
T5 + MLP	0.0120	92.72%
GAU + MLP	0.0133	92.82%
LSTM + MLP	0.0179	92.61%
CNN + MLP	0.0214	92.72%
Table 6: Ablation studies for RETVec architecture (left), word embedding dimension (middle), and word length (right). **Bold** denotes the hyperparameter selected for the final RETVec model.
Emb Dim	Loss	Acc
100	0.0267	92.6%
128	0.0260	92.7%
200	0.0253	92.7%
256	0.0248	92.9%
300	0.0245	92.8%
384	0.0237	92.8%
Word Len	Loss	Acc
12	0.0267	92.8%
14	0.0259	92.8%
16	0.0248	92.9%
20	0.0242	92.7%
24	0.0234	92.9%
32	0.0235	92.8%
Model architecture Despite considerable efforts, we couldn't find a more complex architecture that outperformed the simple MLP architecture (Table 6). The best transformer-based architecture which uses 2 encoder blocks of dimension 128 resulted in significantly lower loss regardless of the transformer block used (BERT [7], GAU [13], T5 [32]). However, as alluded to earlier, the significantly lower loss did not translate to better performance on the benchmarks. We hypothesize that the fitting the text manifold is an easy enough task to solve with any reasonable architecture.
Embedding Size We train RETVec with different embedding dimensions and report the results in Table 6. Classification benchmarks show little difference in performance for embedding dimensions between 128 and 512, with 256 embedding dimension having the slightest edge.
Max Input Character length To select the optimal max word length for RETVec, we started by looking the fastText dataset's word length distribution. More than  $95\%$  of the words are less than 16 characters long and the median word length is 7.9 characters (Appendix I). Using more than 16 characters didn't led to any accuracy improvements despite those longer inputs exhibiting a lower loss, as reported in Table 6.
11 Discussion and Future Work
In this paper, we presented RETVec, a new multilingual text vectorizer that combines a novel character encoding with an small model to project words into a compact, 256-dimensional embedding. Through extensive evaluations, we demonstrated that models trained using RETVec achieve state-of-the-art classification performance and provide a  $10 - 15\%$  increase in resilience against typos and adversarial attacks compared to popular text vectorizers and word embeddings.
The key outstanding question is how to best utilize RETVec for generative tasks and in large language models (LLMs), which could enable us to train LLMs with stronger multilingual capabilities, improved adversarial robustness, and reduced model size and computational costs. In particular, for "small" LLMs (less than or around 1 billion total parameters), the vocabulary embedding layer can be more than  $20\%$  of the total parameters [3], which would be eliminated when using RETVec.
We discovered that the main difficulty when training generative models using RETVec is that RETVec's 256-float embedding cannot be directly converted into a softmax output, unlike other tokenizers which represent text as integer token IDs. In order to effectively use RETVec in decoder-only models, we expect that a new training procedure for RETVec-based models which is compatible with text generation tasks needs to be invented, e.g. a pre-training procedure which does not rely on directly predicting the next token. We briefly experimented with ideas including decoding the RETVec representation character-by-character or using a VQ-VAE [28] model to quantize the output, but the results so far were inconclusive. We hope to explore this area and address this limitation in future work. Other potential directions of future work include using RETVec as a word embedding in various applications in place of GloVe [22] and word2vec [20], and using the RETVec character encoder and training procedure to train text similarity models.
[21] J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, Oct. 2020.
[22] J. Pennington, R. Socher, and C. Manning. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar, Oct. 2014. Association for Computational Linguistics.
[23] D. Pruthi, B. Dhingra, and Z. C. Lipton. Combating Adversarial Misspellings with Robust Word Recognition, Aug. 2019.
[24] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, July 2020.
[25] F. Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823, June 2015.
[26] R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units, June 2016.
[27] Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei. Circle Loss: A Unified Perspective of Pair Similarity Optimization, June 2020.
[28] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, 2018.
[29] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[30] X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott. Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning, Mar. 2020.
[31] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell. Distance metric learning, with application to clustering with side-information.
[32] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models, Mar. 2022.
[33] X. Zhang, J. Zhao, and Y. LeCun. Character-level Convolutional Networks for Text Classification, Apr. 2016.
[34] Z. Zhang and V. Saligrama. Zero-Shot Learning via Joint Latent Similarity Embedding, Aug. 2016.
[35] S. Zhuang and G. Zuccon. Dealing with Typos for BERT-based Passage Retrieval and Ranking, Sept. 2021.
A RETVec Model Details
Table 7 details the hyperparameter settings for the RETVec model architecture, as described in Section 3.
Hyperparameter	RETVec
Max word length	16
Per-character encoding dim	24
Activation	GeLU
# of projection layers	1
Projection layer dim	32
# of fully-connected layers	2
Fully-connected layer dim	256
Spatial dropout rate	0.0625
Dropout rate	0
Embedding activation	Tanh
Embedding dim	256
Similarity dim	256
Table 7: RETVec model hyperparameter details.
B Benchmarking Models
In this section, we provide more detailed model hyperparameters for the evaluation models used in Section 5 and Section 8.
RNN We used a Stacked-LSTM architecture, with the following hyperparameters:
Dim: 256
Layers: 4
- Dropout rate: 0.1
DPCNN We use the architecture described in [15], with the following hyperparameters:
- Filters: 256
Layers: 6
- Kernel size: 3
- Final dropout: 0.5
- Activation: ReLU
BERT-Mini We use the architecture as described in [7] for BERT-Mini, with the following hyperparameters:
Layers: 4
Hidden dim: 256
Intermediate dim: 1024
Self-attention heads: 4
- Dropout rate: 0.1
- Activation: GeLU
BERT-Base We use the architecture described in [7] for BERT-Base, with the following hyperparameters:
Layers: 12
- Hidden dim: 768
Intermediate dim: 3072
- Self-attention heads: 12
- Dropout rate: 0.1
- Activation: GeLU
C RETVec Ablation Studies
In this section, we present our ablation study results for the RETVec model design and hyperparameter selection. Results are reported on the Multilingual Amazon Reviews dataset following the methodology described in Section 10.
C.1 Embedding Dimension
Detailed results on how RETVec's embedding layer dimension affects classification performance are reported in Table 8.
Embedding Dim	Pre-training Loss	# Params	Test Accuracy
RNN	CNN	BERT	AVG
64	0.0284	181k	92.8%	91.8%	92.5%	92.4%
100	0.0267	190k	93.2%	92.1%	92.4%	92.6%
128	0.0260	197k	93.3%	92.2%	92.5%	92.7%
200	0.0253	216k	93.4%	92.3%	92.6%	92.7%
256	0.0248	230k	93.6%	92.3%	92.8%	92.9%
300	0.0245	241k	93.6%	92.3%	92.6%	92.8%
384	0.0237	263k	93.6%	92.2%	92.6%	92.8%
512	0.0225	296k	93.7%	92.3%	92.8%	92.9%
C.2 Model Architecture
Detailed results on the effect of RETVec architecture type on classification performance are reported in Table 9.
Table 8: Ablation study results on the effect of the RETVec pre-trained model's embedding dimension on classification performance. Bold denotes the hyperparameter selected for the final RETVec model.
Architecture	Pre-training Loss	Test Accuracy
RNN	CNN	BERT	AVG
MLP	0.0248	93.6%	92.3%	92.8%	92.9%
MLP + BERT	0.0129	93.2%	92.0%	92.4%	92.5%
MLP + T5	0.0120	93.5%	92.2%	92.5%	92.7%
MLP + GAU	0.0133	93.5%	92.2%	92.8%	92.8%
MLP + LSTM	0.0179	93.4%	91.9%	92.5%	92.6%
MLP + CNN	0.0214	93.5%	92.1%	92.6%	92.7%
Table 9: Ablation study results on RETVec model architecture type on classification performance. Bold denotes the hyperparameter selected for the final RETVec model.
C.3 Maximum Word length
Detailed results on how RETVec's maximum input word length affects classification performance are reported in Table 10.
Word Len	Pre-training Loss	# Params	Test Accuracy
RNN	CNN	BERT	AVG
8	0.0428	181k	93.6%	92.3%	92.7%	92.8%
10	0.0326	193k	93.5%	92.4%	92.6%	92.8%
12	0.0267	206k	93.5%	92.2%	92.7%	92.8%
14	0.0259	218k	93.5%	92.2%	92.6%	92.8%
16	0.0248	230k	93.6%	92.3%	92.8%	92.9%
20	0.0242	255k	93.5%	92.2%	92.6%	92.7%
24	0.0234	280k	93.6%	92.3%	92.7%	92.9%
28	0.0246	304k	93.5%	92.2%	92.7%	92.8%
32	0.0235	328k	93.6%	92.2%	92.6%	92.8%
C.4 Pre-training Loss Hyperparameters
Detailed results on the effect of Multi-Similarity loss hyperparameters on RETVec classification performance are reported in Table 11. We also experimented with Circle Loss [27] and report the results in Table 12.
Table 10: Ablation study results on the effect of maximum word length on RETVec classification performance. **Bold** denotes the hyperparameter selected for the final RETVec model.
Hyperparameter	Test Accuracy
α	β	λ	Pre-training Loss	RNN	CNN	BERT	AVG
2	20	0.5	0.0432	93.5%	92.1%	92.4%	92.7%
2	20	1.0	0.2643	92.9%	91.2%	92.6%	92.2%
2	40	0.5	0.0455	93.6%	92.1%	92.6%	92.7%
2	40	1.0	0.3610	92.9%	91.2%	92.4%	92.2%
2	80	0.5	0.0464	93.5%	92.3%	92.6%	92.8%
2	80	1.0	0.3583	92.6%	91.1%	92.4%	92.0%
4	20	0.5	0.0270	93.4%	92.0%	92.6%	92.7%
4	20	1.0	0.1537	93.0%	91.3%	92.4%	92.2%
4	40	0.5	0.0242	93.5%	92.4%	92.6%	92.8%
4	40	1.0	0.1919	92.8%	91.3%	92.4%	92.2%
4	80	0.5	0.0248	93.6%	92.3%	92.8%	92.9%
4	80	1.0	0.1851	92.8%	91.2%	92.4%	92.1%
Table 11: Ablation study results on the effect of Multi-Similarity loss hyperparameters on RETVec classification performance.  $\epsilon = 0.1$  is fixed for all experiments. Bold denote the hyperparameters selected for the final RETVec model.
Circle-Loss Hyperparameter	Test Accuracy
Scale Factor γ	Relaxation Factor m	Pre-training Loss	RNN	CNN	BERT	AVG
64	0.3	7.55	93.3%	91.9%	92.6%	92.6%
64	0.4	2.77	93.5%	92.0%	92.4%	92.7%
64	0.5	0.63	93.6%	92.4%	92.6%	92.9%
128	0.3	12.85	93.3%	91.9%	92.6%	92.6%
128	0.4	5.06	93.6%	92.1%	92.6%	92.8%
128	0.5	1.18	93.6%	92.3%	92.6%	92.9%
256	0.3	24.97	93.3%	91.9%	92.5%	92.5%
256	0.4	9.49	93.5%	92.0%	92.4%	92.7%
256	0.5	2.59	93.7%	92.4%	92.5%	92.9%
512	0.3	49.01	93.2%	92.0%	92.6%	92.6%
512	0.4	19.88	93.3%	92.2%	92.7%	92.7%
512	0.5	5.26	93.6%	92.4%	92.7%	92.9%
Table 12: Ablation study results for pre-training RETVec with various Circle Loss [27] hyperparameter settings.
C.5 Model Capacity
Detailed results on how the number and dimension of fully-connected dense layers in the RETVec model affects classification performance are presented in Table 13.
Model Capacity	# Params	Pre-training Loss	Test Accuracy
Dense Layers	Dense Layer Dim	RNN	CNN	BERT	AVG
0	-	99k	0.0445	92.8%	91.6%	92.3%	92.2%
1	128	82k	0.0383	93.5%	91.8%	92.4%	92.6%
1	256	164k	0.0312	93.5%	91.8%	92.4%	92.6%
1	384	246k	0.0284	93.6%	92.0%	92.6%	92.7%
1	512	328k	0.0258	93.6%	92.2%	92.8%	92.8%
2	128	989k	0.0334	93.4%	91.9%	92.6%	92.6%
2	256	230k	0.0248	93.6%	92.3%	92.8%	92.9%
2	384	394k	0.0201	93.5%	92.3%	92.6%	92.8%
2	512	591k	0.0177	93.7%	92.3%	92.6%	92.9%
3	128	115k	0.0314	93.4%	91.9%	92.5%	92.6%
3	256	296k	0.0213	93.6%	92.4%	92.5%	92.8%
3	384	542k	0.0175	93.6%	92.2%	92.7%	92.8%
3	512	854k	0.0149	93.5%	92.4%	92.6%	92.8%
C.6 Spatial Dropout Rate
Detailed ablation study results on the amount of spatial dropout in the RETVec model and its effect on classification performance are presented in Table 14. Increments of  $1/16$  were used because it corresponds to dropping out one character of the input on average, since RETVec's model accepts an input of up to 16 characters per word.
Table 13: Ablation study results on the effect of RETVec model capacity (number and dimension of the fully-connected layers) on classification performance. Bold denote the hyperparameters selected for the final RETVec model.
Spatial Dropout	Pre-training Loss	Test Accuracy
RNN	CNN	BERT	AVG
0.00%	0.0122	93.4%	91.4%	92.5%	92.4%
6.25%	0.0248	93.6%	92.3%	92.8%	92.9%
12.50%	0.0465	93.4%	92.0%	92.3%	92.6%
18.75%	0.0722	92.7%	91.5%	91.8%	92.0%
25.00%	0.0967	92.7%	91.4%	91.2%	91.8%
C.7 Pre-Training Objectives
We evaluated combining RETVec's pre-training objective (Multi-Similarity loss) with other objective functions and pre-training tasks as well. Specifically, we experimented with the following objectives: 1) augmentation position prediction, 2) augmentation position and type prediction, 3) decoding (predicting the character encoding of the input token), and 4) denoising (predicting the character encoding of the original, non-augmented token). Table 15 reports the results of our experiments on different pre-training objectives.
Table 14: Ablation study results on the effect of spatial dropout rate on the RETVec input character encoding. **Bold** denotes the value selected for the final RETVec model.
Objectives	Similarity Loss	Total Loss	RNN	CNN	BERT	AVG
Similarity, Augmentation Position Detection	0.0261	0.2228	93.4%	92.0%	92.8%	92.7%
Similarity, Augmentation Position and Type Prediction	0.0238	0.0970	93.2%	92.2%	92.6%	92.7%
Similarity, Decoding	0.0278	0.0384	93.5%	92.1%	92.6%	92.8%
Similarity, Denoising	0.0241	0.1088	93.3%	91.8%	92.7%	92.6%
Similarity	0.0248	0.0248	93.6%	92.3%	92.8%	92.9%
Table 15: Ablation study results on combining different pre-training objectives with similarity loss for RETVec pre-training.
D RETVec Pre-training Dataset Augmentations
Below, we provide the full list of character-level augmentations (broken down into four categories) used to generate typo-augmented words for the RETVec pre-training dataset, as described in Section 3.3.
- Deletion
- Insertion
- Repeated character insertion
- n-grams based prefix insertion for  $n = 3,4,5$
- n-grams based suffix insertion for  $n = 3,4,5$
- Random ASCII character insertion
- Language alphabet-based random character insertion
- Random punctuation insertion
- Random punctuation prefix
- Random punctuation suffix
- Random BMP Unicode insertion
- Random emoji prefix
- Random emoji suffix
- Substitution
Case substitution
- n-grams based substitution for  $n = 3,4,5$
- QWERTY keyboard typo substitution
- Homoglyphs substitution
- Random ASCII character substitution
- Language alphabet-based random character substitution
- Random puctuation substitution
- Random BMP Unicode substitution
- Transposition
- Neighboring character swap
- 3-character block random shuffle
E RETVec Pre-training Hyperparameters
We train RETVec using Multi-Similarity loss with hyperparameters  $\alpha = 4$ ,  $\beta = 40$ ,  $\epsilon = 0.1$  and  $\lambda = 0.5$ . Detailed pre-training hyperparameters are reported in Table 16.
Hyperparameter	Pre-training
Training steps	500k
Batch size	1024
Adam ε	1.00e-7
Adam β1	0.9
Adam β2	0.999
Weight decay	0
Peak learning rate	0.001
End learning rate	0.0001
Warmup steps	10000
Decay function	Cosine
Table 16: RETVec pre-training optimizer hyperparameters.