SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Elie Bursztein
Google elieb@google.com
Marina Zhang
Google marinazh@google.com
Owen Vallis
Google ovellis@google.com
Xinyu Jia
Google jiaxinyu@google.com
Alexey Kurakin
Google kurakin@google.com
This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pretrained using pair-wise metric learning to be robust against typos and character-level adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec.
Researchers have proposed many techniques for converting text into dense representations suitable for training neural networks, from the early use of character bi-grams and tri-grams to more modern semantic word embeddings and state-of-the-art subword vectorizers. Each of these approaches aims to mitigate issues caused by out-of-vocabulary (OOV) tokens, including intentional text perturbations such as adversarial attacks and unintentional ones such as typos. These approaches leverage techniques such as subword-level tokenization [17] and decomposing unknown words into n-grams [4].
However, we find that each of these approaches suffers from at least one of following drawbacks:
- They have insufficient resilience against typos and adversarial attacks [21].
- They require large dictionaries and embedding lookup tables.
- They exhibit poor performance on certain languages or in multilingual settings.
RETVec addresses these issues by combining a novel UTF-8 character encoder with an optional small model (230k parameters). This model projects encoded words into a 256 dimensional syntactic metric embedding as detailed in Section 3. RETVec embeddings are trained using pair-wise metric learning [30], ensuring that words containing typos are embedded close to the original word.
RETVec does not require dataset pre-processing and does not have OOV tokens because it accepts all valid UTF-8 characters. RETVec's embedding model is trained on a word dataset with more than 157 languages. RETVec is also space-efficient (<1MB) since it does not require a large embedding lookup table. This reduces model memory footprints, making RETVec-based models ideal candidates for on-device model deployment where storage, bandwidth, and memory resources are scarce. Through a series of extensive experiments on different model architectures, and on a wide range of datasets, we demonstrate that RETVec outperforms or is comparable to state-of-the-art vectorizers and word
embeddings including BPE [26], SentencePiece [17], and fastText [4]. Overall, RETVec outperforms other vectorizers on text classification tasks by about  $1\%$ , while being up to  $15\%$  more resilient to typos at  $20\%$  word typo rate, and less susceptible to character-level adversarial attacks by over  $10\%$ .
- We introduce RETVec, a resilient, efficient, and multilingual text vectorizer designed for neural-based text processing.
- We show that RETVec is faster and less memory intensive than other vectorizers on multicore CPUs and on GPUs.
- We demonstrate that models trained with RETVec have slightly higher accuracy, greater resilience to typos, and significantly better resilience to adversarial attacks compared to models trained with the other vectorizers.
- We provide a TensorFlow implementation of RETVec, including its pre-trained models, alongside the code to reproduce our benchmarks under the Apache 2 license at https://github.com/google-research/retvec.
Text Vectorizers Text vectorizers play a major role in deep neural networks' (DNN) performance by ensuring that the input text is properly segmented into tokens (typically words or sub-words) and embedded into a dense, float-point representation for the model to use. The traditional approach for text vectorization splits texts into words using whitespace and punctuation and uses a word embedding lookup table to map each word into a dense vector. The embedding lookup table can be constructed using algorithms such as Word2Vec [20], GloVe [22] and fastText [4], or by randomly initializing an embedding table and training it as a part of the LLM pre-training process (like GPT-2 [2]). Embedding lookup tables are typically large and limited to known, in-vocabulary tokens which significantly reduces their representational capabilities in the presence of typos and adversarial attacks.
Subword tokenizers such as WordPiece [7], BPE [26] and SentencePiece [17]) split text into subwords to reduce vocabulary size and mitigate OOV issues. SentencePiece is used by many state-of-the-art text models [24, 5]. All of these vectorizers require a separate training phase to adapt to the targeted dataset and require shipping extra files with the models to be used at inference time.
Pair-based learning The RETVec model is trained using pair-based metric learning. Pair-based learning aims to learn an embedding space, where embedded vectors of similar samples are encouraged to be closer together while dissimilar ones are pushed apart from each other [31]. This approach has been successfully applied to various tasks including image retrieval [12, 30, 27], face recognition [25], and zero-shot learning [34].
Typos and Adversarial Attacks One challenge faced by text vectorizers is the existence of typographical errors that might not appear in the training dataset. At inference time, these typographical errors can lead the vectorizer to incorrectly output OOV ids for valid words. This behavior ultimately leads to lower accuracy, particularly for retrieval tasks where human queries are known to exhibit between  $16\%$  and  $19\%$  typos rate [35, 11]. In many anti-abuse settings such as spam classification, the typo rate used by attackers in attempt to evade defense systems is much higher [6]. It is also well-known that text models are vulnerable to different types of adversarial attacks [21]. Many character-level adversarial attacks, which can be viewed as the most severe form of intentional typos, have been proposed in the literature ([18], [9], [23]).
3 RETVec
In this section, we describe how RETVec works. We start off by introducing its novel character encoding scheme, detail the design of its model, and finally describe its pre-training procedure.
3.1 RETVec Character Encoder
The RETVec character encoder, as shown in Figure 1, is composed of two layers: an Integerizer layer and a Binarizer layer. The character encoder enables RETVec to encode all UTF-8 characters
Figure 1: RETVec architecture overview - the output shape of each layer is in parenthesis. The clen indicates the number of characters used per word - 16 characters by default. The batch and word sequence length dimensions are omitted.
in an efficient and stateless manner. Encoding a word is achieved by first converting its characters into their UTF-8 codepoints (Integerizer) and then converting this integer representation into a flattened, binary (little-endian) representation (Binarizer). The Binarizer uses a compact 24-bit binary character representation to encode all valid UTF-8 characters. Additionally, RETVec outputs a fixed-length, dense word embedding using a maximum of 16 characters per word. In the ablation study (Section 10), we empirically find that 16 characters per word works well and increasing the max word length provides no further accuracy gains.
The Integerizer is similar to the one used in ByT5 [32] which also encodes characters into their UTF-8 codepoints. What makes the RETVec encoder unique is that the codepoints are first converted to their binary representation and then flattened to create the word representation. This novel binary representation is both easy to learn and significantly more compact than a one-hot representation. We tested many other approaches to convert the Integerizer output into a more compact form, including the use of a single float per character, but these more compact representations did not appear to be learnable. Additionally, a key design choice is to keep the Integerizer and the Binarizer separated for efficiency reasons: when using a remote accelerator, such as a TPU, it is more efficient to send the output of the Integerizer  $(16 \times \text{int} 32)$  across the network rather than directly sending the binarization representation  $(16 \times 24 \text{float} 32)$ .
3.2 RETVec Model Architecture
While the RETVec Binarizer provides an efficient word representation, it is not competitive with state-of-the-art vectorizers. To address this, we add a small model on top of the Binarizer output. This improves accuracy and enables RETVec to outperform other vectorizers (see Section 5). Additionally, this reduces the embedding size from 384 to 256 float32s and further improves resilience to adversarial attacks (see Section 7). The performance gains only incur a small increase in computational cost, making the use of the pre-trained model a worthwhile trade-off. To better understand what gains can be attributed to the character encoder, the remainder of the paper reports the performance for "RETVec with the model" as RETVec, and to understand what can be attributed to the model we report "RETVec without the model" as RETVec-raw.
The RETVec model, as visible in Figure 1, is composed of a dense projection layer, a flatten layer, a dense compression layer, and the embedding layer. The embedding layer is a dense layer with a tanh activation to scale the embedding values between 0 and 1, while the rest of the network uses gelu activations. Increasing capacity by adding additional layers or using more powerful architectures did not improve performance on our benchmarks (Section 10). Detailed ablation studies on loss hyperparameters, dropout rates, activation functions, and model capacity can be found in Appendix C.
3.3 RETVec Pre-training Procedure
Dataset The RETVec model is pre-trained on a typo-augmented version of the 157-language fastText words datasets [10]. The original words are extracted from the Common Crawl corpus. Our dataset then takes the set of words over the union of all 157 languages, yielding 88.8M unique
tokens. To help with generalization, we supplement this dataset by adding 8.88M (10%) randomly generated tokens, where the length of a token is randomized between 1 and 16. Each token consists of a random set of UTF-8 characters. Finally, we create 20 versions of each token (16 augmented and 4 non-augmented versions) such that  $80\%$  of the tokens in the training set are typo-augmented. At the end of the generation process we end-up with a combined 1.9B token dataset.
Augmentations Token augmentation consists of randomly inserting up to 4 typos per token up to  $25\%$  of the token length. This is consistent with an observed maximum human error frequency of around  $20\%$  [11]. We use 22 distinct typo augmentations, which can be grouped into four categories: deletion, insertion, substitution, and transposition. For each token, we randomly select a target augmentation percentage between  $0 - 25\%$ , and for each augmentation step we randomly apply an augmentation from one of the four typo categories. The full list of augmentations used is reported in Appendix D.
Training RETVec model is trained using pair-wise learning and uses the Multi-Similarity Loss [30]  $(\alpha = 4, \beta = 40, \lambda = 0.5,$  and  $\epsilon = 0.1)$ . We construct example pairs by creating batches that always contain two variations of the same word. The model is trained for  $500k$  steps with batch size  $= 1024$ , using Adam with max learning rate  $= 0.001$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and cosine decaying the learning rate to 0.0001 during training. Full training hyperparameters can be found in Appendix E. We experimented with additional losses and other self-supervised tasks, reported in Appendix C, but none of them yielded better results.
Figure 2: The cosine similarity distributions of RETVec embeddings for 1000 pairs of augmented and non-augmented versions of words, selected languages shown. 'Random' language refers to randomly-generated UTF-8 strings.
Embedding Quality To visualize and evaluate the training quality and embedding consistency across languages, we randomly sample pairs of augmented and non-augmented words from select languages and compare the distribution of their cosine similarities. As shown in Figure 2, we observe that the mean and variance of the cosine similarity distributions are fairly consistent across various languages, which suggests that the model has learned near-uniformly across languages. The mean cosine similarity between word pairs in Chinese is slightly lower, which could be attributed to the fact that the average length of words is significantly shorter in Chinese. The distribution for random strings is also consistent with other languages' distributions, suggesting that RETVec is able to generalize to previously unseen words and embed words with rare UTF-8 characters meaningfully, which would greatly contribute towards its adversarial resilience. In Appendix J, we used the TensorBoard Embedding Projector to visualize RETVec embeddings in a 3D space and demonstrate that syntactically-similar words are clustered together, adding further evidence that RETVec is meaningfully projecting tokens into the embedding space.
4 Evaluation: Speed
In this section, we evaluate how quickly RETVec can process and vectorize datasets compared to the other commonly used vectorizers. We considered SentencePiece [17], BPE [26], a simple
Name	Embedding Size	Vocabulary Size	Preprocessing Time	System Memory (GB)	Model Params	CPU	GPU
Wall time (s)	Usage	Core Sec	Wall (s)	Usage	Memory (GB)
SentencePiece	256	32k	749	10.9	8M	877	124%	1087	870	100%	0.5
BPE	256	32k	861	3.6	8M	1062	169%	1795	1050	100%	0.5
FastText	300	-	420	43.9	0	845	52%	439	-	-	-
Whitespace	256	32k	132	1.7	8M	496	126%	625	396	100%	0.5
RETVec-raw	384	-	0	1.9	0	225	451%	1015	235	20%	0.4
RETVec	256	-	0	2.1	230K	570	840%	4789	290	40%	0.5
Table 1: Comparison of speed, preprocessing time, memory usage, and CPU/GPU usage for all benchmarked vectorizers when vectorizing the Multilingual Amazon Reviews training dataset.
Name	Train Size	Test Size	# Classes	# Languages	OOV Words	Avg Sentence Length	Avg Word Length
AG News [33]	120k	7.6k	4	1	9.9%	45	4.4
Yelp Reviews (Polarity) [33]	560k	38k	2	1	17.4%	160	3.7
Multilingual Amazon Reviews (Polarity) [19]	960k	24k	2	6	35.2%	27	5.0
MASSIVE (Intent Classification) [8]	587k	152k	60	51	35.3%	6	5.1
Table 2: List of datasets used for classification evaluation.
whitespace vectorizer, and fastText [4]. We use the HuggingFace implementation of BPE and the official SentencePiece implementation.
Setup We use the Multilingual Amazon Reviews [19] dataset for the vectorization speed evaluations. The corpus is composed of 1.2M tokens - 200k tokens for 6 languages (English, French, German, Spanish, Chinese, Japanese). All the experiments are run on a standard Google Cloud VM with 16 CPU cores and a V100 Nvidia GPU. We disable the GPU visibility to perform the CPU measurements and use the GNU time command to measure the total wall time, CPU usage, and system memory. We use nvidia-smi to record GPU usage and memory. We report both the time it takes to adapt to the datasets and how long it takes to vectorize the entire dataset end-to-end, with and without GPU acceleration. We also report normalized CPU-core times as RETVec (and other vectorizers to a lesser extent) benefits from multi-threaded acceleration. Finally, we report memory usage for both system and GPU, since they are important metrics when developing models for memory-constrained devices such as IoT devices and smartphones.
Results We found that RETVec-raw and RETVec are the fastest vectorizers when a GPU is available and are in the top three when multi-core CPU are available (Table 1), which is common on current devices including smartphones. RETVec's performance on CPU stems from its ability to efficiently use the available cores, as visible in the Core Sec column – we expect that more optimized versions of SentencePiece and BPE could achieve similar performance.
5 Evaluation: Classification
In this section, we compare the classification training performance of RETVec against other vectorizers and word embeddings. We train the models from scratch and report classification accuracy on a wide range of datasets to benchmark the generality of the approach.
Setup In order to perform a comprehensive evaluation, we evaluate classification performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths, as summarized in Table 2. For example, the MASSIVE [8] intent classification dataset has very short sentences but a lot of languages (51), whereas the Yelp Reviews dataset has significantly longer texts but is monolingual.
We benchmark performance on three of the most common model architectures for text classification: Transformer (BERT-Mini [7]), CNN (DPCNN [15]) and LSTM (Stacked-LSTM [1]), details provided in Appendix B. To keep evaluation fair and consistent when comparing various vectorizers, we keep the models the same and only swap the vectorizers. All models are implemented in TensorFlow 2.11 and training is conducted on a Google Cloud VM using a single NVidia V100 GPU.
All models are trained with Adam optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and a max learning rate of 5e-4, with the exception of fastText on multilingual datasets. We found that a max learning rate of 5e-4 was too high and led to divergence, so we used 1e-4 instead for fastText on MASSIVE and Multilingual Amazon Reviews. For BERT, we linearly warmup the learning rate for 5k steps. All
models are trained for 100k steps with batch size 256 and cosine learning rate decay to 0. The test accuracy of each model averaged over 3 trials with different random seeds is reported in Table 3.
Dataset	Model	Whitespace	SentencePiece	BPE	fastText	RETVec-raw	RETVec
AG News	RNN	91.6%	91.3%	91.3%	92.7%	91.3%	92.6%
CNN	92.5%	92.4%	90.7%	92.3%	88.2%	91.2%
BERT	91.5%	91.5%	92.1%	93.4%	93.1%	93.5%
AVG	91.9%	91.7%	91.4%	92.8%	90.9%	92.4%
Yelp P.	RNN	93.9%	93.3%	93.4%	94.7%	94.1%	94.7%
CNN	94.4%	93.8%	93.0%	93.9%	92.5%	93.4%
BERT	91.6%	91.3%	91.6%	93.5%	93.2%	92.8%
AVG	93.3%	92.8%	92.7%	94.0%	93.3%	93.6%
Multilingual Amazon P.	RNN	93.2%	90.7%	90.3%	87.0%	92.9%	93.5%
CNN	93.2%	90.8%	89.2%	84.4%	91.1%	92.2%
BERT	91.8%	87.3%	87.2%	86.6%	92.7%	92.6%
AVG	92.8%	89.6%	88.9%	86.0%	92.2%	92.8%
MASSIVE	RNN	70.3%	69.6%	68.6%	13.5%	71.0%	73.8%
CNN	70.4%	69.2%	59.2%	12.6%	59.8%	66.7%
BERT	70.6%	67.8%	66.2%	23.9%	78.1%	78.6%
AVG	70.4%	68.9%	64.7%	16.7%	69.6%	73.0%
Average	RNN	87.3%	86.2%	85.9%	72.0%	87.3%	88.7%
CNN	87.6%	86.5%	83.0%	70.8%	82.9%	85.9%
BERT	86.4%	84.5%	84.3%	74.4%	89.3%	89.4%
AVG	87.1%	85.8%	84.4%	72.4%	86.5%	88.0%
Table 3: Detailed classification results comparing different vectorizers when used to train models from scratch. **Bold** indicates best results, **underline** indicates second best.
Figure 3: Classification performance on Multilingual Amazon Reviews broken down by language.
Results Overall, averaging over all datasets and model architectures, RETVec is the best performing vectorizer by some margin (0.9%). We observe that RETVec performs best when paired with Transformer and RNN architectures, but lags slightly behind for the CNN. Compared to RETVec-raw, RETVec's pre-trained model offers a significant performance improvement across all datasets and model architectures.
fastText performs the best on average for the English-only datasets, but does not perform well on the two multilingual datasets (especially MASSIVE), which explains its overall poor performance. Given that we are using the same code and models for all datasets and the fact that fastText results on English-only datasets match the ones reported in the original paper [16], we believe that fastText's poor performance on multilingual datasets is due to the fact that fastText vectors from different languages may not be compatible with each other when used as inputs to a multilingual model. However, we didn't find any mention of such an issue in the literature.
Delving deeper into multilingual capabilities, as plotted in Figure 3, we observe that RETVec consistently outperforms other vectorizers regardless of the language. We also observe that RETVec has a wider lead on non-Latin languages, which we attribute at least partially to our novel character encoder, given that RETVec and RETVec-raw achieve similar performance on them.
Figure 4: Comparison of various tokenizers' resilience against mixed random typos (left) and common human typos (right) when training classification models from scratch.
Figure 5: Comparison of RETVec resilience against various types of typos. RETVec-raw on the left, RETVec on the right.
6 Evaluation: Tipo Resilience
In this section, we evaluate the resilience of the various vectorizers against both random typos and common human typos.
Setup We rely on two types of typo injection methods to evaluate the tokenizers' typo resilience: random typos and common human typos. For random typos, evaluation is performed on every model and dataset used in Section 5 and average accuracy on the typo-augmented test splits is recorded. For common human typos, because we only have them for English, we take the average accuracy across all three models on only AG News. We evaluate how each tokenizer's resilience degrades as the number of typos increases by gradually increasing the percentage of words with typos in the test set by increments of  $10\%$ .
Random typos are created by using a combination of insertion, deletion, substitution, neighboring swap, and keyboard-based typos. Each typo is applied with a block size of 1 or 2 characters. Words are selected at random and each word can only have one typo. To create human-like typos, we use the word replacement noiser from the Neusspell [14] package. The noiser uses  $109\mathrm{k}$  common misspellings of  $17\mathrm{k}$  popular English words. On the AG News test set, it can only inject typos into  $55\%$  of the words, so we limit the evaluation to  $50\%$  words with common human typos.
Results As reported in Figure 4, the performance of RETVec and RETVec-raw decreases significantly slower than the other vectorizers as the amount of typos increase. RETVec's pre-trained model increases RETVec resilience by up to  $15\%$  compared to RETVec-raw, demonstrating the effectiveness of RETVec's pre-training technique of using pair-wise metric learning to create syntactically robust word embeddings. SentencePiece and BPE perform about the same, with a gradual decline which keeps them competitive with RETVec-raw until about  $60\%$  random typo rate. We note that fastText and Whitespace provide more typo resilience than SentencePiece and BPE, which is expected because it has been previously noted that word-level vectorizers are typically more robust than character-level
Tokenizer	Original Acc	Acc under Atk	Atk Success %
Whitespace	89.4%	32.3%	63.9%
SentencePiece	89.8%	25.8%	71.3%
BPE	90.8%	29.2%	67.8%
fastText	92.6%	40.5%	56.3%
RETVec-raw	93.0%	50.6%	45.6%
RETVec	93.7%	51.7%	44.8%
Table 4: Adversarial resilience evaluation for BERT-Mini trained on AG News with different vectorizers. Adversarial attack results are averaged across three types of character-level adversarial attacks: TextBugger [18], DeepWordBug [9], and Pruthi [23]. Bold indicates best results, underline indicates second best.
or subword-level vectorizers [23]. We see similar trends on the common human typo results, with RETVec-raw and RETVec being noticeably more resilient than baseline vectorizers when the typo rate is  $>30\%$ .