Title: RETVec: Resilient and Efficient Text Vectorizer
Venue: NeurIPS 2023
Accepted: yes
Keywords: language modeling

text embedding

adversarial text attack

text vectorization
Abstract: This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning to be robust against typos and character-level adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec.

RETVec: Resilient and Efficient Text Vectorizer

Abstract

This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pretrained using pair-wise metric learning to be robust against typos and characterlevel adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec .

1 Introduction

Researchers have proposed many techniques for converting text into dense representations suitable for training neural networks, from the early use of character bi-grams and tri-grams to more modern semantic word embeddings and state-of-the-art subword vectorizers. Each of these approaches aims to mitigate issues caused by out-of-vocabulary (OOV) tokens, including intentional text perturbations such as adversarial attacks and unintentional ones such as typos. These approaches leverage techniques such as subword-level tokenization [17] and decomposing unknown words into n-grams [4].

However, we find that each of these approaches suffers from at least one of following drawbacks:

They have insufficient resilience against typos and adversarial attacks [21].

They require large dictionaries and embedding lookup tables.

They exhibit poor performance on certain languages or in multilingual settings.

RETVec addresses these issues by combining a novel UTF-8 character encoder with an optional small model (230k parameters). This model projects encoded words into a 256 dimensional syntactic metric embedding as detailed in Section 3. RETVec embeddings are trained using pair-wise metric learning [30], ensuring that words containing typos are embedded close to the the original word.

RETVec does not require dataset pre-processing and does not have OOV tokens because it accepts all valid UTF-8 characters. RETVec's embedding model is trained on a word dataset with more than 157 languages. RETVec is also space-efficient (<1MB) since it does not require a large embedding lookup table. This reduces model memory footprints, making RETVec-based models ideal candidates for on-device model deployment where storage, bandwidth, and memory resources are scarce. Through a series of extensive experiments on different model architectures, and on a wide range of datasets, we demonstrate that RETVec outperforms or is comparable to state-of-the-art vectorizers and word

embeddings including BPE [26], SentencePiece [17], and fastText [4]. Overall, RETVec outperforms other vectorizers on text classification tasks by about 1%, while being up to 15% more resilient to typos at 20% word typo rate, and less susceptible to character-level adversarial attacks by over 10%.

Our paper makes the following contributions:

We introduce RETVec, a resilient, efficient, and multilingual text vectorizer designed for neural-based text processing.

We show that RETVec is faster and less memory intensive than other vectorizers on multicore CPUs and on GPUs.

We demonstrate that models trained with RETVec have slightly higher accuracy, greater resilience to typos, and significantly better resilience to adversarial attacks compared to models trained with the other vectorizers.

We provide a TensorFlow implementation of RETVec, including its pre-trained models, alongside the code to reproduce our benchmarks under the Apache 2 license at https://github.com/google-research/retvec .

2 Background

Text Vectorizers Text vectorizers play a major role in deep neural networks' (DNN) performance by ensuring that the input text is properly segmented into tokens (typically words or sub-words) and embedded into a dense, float-point representation for the model to use. The traditional approach for text vectorization splits texts into words using whitespace and punctuation and uses a word embedding lookup table to map each word into a dense vector. The embedding lookup table can be constructed using algorithms such as Word2Vec [20], GloVe [22] and fastText [4], or by randomly initializing an embedding table and training it as a part of the LLM pre-training process (like GPT-2 [2]). Embedding lookup tables are typically large and limited to known, in-vocabulary tokens which significantly reduces their representational capabilities in the presence of typos and adversarial attacks.

Subword tokenizers such as WordPiece [7], BPE [26] and SentencePiece [17]) split text into subwords to reduce vocabulary size and mitigate OOV issues. SentencePiece is used by many state-of-the-art text models [24, 5]. All of these vectorizers require a separate training phase to adapt to the targeted dataset and require shipping extra files with the models to be used at inference time.

Pair-based learning The RETVec model is trained using pair-based metric learning. Pair-based learning aims to learn an embedding space, where embedded vectors of similar samples are encouraged to be closer together while dissimilar ones are pushed apart from each other [31]. This approach has been successfully applied to various tasks including image retrieval [12, 30, 27], face recognition [25], and zero-shot learning [34].

Typos and Adversarial Attacks One challenge faced by text vectorizers is the existence of typographical errors that might not appear in the training dataset. At inference time, these typographical errors can lead the vectorizer to incorrectly output OOV ids for valid words. This behavior ultimately leads to lower accuracy, particularly for retrieval tasks where human queries are known to exhibit between 16% and 19% typos rate [35, 11]. In many anti-abuse settings such as spam classification, the typo rate used by attackers in attempt to evade defense systems is much higher [6]. It is also well-known that text models are vulnerable to different types of adversarial attacks [21]. Many character-level adversarial attacks, which can be viewed as the most severe form of intentional typos, have been proposed in the literature ( [18], [9], [23]).

3 RETVec

In this section, we describe how RETVec works. We start off by introducing its novel character encoding scheme, detail the design of its model, and finally describe its pre-training procedure.

3.1 RETVec Character Encoder

The RETVec character encoder, as shown in Figure 1, is composed of two layers: an Integerizer layer and a Binarizer layer. The character encoder enables RETVec to encode all UTF-8 characters

Integerizer

Removed

after training

(clen, 1)

int32

Model

(clen, 24)

float32

(384)

float32

Embedding

(256)

float32

Word

Character encoder

D(24)

(24)

(24)

...

Dense (384)

Dense (384)

Embedding

Similarity L2 Dense

in an efficient and stateless manner. Encoding a word is achieved by first converting its characters into their UTF-8 codepoints ( Integerizer ) and then converting this integer representation into a flattened, binary (little-endian) representation ( Binarizer ). The Binarizer uses a compact 24-bit binary character representation to encode all valid UTF-8 characters. Additionally, RETVec outputs a fixed-length, dense word embedding using a maximum of 16 characters per word. In the ablation study (Section 10), we empirically find that 16 characters per word works well and increasing the max word length provides no further accuracy gains.

The Integerizer is similar to the one used in ByT5 [32] which also encodes characters into their UTF-8 codepoints. What makes the RETVec encoder unique is that the codepoints are first converted to their binary representation and then flattened to create the word representation. This novel binary representation is both easy to learn and significantly more compact than a one-hot representation. We tested many other approaches to convert the Integerizer output into a more compact form, including the use of a single float per character, but these more compact representations did not appear to be learnable. Additionally, a key design choice is to keep the Integerizer and the Binarizer separated for efficiency reasons: when using a remote accelerator, such as a TPU, it is more efficient to send the output of the Integerizer ( 16 × int32) across the network rather than directly sending the binarization representation ( 16 × 24 float32)).

3.2 RETVec Model Architecture

While the RETVec Binarizer provides an efficient word representation, it is not competitive with state-of-the-art vectorizers. To address this, we add a small model on top of the Binarizer output. This improves accuracy and enables RETVec to outperform other vectorizers (see Section 5). Additionally, this reduces the embedding size from 384 to 256 float32 s and further improves resilience to adversarial attacks (see Section 7). The performance gains only incur a small increase in computational cost, making the use of the pre-trained model a worthwhile trade-off. To better understand what gains can be attributed to the character encoder, the remainder of the paper reports the performance for 'RETVec with the model' as RETVec, and to understand what can be attributed to the model we report 'RETVec without the model' as RETVec-raw.

The RETVec model, as visible in Figure 1, is composed of a dense projection layer, a flatten layer, a dense compression layer, and the embedding layer. The embedding layer is a dense layer with a tanh activation to scale the embedding values between 0 and 1, while the rest of the network uses gelu activations. Increasing capacity by adding additional layers or using more powerful architectures did not improve performance on our benchmarks (Section 10). Detailed ablation studies on loss hyperparameters, dropout rates, activation functions, and model capacity can be found in Appendix C.

3.3 RETVec Pre-training Procedure

Dataset The RETVec model is pre-trained on a typo-augmented version of the 157-language fastText words datasets [10]. The original words are extracted from the Common Crawl corpus. Our dataset then takes the set of words over the union of all 157 languages, yielding 88.8M unique

tokens. To help with generalization, we supplement this dataset by adding 8.88M (10%) randomly generated tokens, where the length of a token is randomized between 1 and 16. Each token consists of a random set of UTF-8 characters. Finally, we create 20 versions of each token (16 augmented and 4 non-augmented versions) such that 80% of the tokens in the training set are typo-augmented. At the end of the generation process we end-up with a combined 1.9B token dataset.

Augmentations Token augmentation consists of randomly inserting up to 4 typos per token up to 25% of the token length. This is consistent with an observed maximum human error frequency of around 20% [11]. We use 22 distinct typo augmentations, which can be grouped into four categories: deletion, insertion, substitution, and transposition. For each token, we randomly select a target augmentation percentage between 0-25%, and for each augmentation step we randomly apply an augmentation from one of the four typo categories. The full list of augmentations used is reported in Appendix D.

Training RETVec model is trained using pair-wise learning and uses the Multi-Similarity Loss [30] ( α = 4 , β = 40 , λ = 0 . 5 , and glyph[epsilon1] = 0 . 1 ). We construct example pairs by creating batches that always contain two variations of the same word. The model is trained for 500k steps with batch size = 1024, using Adam with max learning rate = 0.001, β 1 = 0 . 9 , β 2 = 0 . 999 , and cosine decaying the learning rate to 0.0001 during training. Full training hyperparameters can be found in Appendix E. We experimented with additional losses and other self-supervised tasks, reported in Appendix C, but none of them yielded better results.

Language

English

Cosine Similarity

0.9

French

German

0.8

Spanish

Chinese

0.7

Japanese

Random

0.6

0.5

English

French

German

Spanish

Chinese

Japanese

Random

Language

Embedding Quality To visualize and evaluate the training quality and embedding consistency across languages, we randomly sample pairs of augmented and non-augmented words from select languages and compare the distribution of their cosine similarities. As shown in Figure 2, we observe that the mean and variance of the cosine similarity distributions are fairly consistent across various languages, which suggests that the model has learned near-uniformly across languages. The mean cosine similarity between word pairs in Chinese is slightly lower, which could be attributed to the fact that the average length of words is significantly shorter in Chinese. The distribution for random strings is also consistent with other languages' distributions, suggesting that RETVec is able to generalize to previously unseen words and embed words with rare UTF-8 characters meaningfully, which would greatly contribute towards its adversarial resilience. In Appendix J, we used the TensorBoard Embedding Projector to visualize RETVec embeddings in a 3D space and demonstrate that syntactically-similar words are clustered together, adding further evidence that RETVec is meaningfully projecting tokens into the embedding space.

4 Evaluation: Speed

In this section, we evaluate how quickly RETVec can process and vectorize datasets compared to the other commonly used vectorizers. We considered SentencePiece [17], BPE [26], a simple

whitespace vectorizer, and fastText [4]. We use the HuggingFace implementation of BPE and the official SentencePiece implementation.

Setup Weuse the Multilingual Amazon Reviews [19] dataset for the vectorization speed evaluations. The corpus is composed of 1.2M tokens - 200k tokens for 6 languages (English, French, German, Spanish, Chinese, Japanese). All the experiments are run on a standard Google Cloud VM with 16 CPU cores and a V100 Nvidia GPU. We disable the GPU visibility to perform the CPU measurements and use the GNU time command to measure the total wall time, CPU usage, and system memory. We use nvidia-smi to record GPU usage and memory. We report both the time it takes to adapt to the datasets and how long it takes to vectorize the entire dataset end-to-end, with and without GPU acceleration. We also report normalized CPU-core times as RETVec (and other vectorizers to a lesser extent) benefits from multi-threaded acceleration. Finally, we report memory usage for both system and GPU, since they are important metrics when developing models for memory-constrained devices such as IoT devices and smartphones.

Results Wefound that RETVec-raw and RETVec are the fastest vectorizers when a GPU is available and are in the top three when multi-core CPU are available (Table 1), which is common on current devices including smartphones. RETVec's performance on CPU stems from its ability to efficiently use the available cores, as visible in the Core Sec column - we expect that more optimized versions of SentencePiece and BPE could achieve similar performance.

5 Evaluation: Classification

In this section, we compare the classification training performance of RETVec against other vectorizors and word embeddings. We train the models from scratch and report classification accuracy on a wide range of datasets to benchmark the generality of the approach.

Setup In order to perform a comprehensive evaluation, we evaluate classification performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths, as summarized in Table 2. For example, the MASSIVE [8] intent classification dataset has very short sentences but a lot of languages (51), whereas the Yelp Reviews dataset has significantly longer texts but is monolingual.

We benchmark performance on three of the most common model architectures for text classification: Transformer (BERT-Mini [7]), CNN (DPCNN [15] ) and LSTM (Stacked-LSTM [1]), details provided in Appendix B. To keep evaluation fair and consistent when comparing various vectorizers, we keep the models the same and only swap the vectorizers. All models are implemented in TensorFlow 2.11 and training is conducted on a Google Cloud VM using a single NVidia V100 GPU.

All models are trained with Adam optimizer with β 1 = 0 . 9 , β 2 = 0 . 999 and a max learning rate of 5e-4, with the exception of fastText on multilingual datasets. We found that a max learning rate of 5e-4 was too high and led to divergence, so we used 1e-4 instead for fastText on MASSIVE and Multilingual Amazon Reviews. For BERT, we linearly warmup the learning rate for 5k steps. All

models are trained for 100k steps with batch size 256 and cosine learning rate decay to 0. The test accuracy of each model averaged over 3 trials with different random seeds is reported in Table 3.

Language

75.0%

80.0%

85.0%

90.0%

95.0%

English

German

French

Spanish

Chinese

Japanese

RETVec

RETVec-raw

Whitespace

SentencePiece

BPE

fastText

Results Overall, averaging over all datasets and model architectures, RETVec is the best performing vectorizer by some margin (0.9%). We observe that RETVec performs best when paired with Transformer and RNN architectures, but lags slightly behind for the CNN. Compared to RETVec-raw, RETVec's pre-trained model offers a significant performance improvement across all datasets and model architectures.

fastText performs the best on average for the English-only datasets, but does not perform well on the two multilingual datasets (especially MASSIVE), which explains its overall poor performance. Given that we are using the same code and models for all datasets and the fact that fastText results on English-only datasets match the ones reported in the original paper [16], we believe that fastText's poor performance on multilingual datasets is due to the fact that fastText vectors from different languages may not be compatible with each other when used as inputs to a multilingual model. However, we didn't find any mention of such an issue in the literature.

Delving deeper into multilingual capabilities, as plotted in Figure 3, we observe that RETVec consistently outperforms other vectorizers regardless of the language. We also observe that RETVec has a wider lead on non-Latin languages, which we attribute at least partially to our novel character encoder, given that RETVec and RETVec-raw achieve similar performance on them.

% of Words with Random Mixed Typos

50.0%

60.0%

70.0%

80.0%

90.0%

0%

25%

50%

75%

100%

RETVec

RETVec-raw

Whitespace

SentencePiece

BPE

fastText

80.0%

85.0%

90.0%

95.0%

0%

10%

20%

30%

40%

50%

RETVec

RETVec-raw

Whitespace

SentencePiece

BPE

fastText

% of Words with Typos

50.0%

60.0%

70.0%

80.0%

90.0%

0.00%

25.00%

50.00%

75.00%

100.00%

Mixed

Delete

Substitute

Typo

Swap

% of Words with Typos

50.0%

60.0%

70.0%

80.0%

90.0%

0.00%

25.00%

50.00%

75.00%

100.00%

Mixed

Delete

Substitute

Typo

Swap

6 Evaluation: Typo Resilience

In this section, we evaluate the resilience of the various vectorizers against both random typos and common human typos.

Setup We rely on two types of typo injection methods to evaluate the tokenizers' typo resilience: random typos and common human typos . For random typos, evaluation is performed on every model and dataset used in Section 5 and average accuracy on the typo-augmented test splits is recorded. For common human typos, because we only have them for English, we take the average accuracy across all three models on only AG News. We evaluate how each tokenizer's resilience degrades as the number of typos increases by gradually increasing the percentage of words with typos in the test set by increments of 10%.

Random typos are created by using a combination of insertion, deletion, substitution, neighboring swap, and keyboard-based typos. Each typo is applied with a block size of 1 or 2 characters. Words are selected at random and each word can only have one typo. To create human-like typos, we use the word replacement noiser from the Neuspell [14] package. The noiser uses 109k common misspellings of 17k popular English words. On the AG News test set, it can only inject typos into 55% of the words, so we limit the evaluation to 50% words with common human typos.

Results As reported in Figure 4, the performance of RETVec and RETVec-raw decreases significantly slower than the other vectorizers as the amount of typos increase. RETVec's pre-trained model increases RETVec resilience by up to 15% compared to RETVec-raw, demonstrating the effectiveness of RETVec's pre-training technique of using pair-wise metric learning to create syntactically robust word embeddings. SentencePiece and BPE perform about the same, with a gradual decline which keeps them competitive with RETVec-raw until about 60% random typo rate. We note that fastText and Whitespace provide more typo resilience than SentencePiece and BPE, which is expected because it has been previously noted that word-level vectorizers are typically more robust than character-level

or subword-level vectorizers [23]. We see similar trends on the common human typo results, with RETVec-raw and RETVec being noticeably more resilient than baseline vectorizers when the typo rate is >30%.

Delving deeper into RETVec's typo resilience capabilities, as reported in Figure 5, character deletion is the hardest form of typo for RETVec to deal with. Without the pre-trained model, RETVec-raw also struggles with character insertion and RETVec's model also greatly helps nullify the impact of character swapping.

7 Evaluation: Adversarial Attack Resilience

Setup We use the TextAttack framework [21] to evaluate the resilience of RETVec and other vectorizers against character-level adversarial attacks, specifically: TextBugger [18], DeepWordBug [9], and Pruthi [23]. For each model and vectorizer, the attacks are carried out on the same 1000 examples drawn randomly from the AG News test dataset.

Results Table 4 demonstrates that RETVec models are significantly more resilient to adversarial text attacks than other embedding schemes. In particular, the average attack success rate against RETVec-raw and RETVec models are 45.6% and 44.8% respectively, which is significantly lower than for SentencePiece (71.3%) and BPE (67.8%). The best performing, non-RETVec, vectorizer is fastText (56.3% attack success rate), which makes sense because fastText handles OOV tokens by averaging the vector representations of the token's n-grams. Detailed results broken down by adversarial attack algorithm are reported in Appendix G.

8 Evaluation: Pre-training BERT

In this section, we evaluate RETVec's performance when used to pre-train transformers. We evaluate both RETVec's competitiveness on the GLUE benchmark [29] and typo resilience.

Setup We use BERT-Base [7] for all experiments. Given that RETVec outputs word-level embeddings and is not restricted by a vocabulary size, we cannot directly use the standard BERT masked-language modeling (MLM) task and predict token ids using a softmax layer. Instead, we use an approximate MLM task for RETVec-based models by only masking and predicting input tokens that are in the top 100k words. Other than this, we follow the standard MLM procedure by selecting 15% of tokens at random and replacing a selected token with (1) the [MASK] token with 80% chance (2) a random token with 10% chance (3) the same token unchanged with 10% chance. For the baseline, we use SentencePiece with a vocabulary size of 32k and the standard MLM task described in [7].

For each model, we pre-train for 1M steps with batch size 64 on the English C4 dataset [32]. We use Adam with a max learning rate of 5e-5, β 1 = 0 . 9 , β 2 = 0 . 999 , and 0 . 01 L2 weight decay. We warmup for 10000 steps and linearly decay the learning rate. We fine-tune all models for 20 epochs on GLUE using 3 different random seeds and report the best result for each dataset. We pre-train and fine-tune each model using 8 NVidia V100s. Detailed pre-training and fine-tuning hyperparameters can be found in Appendix H.

% of Words with Random Mixed Typos

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

0%

25%

50%

75%

100%

RETVec

RETVec-raw

SentencePiece

90.0%

92.0%

94.0%

96.0%

0%

10%

20%

30%

40%

50%

RETVec

RETVec-raw

SentencePiece

Results Overall, as reported in Table 5, BERT pre-trained with RETVec and RETVec-raw using MLMon 100k words achieves competitive results compared to SentencePiece. Additionally, using RETVec reduces the pre-training time of BERT by around 10% in our experiment, when training for an equivalent number of steps. Surprisingly, RETVec-raw is more competitive than RETVec, which we attribute to the fact that BERT-Base has more than enough capacity to learn a competitive representation without RETVec's word embedding model. However, this increased competitiveness for RETVec-raw comes at the cost of decreased typo resilience, as reported in Figure 6, although the difference in typo resilience between pre-trained BERT models using RETVec-raw versus RETVec is only significant when the percentage of injected typos exceeds 50% of the words. As such, it seems that using RETVec-raw when pre-training large models is the best choice. How to optimally do so is left for future work.

9 Evaluation: In the Wild

As a final evaluation, to ensure RETVec will work in practice against adversarial content, we trained a version of a transformer model we use as part of our spam email filtering system at Google. As with previous experiments, we kept everything constant besides swapping SentencePiece with RETVec. Benchmarking on our internal production evaluation dataset, we found that RETVec improved the recall at 0.80% false positive rate by ∼ 0.51%, helping detect 38% of the spam emails that would have been missed by the baseline model while reducing model serving latency by 30%. This result gives us confidence that RETVec is competitive for real-world classification tasks, especially in the face of adversarial content.

10 Ablation Study

In this section, we report how key hyperparameters variations affect RETVec's performance. An unforeseen challenge we discovered while developing RETVec models is that a lower pre-training loss didn't directly translate into meaningful performance on our benchmarks. To overcome this

challenge, we had to include an extra validation step that trains and evaluates models on the Multilingual Amazon Reviews dataset [19] using the methodology described in Section 5 to measure downstream improvement. We choose the Amazon dataset due to its multilingual nature and difficulty. Detailed results on classification performance by model type, pre-training loss, and model size for the ablation studies can be found in Appendix C, along with additional experiments on less impactful hyperparameters.

Model architecture Despite considerable efforts, we couldn't find a more complex architecture that outperformed the simple MLP architecture (Table 6). The best transformer-based architecture which uses 2 encoder blocks of dimension 128 resulted in significantly lower loss regardless of the transformer block used (BERT [7], GAU [13], T5 [32]). However, as alluded to earlier, the significantly lower loss did not translate to better performance on the benchmarks. We hypothesize that the fitting the text manifold is an easy enough task to solve with any reasonable architecture.

Embedding Size We train RETVec with different embedding dimensions and report the results in Table 6. Classification benchmarks show little difference in performance for embedding dimensions between 128 and 512, with 256 embedding dimension having the slightest edge.

Max Input Character length To select the optimal max word length for RETVec, we started by looking the fastText dataset's word length distribution. More than 95% of the words are less than 16 characters long and the median word length is 7.9 characters (Appendix I). Using more than 16 characters didn't led to any accuracy improvements despite those longer inputs exhibiting a lower loss, as reported in Table 6.

11 Discussion and Future Work

In this paper, we presented RETVec, a new multilingual text vectorizer that combines a novel character encoding with an small model to project words into a compact, 256-dimensional embedding. Through extensive evaluations, we demonstrated that models trained using RETVec achieve state-of-the-art classification performance and provide a 10-15% increase in resilience against typos and adversarial attacks compared to popular text vectorizers and word embeddings.

The key outstanding question is how to best utilize RETVec for generative tasks and in large language models (LLMs), which could enable us to train LLMs with stronger multilingual capabilities, improved adversarial robustness, and reduced model size and computational costs. In particular, for 'small' LLMs (less than or around 1 billion total parameters), the vocabulary embedding layer can be more than 20% of the total parameters [3], which would be eliminated when using RETVec.

We discovered that the main difficulty when training generative models using RETVec is that RETVec's 256-float embedding cannot be directly converted into a softmax output, unlike other tokenizers which represent text as integer token IDs. In order to effectively use RETVec in decoderonly models, we expect that a new training procedure for RETVec-based models which is compatible with text generation tasks needs to be invented, e.g. a pre-training procedure which does not rely on directly predicting the next token. We briefly experimented with ideas including decoding the RETVec representation character-by-character or using a VQ-VAE [28] model to quantize the output, but the results so far were inconclusive. We hope to explore this area and address this limitation in future work. Other potential directions of future work include using RETVec as a word embedding in various applications in place of GloVe [22] and word2vec [20], and using the RETVec character encoder and training procedure to train text similarity models.

References

Deep Learning with Python. https://www.manning.com/books/deep-learning-with-python.

Better Language Models and Their Implications. https://openai.com/blog/better-languagemodels/, Feb. 2019.

S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching Word Vectors with Subword Information, June 2017.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, and M. et al. PaLM: Scaling Language Modeling with Pathways, Oct. 2022.

E. G. Dada, J. S. Bassi, H. Chiroma, S. M. Abdulhamid, A. O. Adetunmbi, and O. E. Ajibuwa. Machine learning for email spam filtering: review, approaches and open research problems. Heliyon, 5(6):e01802, 2019.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019.

J. FitzGerald, C. Hench, C. Peris, S. Mackie, K. Rottmann, A. Sanchez, A. Nash, L. Urbach, V. Kakarala, R. Singh, S. Ranganath, L. Crist, M. Britan, W. Leeuwis, G. Tur, and P. Natarajan. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages, June 2022.

J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers, May 2018.

E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. Learning Word Vectors for 157 Languages, Mar. 2018.

M. Hagen, M. Potthast, M. Gohsen, A. Rathgeber, and B. Stein. A Large-Scale Query Spelling Correction Corpus. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 1261-1264, New York, NY, USA, Aug. 2017. Association for Computing Machinery.

X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai. Triplet-Center Loss for Multi-View 3D Object Retrieval, Mar. 2018.

W. Hua, Z. Dai, H. Liu, and Q. V. Le. Transformer Quality in Linear Time, June 2022.

S. M. Jayanthi, D. Pruthi, and G. Neubig. NeuSpell: A Neural Spelling Correction Toolkit. Comment: Accepted at EMNLP 2020 (system demonstrations).

R. Johnson and T. Zhang. Deep Pyramid Convolutional Neural Networks for Text Categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562-570, Vancouver, Canada, 2017. Association for Computational Linguistics.

A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain, Apr. 2017. Association for Computational Linguistics.

T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, Aug. 2018.

J. Li, S. Ji, T. Du, B. Li, and T. Wang. TextBugger: Generating Adversarial Text Against Real-world Applications. In Proceedings 2019 Network and Distributed System Security Symposium, 2019.

J. McAuley and J. Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, pages 165-172, Hong Kong China, Oct. 2013. ACM.

T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient Estimation of Word Representations in Vector Space, Sept. 2013.

J. X. Morris, E. Lifland, J. Y . Yoo, J. Grigsby, D. Jin, and Y . Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, Oct. 2020.

J. Pennington, R. Socher, and C. Manning. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar, Oct. 2014. Association for Computational Linguistics.

D. Pruthi, B. Dhingra, and Z. C. Lipton. Combating Adversarial Misspellings with Robust Word Recognition, Aug. 2019.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, July 2020.

F. Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823, June 2015.

R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units, June 2016.

Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei. Circle Loss: A Unified Perspective of Pair Similarity Optimization, June 2020.

A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, 2018.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.

X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott. Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning, Mar. 2020.

E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell. Distance metric learning, with application to clustering with side-information.

L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models, Mar. 2022.

X. Zhang, J. Zhao, and Y. LeCun. Character-level Convolutional Networks for Text Classification, Apr. 2016.

Z. Zhang and V. Saligrama. Zero-Shot Learning via Joint Latent Similarity Embedding, Aug. 2016.

S. Zhuang and G. Zuccon. Dealing with Typos for BERT-based Passage Retrieval and Ranking, Sept. 2021.

Appendix

A RETVec Model Details

Table 7 details the hyperparameter settings for the RETVec model architecture, as described in Section 3.

B Benchmarking Models

In this section, we provide more detailed model hyperparameters for the evaluation models used in Section 5 and Section 8.

RNN We used a Stacked-LSTM architecture, with the following hyperparameters:

Dim: 256

Layers: 4

•

Dropout rate: 0.1

DPCNN We use the architecture described in [15], with the following hyperparameters:

•

Filters: 256

Layers: 6

Kernel size: 3

Final dropout: 0.5

Activation: ReLU

BERT-Mini We use the architecture as described in [7] for BERT-Mini, with the following hyperparameters:

Layers: 4

•

Hidden dim: 256

•

Intermediate dim: 1024

Self-attention heads: 4

Dropout rate: 0.1

Activation: GeLU

BERT-Base We use the architecture described in [7] for BERT-Base, with the following hyperparameters:

Layers: 12

•

Hidden dim: 768

Intermediate dim: 3072

Self-attention heads: 12

Dropout rate: 0.1

•

Activation: GeLU

C RETVec Ablation Studies

In this section, we present our ablation study results for the RETVec model design and hyperparameter selection. Results are reported on the Multilingual Amazon Reviews dataset following the methodology described in Section 10.

C.1 Embedding Dimension

Detailed results on how RETVec's embedding layer dimension affects classification performance are reported in Table 8.

C.2 Model Architecture

Detailed results on the effect of RETVec architecture type on classification performance are reported in Table 9.

C.3 Maximum Word length

Detailed results on how RETVec's maximum input word length affects classification performance are reported in Table 10.

C.4 Pre-training Loss Hyperparameters

Detailed results on the effect of Multi-Similarity loss hyperparameters on RETVec classification performance are reported in Table 11. We also experimented with Circle Loss [27] and report the results in Table 12.

C.5 Model Capacity

Detailed results on how the number and dimension of fully-connected dense layers in the RETVec model affects classification performance are presented in Table 13.

C.6 Spatial Dropout Rate

Detailed ablation study results on the amount of spatial dropout in the RETVec model and its effect on classification performance are presented in Table 14. Increments of 1 / 16 were used because it corresponds to dropping out one character of the input on average, since RETVec's model accepts an input of up to 16 characters per word.

C.7 Pre-Training Objectives

We evaluated combining RETVec's pre-training objective (Multi-Similarity loss) with other objective functions and pre-training tasks as well. Specifically, we experimented with the following objectives: 1) augmentation position prediction, 2) augmentation position and type prediction, 3) decoding (predicting the character encoding of the input token), and 4) denoising (predicting the character encoding of the original, non-augmented token). Table 15 reports the results of our experiments on different pre-training objectives.

D RETVec Pre-training Dataset Augmentations

Below, we provide the full list of character-level augmentations (broken down into four categories) used to generate typo-augmented words for the RETVec pre-training dataset, as described in Section 3.3.

Deletion

Insertion

-Repeated character insertion

-n-grams based prefix insertion for n = 3 , 4 , 5

-n-grams based suffix insertion for n = 3 , 4 , 5

-Random ASCII character insertion

-Language alphabet-based random character insertion

-Random punctuation insertion

-Random punctuation prefix

-Random punctuation suffix

-Random BMP Unicode insertion

-Random emoji prefix

-Random emoji suffix

Substitution

-Case substitution

-n-grams based substitution for n = 3 , 4 , 5

-QWERTY keyboard typo substitution

-Homoglyphs substition

-Random ASCII character substitution

-Language alphabet-based random character subsitution

-Random puctuation substitution

-Random BMP Unicode substitution

Transposition

-Neighboring character swap

-3-character block random shuffle

E RETVec Pre-training Hyperparameters

We train RETVec using Multi-Similarity loss with hyperparameters α = 4 , β = 40 , glyph[epsilon1] = 0 . 1 and λ = 0 . 5 . Detailed pre-training hyperparameters are reported in Table 16.

F Typo Resilience Evaluation

Detailed results for random mixed typo resilience across every dataset and vectorizer can be found in Table 17.

G Adversarial Resilience Evaluation

We report adversarial attack resilience results for all vectorizers, classification models, and adversarial attack algorithms we benchmarked in Table 18. The TextAttack [21] framework was used to conduct all three types of adversarial attacks.

Model

Vectorizer

TextBugger

Pruthi

DeepWordBug

Original Acc

Acc under Atk

Original Acc

Acc under Atk

Original Acc

Acc under Atk

LSTM

Whitespace

90.6%

9.9%

89.1%

90.6%

84.2%

7.1%

90.6%

9.9%

89.1%

SentencePiece

90.3%

0.8%

99.1%

90.3%

68.1%

24.6%

90.3%

0.8%

99.1%

BPE

88.1%

3.3%

96.3%

88.1%

73.3%

16.8%

88.1%

3.3%

96.3%

fastText

92.7%

14.4%

84.5%

92.7%

83.3%

10.1%

92.7%

14.4%

84.5%

RETVec-raw

90.8%

22.3%

75.4%

90.8%

74.8%

17.6%

90.8%

22.3%

75.4%

RETVec

91.8%

23.7%

74.2%

91.8%

80.9%

11.9%

91.8%

23.7%

74.2%

CNN

Whitespace

90.9%

17.6%

80.6%

90.9%

83.8%

7.8%

90.9%

9.0%

90.1%

SentencePiece

90.3%

2.9%

96.8%

90.3%

72.2%

20.0%

90.3%

3.5%

96.1%

BPE

89.3%

31.8%

64.4%

89.3%

54.1%

39.4%

89.3%

43.5%

51.3%

fastText

91.9%

17.3%

81.2%

91.9%

74.9%

18.5%

91.9%

15.4%

83.2%

RETVec-raw

86.9%

30.1%

65.4%

86.9%

59.6%

31.4%

86.9%

37.7%

56.6%

RETVec

91.4%

34.3%

62.5%

91.4%

77.4%

15.3%

91.4%

45.0%

50.8%

BERT

Whitespace

89.4%

9.8%

89.0%

89.4%

83.6%

6.5%

89.4%

3.3%

96.3%

SentencePiece

89.8%

2.8%

96.9%

89.8%

70.8%

21.2%

89.8%

3.7%

95.9%

BPE

90.8%

8.2%

91.0%

90.8%

78.2%

13.9%

90.8%

1.2%

98.7%

fastText

92.6%

22.9%

75.3%

92.6%

80.5%

13.1%

92.6%

18.1%

80.5%

RETVec-raw

93.0%

30.8%

66.9%

93.0%

82.2%

11.6%

93.0%

38.9%

58.2%

RETVec

93.7%

30.1%

67.9%

93.7%

84.6%

9.7%

93.7%

40.5%

56.8%

H Pre-training and Fine-tuning BERT

Table 19 details the hyperparameter settings used for pre-training and fine-tuning BERT-Base models. Table 20 shows detailed results on the GLUE benchmark, including the models' average performance and standard deviation for each GLUE task.

I fastText Word Dataset

Table 21 contains statistics on word length computed on the fastText word dataset using words from all 157 available languages.

J Embedding Visualization

We use the TensorBoard Embedding Projector to visualize RETVec embeddings for words, as shown in Figure J.

distance

COSINE

EUCLIDEAN

Nearest points in the original space:

discovers

0.078

discovery

0.145

discovered

0.177

discoveis

0.238

discoveries

0.278

discover

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RETVec, a resilient and multilingual text vectorizer designed for neural-based text processing. RETVec combines a unique character encoding with an optional small model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning, making it robust against typos. RETVec does not require dataset pre-processing and does not have out-of-vocabulary (OOV) tokens, as it accepts all valid UTF-8 characters. The authors provide a comprehensive evaluation of RETVec, demonstrating that it is faster and less memory-intensive than other vectorizers on multi-core CPUs and GPUs. Models trained with RETVec have slightly higher accuracy, greater resilience to typos, and better resilience to adversarial attacks compared to models trained with other vectorizers.\n\nSTRENGTHS:\n- The paper introduces RETVec, a text vectorizer that combines a unique character encoding with an optional small model, which addresses several challenges associated with existing text vectorizers.\n- The paper provides a comprehensive evaluation of RETVec, demonstrating its performance in terms of speed, memory usage, and resilience to typos and adversarial attacks. Also, RETVec has the potential to significantly improve the performance of neural-based text processing, particularly in multilingual settings and in situations where typos and adversarial attacks are common.\n- The authors provide code for their method, and they promise to make it open-source to the public, which will further facilitate the community.\n\nWEAKNESSES:\n- Though the proposed method exhibits significant improvements on several evaluations, it does not outperform sentencepiece overall when used for training pre-trained language model (BERT), which is a main-stream approach for many NLP tasks. \n- It is vague how to effectively combine the proposed method with pre-trained language model, especially in light of the recent emergence of large language models. The paper would have been strengthened by discussing these aspects.\n\nQUESTIONS:\nMajor:\n1. In light of the recent advances in large language models (LLMs), it would be better to discuss the potential usage of the proposed method.\n2. Regarding the experiments on pre-training BERT, it seems that the proposed method will change the traditional training routine of these pre-trained models. Unfortunately, the proposed method does not exhibit significant improvements over sentencepiece tokenizer. Besides the results in Figure 5, it would be better to illustrate the potential usage of the proposed method combined with pre-trained models (and LLMs, of course). Also, does the proposed method boost the training speed or convergence speed for BERT? \n\nMinor:\n1. line 53: Glove -> GloVe\n2. line 240: missing reference to GLUE benchmark\n\nLIMITATIONS:\nA more explicit discussion of the limitations of RETVec would strengthen the paper and provide a more balanced view of its potential applications and implications. For example, a discussion on its inferior results on training BERT, and talk about what would be the best way or recommended way to use the porposed approach with BERT (or similar)?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RETVec, a resilient and efficient text vectorizer designed for neural-based text processing. RETVec is a multilingual tool that combines a novel character encoding with a pre-trained embedding model to create a 256-dimensional vector space. The vectorizer is significantly more resilient to typos and adversarial text attacks than other state-of-the-art vectorizers.The results show that RETVec outperforms other vectorizers in terms of accuracy and resilience to typos and adversarial attacks.\n\nSTRENGTHS:\nRETVec is a novel and efficient text vectorizer that is designed to be resilient to typos and adversarial text attacks. This is a significant contribution to the field of natural language processing, as text vectorization is a critical component of many NLP tasks.\n\nThe paper provides a detailed description of RETVec's architecture and evaluation methodology. This makes it easier for other researchers to understand and replicate the results of the paper.\n\nThe paper evaluates RETVec's performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths. This demonstrates the versatility and effectiveness of RETVec across a wide range of NLP tasks and datasets.\n\nThe results show that RETVec outperforms other vectorizers in terms of accuracy and resilience to typos and adversarial attacks. This is a significant finding, as it suggests that RETVec could be a valuable tool for real-world NLP applications where robustness to errors and attacks is critical.\n\nWEAKNESSES:\nThe paper does not provide a detailed analysis of RETVec's limitations and potential failure cases. While the paper does mention some of the challenges faced during the development of RETVec, a more thorough analysis of its limitations and potential failure cases would have been helpful to better understand the scope of its applicability.\n\nThe paper conducts experiments on classification tasks. I'm curious how it performs on text generation tasks, like machine translation. Since this determines the scalibility of a practical text vectorizer.\n\nQUESTIONS:\nPlease refer to the weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RETVec, a resilient, efficient, and multilingual text vectorizer designed for neural-based text processing. It addresses the limitations of existing approaches by combining a novel UTF-8 character encoder with a small model. RETVec does not require dataset pre-processing and accepts all valid UTF-8 characters, eliminating the need for out-of-vocabulary tokens. The embeddings are trained using pair-wise metric learning, ensuring that words with typos are embedded close to the original word. RETVec outperforms other vectorizers on text classification tasks, exhibiting higher accuracy, greater resilience to typos, and better resilience to adversarial attacks. The paper provides a TensorFlow implementation of RETVec, along with pre-trained models.\n\nSTRENGTHS:\n1) Addresses the limitations of existing text vectorization approaches.\n2) Combines a novel UTF-8 character encoder with a small model.\n3) Does not require dataset pre-processing and eliminates the need for out-of-vocabulary tokens.\n4) Trained on a word dataset with more than 157 languages.\n5) Space-efficient and suitable for on-device model deployment.\n6) Outperforms other vectorizers on text classification tasks, with improved accuracy and resilience to typos and adversarial attacks.\n7) Provides a TensorFlow implementation and pre-trained models.\n\nWEAKNESSES:\nThe paper does not provide detailed comparison results with other vectorizers on different languages and multilingual settings.\n\nQUESTIONS:\n1) How does the character encoder handle rare or uncommon characters in the UTF-8 character set?\n2) Are there any limitations or performance trade-offs when using RETVec with extremely long words?\n\nLIMITATIONS:\n1) The paper focuses on text classification tasks and does not explore other natural language processing tasks.\n2) The paper does not provide insights into the interpretability of RETVec embeddings and their usefulness in downstream tasks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}