RETVec: Resilient and Efficient Text Vectorizer
Authors: Elie Bursztein
Marina Zhang
Owen Skipper Vallis
Xinyu Jia
Alexey Kurakin
Venue: NeurIPS 2023 poster (2023)
Keywords: language modeling
text embedding
adversarial text attack
text vectorization
TL;DR: RETVec is a new multilingual, typo-resilient text vectorizer which achieves state-of-the-art performance on various benchmarks

Abstract: This paper describes RETVec, an efficient, resilient, and multilingual text vectorizer designed for neural-based text processing. RETVec combines a novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning to be robust against typos and character-level adversarial attacks. In this paper, we evaluate and compare RETVec to state-of-the-art vectorizers and word embeddings on popular model architectures and datasets. These comparisons demonstrate that RETVec leads to competitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVec is available under the Apache 2 license at https://github.com/google-research/retvec.

RETVec: Resilient and Efficient Text Vectorizer
Elie Bursztein
Marina Zhang Google mar marinazh@google.c com
Owen Vallis
Google el elieb&google.co com
Google ovallis@google.com com
Xinyu Jia
Alexey Kurakin
Google jiaxinu@google..om
Google kurakin@google..com com
Abstract
This paper describes RETVec, an efficient, resilient, and multilingual text vec- torizer designed for neural-based text processing. RETVec combines novel character encoding with an optional small embedding model to embed words into a 256-dimensional vector space. The RETVec embedding model pre- trained using pair-wise metric learning to be robust against typos and character- level adversarial attacks. this paper, we evaluate and compare RETVec state-of-the-art vectorizers and word embeddings on popular model architec- tures and datasets. These comparisons demonstrate that RETVee leads to com- petitive, multilingual models that are significantly more resilient to typos and adversarial text attacks. RETVee is available under the Apache 2 license at https://githuu: wom//ooogleerreeerrr/rrreee.
Introduction
Researchers have proposed many techniques for converting text into dense representations suitable for training neural networks, from the early use of character bi-grams and tri-grams to more modern semantic word embeddings and state-of-the-art subword vectorizers. Each of these approaches aims to mitigate issues caused by out-of-vocabulary (OOV) tokens, including intentional text perturbations such adversarial attacks and unintentional ones such as typos. These approaches leverage tech- niques such as subword-level tokenization [7 and decomposing unknown words into n-grams [4]
However, we find that each of these approaches suffers from at least one of following drawbacks:
They have insufficient resilience against typos and adversarial attacks [21].
They require large dictionaries and embedding lookup tables.
They exhibit poor performance certain languages in multilingual settings.
RETVec addresses these issues by combining a novel UTF-8 character encoder with an optional small model (230k parameters). This model projects encoded words into 256 dimensional syntactic metric embedding as detailed in Section 3 RETVec embeddings are trained using pair-wise metric learning (30], ensuring that words containing typos are embedded close the the original word.
RET does not require dataset pre- pre-processing and does not have OOV tokens because accepts all valid UTF-8 characters. RETVec's 's embedding model trained languages. RETVec is also space-efficient (<IMB) since it does not require large embedding lookup table. This reduces model memory footprints, making -ec-baedd models ideal candidates for on-device model deployment where storage, bandwidth, and memory resources are scarce. Through series of extensive experiments different model architectures, and wide range of datasets, we demonstrate that RETVee outperforms or is comparable to state-of-the-art vectorizers and word
word dataset with more than 157
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Overall, RETVec outperforms other vectorizers on text classification tasks by about 1%, while being up to 15% more resilient to typos at 20% word typo rate, and less susceptible to character-level adversarial attacks by over 10%.
embeddings including BPE 26). SentencePiece [17], and fastText 4.
Our paper makes the following contributions:
We introduce ReT'Vec, a resilient, efficient, and multilingual text vectorizer designed for neural-based text processing.
We show that RET'vcc is faster and less memory intensive than other vectorizers multi- core CPUs and on GPUs.
We demonstrate that models trained with RETVec have slightly higher accuracy, greater resilience to typos, and significantly better resilience to adversarial attacks compared to models trained with the other vectorizers.
We provide a TensorFlow implementation of RETVee, including its pre-trained models, alongside the code to reproduce our benchmarks under the Apache license at research/retved
https://gitthne com/
2 Background
Text Vectorizers Text vectorizers play major role in deep neural networks' (DNN) performance by ensuring that the input text is properly segmented into tokens (typically words or sub-words) and embedded into dense, float-point representation for the model to use. The traditional approach for text vectorization splits texts into words using whitespace and punctuation and uses a word embedding lookup table map each word into dense vector. The embedding lookup table be constructed using algorithms such as Word2Vec (20). GloVe [22) and fastText 44. or by randomly initializing an
embedding table and training lookup tables are typically large and limited to known, in-vocabulary tokens which significantly reduces their representational capabilities in the presence of typos and adversarial attacks.
a part of the LLM pre-training process (like GPT-2 (2). Embedding
Subword tokenizers such as WordPiece 77. BPE (26] and SentencePiece (17) split text into subwords to reduce vocabulary size and mitigate OOV issues. SentencePiece is used by many state-of-ther text models [24] 5). All of these vectorizers require separate training phase to adapt to the targeted dataset and require shipping extra files with the models be used at inference time.
Pair-based learning The RETVec model trained using pair-based metric learning. Pair-based learning aims to learn embedding space, where embedded vectors of similar samples couraged to be closer together while dissimilar ones are pushed apart from each other [31] This approach has been successfully applied various tasks including image retrieval [2 30
face
recognition 25 and zero-shot learning 34
Typos and Adversarial Attacks One challenge faced by text vectorizers is the existence of typo- graphical errors that might not appear in the training dataset. At inference time, these typographical errors can lead the vectorizer to incorrectly output OOV ids for valid words. This behavior ultimately leads to lower accuracy, particularly for retrieval tasks where human queries are known to exhibit between 16% and 19% typos rate [35 [T]. In many anti-abuse settings such as spam classification, the typo rate used by attackers in attempt to evade defense systems is much higher [6] It is also
well-known that text models vulnerable to different types of adversarial attacks (21] Many
character- adversarial attacks, which can be viewed as the most severe form of intentional typos, (23).
have been proposed in the literature (18).
3 RETVec
In this section, we describe how RETVec works. We start off by introducing its novel character encoding scheme, detail the design of its model, and finally describe its pre-training procedure.
3.1 RETVec Character Encoder
composed of two layers: an Integerizer The RET'Vec character encoder, as shown in Figure D
layer and Binarizer layer. The character encoder enables RET to encode all UTF-8 characters

---

Einieddiig g
(die, 2) 11E2
(clen, 24) flaa32
(384) float32
Entedding
(30:0)
Word
Ineger
Medel
Sinkkany
Dense (384)
D(24)
(244)
Choracter encodder
Binorizer
Figure I: RETVec architecture overview the output shape of each layer is in parenthesis. The clen indicates the number of characters used per word 16 characters by default. The batch and word sequence length dimensions are omitted.
in an efficient and stateless manner. Encoding a word is achieved by first converting its characters into their UTF-8 codepoints (Integerizer) and then converting this integer representation into flattened, binary (little-endian) representation (Binarizer). The Binarizer uses a compact 24-bit binary character representation encode all valid UTF-8 characters. Additionally, ReTVee outputs fixed-length, dense word embedding using a maximum of 16 characters per word. In the ablation study (Section 10. we empirically find that 16 characters per word works well and increasing the max word length provides no further accuracy gains.
The Integerizer is similar to the used in ByT5 32 which also encodes characters into their UTF-8 codepoints. What makes the RET encoder unique is that the codepoints are first converted to their binary representation and then flattened create the word representation. This novel binary representation is both easy to learn and significantly more compact than a one-hot representation. We tested many other approaches to convert the Integerizer output into including the use of a single float per character, but these more compact representations did not appear to be learnable. Additionally, a key design choice is to keep the Integerizer and the Binarizer separated for efficiency reasons: when using remote accelerator, such as a TPU, it is more efficient send the output of the Integerizer (16x int32) across the network rather than directly sending the binarization representation (16 x 24 float32)).
compact form,
3.2 RETVe Model Architecture
While the RETVee Binarizer provides an efficient word representation, it is not competitive with sate-of-theearr vectorizers. To address this, add small model on top of the Binarizer output. This improves accuracy and enables RETVec to outperform other vectorizers (see Section S) Additionally, this reduces the embedding size from 384 to 256 float32s and further improves resilience to adversarial attacks (see Section 77. The performance gains only incur a small increase in computational cost, making the of the pre-trained model worthwhile trade-off. To better understand what gains can be attributed to the character encoder, the remainder of the paper reports the performance for "RETVec with the model" RETVve, and understand what be attributed to the model we report "RETVec without the model" as RETVec-raw.
The ReTVee model, as visible in Figure is composed of a dense projection layer, a flatten layer, a dense compression layer, and the embedding layer. The embedding layer is dense layer with tanh activation to scale the embedding values between and 1, while the rest of the network uses gelu activations. Increasing capacity by adding additional layers or using more powerful architectures did not improve performance on our benchmarks (Section TOp. Detailed ablation studies on loss hyperparameters, dropout rates, activation functions, and model capacity can be found in Appendix C
3.3 RETVee Pre-training Procedure
The RETVec model is pre-trained on typo-augmented version of the 157-language fast words datasets [10]. The original words are extracted from the Common Crawl corpus. Our dataset then takes the set of words over the union of all 157 languages, yielding 88.8M unique
Dataset
3

---

tokens. To help with generalization, we supplement this dataset by adding 8.88M (10%) randomly generated tokens, where the length token randomized between 1 and 16. Each token consists of random set of UTF-8 characters. Finally, we create 20 versions of each token (16 augmented and 4 non-augmented versions) such that 80% of the tokens in the training set are erpo-augmented. At the end of the generation process we end-up with combined 1.9B token dataset.
Augmentations Token augmentation consists of randomly inserting up typos per token up to 25% of the token length. This is consistent with an observed maximum human error frequency of around 20% [T]]. We use distinct typo augmentations, which can be grouped into four categories: deletion, insertion, substitution, and transposition. For each token, we randomly select a target augmentation percentage between 0-25%, and for each augmentation step we randomly apply an augmentation from one of the four typo categories. The full list of augmentations used is reported in Appendix D
Training RETVec model is trained using pair wise learning and uses the Multi-Similarity Loss 30 (a 4, = 40, À 0.5, and 0.1). We construct example pairs by creating batches that always contain two variations of the same word. The model is trained for 500k steps with batch size 1024, using Adam with learning rate 0.001, 31 0.9, 32 = 0.999, and cosine decaying the learning rate to 0.0001 during training. Full training hyperparameterss can be found in Appendix E We experimented with additional losses and other self-supervised tasks, reported in Appendix C but none of them yielded better results.
Language
Cosine Similarity
French
Spanish
0.7
Japanese
0.6
0.5
English
Spanish
Language
Figure 2: The cosine similarity distributions RETVee embeddings for 1000 pairs of augmented and non-augmented versions of words, selected languages shown. 'Random' language refers to randomly-generated UTF-8 strings.
Embedding Quality To visualize and evaluate the training quality and embedding consistency across languages, we randomly sample pairs of augmented and non-augmented words from select languages and compare the distribution of their cosine similarities. As shown in Figure 2 we observe that the mean and variance of the cosine similarity distributions are fairly consistent across various languages, which suggests that the model has learned near-uniformly across languages. The mean cosine similarity between word pairs in Chinese slightly lower, which could attributed to the fact that the average length of words is significantly shorter in Chinese. The distribution for random strings is also consistent with other languages' distributions, suggesting that RETVec is able to generalize to previously unseen words and embed words with rare UTF-8 characters meaningfully, which would greatly contribute towards its adversarial resilience. In Appendix[] we used the TensorBoard Embedding Projector visualize RETVee embeddings in 3ID space and demonstrate that syntactically-similaa words are clustered together, adding further evidence that RETVec is meaningfully projecting tokens into the embedding space.
4 Evaluation: Speed
In this section, we evaluate how quickly RETVee can process and vectorize datasets compared to the other commonly used vectorizers. We considered SentencePiece [17), BPE (26), simple

---

GPU
Name
Finhodding
Size
Vacahelary Size
Time
Prepmesssing
System Memory (GB)
Madel Params
Wall time (+)
Usage
Core
Sec
Wall
Usage
Momory (GB)
Sentence Piece
256
32k
749
R09
BM
877
124%
1087
870
10P4
256
100%
Winlesguee
256
32K
132
17
8M
496
126%
635
396
1004
03
154
19
0
225
4518
1013
233
20%
01
KET
256
230K
840%
250
40%
Table I: Comparison of speed, preprocessing time, memory usage, and CPU/GPU usage for all benchmarked vectorizers when vectorizing the Multilingual Amazon Reviews training dataset.
Name
Trii Stee
Test Sioe
# Classos
Langunges
GOV Words
Arg Senience Length
Avg Woml Lengh
AXT News 1x
120K
7.8
9096
41
4.4
Keviews (Polurit) [33]
36lK
17.4%
160
Multlingaal Amazon Keviews (Polarity)
MASSIVE (lotent Classification)
3
58%
1526
60
31
35.3%
Table 2: List of datasets used for classification evaluation.
whitespace vectorizer, and fastText []. We use the HuggingFace implementation of BPE and the official SentencePiece implementation.
Setup We use the Multilingual Amazon Reviews 19 dataset for the vectorization speed evaluations.
The corpus is composed of 1.2M tokens 200k tokens for languages (English, French, German, Spanish, Chinese, Japanese). All the experiments are run on a standard Google Cloud VM with 16 CPU cores and V100 Nvidia GPU. We disable the GPU visibility to perform the CPU measurements and use the GNU time command to measure the total wall time, CPU usage, and system memory. We use nvidia--iiie smi to record GPU usage and memory. We report both the time it takes to adapt to the datasets and how long takes to vectorize the entire dataset end-to-end, with and without GPU accelcration. We also report normalized CPU-core times as RETVec (and other vectorizers to a lesser extent) benefits from multi multi-threaded acceleration. Finally, report memory usage for both system and GPU, since they are important metrics when developing models for memory- constrained devices such loT devices and smartphones.
Results We found that RETVec-rane and RETVec are the fastest vectorizers when a GPU is available and are in the top three when multi-core CPU are available (Table which is common on current devices including smartphones. RETVec's performance on CPU stems from its ability to efficiently use the available cores, as visible in the Core Sec column we expect that more optimized versions of SentencePiece and BPE could achieve similar performance.
5 Evaluation: Classification
In this section, we compare the classification training performance of RETVec against other vectori- zors and word embeddings. We train the models from scratch and report classification accuracy on wide range of datasets to benchmark the generality of the approach.
Setup In order to perform a comprehensive evaluation, we evaluate classification performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths, as summarized in Table For example, the MASSIVE [8] intent classification dataset has very short sentences but lot of languages (51), whereas the Yelp Reviews dataset has significantly longer texts but is monolingual.
We benchmark performance three of the most common model architectures for text classification: Transformer MERR (7)), CNN (DPCNN (15) and LSTM (Stacked-LSTM (]), details provided in Appendix B To keep evaluation fair and consistent when comparing various vectorizers, we keep the models the same and only swap the vectorizers. All models are implemented in TensorFlow 2.11 and training is conducted
Google Cloud VM using single NVidia V100 GPU.
models are trained with Adam optimizer with 31 = 0.9, 32 = 0.999 and max learning rate of 5e-4, with the exception of fastText on multilingual datasets. We found that max learning rate of 5e-4 was too high and led to divergence, so we used le-4 instead for fastText on MASSIVE and Multilingual Amazon Reviews. For BERT, linearly warmup the learning rate for 5k steps. All
5

---

models are trained for 100k steps with batch size 256 and cosine learning rate decay to 0. The test accuracy of each model averaged over 3 trials with different random seeds is reported Table
Dataset
Model
Whitespace
SentencePiece
BPE
fast
VeTVecrraw
RETVee
RNN
91.6%
91.3%
91.3%
92.7%
91.3%
92.6%
CNN
92.5%
92.4%
90.7%
92.3%
88.2%
91.2%
AG News
BERT
91.5%
91.5%
92.1%
93.4%
93.1%
93.5%
AVG
91.9%
91.7%
91.4%
92.8%
90.9%
92.4%
RNN
93.9%
93.3%
93.4%
94.7%
94.1%
94.7% %
CNN
94.4%
93.8%
93.0%
93.9%
92.5%
93.4%
Yelp
BERT
91.6%
91.3%
91.6%
93.5 %
93.2%
92.8%
AVG
93.3%
92.8%
92.7%
94.0 %
93,3%
93.6%
RNN
93.2%
90.7%
90.3%
87.0%
92.9%
93.5%
CNN
93.2%
90.8%
89.2%
84, 34.4%
91.1%
92.2%
Multilingual Amazon P.
BERT
91.8%
87.3%
87.2%
86.6%
92.7%
92.6%
AVG
92.8%
89.6%
88.9%
86.0%
92,2%
92.8%
RNN
70.3%
69.6%
68.6%
13.5%
71.0%
73.8%
CNN
70.4%
69.2%
59.2%
59.8%
66.7%
MASSIVE
BERT
70.6%
67.8% 87.
66.2%
23.9%
78.1%
78.6%
AVG
70.4%
68.9%
64.7%
16.7%
69.6%
73.0 %
RNN
87.3%
86.2%
85,9%
72.0%
87.3%
88.7%
CNN
87.6 %
86.5%
83.0%
70.8%
82.9%
85.9%
Average
BERT
86.4%
84.5%
84.3%
74.4%
89.3%
89.4 %
AVG
87.1%
85.8%
84.4%
72.4%
86.5%
88.0 %
Table 3: Detailed classification results comparing different vectorizers when used train models from scratch. Bold indicates best results, underline indicates second best.
90.0%
Acuunay
00.0%
80.0%
70.0%
Engleh
German
French
Sparteh
Chresee
Japanesn
Language
RETTVe
Whinepaee
SontencePPecs
RPP
IasdToo
Figure Classification performance on Multilingual Amazon Reviews broken down by language.
Overall, averaging over all datasets and model architectures, RETVec is the best performing vectorizer by some margin (0.9%). We observe that RETVee performs best when paired with Transformer and RNN architectures, but lags slightly behind for the CNN. Compared to RETVec-raw, RETVec's pre-trained model offers significant performance improvement across all datasets and model architectures.
Results
fastTText performs the best on average for the English-only datasets, but does not perform well on the two multilingual datasets (especially MASSIVE), which explains its overall poor performance. Given that using the same code and models for all datasets and the fact that fastText results English-only datasets match the ones reported the original paper []], we believe that fastText's poor performance multilingual datasets is due to the fact that fastText vectors from different languages may not be compatible with each other when used as inputs to a multilingual model. However, didn't find any mention of such issue in the literature.
Delving deeper into multilingual capabilities, as plotted Figure we observe that RETVec consistently outperforms other vectorizers regardless of the language. We also observe that RETVec has wider lead on non-Latin languages, which we attribute at least partially our novel character encoder, given that RETVec and RETVec-raw achieve similar performance on them.

---

60.9%
Assunay
70.04
15.75
60.0%
50.04
80.25
of Words wich Rander Nexd Typo
of
Common Human
RETIV::
RETWWOYTA
Wenteepace
SerfansePeeee
RPF
ARTMW
RETW
SersecaPeon
8PC
Figure Comparison of various tokenizers' resilience against mixed random typos (left) and common human typos (right) when training classification models from scratch.
00.0%
60.0%
Kmuny
70.5
73.0%
03.0%
50.0%
63.0%
3.00%
25.00%
50.0%
75.00% 00%
100.00%
0.00%
25.0%%
63
76.00%
100
15 Words with Typos
5: Words with Typza
Maod
Irsot
Delio
Subollite
Ty30
Swep
Meat
lssst
Oeloto
Tyoo
Swop
Figure 5: Comparison of RETVec resilience against various types of typos. RETVec-raw on the left, RETVe on the right.
6 Evaluation: Typo Resilience
In this section, we evaluate the resilience of the various vectorizers against both random typos and common human typos.
Setup We rely on two types of typo injection methods to evaluate the tokenizers' typo resilience: random typos and common human typos. For random typos, evaluation performed on every model and dataset used in Section 3 and average accuracy on the appoaaugmented test splits is recorded. For common human typos, because we only have them for English, we take the average accuracy across all three models on only AG News. We evaluate how each tokenizer's 's resilience degrades as the number of typos increases by gradually increasing the percentage of words with typos in the test set by increments of 10%.
Random typos created by using a combination of insertion, deletion, substitution, neighboring swap, and keyboard-based typos. Each typo is applied with a block size of I or 2 characters. Words selected random and each word can only have one typo. To create human-like typos, we use the word replacement noiser from the Neuspell [14] package. The noiser uses 109k common misspellings of 1 17k popular English words. On the AG News test set, can only inject typos into 55% of the words, so we limit the evaluation to 50% words with common human typos.
As reported in Figure 4 the performance of RETVec and RETVec-raw decreases signifi- cantly slower than the other vectorizers as the amount of typos increase. RET'vec's pre-trained model increases RETVee resilience by up to 15% compared to RETVec-raw. demonstrating the effectiveness of RETVec's pre-training technique of using pair-wise wise metric learning to create syntactically robust word embeddings. Sentence Piece and BPE perform about the same, with a gradual decline which keeps them competitive with RET Vec-raw until about 60% random typo rate. We note that fastText and Whitespace provide more typo resilience than SentencePiece and BPE, which is expected because it has been previously noted that word-level vectorizers are typically more robust than character-level
Results
7

---

Tokenizer
Original Acc
Acc under Atk
Atk Success %
Whitespace
89.4%
32.3%
63.9%
SentencePPees
89 8%
25.8%
71.3%
BPE
90.8%
29.2%
67.8%
Text
92. 92.6%
40.5%
56.3%
RET Vec-raw
93.0%
50.6%
45.6%
RETVec
93.7%
51.7%
44.8%
Table Adversarial resilience evaluation for BERT-Mini trained on AG News with different vec- torizers. Adversarial attack results are averaged across three types of character-leve adversarial
attacks: TextBugger [88) DeepWordBug [9. and Pruthi (23]. Bold indicates best results, underline indicates second best.
or subword-level vectorizers 23 We see similar trends on the common human typo results, with VeTVec-raw and RETVec being noticeably more resilient than baseline vectorizers when the typo rate is >30%.
Delving deeper into RETVec's typo resilience capabilities, as reported in Figure 5 character deletion the hardest form of typo for RETVec to deal with. Without the pre-trained model, RETVec-raw also struggles with character insertion and RETVec's model also greatly helps nullify the impact of character swapping.
7 Evaluation: Adversarial Attack Resilience
Setup We the TextAttack framework 21
evaluate the resilience of RETVee and other vec-
torizers against character-level adversarial attacks, specifically: TextBugger [88. DeepWordBug (9). and Pruthi [23). For each model and vectorizer, the attacks are carried out on the same 1000 examples drawn randomly from the AG News test dataset.
Results Table 4 demonstrates that RETVec models significantly more resilient to adversarial text attacks than other embedding schemes. In particular, the average attack success rate against RETVec-raw and RET'Vec models are 45.6% and 44.8% respectively, which is significantly lower than for SentencePiece (71.3%) and BPE (67.8)) The best performing, non-RET've, vectorizer is fastTText (56.3% attack success rate), which makes sense because fastTexx handles OOV tokens by averaging the vector representations of the token's n-grams. Detailed results broken down by adversarial attack algorithm are reported in Appendix G
8 Evaluation: Pre-training BERT
In this section, evaluate RET've''s performance when used pre-train transformers. We evaluate both RETVec's competitiveness on the GLUE benchmark [29] and typo resilience.
Setup We use BERT-Bast f for all experiments. Given that RETVec outputs word-level em- beddings and not restricted by vocabulary size, we cannot directly use the standard BERT masked-language modeling (MLM) task and predict token ids using softmax layer. Instead, we use an approximate MLM task for RETVec-based models by only masking and predicting input tokens that are in the top 100k words. Other than this, we follow the standard MLM procedure by selecting 15% of tokens at random and replacing a selected token with (I) the [MASK] token with 80% chance (2) random token with 10% chance (3) the same token unchanged with 10% chance. For the baseline, we use SentencePicca with a vocabulary size of 32k and the standard MLM task described in 77
For each model, we pre-train for 1M steps with batch size 64 on the English C4 dataset 32]. We use Adam with a max learning rate of 5e-5, 31 = 0.9, Bg = 0.999, and 0.01 L2 weight decay. We warmup for 10000 steps and linearly decay the learning rate. We fine-tune all models for 20 epochs on GLUE using 3 different random seeds and report the best result for each dataset. We pre-train and fine-tune each model using 8 NVidia V V100s. Detailed pre-training and fine-tuning hyperparameters can be found in Appendix H

---

To benchmark typo resilience of RETVec-based pre-trained BERT models, we fine-tune on AG News to produce comparable results Section and follow the same evaluation methodology against random and common human typos.
Vectorizer SentencoPiece
MNLI
QNLI
QQP
RTE
SST-2
MRPC
CoLA
STS-B
Avg
80.6
88.7
90.1
67.2
91.1
85.8
50.7
82.5
79.6
RETVec-raw raw
82.5
89.6
90.5
65.3
91.5
87.2
49.1
79.3
79.4
RETVec
81.4
89.2
90.4
65.7
90.8
87.3
47.9
79.8
79.1
Table 5: Results on GLUE dev sets. We report matched accuracy for MNLI. Matthews correlation for CoL.A, Pearson correlation for STS-B, F1 score for QQP, and accuracy for all other tasks. Bold indicates best results, underline indicates second best.
100
36.0%
94.0%
70.0%
00.0%
0%
50%
100%
307
Typos
Types
Figure 6: Comparison typo resilience against mixed random typos (left) and common human typos (right) when fine-tuning pre-trained BERT-Base model AG News dataset for RET'vec and SentencePiece.
Results Overall, reported in Table 3 BERT pre-trained with RETVec and RET using MLM on 100k words achieves competitive results compared to SentencePiece, Additionally, using RETVee reduces the pre-training time of BERT by around 10% our experiment, when training for an equivalent number of steps. Surprisingly, RETVec-raw is more competitive than RETVee, which we attribute to the fact that BERT-Base has more than enough capacity to learn a competitive representation without RET Vec's word embedding model. However, this increased competitiveness for RETVec-raw comes at the cost decreased typo resilience, as reported in Figure & although the difference in typo resilience between pre-trained BERT models using RETVec-raw versus RETVec is only significant when the percentage of injected typos exceeds 50% of the words. As such, it seems that using RETVec-raw when pre-trie rre-training large models is the best choice. How optimally do so is left for future work.
Evaluation: In the Wild
As final evaluation, to ensure RET'Ve will work practice against adversarial content, we trained version of transformer model we use as part of our spam email filtering system at Google. As with previous experiments, we kept everything constant besides swapping SentencePiece with RETVec. Benchmarking on our internal production evaluation dataset, found that RETVec improved the recall at 0.80% false positive rate by ~0.51 %, helping detect 38% of the spam emails that would have been missed by the baseline model while reducing model serving latency by 30%. This result gives us confidence that RETVec is competitive for real-world classification tasks, especially in the face of adversarial content.
10 Ablation Study
In this section, we report how key hyperparameters variations affect RETVec's performance. An unforeseen challenge we discovered while developing RETVec models is that lower pre-training loss didn't 't directly translate into meaningful performance benchmarks. To overcome this

---

challenge, we had to include an extra validation step that trains and evaluates models on the Mul- tilingual Amazon Reviews dataset [9] using the methodology described Section to measure downstream improvement. We choose the Amazon dataset due to its multilingual nature and difficulty. Detailed results on classification performance by model type, pre-training loss, and model size for the ablation studies can be found in Appendix C along with additional experiments on less impactful
hyperparameters.
Architecture
Loss
Accuracy
Emb Dim
Loss
Ace
Word Len
Loss
Acc
MLP
0.0248
92.89%
100
0.0267
92.6%
0.0267
92.8%
BERT + MLP
0.0129
92.54%
128
0.0260
92.7%
0.0259
92.8%
T5 + MLP
0.0120
92.72%
200
0.0253
92.7%
0.0248
92.9%
GAU + MLP
0.0133
92.82%
256
0.0248
92.9%
20
0.0242
92.7%
LSTM + MLP
0.0179
92.61%
300
0.0245
92.8%
24
0.0234
92.9%
CNN MLP
0.0214
92.72%
384
0.0237
92.8%
32
0.0235
92.8%
Table 6: Ablation studies for RETVee architecture (left), word embedding dimension (middle), and
word length (right). Bold denotes the hyperparameter selected for the final RETVec model.
complex architecture that outperformed the simple MLP architecture (Table(6). The best waasformer-basee architecture which uses 2 encoder blocks of dimension 128 resulted in significantly lower loss regardless of the transformer block used (BERT 71. GAU 13. T5 (32). However, as alluded to earlier, the significantly lower loss did not translate to better performance on the benchmarks. We hypothesize that the fitting the text manifold is an easy enough task to solve with any reasonable architecture.
Model architecture Despite considerable efforts, couldn't find
Embedding Size We train RETVec with different embedding dimensions and report the results in T Classification benchmarks show little difference in performance for embedding dimensions between 128 and 512, with 256 embedding dimension having the slightest edge.
Max Input Character length select the optimal max word length for RETVee, we started by looking the fastText dataset's word length distribution. More than 95% of the words are less than 16 characters long and the median word length is 7.9 characters (Appendix [D. Using more than 16 characters didn't 't led to any accuracy improvements despite those longer inputs exhibiting lower loss, as reported in Table 6
11 Discussion and Future Work
In this paper, we presented RETVec, a new multilingual text vectorizer that combines a novel character encoding with an small model to project words into a compact, 256-dimensional embedding. Through extensive evaluations, demonstrated that models trained using RET Vec achieve state-of-the-att classification performance and provide 10-15% increase in resilience against typos and adversarial attacks compared to popular text vectorizers and word embeddings.
The key outstanding question is how to best utilize RETVec for generative tasks and in large language models (LLMs), which could enable us to train LLMs with stronger multilingual capabilities, improved adversarial robustness, and reduced model size and computational costs. In particular, for "small" LLMs (less than or around billion total parameters), the vocabulary embedding layer can be more than 20% of the total parameters 3 which would be eliminated when using RETVee.
We discovered that the main difficulty when training generative models using RET'Ve is that RETVec's 256-float embedding cannot be directly converted into a softmax output, unlike other tokenizers which represent text as integer token IDs. In order to effectively use RETVve in decoder- only models, we expect that new training procedure for RETVec-based models which compatible with text generation tasks needs to be invented, e.g. pre-training procedure which does not rely on directly predicting the next token. We briefly experimented with ideas including decoding the RETVec representation character-by by-htrrrr--chhaeteerr or using VQ-VAE 28 model to quantize the output, but the results so far were inconclusive. We hope to explore this area and address this limitation in future work. Other potential directions of future work include using RETVVc as a word embedding in various applications in place of GloVe [2 and word2vee (20, and using the RETVee character encoder and training procedure to train text similarity models.
10

---

References
[1] Deep Learning with Python.
httpphhwwwwwp///......a..noooooo htt.aaamnii...oooooo///e/eeeannn withiiinggwttthtttteee with-python.
[2] Better Language Models and Their Implications. http:://opomniiccoom//bbbbtttttp models/, Feb. 2019.
[3] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O' O'Brien, E. Hallahan, M. A. Khan, Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[4] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching Word Vectors with Subword Information, June 2017.
[5] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, and M. al. PaLM: Scaling Language Modeling with Pathways, Oct. 2022.
[6] E. G. Dada, J. S. Bassi, H. Chiroma, S. M. Abdulhamid, A. O. Adetunmbi, and O. E. Ajibuwa. Machine learning for email spam filtering: review, approaches and open research problems. Heliyon, 5(6)xe01002, 2019.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019.
[8] J. FitzGerald, C. Hench, C. Peris, S. Mackie, K. Rottmann, A. Sanchez, A. Nash, L. Urbach,
V. Kakarala, R. Singh, S. Ranganath, L. Crist, M. Britan, W. Leeuwis, G. Tur, and P. Natarajan. MASSIVE: A IM-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Dieer Diverrs Languages, June 2022.
[9] J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers, May 2018.
[10] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov. Learning Word Vectors for 157 Languages, Mar. 2018.
[11] M. Hagen, M. Potthast, M. Gohsen, A. Rathgeber, and B. Stein. A Large-Scale Query Spelling Correction Corpus. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 1261-1264, New York, NY, USA, Aug. 2017. Association for Computing Machinery.
[12] X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai. Triplet-Center Loss for Multi- View 3D Object Retrieval, Mar. 2018.
[13] W. Hua, Z. Dai, H. Liu, and Q V. Le. Transformer Quality in Linear Time, June 2022.
[14] S. M. Jayanthi, D. Pruthi, and G. Neubig. NeuSpell: A Neural Spelling Correction Toolkit. Comment: Accepted at EMNLP 2020 (system demonstrationss.
[15] R. Johnson and T. Zhang. Deep Pyramid Convolutional Neural Networks for Text Categorization. In Proceedings of the 55th Annual Mecting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562-570, Vancouver, Canada, 2017. Association for Computa- tional Linguistics.
[16] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of Tricks for Efficient Text Classi- fication. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain, Apr. 2017. Association for Computational Linguistics.
[17] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, Aug. 2018
[18] Li, S. Ji, T. Du, B. Li, and T. Wang- TextBugger: Generating Adver- sarial Text Against Real-world Applications. In Proceedings 2019 Network and Distributed System Security Symposium, 2019.
[19] McAuley and Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, pages 165-172, Hong Kong China, Oct. 2013. ACM.
[20] Mikolov, Chen, Corrado, and J. Dean, Efficient Estimation of Word Representations in Vector Space, Sept. 2013.

---

[21] J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training NLP, Oct. 2020.
[22]
Pennington, Socher, and C. Manning. GloVe: Global Vectors for Word Representation. In Proceedings the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar, Oct. 2014. Association for Computational Linguis- tics.
[23] D. Pruthi, B. Dhingra, and Z. C. Lipton. Combating Adversarial Misspellings with Robust Word Recognition, Aug. 2019.
[24] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the Limits Transfer Learning with Unified Text-to-Text Transformer, July 2020.
[25) Schroff, D. Kalenichenko, and J. Philbin. FaceNet: Unified Embedding for Face Recog- nition and Clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815-823, June 2015.
[26] R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units, June 2016.
[27] Y. Sun, C. Cheng, Zhang, C. Zhang, L.. Zheng, Z. Wang, and Y. Wei. Circle Loss: A Unified Perspective of Pair Similarity Optimization, June 2020.
[28] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, 2018.
[29] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
[30] X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott. Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning, Mar. 2020.
[31] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. Russell. Distance metric learning, with application to clustering with side-information.
[32] L.. Xue, A. Barua, N. Constant, R. AI-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel. ByT5:: Towards a token-free future with pre-trained byte-to-byte models, Mar. 2022.
[33] X. Zhang, J. Zhao, and Y. LeCun. Character-level Convolutional Networks for Text Classifica- tion, Apr. 2016.
[34] Z. Zhang and V. Saligrama. Zero-Shot Learning via Joint Latent Similarity Embedding, Aug. 2016.
[35] S. Zhuang and G. Zuccon. Dealing with Typos for BERT-based Passage Retrieval and Ranking, Sept. 2021.
12

---

Appendix
RETVec Model Details
Table details the hyperparameter settings for the RETVec model architecture, described in Section 3
Hyperparameter
RETVee
Max word length
16
Per-character encoding dim
24
Activation
GeLU
# of projection layers
1
Projection layer dim
32
fully-connected layers
2
Fully-connected layer dim
256
Spatial dropout rate
0.0625
Dropout rate
0
Embedding activation
Tanh
Embedding dim
256
Similarity dim
256
Table 7: RETVec model hyperparameter details.
B Benchmarking Models
In this section, we provide more detailed model hyperparameters for the evaluation models used in Section and Section g
RNN
We used Stacked-LSTM architecture, with the following hyperparameters:
Dim: 256
Layers:
Dropout rate: 0.1
DPCNN We use the architecture described in 15), with the following hyperparameters:
Filters: 256
Layers:
Kernel size: 3
Final dropout: 0.5
Activation: ReL.U
BERT-Mini
We use the architecture as described in for BERT-Mini, with the following hyper-
parameters:
Layers:
Hidden dim: 256
Intermediate dim: 1024
Self-attention heads: 4
Dropout rate: 0.1
Activation: GeLU
13

---

BERT-Base
We use the architecture described in 7 for BERT-Base, with the following hyperpa-
rameters:
Layers: 12
Hidden dim: 768
Intermediate dim: 3072
Self-attention heads: 12
Dropout rate: 0.1
Activation: GeL.U
C RETVec Ablation Studies
In this section, we present our ablation study results for the RETVec model design and hyperpa- rameter selection. Results are reported on the Multilinguul Amazon Reviews dataset following the methodology described in Section TO)
C.1 Embedding Dimension
Detailed results how RET've''s embedding layer dimension affects classification performance reported in Table g
Embedding Dim
Pre-training Loss
Test Accuracy
RNN
CNN
BERT
AVG
0.0284
181k
92.8%
91.8%
92.3%
92.4%
100
0.0267
190k
93.2%
128
0.0260
197k
43.3%
92.2%
92.5%
92.7%
200
0.0253
216k
93.4%
92.3%
92.6%
92.7%
256
0.0248
230k
93.6%
92.3%
92.8%
92.9%
300
0.0245
241k
93.6%
92.3%
92.6%
92.8%
93.6%
92.2%
92.6%
512
0.0225
2966
93.7%
92.3%
92.8%
92.9%
Table 8: Ablation study results on the effect of the RET pre-trained model's embedding dimension classification performance. Bold denotes the hyperparameter selected for the final RET'Vec model.
C.2 Model Architecture
Detailed results on the effect of RETVec architecture type on classification performance are reported
in Table
Test Accuracy
Architecture
Pre-training Loss
RNN
CNN
BERT
AVG
ML.P
0.0248
93.6%
92.3%
92.8%
92.9%
MLP + BERT
0.0129
93.2%
92.0%
92.4%
92.5%
MLP+ T5
0.0120
93.5%
92.2%
92.5%
92.7%
MLP + GAU
0.0133
93.5%
92.2%
92.8%
92.8%
MLP + LSTM
0.0179
93.4%
91.9%
92.5%
92.6%
M
0.0214
93.5%
92.1%
92.6%
92.7%
Table 9: Ablation study results on RETVec model architecture type on classification performance. Bold denotes the hyperparameter selected for the final RETVec model.
C.3
Maximum Word length
Detailed results on how RETVee's maximum input word length affects classification performance are reported in Table I0
14

---

Test Accuracy
Word Len
Pre-training Loss
# Params
RNN
CNN
BERT
AVG
0.0428
93.6%
92.3%
92.7%
92.8%
193K
93.5%
12
0.0267
206k
93.5%
92.2%
92.7%
92.8%
14
0.0259
218K
93.5%
92.2%
92.6%
92.8%
16
0.0248
230k
93.6%
92.3%
92.8%
92.9%
20
0.0242
255k
93.5%
92.2%
92.6%
92.7%
24
0.0234
280k
93.6%
92.3%
92.7%
92.9%
0.0246
93.5%
92.2%
92.7%
92.8%
32
0.0235
32NK
93.6%
92.22%
92.6%
92.8%
Table 10: Ablation study results on the effect of maximum word length on RETVec classification performance. Bold denotes the hyperparameter selected for the final RETVec model.
C.4 Pre-training Loss Hyperparameters
Detailed results on the effect of Multi- Multi-Similarity loss hyperparameters on RETVec classification We also experimented with Circle Loss (27 and report the
performance are reported in Table I1 results in Table 12
Hy Hyperparameter
Test Accuracy
À
Pre-training Loss
RNN
CNN
BERT
AVG
20
0.5
0.0432
93.5%
92.1%
92.4%
92.7%
0.2643
92.9%
91.2%
92.6%
92.2%
40
0.5
0.0455
93.6%
92.1%
92.6%
92.7%
40
1.0
0.3610
92.9%
91.2%
92.4%
92.2%
0.5
0.0464
93.5%
92.3%
92.6%
29.8%
80
1.0
0,3583
92.6%
91.1%
92.4%
92.0%
4
20
0.5
0.0270
93.4%
92.0%
92.6%
92.7%
20
1.0
0.1537
93.0%
91.3%
92.4%
92.2%
40
0.5
0.0242
93.5%
92.4%
92.6%
92.8%
4
40
1.0
0.1919
92.8%
91.3%
92.4%
92.2%
80
0.5
0.0248
93.6%
92.3%
92.8%
92.9%
80
1.0
0.1851
92.8%
91.2%
92.4%
92.1%
Table 11: Ablation study results the effect of Multi-Similarity loss hyperparameters RETVec classification performance. 0.1 is fixed for all experiments. Bold denote the hyperparameters selected for the final RETVec model.
Circle-Loss
Hyperparameter
Test Accuracy
Scale Factor
Relaxation Factor
Pre-training Loss
RNN
CNN
BERT
AVG
0.3
7.55
93.3%
91.9%
92.6%
92.6%
93.3%
92.4%
92.7%
64
0.
0.63
93.6%
92.4%
92.6%
92.9%%
92.6%
128
0.4
5.06
93.6%
92.1%
92.6%
92.8%
128
0.5
1.18
93.6%
92.39
92.6%
92.9%%
256
033
24.97
93..5%
91.99%
92.35%
92.55%
256
0.4
9.49
93.3%
92.0%
92,4%
92.7%
512
0.3
49.01
93.2%
92.0%
92.6%
92.6%
512
0.4
19.88
93.3%
92.
92.7%
92.7%
512
0.5
5.26
93.6%
92.4%%
92.7%
92.9%
Table 12: Ablation study results for pre-training RETVve with various Circle Loss (2 hyperparame- ter settings.
C.5 Model Capacity
Detailed results on how the number and dimension of fully-connected dense layers the RETVee model affects classification performance presented in Table B3
15

---

Dense Layers
Model Capacity
Dense Layer 1km
Params
Pre-training Loss
RNN
CNN
Jest Accuracy
BERT
AVG
991
128
K2k
0,0383
93.5%
91.8%
92.4%
92.66%
256
164k
0,0312
93.5%
91.8%
92.4%
92.6%
246k
93.6%
92.25
512
128K
0.0258
93.0%
92.2%
92.8%
92.8%
256
230k
0.0248
93.6%
92.33
92.8%
92.9%
384
394K
0.0201
93.5%
92.33
92.6%
92.8%
591k
93.75
92.9%
128
115K
0.0314
93.495
91.9%
92.5%
92.6%
206k
384
512K
0.0175
93.0%
92.2%
92.7%
92.8%5
512
854K
0.0149
93.5%
92.4%
92.6%%
92.8%
Table 13:
Ablation study results the effect of RETVe model capacity (number and dimension of the fully-connected layers) on classification performance. Bold denote the hyperparameters selected for the final RETVec model.
C.6 Spatial Dropout Rate
Detailed ablation study results on the amount of spatial dropout in the RETVec model and its effect on classification performance are presented Table [4] Increments of 1/16 were used because corresponds to dropping out character of the input average. since RETVec's model accepts an input of up to 16 characters per word.
Test Accuracy
Spatial Dropout
Pre-training Loss
RNN
CNN
BERT
AVG
0.00%
0.0122
93.4%
91.4%
92.5%
92.4%
6.25%
0.0248
93.6%
92.3%
92.8%
92.9%
12.50%
0.0465
93.4%
92.0%
92.3%
92.6%
18%75%
0.0722
92.7%
91.5%
91.8%
92.0%
25.00%
0.0967
92.7%
91.4%
91.2%
91.8%
Table 14: Ablation study results on the effect of spatial dropout rate on the RETVec input character encoding. Bold denotes the value selected for the final RETVec model.
C.7 Pre-Training Objectives
We evaluated combining RETVec's pre-training objective (Multi-Similarity loss) with other objective functions and pre-training tasks well. Specifically, we experimented with the following objectives: 1) augmentation position prediction, 2) augmentation position and type prediction, 3) decoding (predicting the character encoding of the input token), and 4) denoising (predicting the character encoding the original, non aonneneeented token). Table 15 reports the results of our experiments on different pre- training objectives.
Objectives
Similarity
Total
RNN
AVG
Similarity, Augmentation Position Detection
0.026
0.2228
93.4%
92.0%
92.8%
92.7%
Similarity, Augmentation Position and Type Prediction 0.0238
0.0970
93.2%
92.2%
92.6%
92.7%
Similarity, Decoding
0.0278
0.0384
93.5%
92.1%
92.6%
92.8%
Similarity, Denoising
0.0241
0.1088
93.3%
91.8%
92.7%
92.6%
Similarity
0.0248
0.0248
93.6%
92.3%
92.8%
92.9%
Table 15: Ablation study results on combining different pre-training objectives with similarity loss for RETVec pre-training.
16

---

D RETVec Pre-training Dataset Augmentations
Below, we provide the full list of character-level augmentations (broken down into four categories) used to generate typo-augmented words for the RETVec pre-training dataset, as described in Sec- tion
Deletion
Insertion
Repeated character insertion
g-gaams based prefix insertion for n = 3, 4,5
n-grams based suffix insertion for 3,4,5
Random ASCII character insertion
Language alphabet-based random character insertion
Random punctuation insertion
Random punctuation prefix
Random punctuation suffix
Random BMP Unicode insertion
Random emoji prefix
Random emoji suffix
Substitution
Case substitution
n-grams based substitution for 3,4,5
QWERTY keyboard typo substitution
Homoglyphs substition
Random ASCII character substitution
Language alphabet- based random character subsitution
Random puctuation substitution
Random BMP Unicode substitution
Transposition
Neighboring character swap
3-character block random shuffle
E RETVec Pre-training Hyperparameters
We train RETVec using Multi-Similarity loss with hyperparameters 4, 3 40, =0.1 and 0.5. Detailed pre-training hyperparameters are reported in Table [6
Hyperparameter
Pre-training
Training steps
500k
Batch size
1024
Adam
1.00e-7
Adam
0.9
Adam Bg
0.999
Weight decay
Peak learning rate
0.001
End learning rate
0.0001
Warmup steps
10000
Decay function
Cosine
Table 16: RETVee pre-training optimizer hyperparameters.
17

---

Dataset
Vectorizer
0%
10%
20%
30%
400%
50%
00%
70%
80%
90%
100%
Whiliespace
92.0%
91.6%
91.0%
58.6%
86.8%
83.85
79.3%
72.3%
63. 57.5%
49.1%
SentencaPicoe
AG News
BPE
TastTest
92.8%
91.6%
92.1%
90.5%
89.1%
91. 666
90.7%%
87.6%
89.1%
84.8%
87.85%
80.6%
85,
76.1%
82.5%
70.25%
77.25
63.8%
70.5%
56.2%
59.7%
46.31
RETVec
91.0%
90.3%
89.6%
68.3%
87.3%
85.5%
63.7%
81.8%
78. 78.2%
74.7%
68.55
RFTVee
92.6%
901.555
95
80.7%
86.055
81 81.25%
Whiterpace
93.2%
92.55%
91.6%
90.55%
55.0%
14.11%
$0.45%
75.8%
69.7%
60.9%
SentenccPieee
93.1%
91.5%
89.7%
67.7%
85.1%
81.9%
78.7%
75.1%
71.2%
67.4%
62.66%
Yelp P.
TastTost
93.0%%
94.15%
93.5%%
92. 98
92.00%
88,
50..7%
$6,25
89. 5.%
87.9%
80.56%
85.89%
77.25%
83.1%
80.2%
75.6%
RETVec-raw
92.4%
91.7%%
90.9%
89.8%
$8.61%
87.2%
65.7%
83.7%
81.2%
78.6%
74.5%
RETVee
92.27%
Whillsspace
92.75%
92.1%
91 66%
91.1%%
501.21%
891 1.5
87.9%
86. 215
83.75%
800 19.
75.0%
SentenocPreee
89.6%
$5.55%
87.5%
86. 86.2%
$4.55%
82.66
80.7%
78.7%
76.15
73.9%
70.7%
Muhtlingual Amazon
80.15
fastToxt
86.21
85.5%
84.8%
84.0%
$2.9%
81.0%
80.1%
77.9%
75.9%
73.9%
70.2%
RETVee-no
92.3%
91.66
90.8%
90.0%%
89, .0%
87.75
86.3%
84.8%
82.77
80.64
77.0%
RETVec
92.9%
92.5%
92. 0%
91.7%
91.3%
90.6%%
90.2%%
89.5%
85, 65.6%%
87.7%
86.1%
Whitespace
70.0%%
55.99%
55.3%
38.71%
35,8%6
31.7%
79,27%
43.9%
37.1%
31.35%
77.3%
SentencePlece
69.0%
58.5%
58.45
57.8%
55.8%
52.74%
50.7%
46.7%
41.8%
40.9%
29.6% 69.6
BPE
65.5%
54.6%
54.4%
54.1% 1%
52.0%
48.6%
46.9%
42.7%
38.4%
37.0%%
26.4%
16.2%
14.8%
13.
RETVee-aw 69.66%
59.7%
59.7%
59
57 44%
54.4%
52.7%
49 109
44.
43.69%
32.5%%
RETVee
73.2%
65.7%
65.7%
65.598
63.9%
61.8%
60.4%
57.5%
54.1%
52.9%
43.5%
Table 17: Random mixed typo resilience results (0% 100% word typo rate) for each classification dataset and vectorizer. Following the methodology described in Section 6 test accuracy on each dataset is reported and results averaged across the three model architectures benchmarked in
F
Typo Resilience Evaluation
Detailed results for random mixed typo resilience across every dataset and vectorizer can be found in
Table 17
G Adversarial Resilience Evaluation
We report adversarial attack resilience results for all vectorizers, classification models, and adversarial attack algorithms we benchmarked in Table 18 The TextAttack [2] framework was used conduct all three types of adversarial attacks.
Model
Vecterieer
Original
Alk
Aer
Ace under
Success
Ohiginal
DoppWaelBor under Alk
Succons
Whitesuace
90.6%
$3%
69.1%
90.6
$4.2%
7.1%
90.65
9.9%
$5.15
19.19
0.75
10
76.12
31.0
11305
30.10
HPE
$5.1%
3.3%
$6.356
$5.155
75.2%
10.0%
10.15
3.3%
26.35
22.39
XIF
0130
33.75
1009
11.35
11.4
23.75
14.35
Whilcqus
0..9%
17.85
10.9
90.9
$3.84
7.8%
00.9
9.0%
90.194
10.70
2375
15.00
00.9%
72.23
20175
10.22
3.3%
15.75
NPE
43.5%
$1.36
91.9%
XIT
07:10
11.0
10
17405
15.10
01.35
2100
W While-pae
39.44
9.3%
19.94
$0.05
63%
09.45
3.3%
96.39
135
16.9%
10395
10.0
21.75
175
05.95
RET
METTSE
67.5%
$5.75
Table 18: Detailed adversarial resilience results AG News. Results are reported on the same randomly selected 1000 examples from the AG News test split, following the methodology described
in Section 7
H Pre-training and Fine-tuning BERT
Table 19 details the hyperparameter settings used for pre-training and fine-tuning BERT-Base models. Table shows detailed results on the GLUE benchmark, including the models' average performance and standard deviation for each GLUE task.
18

---

Hyperparameter
Pre-training
Fine-tuuing
Training steps
100k steps
20 epochs
Batch size
64
Sequence length
512
512
Adam
le-8
1e-8
Adam 31
0.9
0.9
Adam 3g
0.999
0,999
Weight decay
0.01
0.01
Max learning rate
5c-5
2e-5
End learning rate
0
0
Warmup steps
10000
First 5% of steps None
Decay function
Linear
Table 19: Pre-training and fine-tuning hyperparameters for BERT- models described in Section g
Vectorizer
MNLI
QNLI
QQP
KTE
SST-2
MRPC
CoLA
STS-B
GLUE Avg
SentencePiece 80.6 (0.1)
88.4 (0.3)
90.1 (0.0)
66.1 (1.0)
(0.3
85.4 (0.5)
(0.4)
(0.5)
79.2 (0.3)
VoTVaeww (2.0.0.5)
89.5
90.4
86.3
(9.1 (LD)
78.9 (0.4)
RET ML9 (0.4)
(8.9
90.4 (0.1)
(0.7
90.7 (0.2)
86.9 (0.4)
(0.7
(9.6
78.7 (0.3)
Table 20: Detailed results on GLUE Benchmark for pre-trained BERT-Base models using RETVec compared to SentencePiece. Each model is trained three times with different seeds, and the average and standard deviation reported here. Bold indicates best results, underline indicates second best.
I fast Word Dataset
Table 21 contains statistics on word length computed on the fastText word dataset using words from all 157 available languages.
Avg Median Std
p90
p95
p99
p99.9
13.0
15.0
20.8
36.1
Table 21: Word length statisties computed on all fastText words from 157 languages. p90 denotes the 90th percentile, p95 denotes the 95th percentile, and so on.
J Embedding Visualization
We use the TensorBoard Embedding Projector to visualize RETVec embeddings for words, as shown in Figure
19

---

Nearost points the original opass:
clacones
chroneed
63377
Figure 7: RETVec embedding visualization (using the TensorBoard Embedding Projector) of the most common 10000 English words and a typo-laden version of each word. The word 'discover' is selected as example, and the nearest neighbors and their cosine distances are shown.
20


Reviews:
[{"review_id": "9cQswPJcJL", "reviewer": "Reviewer_aZHy", "summary": "The paper introduces RETVec, a resilient, efficient, and multilingual text vectorizer designed for neural-based text processing. It addresses the limitations of existing approaches by combining a novel UTF-8 character encoder with a small model. RETVec does not require dataset pre-processing and accepts all valid UTF-8 characters, eliminating the need for out-of-vocabulary tokens. The embeddings are trained using pair-wise metric learning, ensuring that words with typos are embedded close to the original word. RETVec outperforms other vectorizers on text classification tasks, exhibiting higher accuracy, greater resilience to typos, and better resilience to adversarial attacks. The paper provides a TensorFlow implementation of RETVec, along with pre-trained models.", "questions": "1) How does the character encoder handle rare or uncommon characters in the UTF-8 character set?\n2) Are there any limitations or performance trade-offs when using RETVec with extremely long words?", "limitations": "1) The paper focuses on text classification tasks and does not explore other natural language processing tasks.\n2) The paper does not provide insights into the interpretability of RETVec embeddings and their usefulness in downstream tasks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Addresses the limitations of existing text vectorization approaches.\n2) Combines a novel UTF-8 character encoder with a small model.\n3) Does not require dataset pre-processing and eliminates the need for out-of-vocabulary tokens.\n4) Trained on a word dataset with more than 157 languages.\n5) Space-efficient and suitable for on-device model deployment.\n6) Outperforms other vectorizers on text classification tasks, with improved accuracy and resilience to typos and adversarial attacks.\n7) Provides a TensorFlow implementation and pre-trained models.", "weaknesses": "The paper does not provide detailed comparison results with other vectorizers on different languages and multilingual settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and constructive feedback. Please find our responses below.\n\n> **Q1:** The paper does not provide detailed comparison results with other vectorizers on different languages and multilingual settings. \n\n**A1:** We evaluated RETVec on the Amazon Multilingual Reviews Corpus and reported the results in Figure 2, broken down per language. Figure 2 shows that RETVec outperforms the four baseline text vectorizers including SentencePiece and BPE on all 6 languages in the dataset, with a strong lead in Chinese and Japanese. We will add results on another dataset with more languages for the revised version.\n\n> **Q2:** How does the character encoder handle rare or uncommon characters in the UTF-8 character set?\n\n**A2:** RETVec’s character encoder uses 24 bits per character. The character encoder converts every valid UTF-8 character into its integer codepoint before converting it into a binary representation, which ensures that 100% of the UTF-8 character set can be uniquely represented. Additionally, for the RETVec model, we ensure that all UTF-8 characters are seen during training by including 10% random UTF-8 character strings in the training dataset and applying random character insertion and substitution augmentations.\n\nIn order to visualize how RETVec handles uncommon words with rare UTF-8 characters, we will add a plot of the similarity distance between words and their typo-laden versions for both a set of common words and a set of random strings/uncommon words using the same typos, and show that they are comparable. This will demonstrate that every token, including those containing rare UTF-8 characters, is handled in a similar fashion by RETVec.\n\n> **Q3:** Are there any limitations or performance trade-offs when using RETVec with extremely long words?\n\n**A3:** In the ablation study, we trained RETVec models with input word lengths ranging from 12 to 32, but we did not see any performance improvements by increasing the word length above 16 characters per word (Table 6). Furthermore, increasing the input word length also increases RETVec’s model size and latency – we will add these metrics to Table 6 as well and discuss them in further detail in Section 10.\n\n> **Q4:** The paper focuses on text classification tasks and does not explore other natural language processing tasks.\n\n**A4:** RETVec is designed for adversarially resilient text classification and with on-device use-cases in mind, which is why most of our benchmarks are focused around classification performance and adversarial robustness. We realized that we should have devoted more space to discuss other use-cases such as text generation in future work and plan to use the extra space in the revised version to correct this.\n\n> **Q5:**  The paper does not provide insights into the interpretability of RETVec embeddings and their usefulness in downstream tasks.\n\n**A5:**  We will try to display the clusters of word embeddings using an embedding projector (https://projector.tensorflow.org/) and add it to the GitHub repository and paper appendix. This visualization will help demonstrate that syntactically similar words (e.g. a word and a typo version of a word) are clustered closer together while the embeddings of different and dissimilar words are further apart. This will hopefully provide some intuition on the RETVec embedding space and offer insights into the interpretability of RETVec embeddings. We are unsure how to provide insights into the embeddings’ usefulness in downstream tasks.\n\n\n\nPlease let us know if you have any additional questions or feedback, we would be happy to incorporate any further feedback into the revision of the paper. Thank you!"}, {"review_id": "Zo3SZP26Hd", "reviewer": "Reviewer_AFvP", "summary": "This paper introduces RETVec, a resilient and efficient text vectorizer designed for neural-based text processing. RETVec is a multilingual tool that combines a novel character encoding with a pre-trained embedding model to create a 256-dimensional vector space. The vectorizer is significantly more resilient to typos and adversarial text attacks than other state-of-the-art vectorizers.The results show that RETVec outperforms other vectorizers in terms of accuracy and resilience to typos and adversarial attacks.", "questions": "Please refer to the weakness", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "RETVec is a novel and efficient text vectorizer that is designed to be resilient to typos and adversarial text attacks. This is a significant contribution to the field of natural language processing, as text vectorization is a critical component of many NLP tasks.\n\nThe paper provides a detailed description of RETVec's architecture and evaluation methodology. This makes it easier for other researchers to understand and replicate the results of the paper.\n\nThe paper evaluates RETVec's performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths. This demonstrates the versatility and effectiveness of RETVec across a wide range of NLP tasks and datasets.\n\nThe results show that RETVec outperforms other vectorizers in terms of accuracy and resilience to typos and adversarial attacks. This is a significant finding, as it suggests that RETVec could be a valuable tool for real-world NLP applications where robustness to errors and attacks is critical.", "weaknesses": "The paper does not provide a detailed analysis of RETVec's limitations and potential failure cases. While the paper does mention some of the challenges faced during the development of RETVec, a more thorough analysis of its limitations and potential failure cases would have been helpful to better understand the scope of its applicability.\n\nThe paper conducts experiments on classification tasks. I'm curious how it performs on text generation tasks, like machine translation. Since this determines the scalibility of a practical text vectorizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and insightful comments. Please find our response below.\n\n> **Q1:** The paper does not provide a detailed analysis of RETVec's limitations and potential failure cases. While the paper does mention some of the challenges faced during the development of RETVec, a more thorough analysis of its limitations and potential failure cases would have been helpful to better understand the scope of its applicability. The paper conducts experiments on classification tasks. I'm curious how it performs on text generation tasks, like machine translation. Since this determines the scalibility of a practical text vectorizer.\n\n**A1:** Through this review, we realized that we have been overly focused on classification tasks and adversarial resilience, which were the main goals of RETVec. We plan on fixing this in the revised version by adding an in-depth discussion on the challenges and potential approaches for applying RETVec to other tasks, including text generation tasks, other sequence-to-sequence tasks, and pre-training large language models. Generative tasks are challenging because the 256-float embedding returned by RETVec cannot be converted into a softmax output like in the case of token IDs outputted by other vectorizers such as SentencePiece. As a result, there is no straightforward way of training a generative model which predicts the next token ID in the sequence, and we have to experiment with alternative forms of pre-training such as predicting the top N words, training a decoder for the RETVec embedding model, decoding character-by-character, or using a VQ-VAE model [1]. There has been limited work on the area of combining contrastively pre-trained word embeddings with pre-trained language models on text generation tasks. Thus, we are still unsure which methodology is the best, and we plan on using the extra page in the final version to discuss these challenges, potential applications, our initial results, and outline the need for future work in this direction.\n\nPlease let us know if you have any additional questions or feedback. Thank you for the review!\n\n[1] Van den Oord et al. “Neural Discrete Representation Learning.” NIPS 2017. arXiv:1711.00937.\n"}, {"review_id": "X4lCPAUZek", "reviewer": "Reviewer_LKoT", "summary": "This paper introduces RETVec, a resilient and multilingual text vectorizer designed for neural-based text processing. RETVec combines a unique character encoding with an optional small model to embed words into a 256-dimensional vector space. The RETVec embedding model is pre-trained using pair-wise metric learning, making it robust against typos. RETVec does not require dataset pre-processing and does not have out-of-vocabulary (OOV) tokens, as it accepts all valid UTF-8 characters. The authors provide a comprehensive evaluation of RETVec, demonstrating that it is faster and less memory-intensive than other vectorizers on multi-core CPUs and GPUs. Models trained with RETVec have slightly higher accuracy, greater resilience to typos, and better resilience to adversarial attacks compared to models trained with other vectorizers.", "questions": "Major:\n1. In light of the recent advances in large language models (LLMs), it would be better to discuss the potential usage of the proposed method.\n2. Regarding the experiments on pre-training BERT, it seems that the proposed method will change the traditional training routine of these pre-trained models. Unfortunately, the proposed method does not exhibit significant improvements over sentencepiece tokenizer. Besides the results in Figure 5, it would be better to illustrate the potential usage of the proposed method combined with pre-trained models (and LLMs, of course). Also, does the proposed method boost the training speed or convergence speed for BERT? \n\nMinor:\n1. line 53: Glove -> GloVe\n2. line 240: missing reference to GLUE benchmark", "limitations": "A more explicit discussion of the limitations of RETVec would strengthen the paper and provide a more balanced view of its potential applications and implications. For example, a discussion on its inferior results on training BERT, and talk about what would be the best way or recommended way to use the porposed approach with BERT (or similar)?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces RETVec, a text vectorizer that combines a unique character encoding with an optional small model, which addresses several challenges associated with existing text vectorizers.\n- The paper provides a comprehensive evaluation of RETVec, demonstrating its performance in terms of speed, memory usage, and resilience to typos and adversarial attacks. Also, RETVec has the potential to significantly improve the performance of neural-based text processing, particularly in multilingual settings and in situations where typos and adversarial attacks are common.\n- The authors provide code for their method, and they promise to make it open-source to the public, which will further facilitate the community.", "weaknesses": "- Though the proposed method exhibits significant improvements on several evaluations, it does not outperform sentencepiece overall when used for training pre-trained language model (BERT), which is a main-stream approach for many NLP tasks. \n- It is vague how to effectively combine the proposed method with pre-trained language model, especially in light of the recent emergence of large language models. The paper would have been strengthened by discussing these aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and constructive feedback. Please find our responses below.\n\n> **Q1:** Though the proposed method exhibits significant improvements on several evaluations, it does not outperform sentencepiece overall when used for training pre-trained language model (BERT)...\n\n**A1:**  Currently, we have not found a way to leverage RETVec to outperform other vectorizers across all tasks for pre-trained BERT models while providing increased robustness and efficiency. RETVec’s resilience comes from its learned embeddings which are pre-trained using a deep similarity loss, which means that the vocabulary size is not limited, as is in the case of standard tokenizers like SentencePiece. The major downside of not having a restricted set of token IDs in the vocabulary is that RETVec embeddings cannot be converted into a softmax output, which makes the standard pre-training tasks such as masked language modeling (predicting masked token IDs) unusable as-is. As a result, we needed to use alternative pre-training tasks to pre-train language models such as BERT. We felt that it was important to report this shortcoming as it emphasizes the current limitations for RETVec.\n\nWe plan to use the extra space in the final version to discuss this challenge in more detail, describe interesting directions to explore in future work for combining RETVec with pre-trained language models, as well as add additional references that highlight the potential tradeoffs between robustness and accuracy [1, 2].\n\n[1] Zhang et al. “Theoretically Principled Trade-off between Robustness and Accuracy.” ICML, 2019.\n\n[2] Tsipras et al. “Robustness May Be at Odds with Accuracy.” ICLR, 2019.\n\n> **Q2:** It is vague how to effectively combine the proposed method with pre-trained language model...\n\n**A2:** We experimented with various approaches and reported the most successful one we found so far in the paper in Section 8. As discussed above, we don’t have a definitive answer yet on what is the best way to pre-train large language models using RETVec, especially since combining contrastively pre-trained word embeddings with LLMs is a largely unexplored area of study. We will make sure to include a discussion on this topic in the future work section, as we think the question deserves its own research given its complexity. \n\n> **Q3:** In light of the recent advances in large language models (LLMs), it would be better to discuss the potential usage of the proposed method.\n\n**A3:**  We will devote the extra space in the revised paper to address this point and emphasize that future work is needed to develop a better pre-training methodology that is more compatible with RETVec, due to the unique challenges of using pre-trained word embeddings instead of token IDs to represent text. As part of this discussion, we will make sure to highlight the potential benefits of using RETVec as the vectorizer for LLMs, including better multilingual capabilities, adversarial robustness, and smaller model size. In particular, for “small” or “medium-scale” LLMs (less than or around 1 billion total parameters), the embedding layers' parameters often accounts for more than 20% of the total parameters [3], which could be potentially reclaimed by using RETVec.\n\n[3] Biderman et al. “Pythia: A Suite for Analyzing Large Language Models.” arXiv:2304.01373.\n\n> **Q4:** Regarding the experiments on pre-training BERT, it seems that the proposed method will change the traditional training routine of these pre-trained models...it would be better to illustrate the potential usage of the proposed method combined with pre-trained models (and LLMs, of course).\n\n**A4:**  Pre-trained BERT with RETVec exhibits higher resilience to adversarial attacks and typos compared to SentencePiece (Figure 5) while remaining competitive (RETVec surpasses SentencePiece performance on 5/8 of the GLUE tasks, as shown in Table 5). As discussed above, we have not yet found the best pre-training methodology compatible with RETVec which will provide these benefits (efficiency and robustness) while offering equal or better performance on all tasks. Finding a more effective approach for pre-training RETVec-based models is the clear next step for our future work. For now, we reported this tradeoff and provided an in-the-wild study of the practical benefits of using RETVec with BERT when classifying adversarial text content such as spam (Section 9). We will use the extra space we have to discuss the usage and potential applications of RETVec in large pre-trained models including LLMs and our plans for future work.\n\n> **Q5:** Also, does the proposed method boost the training speed or convergence speed for BERT?\n\n**A5:** Yes, the training speed of BERT with RETVec is slightly faster than BERT with SentencePiece. The same number of training steps are needed so the convergence speed remains the same (the same number of training steps were also used to ensure a fair comparison in our benchmarks). We will make sure to highlight this fact in Section 8 and report overall pre-training and fine-tuning computational costs. \n\n> Minor: line 53: Glove -> GloVe; line 240: missing reference to GLUE benchmark\n\nThank you for spotting those, we will correct them.\n\n> **Q6:** A more explicit discussion of the limitations of RETVec would strengthen the paper and provide a more balanced view of its potential applications and implications...\n\n**A6:** \nWe thank the reviewer for their insightful comments, and recognize the importance of such discussion. We will make sure to devote the extra space in the final version to discuss limitations and challenges in the existing pre-training methodology for BERT, as well as potential applications and future work to adapt RETVec to large pre-trained language models.\n\nWe hope that our responses helped address your questions. Please let us know if you have any further questions or feedback. Thank you for the review!"}]

Author rebuttal:

Meta-review: The proposed RETVec is a robust and effective multilingual text vectorization method, as acknowledged by all the reviewers. Reviewers also appreciated the authors' response in clarifying some questions. I'm in favor of accepting the paper
Decision: Accept (poster)