Tokenization with Factorized Subword Encoding
Year: 2023

Abstract:
In recent years, language models have become increasingly larger and more complex. However, the input representations for these models continue to rely on simple and greedy subword tokenization methods. In this paper, we propose a novel tokenization method that factorizes subwords onto discrete triplets using a VQ-VAE model. The effectiveness of the proposed tokenization method, referred to as the FACTORIZER, is evaluated on language modeling and morpho-syntactic tasks for 7 diverse languages. Results indicate that this method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm.

heading: 1 Introduction
text: A typical subword tokenizer consists of a vocabulary of 10 000s of subwords, each of them mapped onto a single and independent index. Instead, we propose a method that learns to project subwords onto triplets of 256 indices:
melon → [30, 255, 209].
This mapping is learned by a vector-quantized variational auto-encoder (VQ-VAE; van den Oord et al., 2017) from a large word-frequency list, resulting in a projection where orthographically different words use different indices and similar words share similar indices:
melons → [261, 255, 209], water → [96, 235, 109].
Modeling this projection with a VQ-VAE also automatically gives the estimated probability of every subword given an index triplet. Maximizing the joint probability allows for an optimal subword tokenization of words not contained in the vocabulary.
water | melon → [208, 235, 109], [45, 255, 209].
In this paper, we present FACTORIZER, a subword encoding method that serves as a drop-in replacement for any subword tokenizer used in modern NLP pipelines. We release the source code, trained models and ready-to-use tokenizers online.1 Our approach demonstrates the following advantages:
1. Improved performance. We evaluate the performance of our FACTORIZER on masked language models of seven linguistically diverse languages: Arabic, Chinese, Czech, English, Norwegian, Scottish Gaelic and Turkish. These models are evaluated on part-of-speech tagging, dependency parsing, and lemmatization tasks and demonstrate a substantial improvement in performance.
2. Increased robustness. Additionally, we investigate the robustness of our FACTORIZER to random noise during inference as well as the robustness to data scarcity during pretraining. We measure performance with increasing levels of noise and data scarcity and demonstrate that our FACTORIZER improves robustness to these factors.
3. More effective use of parameters. Traditional subword tokenizers require large vocabularies to cover most word forms, which results in a substantial portion of learnable parameters being consumed by the subword embedding layer. For example, the embedding layer of BERTbase uses more than 20% of its parameters, totaling over 23 million. In contrast, the memory footprint of our FACTORIZER embedding is substantially lower as it only requires about 0.6 million parameters.2
The remaining parameters can be then allocated more effectively in self-attention layers.
1https://github.com/ltgoslo/factorizer 2It uses embedding matrices for 3 × 256 indices, where
each embedding vector has length 768, as in BERTbase.
14143

heading: 2 Background: VQ-VAE
text: In this paper, we propose a novel tokenizer that utilizes a vector-quantized variational auto-encoder (VQ-VAE; van den Oord et al., 2017) as its central component. VQ-VAE is a powerful technique for learning discrete latent variables that can encode words and reconstruct them back to their original form. This process can be broken down into three main steps, as illustrated in Figure 1:
1. The encoder maps the data samples (subwords) x to continuous latent vectors e(x).
2. The codebook is made up of K codebook vectors zk, k ∈ 1 . . .K. Each latent vector e(x) is quantized to its nearest codebook vector zk, effectively mapping each input sample x onto a discrete variable k. The encoder and codebook downsample and compress the information in x, serving as an information bottleneck (Tishby et al., 1999).
3. The decoder reconstructs the symbols z back into the original input distribution, modeling the distribution p(x|z) as d(z).
Gradient backpropagation. The model is optimized jointly with the backpropagation algorithm. However, special attention must be given to the codebook quantization q as this operation is not differentiable:
q(e(x)) = zk, where
k = argmin i ∥e(x)− zi∥2 (1)
During the forward phase, the quantized codebook vectors are simply passed to the decoder. However, during the backward phase, the gradient of the loss function ∇dL is passed directly from the decoder to the encoder. This technique, known as straightthrough gradient estimation, is possible because the output of the encoder and the input of the decoder share the same latent space. The output e(x) is sufficiently similar to the decoder input z, such that the gradient carries information about how to change e(x) to lower the reconstruction loss.
Loss terms. In addition to the standard reconstruction loss Lr = log p(x|z), which measures the auto-encoding performance, the VQ-VAE model incorporates two auxiliary loss terms that align the encoder outputs e(x) with the nearest codebook vectors. Specifically, the codebook loss
is defined as Lq = ∥sg(e(x))− zk∥2 and the commitment loss as Le = ∥e(x)−sg(zk)∥2, where ‘sg’ is the stop gradient operation and zk is the codebook vector defined in Equation (1). The overall loss of a VQ-VAE model is computed as:
L = Lr + Lq + βLe,
where β is a hyperparameter that balances the focus between reconstruction and codebook alignment.
Codebook EMA. An alternative approach to updating the codebook, proposed by van den Oord et al. (2017), is to update it as an exponential moving average (EMA) of the latent variables e(x) – instead of incorporating the codebook lossLq. This update consists of updating two variables: the codebook usage counts ck and the codebook vectors zk, where λ is the EMA decay hyperparameter:
ck ← λck + (1− λ) ∑
i
1[q(e(xi))=zk]
zk ← λzk + (1− λ)
ck
∑
i
e(xi)1[q(e(xi))=zk]
This approach leads to more stable training (Kaiser et al., 2018) and it allows us to mitigate the codebook collapse, which occurs when the usage count of a vector drops to zero and the vector is then never updated (Kaiser et al., 2018). Thus, whenever a codebook vector zi has usage ci lower than cmin, it is reassigned to a random latent vector zi ← e(xj) and the usage count is reset ci ← 1, similar to Williams et al. (2020) or Dhariwal et al. (2020).

heading: 3 FACTORIZER
text: We utilize the VQ-VAE architecture to train a model capable of mapping discrete triplet representations z to subword strings w (and vice versa). Furthermore, we employ the VQ-VAE decoder to estimate the probabilities of the subword strings p(w|z). After the model is trained, we infer its vocabulary, consisting of a set of tuples ⟨wi, zi, log p(wi|zi)⟩. Finally, we use this vocabulary to perform optimal (in regards to the logprobabilities) subword tokenization.

heading: 3.1 VQ-VAE subword encoding
text: Training data. The auto-encoder is trained on a word-frequency list obtained by word-tokenizing a large text corpus. Let us denote the frequency of word w by fw. Note that while this data representation simplifies the model by discarding any contextual information, it still enables proper estimation of p(wi|zi) by following the word frequencies fw. Word representation. In this study, words are represented as sequences of UTF-8 bytes. To ensure proper handling of word boundaries, the word sequences start with a special “beginning-of-word” token and end with an “end-of-word” token; both tokens are depicted as the symbol ‘ ’ in this text.
Data sampling. In theory, the correct way of sampling the training data is to directly follow the frequencies fw. However, in practice, the distribution of words in a natural language follows Zipf’s law, resulting in a skewed distribution that collapses the training process. To address this issue, we sample the data according to a more balanced distribution
psample(w) ∝ fw
log(fw + 1) .
To compensate for this alteration and accurately model the true word distribution, we incorporate the denominator into the loss function by weighting it as follows:
L = ∑
w
log(fw + 1)L(w).
Subword splitting. Up to this point, we have only discussed word encoding. To also encode subwords, we randomly split some of the sampled words into subwords. Specifically, as the more frequent words should be split less frequently, we keep the word as-is with the probability
pnot-split(w) = log(fw + 1)
maxx log(fx + 1) .
Factorized codebooks. To capture fine-grained information about the characters inside words, we employ separate codebooks, each within a unique latent space. In addition, to further reduce the issue of codebook collapse, each codebook consists only of K = 256 codebook vectors.3
Backbone architecture. The auto-encoder model is based on an encoder-decoder transformer architecture (Vaswani et al., 2017) with three quantization bottlenecks. Specifically, we first pad the byte-tokens w with special R, G, B prior tokens, then embed and encode each sequence with a transformer encoder. The contextualized embedding vectors for the first three tokens serve as the encoding e(w). These three vectors are then quantized and prepended to the subword byte-tokens w, which are finally input into the autoregressive transformer decoder d(z,w), modeling p(w|z). Training details. All VQ-VAE models in this study utilize a transformer with 6 layers and 256 hidden size for both the encoder and decoder. The models are trained for 50 000 steps with batch size of 4 096 and optimized with adaptive sharpnessaware minimization (ASAM; Kwon et al., 2021) to ensure better generalization to unseen words, with AdamW (Loshchilov and Hutter, 2019) as the underlying optimizer. To further improve generalization, we calculate the exponential moving average of the parameters (with a decay of 0.999) and use this average for inference. For more details about the transformer architecture and hyperparameters, please refer to Samuel et al. (2023) and Appendix D.
Vocabulary construction. The VQ-VAE is not used directly for tokenization as that would greatly slow it down. Instead, we infer its static vocabulary by iterating through all 2563 instances of the RGB codebooks. The subword vocabulary is decoded as W = { argmax
w p(w|z)
∣∣∣∣ z ∈ [256]3, p(z)>0 } ,
where the prior distribution p(z) is calculated by counting the usage of all codebook triplets throughout training. Finally, the full vocabulary V is a set of tuples ⟨wi, zi, log p(wi|zi)⟩, where wi ∈ W and zi = argmaxz p(wi|z).
3Note that this is reminiscent of the standard 24-bit RGB color encoding, where every codebook triplet can be viewed as an RGB color. FACTORIZER essentially projects subwords into a color space, see Appendix A for more details.

heading: 3.2 Subword tokenizer
text: Optimal split search. After inferring the vocabulary V , we can search for the optimal tokenization of a word x into subwords w1,w2, . . .wk:
tokenize(x) = argmin w1w2...wk =x
k∑
i=1
score(wi),
where for each subword wi from the vocabulary triplets ⟨wi, zi, log p(wi|zi)⟩ ∈ V , its score is
score(wi) = −log p(wi|zi) + αsplit.
The parameter αsplit allows for a smooth change of the amount of splits per word, as shown in Figure 3. We use αsplit = 0.1 for all experiments in this work.
The tokenize function is implemented via the shortest path search in a tokenization graph, as illustrated in Figure 2. Specifically, the search uses Dijkstra’s algorithm (Dijkstra, 1959) and the forward edges from each node are efficiently iterated with the prefix search in a directed acyclic word graph (DAWG; Blumer et al., 1985) of all subwords from V . A simplified pseudocode of the search algorithm is given below, the optimal tokenization is then obtained by reverse iteration of the returned prev dictionary.
def search(word, dawg_trie): Q.append(word) priority queue of suffixes cost[word] = 0.0 cost returns +inf by default prev[word] = none pointers to predecesors
while Q is not empty: suffix = suffix from Q with minimal cost remove suffix from Q if suffix is empty: break
for every prefix of suffix in dawg_trie: new_cost = cost[suffix] + score(prefix) new_suffix = suffix[len(prefix):] if new_cost < cost[new_suffix]:
Q.append(new_suffix) cost[new_suffix] = new_cost prev[new_suffix] = min_suffix
return prev
Sampling. Our formulation of the tokenizer allows for a straightforward modification for sampling different subword splits. This is similar to BPE dropout (Provilkov et al., 2020), a powerful regularization technique. The sampling works by modifying the score function as follows, making sure that all scores are non-negative so that the correctness of the search method holds:
score(wi) = −log p(wi|zi) + αsplit + |wi|exp(ϵ) ϵ ∼ N (0, σ2sample)
We set the parameter σsample to 0.02 in all sampling experiments.

heading: 3.3 Application
text: Embedding layer. The three FACTORIZER subword indices are embedded with separate embedding layers, summed and transformed with a GeLU non-linearity. In the end, we get a single embedding vector for each subword, which makes it a simple drop-in replacement for standard embedding layers.
Averaging. The random subword sampling can be utilized for more robust inference by sampling multiple tokenizations of each data instance and averaging the predictions for each of them.

heading: 4 Experiments
text: The effectiveness of FACTORIZER is experimentally verified on a typologically diverse set of languages. We train masked language models on each language and then evaluate it on morpho-syntactic tasks from the Universal Dependencies (UD) treebanks (Nivre et al., 2016). Additionally, we demonstrate that the FACTORIZER-based language models exhibit greater robustness to noise and superior performance in low-resource settings. We present four sets of experiments in this section:
1. In order to provide some initial observations, we evaluate simple parsing models that do not rely on any pretrained embeddings.
2. Then, we ablate different tokenizer configurations with English language modeling and UD finetuning.
3. As the main experiment, we pretrain and finetune language models on 7 typologically diverse languages.
4. Lastly, controlled experiments on English examine the robustness of our method to noise and data scarcity.
Languages. In total, our method is evaluated on 7 different languages to investigate its performance on different morphological typologies. For the most part, we follow Vania and Lopez (2017) in their selection of languages according to the traditional categories of morphological systems. The text corpus for each language is drawn from the respective part of the mC4 corpora (Xue et al., 2021), unless specified otherwise. Note that we use the same corpus for training FACTORIZER models (where we extract a word-frequency list for every language) as for training language models. The languages chosen for evaluation are:
• Arabic – The first distinct language type used for evaluation are introflexive languages. We decided to use Arabic as a representative of introflexive languages because it is arguably the most wide-spread language in this class. Arabic also tests how a tokenizer can handle a non-Latin script.
• Chinese – A purely analytical language; evaluation on Chinese tests performance on a completely different morphological typology than the other 6 languages.
• Czech – An Indo-European language which exhibits fusional features and a rich system of morphology, typical for all Slavic languages.
• English – In principle, also a fusional language, however, one of the most analytical languages in this class. The English corpus is made of the publicly available replications of BookCorpus and OpenWebText (because of the unnecessarily large size of the English part of mC4 for our purposes).45
• Norwegian – Another Germanic language, typologically close to English, but has different morphological properties, in particular highly productive use of morphological compounding, which makes it a fitting choice for evaluation of tokenizers.
• Scottish Gaelic – Also a fusional language. In order to evaluate performance on lowresource languages, this language, which has about 57 000 fluent speakers, was chosen as it is the smallest language from mC4 with a full train-dev-test split in Universal Dependencies.
• Turkish – As an agglutinative language, Turkish is on the other end of the ‘morphemes-perword spectrum’ than an analytical language like Chinese.
BPE tokenizer baseline. We compare the performance of FACTORIZER with the byte-pair encoding compression algorithm (BPE; Gage, 1994; Sennrich et al., 2016) as the most commonly used subword tokenizer. Following recent language modeling improvements (Radford et al., 2019), we use BPE directly on UTF-8 bytes instead of unicode characters – similarly to FACTORIZER, which also utilizes UTF-8 bytes as the atomic unit. The training itself uses the open implementation from tokenizers library6 and utilizes the same text corpora as the corresponding FACTORIZER models. The choice of BPE also allows for a comparison between FACTORIZER sampling and BPE dropout (Provilkov et al., 2020).

heading: 4.1 Experiment 1: UD parser ‘from scratch’
text: Before training large language models and evaluating the tokenizers in a more resource-intensive
4https://the-eye.eu/public/AI/pile_ preliminary_components/books1.tar.gz
5https://openwebtext2.readthedocs.io 6https://github.com/huggingface/
tokenizers
setting, we motivate our proposed method on a simple model that does not utilize any pretrained word embeddings.
UD parser architecture. We base this first experiment on UDPipe 2, a publicly available model for PoS tagging, lemmatization and dependency parsing by Straka et al. (2019). We generally follow the original architecture, only replacing its word embeddings – UDPipe 2 utilizes a combination of pretrained contextualized embeddings, static embeddings and character embeddings – instead, we use only randomly initialized subword embeddings and learn them from scratch; the final contextualized representations for the subwords are average-pooled for each word-token to obtain word-level representations. Otherwise we use the same joint modeling approach based on bidirectional LSTM layers (Hochreiter and Schmidhuber, 1997) followed by individual classification heads for UPOS, XPOS and UFeats tagging, a classification head for relative lemma rules and a final head for dependency parsing (Dozat and Manning, 2017).
Results. The amount of splits per word has a noticeable impact on the performance, we therefore include this dimension in the evaluation, visualized in Figure 4. This shows that FACTORIZER is clearly a Pareto-optimal tokenizer in this comparison. We conjecture that this is caused by its ability to learn a usable embedding for less frequent, or even outof-vocabulary subwords, as illustrated in Figure 5.

heading: 4.2 Experiment 2: Ablation study with English language modeling
text: In order to evaluate the performance of different BPE and FACTORIZER configurations, we conduct a comparative study on English.
Language model pretraining. In practice, the capabilities of a UD parser can be massively improved by utilizing contextualized embeddings from large language models that have been trained on vast amounts of data. In light of this, the main focus of this experimental section is to evaluate the impact of the FACTORIZER on language models and their downstream performance.
We follow the training budget and parameters of the original BERTbase (Devlin et al., 2019) for pretraining the language models, the BPE-based model also uses BERT’s vocabulary size of 32K. But unlike BERTbase, we establish our models on the more efficient LTG-BERT architecture (Samuel et al., 2023), which enhances the models with some recent improvements, such as NormFormer layer normalization (Shleifer and Ott, 2022), disentangled attention with relative positional encoding (He et al., 2021) or span masking (Joshi et al., 2020). In order to reduce training time, pretraining is paralellized over 128 GPUs, the total batch size is increased to 8 192 and the amount of steps is reduced to 31 250, matching the training budget of BERTbase. Please refer Samuel et al. (2023) for details on the architecture and Appendix D for the set of all hyperparameters.
In order to make the comparison between both tokenizers fair, we approximately match the num-
ber of trainable parameters in all language models. FACTORIZER requires relatively small embedding layers, so we move its ‘parameter budget’ to 3 additional transformer layers – then the BPE-based models use 110.8M parameters while the FACTORIZER-based models use 108.1M parameters in total. The effect of this choice is evaluated in Appendix B.
UD finetuning. We employ the same model as in Section 4.1, only replacing the LSTM layers with a convex combination of hidden layers from a pretrained language model, similarly to UDify (Kondratyuk and Straka, 2019). Then we finetune the full model on a UD treebank. The detailed hyperparameter choice is given in the Appendix D.
Results. The comparison is presented in Table 1. On average, FACTORIZER outperforms BPE, especially in lemmatization. The results show that both sampling and averaging techniques (discussed in Section 3) improve performance, with the FACTORIZER particularly benefiting from these features.

heading: 4.3 Experiment 3: Multi-lingual evaluation.
text: Finally, we train 7 BPE-based and FACTORIZERbased language models on the selected languages and finetune them on the respective UD treebanks.
Results. The results, displayed in Table 2, indicate that FACTORIZER clearly achieves better performance than BPE on average. Furthermore, the FACTORIZER is consistently more effective on lemmatization (with statistical significance), suggesting that the factorized subwords are indeed able to carry information about the characters inside the subword-units.
Interestingly, FACTORIZER does not always perform better on Czech and Turkish, even though we hypothesized that these languages should benefit from a better character information given their rich morphology. Neither BPE or FACTORIZER is an overall better choice for these languages.
While the results for dependency parsing in isolation (as measured by UAS and LAS) are more mixed, the parsing results which further take into account morphology (MLAS) and lemmatization (BLEX) largely favor the factorized subword encoding. FACTORIZER outperforms BPE (with statistical significance) in 6 out of 7 languages on BLEX scores.

heading: 4.4 Experiment 4: Robustness
text: Robustness to noise. Being able to maintain performance when dealing with unnormalized and noisy text, is an important quality of any NLP tool deployed in real-world scenarios. In order to evaluate how FACTORIZER deals with unexpected noise, we finetune a UD parser on clean data using the en-ewt corpus and evaluate it on modified development sets with increasing levels of character noise. The noise is introduced by perturbing every character with a set probability pnoise by uniformly choosing from (1) deleting the character, (2) changing its letter case, or (3) repeating the character 1–3 times.
Figure 6 shows the relation between the noise level pnoise and the expected performance. When comparing FACTORIZER with BPE, it is apparent that FACTORIZER is more robust to increased noise on tagging and dependency parsing as the performance drops more slowly with increased noise. The accuracy drops equally on lemmatization, which may be attributed to its formulation as prediction of relative lemma rules (as in UDPipe 2 by Straka et al. (2019)).
Robustness to resource scarcity. The promising results achieved on the very low-resource language of Scottish Gaelic have motivated a more thorough examination of the relationship between the size of a pretraining corpus and the downstream performance. In order to investigate this relationship, we pretrain multiple English language models using random fractions of the full English corpus while maintaining a constant number of training steps.
The performance of the finetuned models is evaluated and plotted in Figure 7. The results demonstrate that our proposed method is more robust to resource scarcity. All language models are able to
maintain performance with 1/64 of the full corpus and after this point, the FACTORIZER-based models are less sensitive to reducing the corpus size even more.

heading: 5 Related work
text: The neural era in NLP has brought about a change in how sentences are typically tokenized into atomic units. Tokenization in current NLP typically involves the segmentation of the sentence into subwords, smaller units than the traditional word tokens of the pre-neural era. The question of what these subword units should consist of has received a fair amount of attention in previous research. Linguistically motivated or rule-based approaches are found (Porter, 1980), however the majority of this work is based on unsupervised morphological segmentation. While this field has a long-standing history (Mielke et al., 2021), subwords have in recent years been based on BPE (Gage, 1994; Sennrich et al., 2016) and variations such as WordPiece (Wu et al., 2016) or SentencePiece (Kudo and Richardson, 2018).
There have been quite a few studies examining the influence of different morphological systems on language modeling performance (Vania and Lopez, 2017; Gerz et al., 2018; Bostrom and Durrett, 2020). In a highly multilingual setting, examining 92 languages, Park et al. (2021) study the influence of morphology on language modeling difficulty, contrasting BPE with the Morfessor system (Creutz and Lagus, 2007) and a rule-based morphological segmenter. There has also been some work comparing BPE with alternative tokenizers for downstream applications, mostly within machine translation and largely with negative results (Ataman and Federico, 2018; Domingo et al., 2018;
Macháček et al., 2018). In this work we instead examine several morpho-syntactic tasks, finding clear improvements over BPE.
The recent pixel-based encoder of language (PIXEL; Rust et al., 2023) reframes language modeling as a visual recognition task, employing a transformer-based encoder-decoder trained to reconstruct the pixels in masked image patches, and disposing of the vocabulary embedding layer completely. The method shows strong results for unseen scripts, however, more modest performance for languages with Latin script, such as English. As in our work, they find morpho-syntactic tasks to benefit the most and more semantic tasks to show more mixed results.

heading: 6 Conclusion
text: We proposed a novel tokenization method where every subword token is factorized into triplets of indices. The main benefit of this factorization is that the subwords-units maintain some characterlevel information without increasing the length of tokenized sequences. In practice, FACTORIZER even slightly decreases the number of subword splits (Figure 3), while noticeably improving the performance of large language models on morphosyntactic tasks, especially on lemmatization (Table 2). Further experiments demonstrated increased robustness to noise and to data scarcity (Figure 6 and 7).
We hope that this work clearly demonstrated the potential of utilizing factorized subwords for language modeling and future work will improve their performance even further.
Limitations
Size. A limiting factor for some low-resource applications might be the size of a saved FACTORIZER file. We only have to store the subword vocabulary, which however takes substantially more space than BPE as it needs to be stored as DAWG trie to keep the tokenization speed similar to BPE. For example, the saved English FACTORIZER takes about 115MB of space while the English BPE with 32K subwords takes just about 1MB of space. We believe that the space requirements are negligable compared to the size of large language models, but they can a limiting factor in some edge cases.
The parameter-efficiency of FACTORIZER follows the basic nature of factorized representations: a sequence of 3 bytes can represent more than 16M values (2563). That’s how we can embed millions of subwords with a negligible parameter count. On the other hand, when we store a vocabulary with millions of subwords on disc, it necessarily requires more space than a BPE vocabulary with 10 000s of subwords.
GLUE performance. While our main interest and focus has been on morpho-syntactic downstream tasks, it is reasonable to ask what is the performance of FACTORIZER-based language models on other tasks, such as natural language understanding. We utilize the pretrained English language models and finetune them on 8 GLUE tasks (Wang et al., 2018). The results in Appendix C indicate that in this setting, our method is comparable to BPE but not better on average, even though both approaches stay within a standard deviation from each other. We hope that these results can be improved in future work.

heading: Acknowledgements
text: We would like to thank Petter Mæhlum, Andrey Kutuzov and Erik Velldal for providing very useful feedback on this work.
The efforts described in the current paper were jointly funded by the HPLT project (High Performance Language Technologies; coordinated by Charles University).
The computations were performed on resources provided through Sigma2 – the national research infrastructure provider for High-Performance Computing and large-scale data storage in Norway.

heading: A Color interpretation
text: Our subword encoding method factorizes every subword-unit into 3× 256 values, which is reminiscent of the standard 24-bit RGB color encoding. Thus every subword can be directly visualized as one pixel with an RGB color. While this interpretation does not have any practical application, it might help to explain our method from a different angle. Note that this analogy is not perfect – our method views all indices as completely independent and thus the factorized ‘R channel’ with index 101 is as close to 102 as to 237.
We illustrate this interpretation in Figure 8, where we show the factorized subwords of the first verse and chorus of the song Suzanne by Leonard Cohen:
Suzanne takes you down to her place near the river You can hear the boats go by, you can spend the night beside her And you know that she’s half-crazy but that’s why you want to be there And she feeds you tea and oranges that come all the way from China And just when you mean to tell her that you have no love to give her Then she gets you on her wavelength And she lets the river answer that you’ve always been her lover
And you want to travel with her, and you want to travel blind And you know that she will trust you For you’ve touched her perfect body with your mind

heading: B Full English ablation results
text: The full results of the comparative study from Section 4.2 (using the full set of UD metrics) are given in Table 3. This table also shows what happens if we keep a constant number of layers between the FACTORIZER and BPE-based language models – as opposed to fixing the parameter count.
Model # params UPOS XPOS UFeats Lemmas UAS LAS MLAS BLEX
BPE 110.8M 97.74±0.02 97.55±0.03 97.72±0.04 98.03±0.03 93.79±0.10 92.06±0.08 86.22±0.11 87.94±0.09 BPE + sampling 110.8M 97.88±0.02 97.71±0.05 97.82±0.05 98.25±0.01 93.94±0.07 92.22±0.05 86.44±0.04 88.35±0.06 BPE + sampling + averaging 110.8M 97.88±0.04 97.72±0.02 97.81±0.05 98.29±0.04 93.89±0.13 92.22±0.12 86.58±0.09 88.47±0.09
FACTORIZER 108.1M 97.76±0.04 97.58±0.04 97.74±0.03 98.01±0.01 94.04±0.13 92.25±0.13 86.26±0.14 88.02±0.20 FACTORIZER + sampling 108.1M 97.94±0.02 97.75±0.06 97.84±0.06 98.40±0.02 93.96±0.06 92.29±0.07 86.62±0.11 88.71±0.12 FACTORIZER + sampling + averaging 108.1M 97.97±0.02 97.77±0.03 97.87±0.02 98.48±0.03 94.06±0.13 92.39±0.14 86.87±0.19 88.92±0.19
FACTORIZER (12L) 86.8M 97.75±0.02 97.54±0.03 97.74±0.03 98.01±0.05 93.72±0.08 91.93±0.08 85.99±0.19 87.69±0.20 FACTORIZER (12L) + sampling 86.8M 97.80±0.03 97.68±0.04 97.72±0.07 98.24±0.03 93.45±0.11 91.66±0.08 85.46±0.09 87.61±0.07 FACTORIZER (12L) + sampling + averaging 86.8M 97.88±0.04 97.70±0.04 97.76±0.04 98.31±0.03 93.75±0.06 92.00±0.09 86.06±0.20 88.19±0.10
Table 3: Full comparison of BPE and FACTORIZER configurations. ‘FACTORIZER (12L)’ denote the configuration without any additional transformer layers. The metrics are measured on the development split of en-ewt. We show the mean and standard deviation statistics over 5 random seeds, the best results are displayed in bold.

heading: C GLUE results
text: As noted in the Limitations section, our method is comparable to BPE but not better on average, when evaluated on this NLU benchmark suite. The detailed results are given in Table 4.

heading: D Hyperparameters
text: All hyperparameters are listed below: VQ-VAE hyperparameters in Table 5, pretraining hyperparameters in Table 6 and finetuning hyperparameters in Table 7. Note that the the full PyTorch (Paszke et al., 2019) source code can be found in https://github.com/ltgoslo/factorizer.
The training was performed on 128 AMD MI250X GPUs (distributed over 16 compute nodes) and took approximately 8 hours per model in a mixed precision mode.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section at the end of the paper
7 A2. Did you discuss any potential risks of your work? Our work is about subword tokenization and performance on morpho-syntactic analysis, we do not see any potential risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1: Introduction
3 A4. Have you used AI writing assistants when working on this paper? We used ChatGPT for grammatical error correction of the main sections from the paper.
B 3 Did you use or create scientific artifacts? Sections 3 and 4
3 B1. Did you cite the creators of artifacts you used? Sections 3 and 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that this work clearly demonstrated the potential of utilizing factorized subwords for language modeling and future work will improve their performance even further. Limitations
Size. A limiting factor for some low-resource applications might be the size of a saved FACTORIZER file. We only have to store the subword vocabulary, which however takes substantially more space than BPE as it needs to be stored as DAWG trie to keep the tokenization speed similar to BPE. For example, the saved English FACTORIZER takes about 115MB of space while the English BPE with 32K subwords takes just about 1MB of space. We believe that the space requirements are negligable compared to the size of large language models, but they can a limiting factor in some edge cases. The parameter-efficiency of FACTORIZER follows the basic nature of factorized representations: a sequence of 3 bytes can represent more than 16M values (2563). That’s how we can embed millions of subwords with a negligible parameter count. On the other hand, when we store a vocabulary with millions of subwords on disc, it necessarily requires more space than a BPE vocabulary with 10 000s of subwords. GLUE performance. While our main interest and focus has been on morpho-syntactic downstream tasks, it is reasonable to ask what is the performance of FACTORIZER-based language models on other tasks, such as natural language understanding. We utilize the pretrained English language models and finetune them on 8 GLUE tasks (Wang et al. , 2018). The results in Appendix C indicate that in this setting, our method is comparable to BPE but not better on average, even though both approaches stay within a standard deviation from each other. We hope that these results can be improved in future work