Tokenization and the Noiseless Channel
Year: 2023

Abstract:
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performance than others. We propose that good tokenizers lead to efficient channel usage, where the channel is the means by which some input is conveyed to the model and efficiency can be quantified in information-theoretic terms as the ratio of the Shannon entropy to the maximum possible entropy of the token distribution. Yet, an optimal encoding according to Shannon entropy assigns extremely long codes to low-frequency tokens and very short codes to high-frequency tokens. Defining efficiency in terms of Rényi entropy, on the other hand, penalizes distributions with either very high or very low-frequency tokens. In machine translation, we find that across multiple tokenizers, the Rényi entropy with α = 2.5 has a very strong correlation with BLEU: 0.78 in comparison to just −0.32 for compressed length.

heading: 1 Introduction
text: Tokenization, the practice of breaking up text into words or subword pieces, or more generally, tokens,1 is often the first step in an NLP pipeline. A wide variety of tokenization functions have been proposed in the NLP literature (Mermer, 2010; Sennrich et al., 2016; Kudo, 2018). And, indeed, research on developing a good tokenization function continues because how one tokenizes may have a large impact on model performance in the downstream task. For instance, Gowda and May (2020) note BLEU ranges from 28 to 37 just by changing the size of the vocabulary in their machine translation (MT) pipeline. A direct extrinsic evaluation of a tokenization function,
We release the tokenization-scorer package (App. B). 1To avoid ambiguity, we eschew the common expressions word and subword and, instead, adopt the term token to mean an element of the vocabulary after tokenization. We formally define token in §2.
however, is computationally intensive: One first has to retokenize the corpora (generally quick), but then retrain the NLP model (often computationally intensive) to evaluate the effect. For this reason, characterizing the intrinsic properties of a good tokenization function has practical benefits (Gallé, 2019; Gowda and May, 2020).
Our paper takes an information-theoretic approach to characterizing a good tokenization function. Following Gallé (2019), we contend that tokenization may be fruitfully viewed as determining a good dictionary code for a language. Fortunately, dictionary codes are equipped with a natural intrinsic metric of utility (expected code length) whereas there are many ways to extrinsically measure the tokenization quality. For simplicity and in line with previous research, we choose a specific downstream task metric: BLEU (Papineni et al., 2002) in the domain of MT.2
We hypothesize that, ceteris paribus, downstream task metrics should correlate with the expected code length of the unigram token distribution. While not immediately intuitive, the motivation is that there is a theoretical connection between the expected code length (under an optimal encoder) of a token distribution and that distribution’s Shannon entropy: The latter gives us a lower bound on the former. And given a fixed vocabulary size, higher entropy token distributions are more desirable because they are more balanced, i.e., there are fewer tokens that occur too rarely or too frequently. This characteristic should in turn balance a model’s ability to learn representations for the entire vocabulary, which requires exposure to enough instances of each token while also penalizing the use of very frequent character sequences as tokens, which is often inefficient due to their lack of distinct meaning.
Yet when using Shannon entropy as our metric of a distribution’s balance, the optimal token distribution may still include a large number of in-
2In Tab. 2 (Appendix) we replicate the findings with CHRF (Popović, 2015), BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020).
5184
frequent tokens. This behavior may be undesirable for a number of reasons that we subsequently discuss. Accordingly, we formulate the compression principle, which states that downstream task metrics, e.g., BLEU, should correlate with the expected code length subject to a penalty for long codewords (which correspond to infrequent tokens). Consequently, we introduce a more nuanced formulation of efficiency that employs Rényi entropy (Rényi, 1961),3 whose hyperparameter α allows us to penalize the use of long codes to varying degrees.
In the experimental portion of our paper, we predict the performance of MT models. We find that the channel efficiency with Rényi entropy with α = 2.5 yields a Pearson correlation of 0.78 with BLEU on German → English MT (1M parallel sentences from CommonCrawl). This stands in contrast to Shannon entropy or expected sequence length, which yield Pearson correlations of only 0.22 and −0.30, respectively.
We also provide an easy-to-use package to score tokenizations. See App. B for usage instructions.

heading: 2 Tokenization
text: Tokenization is generally defined informally as the breaking up of text into a sequence of tokens which are then encoded into a machine-interpretable format. However, to proceed with our analysis, we require a more formal treatment. First, we assume that there exists an alphabet, a finite, non-empty set of characters Σ. We call a string of characters σ = ⟨σ1σ2 · · ·σN ⟩ ∈ Σ∗ a text. In this formulation, we assume that the alphabet Σ includes all characters, including punctuation and a distinguished white space character. Finally, an unordered multiset of texts {σ1, . . . ,σM} ⊂ Σ∗ is termed a corpus of size M . We denote the true distribution over all texts as pΣ∗ . Every pΣ∗ induces a marginal distribution over Σ, which we call the Σ-unigram distribution:
pΣ(σ) def = ∑
σ∈Σ∗ pΣ∗(σ)
count(σ,σ)
|σ| (1)
where count(σ,σ) returns the number of times the character σ appears in text σ. In general, we do not have access to pΣ∗ but rather only to samples from pΣ∗ with which we can represent an empirical
3Campbell (1965) shows that Rényi gives a lower-bound on the expected length of an optimal code subject to a length penalty, i.e., overly long or short codewords are penalized.
distribution p̂Σ∗ . Our formal analysis, however, will consider pΣ∗ .
Let ∆ be a second alphabet, which we call the tokenization alphabet. We define a tokenization function t : Σ∗ → D ⊆ ∆∗ as a function mapping texts in alphabet Σ to sequences of tokens in D = t(Σ∗), One popular choice in NLP is to have Σ be a set of Unicode characters and ∆ be a set of strings of Unicode characters. In this case, the tokenization function t segments the text σ into tokens corresponding to smaller chunks of text. There are many approaches for devising different t’s; a brief overview of some of them is offered in App. E.
Furthermore, for our purposes, it is useful to restrict tokenization functions to those that are invertible (bijections), i.e., rules where we can undo the tokenization. This way, the original text can be reconstructed and no information is lost during tokenization.
Example 2.1. Any injective tokenization function, i.e., mapping different inputs δ′, δ′′ to different outputs σ′,σ′′, satisfies our requirements. As a counter example, consider a tokenization function t1 for which t1(two_cows) = ⟨two,_, [UNK]⟩ and t1(two_birds) = ⟨two,_, [UNK]⟩. Indeed, t1’s lack of injectivity prevents us from recovering the original text from the token sequence ⟨two, _, [UNK]⟩.
Because of our restriction to invertible tokenization functions, with a change of variable we can convert the distribution over texts in Σ∗ into one over token sequences δ in D in a straightforward manner: p∆∗(δ) = pΣ∗(t−1(δ)). Note that the pushforward p∆∗(δ) induces a distribution over ∆∗ but with support limited to D. In applied NLP, there is currently no widely accepted notion of the intrinsic quality of a tokenization function. Rather, practitioners are generally interested in its extrinsic performance, i.e., the performance of a model trained on a corpus tokenized using a certain tokenization function. Under such an evaluation, given two tokenization functions, the one that enables better performance on the downstream task is taken to be better. However, gauging the quality of a tokenizer function in this manner is computationally expensive. Thus, we develop an information-theoretic intrinsic evaluation.

heading: 3 Communication in a Noiseless Channel
text: Our analysis of tokenization schemes relies on the following framing: Our ultimate goal when tokeniz-
ing a text σ ∼ pΣ∗ is the transmission of this text across a hypothetical channel. To perform this feat, we first tokenize σ into a sequence in D ⊆ ∆∗. We then encode each token in ∆ as a sequence of symbols from the set {1, . . . , b}, where b is determined by the channel. Our goal is to analyze the properties of tokenization schemes that lead to models with good downstream performance.
In the case of a noisy channel, we seek an encoding scheme that will help ensure that σ is resilient to noise in addition to efficiently encoding σ. However, in the noiseless case, we only care about efficiency. We can assume that we are working with a noiseless channel because, in the process of encoding data, no information is ever altered by a stochastic process. In this case, one can equivalently think of noiseless channel encoding as compression. Thus, our analysis proceeds by considering the efficiency of different tokenization functions as if our goal is to use them to communicate over a noiseless channel. To this end, we first discuss the conditions for building such an encoding and then discuss the concept of efficient channel usage.
Definition 3.1. A token-level encoder enc∆ is a function enc∆ : ∆ → {1, . . . , b}∗ that maps every token δ ∈ ∆ to a string of symbols in base b, which we call a codeword. We can naturally lift the tokenlevel encoder to a sequence-level encoder using concatenation as enc∆(δ) = ⊕|δ| n=1 enc∆(δn). 4
In order to be able to uniquely decode a string δ, we further require that enc∆ produces prefix-free5 codes for all tokens in ∆. As an example, Huffman encoding provides a fast and nearly optimal (in a sense to be discussed in the subsequent section) method to construct prefix-free codes (Huffman, 1952).
Example 3.2 (One-hot encoding). Consider a tokenization alphabet ∆. In NLP, when b = 2, the most straightforward way of encoding the nth element of ∆ is a vector of zeroes with length |∆| with 1 on position n.
Example 3.3 (Transmission). We consider an arbitrary encoder enc∆ over a given alphabet ∆ and a channel with b = 2. Now given a text and some tokenization function t(two_cows) = ⟨two_, cow, s⟩ we apply the encoder: enc∆(two_) = 1010101 enc∆(cow) = 101111101 and enc∆(s) = 01010
4Whitespace information is not lost here because it is included in the tokens (see Example 2.1).
5Prefix-free means that no codeword is a prefix of any other codeword.
and for the whole sequence enc∆(t(two_cows)) = 101010110111110101010.
For the remainder of the paper, we will not be interested in any specific enc∆, but rather in an optimal token-level encoder that we can achieve, as measured by expected code length measures.
Definition 3.4. The expected code length Lenc∆ of a token-level encoder enc∆ is defined as
Lenc∆(p∆) = ∑
δ∈∆ p∆(δ)|enc∆(δ)| (2)
A well-known result from information theory tells us that Eq. (2) is bounded by the Shannon entropy of W∆, a ∆-valued random variable with law p∆. To introduce the theorem, we first define Shannon entropy.
Definition 3.5. The Shannon entropy of W∆ is defined as
H(W∆) def = −
∑ δ∈∆ p∆(δ) log p∆(δ) (3)
For channels using b symbols for transmission, the logarithm is of base b. Traditionally in information theory, ones takes b = 2.
Theorem 3.6 (Shannon’s Source Coding Theorem). Let W∆ be a ∆-valued random variable with law p∆ and let enc∆ be an encoder. Then,
H(W∆) ≤ Lenc∆(p∆) (4)
with the optimal token-level encoder enc⋆∆ satisfying
Lenc⋆∆(p∆) ≤ ⌈H(W∆)⌉ (5)
This theorem tells us that if we wish to communicate tokens from the alphabet ∆ through a noiseless channel, their minimum expected length for any possible encoding is bounded by the Shannon entropy of the distribution p∆. An optimal token-level encoder will produce codes with the expected length within those exact bounds. We can prove a very similar result to Shannon’s source coding theorem (Shannon, 1948) that tells us how well we can optimally encode a ∆∗-valued source using only the token-level encoder.
To this end, we first introduce the notions of expected sequence length and average per-token encoding length, and then offer a lower-bound on the compression achievable using only a tokenlevel encoder. We additionally define three random
variables that will prove useful in our analysis; all of them are pushforwards of p∆∗ .
Let L be a random variable whose values range over strings’ lengths, i.e., L(δ) = |δ|. The expected token sequence length E[L] for sequences sampled according to p∆∗ is then
E[L] = ∑
δ∈∆∗ p∆∗(δ)|δ| (6)
where for notational simplicity, we leave the dependence of this expectation on p∆∗ implicit as it will always be clear from context. Let Xδ(δ) = count(δ,δ) |δ| be the unigram random variable, i.e., a function that returns the proportion of δ that consists of a particular δ. Finally, define the random variable Lenc∆(δ) = ∑ δ∈∆Xδ(δ)|enc∆(δ)|.
We now turn to our first major theorem.
Theorem 3.7. Let p∆∗ be a distribution over ∆∗, and let p∆ be the unigram distribution induced by p∆∗ (Eq. (1)). Then, for an optimal token-level encoder enc⋆∆ : ∆ → {1, . . . , b}∗ lifted to the sequence level, the following lower and upper bounds hold:6
H(W∆) ≤ Lenc⋆∆(p∆∗)− Cov(Lenc⋆∆ , L)
E[L] ≤ ⌈H(W∆)⌉
(7)
Proof. The proof is given in App. C. ■
In the special case of Shannon entropy, we additionally arrive at the following stronger inequality:
Lenc⋆(p∆∗) ≤ Lenc⋆∆(p∆∗) (8) This holds because enc⋆ is not constrained to token-level codes and the unconstrained minimum over all codes is naturally lower than the constrained version. As a concrete example, even if two δ′, δ′′ ∈ ∆ always appear together in practice, enc⋆∆ must assign both δ
′ and δ′′ their own unique code. Such a constraint does not apply to enc⋆. We foreshadow that this inequality does not generalize to Rényi entropy, as discussed in §4.
Theorem 3.7 tells us that the expected code length of a sequence-level encoder, based on a token-level encoder, is proportional to the expected code length of the unigram distribution up to an additive covariance factor. This allows us to
6Be careful not to confuse enc⋆∆, an optimal token-level encoder (here lifted to the sequence level), with enc∆∗ , an arbitrary sequence-level encoder, which we usually denote enc when clear from context.
determine both a lower-bound for the expected code length of such an encoder and an upper-bound for the expected code length of a sequence-level encoder based on an optimal token-level encoder.
We are now in the position to return to the main objective of this paper: Assessing the quality of different tokenizers. One natural way of comparing tokenizers would be to compare properties of the distributions over tokens that they each induce. At first glance, Shannon entropy looks like the most obvious candidate for such a property. However, for distributions over ∆ of different sizes, it is not directly comparable. The efficiency of a tokenization function addresses this issue. Definition 3.8. Let pΣ∗ be a distribution over Σ∗, let t : Σ∗ → ∆∗ be a tokenization function, and let p∆∗ be the distribution over ∆∗ induced by t. The efficiency of t is defined as
eff(pΣ∗, t) def = Lenc⋆∆(p∆∗) LencU∆(p∆∗)
(9)
where enc⋆∆ is an optimal token-level encoder and encU∆ a uniform encoder that assigns all tokens in ∆ codes of equal length: ⌈log |∆|⌉. Theorem 3.9. The efficiency of t is upper-bounded by
⌈H(W∆)⌉+ Cov
( Lenc⋆
∆ ,L
)
E[L]
log |∆| ≥ eff(pΣ∗, t) (10)
and lower-bounded by
H(W∆) + Cov
( Lenc⋆
∆ ,L
)
E[L]
⌈log |∆|⌉ ≤ eff(pΣ∗, t) (11)
where W∆ is a ∆-valued random variable with law p∆, the unigram distribution induced by t.
Proof. The proof is given in App. C. ■
Note that the upper bound given in Eq. (10) tells us how efficient the best code could be, which is the more interesting bound for our purposes. Additionally, we note that, by introducing a normalization factor, efficiency provides a better solution than directly comparing distributions’ entropies. We illustrate this in the following example.
Example 3.10. Consider a tokenization function t1 with tokenization alphabet ∆1 where |∆1| = 6. We then introduce a second tokenization function t2 with a tokenization alphabet ∆2 defined to be
∆1 plus an additional 6 tokens that occur very infrequently. The difference between these two distributions is illustrated in Fig. 1. If, for example, we relied solely on the Shannon entropy, which is higher for more uniformly spread-out distributions, we would judge the second distribution to be better (2.50 < 3.08). However, the efficiency tells the opposite story (0.97% > 0.86%).
As Example 3.10 suggests, the measure provided by efficiency is in line with the idea of a more balanced distribution over ∆∗. Informally, we do not want a tokenizer that induces a distribution with very low entropy, as this is indicative of an unbalanced distribution. The efficiency eff provides us with a notion of this imbalance. To relate efficiency back to our metaphor of the noiseless channel, we note that the quantity 1 − eff(p∆∗, t) is known as relative redundancy and corresponds to the maximum data compression ratio (in percentage of how much can data size be reduced) that can be achieved.
4 Rényi Efficiency
Definition 3.8, the standard definition of efficiency, is based on Shannon entropy. Upon closer inspection, we see it linearly penalizes the use of long codes. To see why, consider a case where the distribution changes such that the entropy increases by one. Then, the upper-bound for the expected code length provided by an optimal encoder also increases by one. However, in some cases, we may wish to assign a non-linear cost to code length, e.g., there may be a non-linearly higher cost for decoding longer codes. In the context of choosing the vocabulary for a model, this corresponds to our desire to avoid inducing tokens that occur very infrequently because there may not be enough examples of them in the training data for the model to learn. To add an additional degree of freedom to accommodate such preferences Campbell (1965) generalizes the measure of expected code length to discounted expected code length for a hyperparameter s as follows:7
L(s)enc∆(p∆) def = lim
s′→s
log (∑
δ∈∆ p∆(δ)b s′|enc∆(δ)|
)
s′ (12)
By L’Hôpital’s rule, we can show that
L(0)enc∆(p∆) = ∑
δ∈∆ p∆(δ)|enc∆(δ)| (13)
7Our notation differs slightly from Campbell (1965).
and, additionally, that
L(∞)enc∆(p∆) = maxδ∈∆ |enc∆(δ)| (14)
Beyond the limiting cases, for s ∈ (−1,∞) \ {0}, we further note that L(s)enc∆(p∆) is a monotonically increasing function of s. The larger our value of s, the more disproportionately L(s)enc∆(p∆) increases as a function of the longest codeword, which often corresponds to the encoding of a low-frequency character in a good code because high-frequency tokens are assigned the shorter codes in order to minimize the expected code length. For large enough s, this has the effect of encouraging all codewords to be roughly of equal length. Campbell (1965) sought an analogue of Shannon’s coding theorem for L(s)enc∆(p∆) where s ̸= 0. As it turns out, there is a deep connection with the Rényi entropy.
Definition 4.1. The Rényi entropy of order α > 0 is defined as
Hα(p∆) = lim α′→α
1 1− α′ log (∑
δ∈∆ p∆(δ)
α′ )
(15)
Prima facie, Rényi entropy bears some semblance to L(s)enc∆(p∆). To see this, consider the limiting cases. At α = 0, we have
H0(p∆) = log |∆| (16)
And, at α = ∞, we have
H∞(p∆) = max δ∈∆
− log p∆(δ) (17)
Finally, we have H1(p) = H(p), i.e., α = 1 corresponds to Shannon entropy, another result which can be shown by L’Hôpital’s rule. These examples suggest the correspondence α = (1 + s)−1, which fits the three cases considered, e.g., note that α = 0 when s → ∞. Moreover, this is exactly the intuition we argued for above: When α = 0, we encode tokens with codewords of the same length which
follows from minimizing the length of the longest codeword. On the other hand, when s = −1, we encourage shorter codes for high-probability tokens. This case corresponds to α = ∞. We now prove that, similarly to how H(W∆) provides bounds for Lenc⋆∆(p∆∗) in Theorem 3.7, Hα(W∆) provides bounds for L(s)encs∆(p∆∗), where enc s ∆ is an encoder optimal with respect to a given s = α−1 + 1. We term such an encoder s-optimal. Theorem 4.2 (Generalization of Campbell (1965)). Let Hα be the Rényi entropy of order α and let L(s)enc∆(p∆) (Eq. (12)) be the discounted expected code length for the encoder enc∆, where s = α−1 − 1. Moreover, let W∆ be a ∆-valued random variable with law p∆. Then for an s-optimal tokenlevel encoder encs∆, the following bound holds on the discounted expected code length:
Hα(W∆) ≤ L(s)encs∆(p∆) ≤ ⌈Hα(W∆)⌉ (18) Proof. Proof in App. C. ■
Note that we have further generalized Campbell’s (1965) result by allowing some negative values for s, namely, s > −1. As a result, we can induce additional non-linear weight on too short codes as opposed to only long codes.
Now we generalize the efficiency with respect to Shannon entropy to Rényi entropy. Let encs∆ be an s-optimal token-level encoder over token alphabet ∆. Note that several terms from our prior notation can now be expressed in terms of encs∆, i.e., enc⋆∆ = enc 0 ∆ and enc U ∆ = enc ∞ ∆ . Theorem 4.3. Let α = (1 + s)−1 and p∆∗ be a distribution over ∆∗, and let p∆ be the unigram distribution induced by p∆∗ (Eq. (1)). Then, the following inequality holds
⌈Hα(W∆)⌉ ≥ Lencs∆(p∆∗)− Cov(Lencs∆ , L)
E[L] (19)
for an s-optimal sequence-level encoder encs∆ based on token-level encoder enc∆ : ∆ → {1, . . . , b}∗ . Proof. Proof in App. C. ■
Definition 4.4. Let pΣ∗ be a distribution over Σ∗, let t : Σ∗ → ∆∗ be a tokenization function, and let p∆∗ be the distribution over ∆∗ induced by t. The Rényi efficiency of t at α is defined as
effα(pΣ∗, t) def = Lencs∆(p∆∗) Lenc∞∆ (p∆∗)
(20a)
= Lencs∆(p∆∗) LencU∆(p∆∗)
(20b)
where s = α−1 − 1. The Rényi efficiency can be easily upperbounded in a similar manner to the Shannon efficiency. Theorem 4.5. Let pΣ∗ be a distribution over Σ∗, let t : Σ∗ → ∆∗ be a tokenization function, and let p∆∗ be the distribution over ∆∗ induced by t. Then, for an s-optimal token-level encoder encs∆ lifted to the sequence-level, the Rényi efficiency of t at α is upper-bounded by
⌈Hα(W∆)⌉+ Cov
( Lencs
∆ , L
)
E[L]
log |∆| ≥ effα(pΣ∗, t) (21)
where W∆ is a ∆-valued random variable with law p∆, the unigram distribution induced by t.
Proof. Proof in App. C. ■
To provide more intuition of why the non-linear penalization in Rényi efficiency makes for a good measure of distribution balance, we offer a worked example in Example E.1.

heading: 5 The Compression Principle
text: In previous sections, we discussed how different tokenizers lead to token distributions of varying properties. Now, we add the last piece necessary to link the downstream performance of a system with the choice of a tokenizer.
Hypothesis 5.1 (Compression Principle). Let pΣ∗ be a distribution over texts with characters from alphabet Σ and t be a tokenization function from Σ∗ to ∆∗ for some token alphabet ∆. Let p∆ be the ∆-unigram distribution induced by t. Finally, let PERFORMANCEM (t) be some measure of performance of a system M which uses tokenization t. Then, for some α dependent on M , effα(pΣ∗, t) is a good predictor of PERFORMANCEM (t).
In words, we hypothesize that the efficiency of the tokenization function t is highly correlated with the downstream performance. We will verify this claim experimentally in §6.
Rényi Entropy α. The choice of α for Hα determines the extent to which longer codewords are penalized. On one hand, if we observe that Rényi efficiency with low α correlates the best with performance, we can conclude that longer
codewords and, hence, very low-frequency tokens hurt performance. On the other hand, if we observe that Rényi efficiency with high α, we can conclude that shorter codewords and, hence, very highfrequency tokens hurt performance. Note that most downstream NLP applications do not explicitly use codewords (all token representations are the same size), but long codewords are still a natural way to think about low- and high-frequency tokens.
Learnability. The most intuitive explanation for why some tokenization functions enable good downstream results and some worse is that having many low-frequent tokens will prevent the model from learning their distributional properties. This hypothesis can be related back to the sample complexity of the learning algorithm, i.e., the number of training samples needed by the model in the given setting to learn the function of interest. If we accept that part of the MT task is learning the meaning of all individual vocabulary tokens, then sample complexity could (at least partially) be expressed in terms of the number of instances of each token. This argument is made by Gowda and May (2020), who are concerned with what proportion of δ ∈ ∆ appears at least 100 times in the corpus for the downstream task at hand.
Nevertheless, we will see shortly that the best predictor with Rényi efficiency is for α > 1, meaning that higher weight is given to codewords for more frequent tokens. We therefore hypothesize, that very high-frequency tokens have the most impact in downstream performance.

heading: 6 Experiments
text: We now seek empirical evidence for Hyp. 5.1. We focus on MT, where a standard automatic evalu-
ation metric is BLEU (Papineni et al., 2002). We use the English→German CommonCrawl dataset in all experiments. The specifics of the MT system, data and evaluation are described in App. D. We consider two different experimental manipulations. First, we experiment with various modifications of the popular byte-pair encoding (BPE) tokenizer (Sennrich et al., 2016) to control its compression rate. The details are discussed in §6.1. Second, we experiment with a variety of tokenization schemes: Unigram (Kudo, 2018), WordPiece (Devlin et al., 2019), Lempel–Ziv–Welch (Ziv and Lempel, 1977; Welch, 1984) and Morfessor (Creutz and Lagus, 2007; Virpioja et al., 2013; Smit et al., 2014). The details are discussed in §6.2.
Note that throughout our experiments, we make the simplifying assumption of Cov ( Lencs∆ , L ) = 0. It simplifies the upper bound of eff(pΣ∗, t) (from Theorem 3.9) to ⌈H(W∆)⌉log |∆| and the upper bound of effα(pΣ∗, t) (from Theorem 4.5) to ⌈Hα(W∆)⌉
log |∆| . From
our preliminary results, Cov ( Lencs∆ , L ) is negative and small. We leave its more accurate approximation, which requires a Rényi analogue of Huffman coding as in Jelinek (1968), to future work.

heading: 6.1 Experiment 1
text: In our first experiment, we analyze how predictive various quantitative attributes of a tokenization scheme are of downstream model performance. We consider BPE with 5 different vocabulary sizes: 2k, 4k, 8k, 16k, and 32k. For each vocabulary size, we create multiple tokenization schemes with varying compression rates. As discussed in App. E.1, BPE produces a vocabulary through a greedy compression algorithm. However, in order to achieve a variety of different compression rates,
we inject random noise into the algorithm.8 We achieve this by sampling from a Boltzmann distribution over the pair frequencies with temperature parameter τ ; see App. E.1 for details.9 We then treat each vocabulary size–temperature pair as a single data point in our analysis.
Our main quantitative attribute of interest, i.e., predictor, is Rényi efficiency. Aside from Rényi efficiency, we further consider Shannon and Rényi entropies, Shannon efficiency, and average tokenized sequence length. Further, one popular heuristics for choosing the vocabulary size is given and justified by Gowda and May (2020). It can be summarized as: “Use the highest possible vocabulary size such that 95% of [tokens] occur at least 100 times in the data.” While the constants seem arbitrary, this rule of thumb works well in practice (Gowda et al., 2022; Dramko et al., 2022; Kumar and Thawani, 2022). Nevertheless, it is stated in an algorithmic manner and not as a predictor of performance or learnability. We attempt to turn it into a regressive predictor so as to make it more comparable with the other quantities studied. Given p∆, let fn(p∆) symbolize the frequency of the nth percentile. We then define the quantity Fγ1,γ2(p∆) = ∑ γ1≤n≤γ2 fn(p∆), which in words, is the sum of token frequencies from the γth1 to γth2 percentile. The original work suggests examining the frequency of the 95th percentile, i.e., γ1 = γ2 = 0.95. In contrast, we add an additional degree of freedom as we do not inspect a single percentile frequency but rather a sum across an interval. Later, we scan the whole space for γ1 and γ2 and show that there are better choices that lead to much higher correlations.
We use Pearson and Spearman correlations with
8The empirical effects of choosing BPE merge operations uniformly at random is studied by Sälevä and Lignos (2023).
9Note that the stochasticity in our case is introduced during training and not during inference, as in the popular BPEdropout method (Provilkov et al., 2020).
downstream model performance (measured with BLEU) as our metrics of predictor quality. Recall that Pearson correlation tells us the strength of a linear relationship between two variables. On the other hand, Spearman correlation quantifies the strength of a linear relationship of the ranking.
Results. In order to select α (for effα) as well as γ1 and γ2 ( for Fγ1,γ2), we use half of the data to perform a grid-search, selecting the hyperparameters that lead to the highest Pearson correlation. We show the results of this grid search for Hα/H0 in Fig. 3 (α∗ . = 2.5) and for Fγ1,γ2 in Fig. 4 (γ∗1 . = 0.03, γ∗2 . = 0.83). Unless otherwise stated, we use these values in subsequent experiments. We show the relationship between BLEU, sequence length and Rényi efficiency as approximated by the lower bound (Theorem 4.5) in Fig. 2. A comprehensive comparison for all predictors is shown in Tab. 1. The visualization of the other predictors is in Fig. 6. From these analyses, we can see that the Rényi efficiency provides a significantly better explanation for downstream model performance than any of our other predictors.
When examining which α leads to the highest absolute correlation with BLEU, we can conclude that tokenization schemes that result in fewer very high-frequency tokens are the best for downstream performance. This is evinced by both the relatively high value of α that leads to the best correlation with performance (Fig. 3, α∗ .= 2.5) and by Fig. 4, which shows that frequencies in the top percentile correlate negatively with performance. Importantly, this finding does not contradict Gowda and May’s (2020) rule of thumb, which focuses on low frequency tokens. While very high and very low frequencies produced by a tokenization scheme are not independent, a tokenization scheme may feasibly produce both, neither or only one.
Furthermore, the Pearson correlation between
the efficiency (H2.5/H0) and percentile frequency (F0.03,0.83) is 0.96, which suggests that both predictors are capturing the same underlying effect.

heading: 6.2 Experiment 2
text: In this experiment, we evaluate whether there exist aspects of a tokenization scheme that influence BLEU beyond the Rényi efficiency. Following results in Experiment 1, we focus on Rényi efficiency at α = 2.5. In contrast to the first experiment, we consider different tokenization schemes (BPE, Unigram, WordPiece, LZW, Morfessor). We manipulate their efficiency by lowering the amount of tokenizer training data (2k, 8k, 100k parallel lines) together with varying vocabulary sizes of 4k, 8k, and 16k tokens. We then treat each tokenizationscheme–training-data-size–vocabulary-size triple as a single data point in this analysis. We compare three different linear models (Gelman and Hill, 2006), where BLEU is always the dependent variable: (i) with the tokenization scheme as a random effect, (ii) with Rényi efficiency as a fixed effect, and (iii) with both. Importantly, we treat tokenization scheme as a random effect because the set of tokenization algorithms that we consider does not encompass all possible methods, i.e., only a sample of all possible algorithms are observed.
To compare the ability of these different models to predict BLEU, we look at the average change in log-likelihood of held-out data points under a given model with respect to a baseline model: A model trained with only an intercept term. A larger value of ∆ log-likelihood indicates that the data point is more probable under the comparison model, i.e., the comparison model more closely fits the observed data. We use 10-fold cross-validation to estimate these differences: Our data is split randomly
into 10 folds, where 9 of the folds are used to learn model coefficients and the 10th fold is held back for evaluation. The same process is performed until we have a ∆ log-likelihood value for each data point.
Results. In Fig. 5, we see that Rényi efficiency is a stronger predictor of MT performance than the tokenization scheme alone. Interestingly though, the predictive power of these two predictors seems to be orthogonal, as evinced by the mean ∆ loglikelihood of a model with both predictors. This finding suggests that there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We leave the investigation of such qualities to future work.

heading: 7 Conclusion
text: Our paper presents a new information-theoretic approach to characterizing a good tokenization scheme. We contend that the Rényi efficiency of the unigram distribution that a tokenization scheme produces is a principled measure of the tokenization quality. To test this claim, we evaluate a large set of tokenization schemes, with varying vocabulary sizes and produced by different tokenization schemes. We observe how the Rényi efficiency of these different tokenizations relates to the performance of a downstream MT model. We find that, for an appropriate choice of the parameter α, this new metric has a very strong Pearson correlation with BLEU: 0.78 in comparison to just −0.32 for baseline sequence length. From a theoretical perspective, this property can be connected to a penalization of token distributions that are too unbalanced, having, in particular, very high-frequency tokens. This finding is in line with the more general principle that compression is connected with learnability. Our framework also has practical benefits as it allows for an intrinsic evaluation of tokenization functions.
Limitations
It is possible that there is a hidden effect caused by the language pair direction, model selection, or training data and its size. However, our results bear high statistical significance for cases where we desire high correlation and low statistical significance where we expect low correlation. Assured by this and concerned by the large cost of training a large number of MT systems, we did not experiment with larger data or other language directions apart from limited additional experiments in Tab. 2.

heading: Acknowledgements
text: We would like to acknowledge Eleanor Chodroff and Shayne Sloggett for extensive discussion about the experimental design in Experiment 2. Clara Meister was supported by the Google PhD Fellowship. Juan Luis Gastaldi has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 839730.

heading: A Related Work
text: Prior to the widespread adoption of subword tokenization, large vocabulary sizes (e.g., 500k) were needed to allow for output expressivity and to avoid a high proportion of out-of-vocabulary tokens. Various tricks were devised to tackle the resulting computational issues (Jean et al., 2015; Mi et al., 2016; L’Hostis et al., 2016). On the other side of the spectrum, character-level NMT was also explored (Ling et al., 2015; Costa-jussà and Fonollosa, 2016), though issues arise with large sequence lengths. Mielke et al. (2021) provide an overview of the evolution of NLP tokenization and describe different types of tokenization approaches. They conclude that reasoning about tokenizer choices remains a vital part of modern pipeline preparation. In this context, our work quantifies and hence also automates some of this process by offering a framework to help guide the decision process and hyperparameter selection. Somewhat similar to our work, Ataman and Federico (2018) perform a comparison between BPE and Morfessor, though with only one specific vocabulary size (30k). Similarly to Gowda and May (2020), they suggest that homogeneity of token frequency is an important factor for MT model performance.

heading: B Package Usage
text: The package is open-source10 and can be downloaded via pip and used via the command-line:
$ pip3 install tokenization-scorer $ tokenization-scorer -i en-de.tokenized_with_unigramlm.{en,de} > 0.4826
$ tokenization-scorer -i en-de.tokenized_with_wordpiece.{en,de} > 0.5047
or as a module in Python:
import tokenization_scorer text1 = "pick @@ed pick @@l @@ed pick @@les" tokenization_scorer.score(text1, metric="renyi", power=2.5) > 0.8031528501359657
text2 = "pick @@e @@d pick @@l @@e @@d pick @@l @@e @@s" tokenization_scorer.score(text2, metric="renyi", power=2.5) > 0.9105681923824472
The supported metrics are the ones presented in Tab. 1: renyi_efficiency (default), renyi_entropy, shannon_efficiency, shannon_entropy, percentile_freq, bits, sequence_len. The power in Rényi can be modified using an extra parameter: -e power=2.5. The similar applies to the percentile frequency with -e perc_start=0.03 perc_end=0.83.
10github.com/zouharvi/tokenization-scorer

heading: C Proofs
text: Lemma C.1. Let p∆∗ be a distribution over ∆∗, and let p∆ be the unigram distribution induced by p∆∗ (Eq. (1)). Then, the following equality holds
E[L] · ∑
δ∈∆ p∆(δ)|enc∆(δ)|+Cov
( Lenc∆ , L ) = Lenc∆(p∆∗) (22a)
Proof. Let δ ∈ ∆∗. Define the expected counts as follows
E-count(δ) def= ∑
δ∈∆∗ p∆∗(δ) count(δ, δ) (23a)
= ∑
δ∈∆∗ p∆∗(δ)
count(δ, δ)
|δ| |δ| (23b)
= E δ∼p∆∗ [Xδ(δ) · L] (23c)
We start by manipulating the expected code length
∑
δ∈∆∗ p∆∗(δ)|enc∆(δ)| =
∑
δ∈∆∗ p∆∗(δ)
|δ|∑
n=1
|enc∆(δn)| (24a)
= ∑
δ∈∆∗
|δ|∑
n=1
p∆∗(δ)|enc∆(δn)| (24b)
= ∑
δ∈∆∗
∑ δ∈∆ p∆∗(δ)count(δ, δ)|enc∆(δ)| (24c)
= ∑
δ∈∆ |enc∆(δ)|
(∑
δ∈∆∗ p∆∗(δ)count(δ, δ)
) (24d)
= ∑
δ∈∆ E-count(δ)|enc∆(δ)| (24e)
Now, we proceed with algebraic manipulation.
∑ δ∈∆ E-count(δ)|enc∆(δ)| = ∑ δ∈∆ E δ∼p∆∗ [Xδ(δ) · L] |enc∆(δ)| (25a)
= ∑
δ∈∆
( E
δ∼p∆∗ [Xδ(δ)] E δ∼p∆∗ [L] |enc∆(δ)|+Cov (Xδ(δ), L) |enc∆(δ)|
) (25b)
= ∑
δ∈∆ E δ∼p∆∗ [Xδ(δ)] E δ∼p∆∗ [L] |enc∆(δ)|+
∑ δ∈∆ Cov (Xδ(δ), L) |enc∆(δ)| (25c)
= ∑
δ∈∆ E δ∼p∆∗ [Xδ(δ)] E δ∼p∆∗ [L] |enc∆(δ)|+Cov
(∑
δ∈∆ |enc∆(δ)|Xδ(δ), L
) (25d)
= ∑
δ∈∆ E δ∼p∆∗ [Xδ(δ)] E δ∼p∆∗ [L] |enc∆(δ)|+Cov
(∑
δ∈∆ |enc∆(δ)|Xδ(δ) ︸ ︷︷ ︸ def =Lenc∆ , L
) (25e)
= E[L] · ∑
δ∈∆ p∆(δ)|enc∆(δ)|
︸ ︷︷ ︸ expected unigram code length
+Cov ( Lenc∆ , L ) (25f)
■
Theorem 3.7. Let p∆∗ be a distribution over ∆∗, and let p∆ be the unigram distribution induced by p∆∗ (Eq. (1)). Then, for an optimal token-level encoder enc⋆∆ : ∆ → {1, . . . , b}∗ lifted to the sequence level, the following lower and upper bounds hold:11
H(W∆) ≤ Lenc⋆∆(p∆∗)− Cov(Lenc⋆∆ , L)
E[L] ≤ ⌈H(W∆)⌉
(7)
Proof. By Lemma C.1, we have
E[L] · ∑
δ∈∆ p∆(δ)|enc⋆∆(δ)|+Cov
( Lenc⋆∆ , L ) = Lenc⋆∆(p∆∗) (26)
Now, by applying Theorem 3.6, we achieve
E[L] ·H(W∆) ≤ E[L] ∑
δ∈∆ p∆(δ)|enc⋆∆(δ)| (27a)
E[L] ·H(W∆) + Cov ( Lenc⋆∆ , L ) ≤ E[L] ∑
δ∈∆ p∆(δ)|enc⋆∆(δ)|+Cov
( Lenc⋆∆ , L ) (27b)
E[L] ·H(W∆) + Cov ( Lenc⋆∆ , L ) ≤ Lenc⋆∆(p∆∗) (27c)
Similarly, from Theorem 3.6, we have
E[L] · ⌈H(W∆)⌉ ≥ E[L] ∑
δ∈∆ p∆(δ)|enc⋆∆(δ)| (28a)
E[L] · ⌈H(W∆)⌉+Cov ( Lenc⋆∆ , L ) ≥ E[L] ∑
δ∈∆ p∆(δ)|enc⋆∆(δ)|+Cov
( Lenc⋆∆ , L ) (28b)
E[L] · ⌈H(W∆)⌉+Cov ( Lenc⋆∆ , L ) ≥ Lenc⋆∆(p∆∗) (28c)
Putting these together with some additional algebraic manipulation we get
H(W∆) ≤ Lenc⋆∆(p∆∗)− Cov(Lenc⋆∆ , L)
E[L] ≤ ⌈H(W∆)⌉ (29)
This concludes the proof. ■ 11Be careful not to confuse enc⋆∆, an optimal token-level encoder (here lifted to the sequence level), with enc∆∗ , an arbitrary sequence-level encoder, which we usually denote enc when clear from context.
Theorem 3.9. The efficiency of t is upper-bounded by
⌈H(W∆)⌉+ Cov
( Lenc⋆
∆ ,L
)
E[L]
log |∆| ≥ eff(pΣ∗, t) (10)
and lower-bounded by
H(W∆) + Cov
( Lenc⋆
∆ ,L
)
E[L]
⌈log |∆|⌉ ≤ eff(pΣ∗, t) (11)
where W∆ is a ∆-valued random variable with law p∆, the unigram distribution induced by t.
Proof. Recall that eff(pΣ∗, t) = Lenc⋆ ∆ (p∆∗)
LencU ∆ (p∆∗)
. To bound the denominator we use the fact, that for uniform
distribution, Cov ( LencU∆ , L ) = 0 and obtain E[L] log |∆| ≤ LencU∆(p∆∗) ≤ E[L]⌈log |∆|⌉. Using this fact, we can obtain an upper bound
eff(pΣ∗, t) = Lenc⋆∆(p∆∗) LencU∆(p∆∗)
(30a)
≤ E[L] · ⌈H(W∆)⌉+Cov
( Lenc⋆∆ , L )
LencU∆(p∆∗) (numerator lower-bound from Theorem 3.7) (30b)
≤ E[L] · ⌈H(W∆)⌉+Cov
( Lenc⋆∆ , L )
E[L] log |∆| (denominator upper-bound) (30c)
= ⌈H(W∆)⌉+
Cov ( Lenc⋆
∆ ,L
)
E[L]
log |∆| (30d)
and a corresponding lower bound
eff(pΣ∗, t) = Lenc⋆∆(p∆∗) LencU∆(p∆∗)
(31a)
≥ E[L] ·H(W∆) + Cov
( Lenc⋆∆ , L )
LencU∆(p∆∗) (numerator lower-bound from Theorem 3.7) (31b)
≥ E[L] ·H(W∆) + Cov
( Lenc⋆∆ , L )
E[L]⌈log |∆|⌉ (denominator upper-bound) (31c)
= H(W∆) +
Cov ( Lenc⋆
∆ ,L
)
E[L]
⌈log |∆|⌉ (31d)
■
Theorem C.2 (Generalized Hölder’s inequality; §2 in Aczél and Beckenbach (1980)). Let f , g and h be vectors of positive values and coefficients p, q and r such that all but one are negative and 1p + 1 q + 1 r = 0. Further, if ∀i : figihi = 1 then
∥f∥p ∥g∥q ∥h∥r ≤ 1. (32)
As noted in Aczél and Beckenbach (1980), Theorem C.2 is in fact a simple special case of Theorem 12 in Hardy et al. (1934). We will use Theorem C.2 specifically with r = −1 and hi = figi. This simplifies the requirements for exactly one of p and q to be negative and 1p + 1 q = 1. Eq. (32) can then be restated as the following.
Corollary C.3 (Reverse Hölder’s inequality). If f, g are positive vectors and p, q are such that 1p + 1 q = 1 and exactly one is negative and the other is positive, then
∥f∥p ∥g∥q ≤ ∥fg∥1. (33)
Theorem 4.2 (Generalization of Campbell (1965)). Let Hα be the Rényi entropy of order α and let L(s)enc∆(p∆) (Eq. (12)) be the discounted expected code length for the encoder enc∆, where s = α−1 − 1. Moreover, let W∆ be a ∆-valued random variable with law p∆. Then for an s-optimal token-level encoder encs∆, the following bound holds on the discounted expected code length:
Hα(W∆) ≤ L(s)encs∆(p∆) ≤ ⌈Hα(W∆)⌉ (18)
Proof. Let p = −s and q = 1− α. Let us consider three cases of s:
• s = 0, then α = 1, then Theorem 3.6 applies;
• s ∈ (−1, 0), then α ∈ (1,∞), p ∈ (0, 1) and q ∈ (−∞, 0).
• s ∈ (0,∞), then α ∈ (0, 1), p ∈ (−∞, 0) and q ∈ (0, 1).
In the latter two cases, we have one of p and q is positive and one is negative. Further, our definitions of p and q imply that p−1 + q−1 = 1, which allows us to use the reverse Hölder’s inequality (Corollary C.3). First, we note that the reverse Hölder’s inequality implies that, for finite sequences (xi) and (yi),
(∑ xpi ) 1 p (∑ yqi ) 1 q ≤ ∑ xiyi (34)
(∑ x−si )− 1 s (∑ y1−αi ) 1 1−α ≤ ∑ xiyi (p and q substitution) (35)
Let ℓ(δ) = |enc∆(δ)| be the lengths of the codes given by our encoding. Further, let p∆(δ) be the unigram probabilities and b be the base of our encoding. Now set xi = p∆(δ)− 1 s b−ℓ(δ) and yi = p∆(δ) 1 s . This step is valid because both quantities are positive. Then we proceed with algebraic manipulations
(∑
δ∈∆
( p∆(δ) − 1 s b−ℓ(δ)
)−s )− 1 s (∑
δ∈∆ (p∆(δ)
1 s )1−α
) 1 1−α
≤ ∑
δ∈∆ (p∆(δ)
− 1 s b−ℓ(δ))(p∆(δ) 1 s ) (36a)
(∑
δ∈∆ (p∆(δ)
− 1 s b−ℓ(δ))−s
)− 1 s (∑
δ∈∆ (p∆(δ)
1 s )1−α
) 1 1−α
≤ ∑
δ∈∆ b−ℓ(δ) (algebra) (36b)
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )− 1 s
︸ ︷︷ ︸ def =A
(∑
δ∈∆ p∆(δ)
α
) 1 1−α
︸ ︷︷ ︸ def =B
≤ ∑
δ∈∆ b−ℓ(δ) ︸ ︷︷ ︸ def =C
(algebra) (36c)
Then, let A def= (∑ δ∈∆ p∆(δ)b s·ℓ(δ))− 1s , B def= (∑ δ∈∆ p∆(δ) α ) 1 1−α and C def= ∑ δ∈∆ b −ℓ(δ) in the subsequent proof descriptions. Note that by the Kraft–McMillan inequality (Kraft, 1949), because our code is prefix-free (and b > 0), it must be that 0 < C ≤ 1.
(∑ δ∈∆ p∆(δ) α ) 1
1−α ∑
δ∈∆ b −ℓ(δ) ≤
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) ) 1 s
(divide by AC) (37a)
(∑
δ∈∆ p∆(δ)
α
) 1 1−α ≤ (∑
δ∈∆ p∆(δ)b
s·ℓ(δ) ) 1 s
(0 < C ≤ 1) (37b)
1 1− α log (∑
δ∈∆ p∆(δ)
α ) ≤ 1
s log
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )
(take log) (37c)
Hα(W∆) ≤ L(s)enc(p∆) (37d)
Our constraint on C is met and Eq. (37d) holds with equality when the lengths of our code satisfy the following relationship
b−ℓ(δ) = p∆(δ)
α
∑ δ′∈∆ p∆(δ ′)α (38a)
ℓ(δ) = −α logb p∆(δ) + logb
(∑
δ′∈∆ p∆(δ
′)α )
(38b)
Now we consider a sequence of M tokens δ = ⟨δ1, . . . , δM ⟩, where each δm is sampled according to p∆; note that this is not necessarily equivalent to a sequence δ ∼ p∆∗ , as we do not assume independence between tokens in that setting. Let us first lift p∆ to take sequences, i.e., p∆(δ) = ∏M m=1 p∆(δ), which follows naturally due to the independence of each δm in this setting. Now let
Q = ∑
δ∈∆M p∆(δ)
α =
(∑
δ∈∆ p∆(δ)
α )M (39)
where the later equality follows from the fact that there are |∆|M sequences in ∆M and each δ appears an equal number of times. We will now use Eq. (38b) to reason about the length of an optimal code for δ, where we similarly denote length as ℓ(δ). We will first assume that s ∈ (−1, 0) but later show that a slight modification to the proof makes it viable also for s > 0. Following the result in Eq. (38b), the length ℓ(δ) of an optimal integer-length code for δ should satisfy
−α logb p∆(δ) + logbQ ≤ ℓ(δ) < −α logb p∆(δ) + logbQ+ 1 (40a) −sα logb p∆(δ) + s logbQ ≥ s · ℓ(δ) > s− sα logb p∆(δ) + s logbQ (multiply by s ∈ (−1, 0))
(40b)
p∆(δ) −sαQs ≥ bs·ℓ(δ) > bsp∆(δ)−sαQs (raise to power b)
(40c)
p∆(δ) −sα+1Qs ≥ p∆(δ)bs·ℓ(δ) > bsp∆(δ)−sα+1Qs (multiply by p∆(δ))
(40d)
p∆(δ) αQs ≥ p∆(δ)bs·ℓ(δ) > bsp∆(δ)αQs (sα = 1− α)
(40e) ∑
δ∈∆M p∆(δ)
αQs ≥ ∑
δ∈∆M p∆(δ)b
s·ℓ(δ) > Qsbs ∑
δ∈∆M p∆(δ)
α
(sum across all sequences of length M ) (40f)
Qs+1 ≥ ∑
δ∈∆M p∆(δ)b
s·ℓ(δ) > Qs+1bs (sub. Q)
(40g)
Qs+1 ≥ (∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )M
> Qs+1bs (same logic as Eq. (39))
(40h)
(s+ 1) logbQ ≥ M logb
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )
> (s+ 1) logbQ+ s (take logb)
(40i)
s+ 1
s logbQ ≤
M
s logb
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )
< s+ 1
s logbQ+ 1
(divide by s ∈ (−1, 0)) (40j)
1 1− α logbQ ≤ M s logb
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )
< 1
1− α logbQ+ 1 (substitute α)
(40k)
1
1− α logb
(∑
δ∈∆ p∆(δ)
α )M ≤ M
s logb
(∑
δ∈∆ p∆(δ)b
s·ℓ(δ) )
< 1
1− α logb
(∑
δ∈∆ p∆(δ)
α )M + 1
(definition of Q) (40l)
M ·Hα(W∆) ≤ M · L(s)encs∆(p∆) < M ·Hα(W∆) + 1 (40m)
Hα(W∆) ≤ L(s)encs∆(p∆) < Hα(W∆) + 1 M (divide by M )
(40n)
Note that in Eq. (40b), we multiplied by a negative value and therefore swapped the directions of the inequalities. Then, in Eq. (40j) we divided by a negative value and swapped the inequalities back to their original directions. If s > 0, we would not change the directions of the inequalities and the proof would proceed as before.
For large M , we can get arbitrarily close to Hα. However, if we wish for ℓ(δ) to be an integer, the upper bound becomes ⌈Hα(W∆)⌉. ■ Theorem 4.3. Let α = (1+s)−1 and p∆∗ be a distribution over ∆∗, and let p∆ be the unigram distribution induced by p∆∗ (Eq. (1)). Then, the following inequality holds
⌈Hα(W∆)⌉ ≥ Lencs∆(p∆∗)− Cov(Lencs∆ , L)
E[L] (19)
for an s-optimal sequence-level encoder encs∆ based on token-level encoder enc∆ : ∆ → {1, . . . , b}∗ .
Proof. Let encs∆ be an s-optimal code, i.e., a code that minimizes L(s)enc∆(p∆∗). By Lemma C.1, we have
E[L] · ∑
δ∈∆ p∆(δ)|encs∆(δ)|+Cov
( Lencs∆ , L ) = Lencs∆(p∆∗) (41)
To prove the bound, we proceed as follows. Assume s ̸= 0 as that case is covered by Theorem 3.7. We first start with a simple application of Jensen’s inequality:
⌈Hα(W∆)⌉ ≥ L(s)encs∆(p∆) (Theorem 4.2) (42a)
= logb
(∑ δ∈∆ p∆(δ)b s|encs∆(δ)| )
s (definition) (42b)
≥ ∑
δ∈∆ p∆(δ)s|encs∆(δ)| logb b s
(Jensen’s inequality) (42c)
= ∑
δ∈∆ p∆(δ)|encs∆(δ)| (42d)
Algebraically, combining the above results
⌈Hα(W∆)⌉ ≥ Lencs∆(p∆∗)− Cov(Lencs∆ , L)
E[L] (43)
which proves the theorem. ■
Theorem 4.5. Let pΣ∗ be a distribution over Σ∗, let t : Σ∗ → ∆∗ be a tokenization function, and let p∆∗ be the distribution over ∆∗ induced by t. Then, for an s-optimal token-level encoder encs∆ lifted to the sequence-level, the Rényi efficiency of t at α is upper-bounded by
⌈Hα(W∆)⌉+ Cov
( Lencs
∆ , L
)
E[L]
log |∆| ≥ effα(pΣ∗, t) (21)
where W∆ is a ∆-valued random variable with law p∆, the unigram distribution induced by t.
Proof. Recall from Definition 4.4 that effα(pΣ∗, t) = Lencs ∆ (p∆∗)
LencU ∆ (p∆∗)
, with s = 1a − 1. Again, to bound
the denominator, we use the fact, that for uniform distribution, Cov ( LencU∆ , L )
= 0 and obtain E[L] log |∆| ≤ LencU∆(p∆∗). Then, we proceed as follows
effα(pΣ∗, t) = Lencs∆(p∆∗) LencU∆(p∆∗)
(44a)
≤ E[L] · ⌈Hα(W∆)⌉+Cov
( Lencs∆ , L )
LencU∆(p∆∗) (upper-bound numerator) (44b)
≤ E[L] · ⌈Hα(W∆)⌉+Cov
( Lencs∆ , L )
E[L] · log |∆| (lower-bound denominator) (44c)
= ⌈Hα(W∆)⌉+
Cov ( Lencs
∆ ,L
)
E[L]
log |∆| (44d)
This proves the result. ■

heading: D Model, computation and reproducibility details
text: For the MT model, we use the transformer_iwslt_de_en model architecture in Fairseq (Ott et al., 2019). For data, we use 1M training and 50k dev parallel sentences from English-German CommonCrawl (El-Kishky et al., 2020). BLEU evaluation performed using SacreBLEU (Post, 2018). Details for the model training and the code to reproduce the experiments in this paper will be made publicly available. For the first experiment we trained 5× 6× 9 = 270 MT models. For the second experiment we trained 5 × 5 × 3 × 3 = 225 MT models. We used varying GPU models (GTX 1080, RTX 2080, RTX 3090) based on availability in shared compute cluster. Although different hardware and different tokenizations had an impact on the training time, the average per one configuration was 1.5 days. Overall, we estimate 800 GPU days.

heading: E Tokenization Schemes
text: In this section, we describe the four tokenization schemes used together in the second experiment. Special attention is paid to BPE, which plays a role in the first experiment. Tab. 3 shows how different tokenizers with varying vocabulary sizes tokenize the same word. For simplicity, we do not include variable-length encoding, even though it has been previously applied to MT (Chitnis and DeNero, 2015).
E.1 Byte-Pair Encoding BPE was first discovered by Gage (1994) as a faster compression algorithm alternative to Lempel–Ziv– Welch. It was later adapted by Sennrich et al. (2016) as a tokenizer for MT. The algorithm starts by splitting words of individual tokens of length 1 (characters). Then it repeatedly takes the most frequent pair of adjacent tokens and joins it into a new single token, adding this token to the vocabulary. This procedure is repeated until we fill the predefined vocabulary budget. Zouhar et al. (2023) show that this greedy approach is approximately optimal when searching for the vocabulary (merge sequence). We are however interested in intentionally suboptimal vocabularies.
6 7 8 9 10 11
Entropy
38
39
40
41
D ev
B L
E U
Pearson correlation 0.22 (p=0.281) Spearman correlation 0.12 (p=0.578)
V=2k V=4k V=8k V=16k V=32k
0.4 0.5 0.6 0.7 0.8
Entropy efficiency
38
39
40
41
D ev
B L
E U
Pearson correlation 0.56 (p=0.004) Spearman correlation 0.38 (p=0.060)
V=2k V=4k V=8k V=16k V=32k
Pearson correlation 0.76 (p<0.001) Spearman correlation 0.63 (p<0.001)
Pearson correlation 0.49 (p=0.013) Spearman correlation 0.38 (p=0.062)
Temperature. In order to introduce stochasticity into the process and intentionally alter the tokenization, instead of deterministically merging the most frequent token pair, we randomly sample a pair proportionally to their frequencies, similar to setup of Sälevä and Lignos (2023). We add an additional hyperparameter to this strategy by annealing the frequency distribution with a temperature parameter τ , where annealing is performed via a softmax. We use τ = 0+ (original greedy BPE), 0.2, 0.4, 0.9, 100, −100, −0.9, −0.4, −0.2, and 0−. Progressively, each temperature creates less and less optimal BPE compression model, increasing the encoded length of the data. The last model, with τ = 0−, we dub antigreedy because it always chooses the least frequent pair to merge, practically leading to a character-only model.
E.2 Unigram LM
While BPE repeatedly merges the most frequent pair, Unigram LM (Kudo, 2018) tokenizer modifies this part of the algorithm with a more complex approach. The algorithm then jointly optimizes the token vocabulary and the unigram probability of the tokenized text. Because of this dual objective, the algorithm is done in Expectation-Maximization manner. We start by seeding the token vocabulary with the most frequent substrings. In one step, probability is assigned to individual tokens and also how much worse the
to the are in X and those corresponding to cow in X .
overall probability of the whole text would be if the single token was removed. In the second step, top-η% of the tokens is preserved. These two steps are repeated until the vocabulary size is reduced to size V .
E.3 Linguistically Informed tokens Morfessor (Creutz and Lagus, 2007; Virpioja et al., 2013; Smit et al., 2014) is a family of unsupervised morphological analyzers that work on the minimum description principle, which is not distant from compression. The link to tokenization is clear: Under some minimization objective, segment the input text into a list of morphemes with the constraint of at most V distinct morphemes appearing in the whole text, where V is a hyperparameter. In the case of tokenization, we would use the morphemes as tokens. For our purposes, we use vanilla Morfessor 2.0.6.
E.4 Lempel–Ziv–Welch Compression BPE, a popular tokenization scheme used in many NLP applications, was first invented (Gage, 1994) as a faster compression algorithm alternative to Lempel–Ziv–Welch (LZW). However, there is, in fact, a family of algorithms related to LZW. LZ77 (Ziv and Lempel, 1977) and LZW (Welch, 1984) are two of the most popular variants.
Example E.1. Consider a scenario where we want to encode a lowercased text without punctuation with the restriction of |∆| = 28. Upon adding the 26 lowercased letters of the English alphabet and space, we can add one additional token. Let us consider the tokens: the and cow. Certainly, the former token is much more frequent in natural English distribution and hence a tokenizer fully utilizing this token would result in shorter optimal expected code length. By adding the token the, we are lowering the probability mass of the most frequent English letters: e and t. This does not happen with cow and hence the first distribution is more uniform. For a depiction of this synthetic example, see Fig. 7. If we measure it using eff1, we obtain only 1% difference but if we measure it using eff9, we get 3% difference, showing that the first tokenization is much better.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Appendix B
3 B1. Did you cite the creators of artifacts you used? Appendix B
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Implicitly open-source
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Publicly available and tested data
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B, Figure 4, Figure 6
C 3 Did you run computational experiments? Section 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Figure 3, Figure 5
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 1
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the investigation of such qualities to future work