On the Effect of (Near) Duplicate Subwords in Language Modelling (2024)

Abstract: Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned arbitrary indices before being served to the LM. While typically lossless, however, this process may lead to less sample efficient LM training: as it removes character-level information, it could make it harder for LMs to generalise across similar subwords, such as now and Now. We refer to such subwords as near duplicates. In this paper, we study the impact of near duplicate subwords on LM training efficiency. First, we design an experiment that gives us an upper bound to how much we should expect a model to improve if we could perfectly generalise across near duplicates. We do this by duplicating each subword in our LM’s vocabulary, creating perfectly equivalent classes of subwords. Experimentally, we find that LMs need roughly 17% more data when trained in a fully duplicated setting. Second, we investigate the impact of naturally occurring near duplicates on LMs. Here, we see that merging them considerably hurts LM performance. Therefore, although subword duplication negatively impacts LM training efficiency, naturally occurring near duplicates may not be as similar as anticipated, limiting the potential for performance improvements. antonschafer/duplicate-subwords

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9580–9597 August 11-16, 2024 ©2024 Association for Computational Linguistics
antonschafer/duplicate-subwords

heading: 1 Introduction
text: Most modern language models (LMs) do not have direct access to the bits or characters which make up the text that they must model. Rather, they operate on higher-level units, so-called tokens, which are elements of a finite set of previously defined subwords. This set of subwords is typically obtained as the output of a tokenisation method and forms an LM’s vocabulary (Gage,
*Shared supervision.
1994; Sennrich et al., 2016; Kudo, 2018; Wu et al., 2016). Importantly, most tokenisation algorithms are lossless: the original character sequence is perfectly recoverable from its tokenised version.
A language model’s vocabulary, however, may contain several near duplicate subwords: minimal pairs like now and Now, with roughly the same semantic meaning, but which differ due to typos, whitespace marking, or capitalisation (Stanić et al., 2023). Such near duplicates can make up over 40% of the vocabulary of modern LMs (see Table 1). Intuitively, if the model had access to character-level information, it should trivially generalise what it learns from one of these forms to the other. Given only access to subword-level inputs, however, the model may not be able to do the same, or may require more data to do so.
Previous work tried to address this issue by modelling language directly at the character, byte, or even pixel level (Kim et al., 2016; Clark et al., 2022; Xue et al., 2022; Yu et al., 2023; Rust et al., 2023, inter alia).1 However, while the existing literature has proposed several solutions to improve
1These works are not solely motivated by near duplicates. Other commonly named advantages of character/byte-level models include: the possibility of optimising them end-to-end without relying on a two-stage approach, greater flexibility by not committing to a (potentially suboptimal) tokeniser, and more direct access to word forms which might be relevant in, e.g., word-play related tasks (Rozner et al., 2021).
9580
LMs generalisation across near duplicates, a proper quantification of the issue is still lacking.
In this work, we thus take a step back and assess the actual impact of near duplicate subwords on LMs’ performance. To this end, we first propose a controlled synthetic setting where we duplicate every subword in our LM’s vocabulary, allowing us to (1.) quantify an LM’s ability to generalise across perfectly equivalent duplicates and to (2.) carefully investigate how the generalisation happens. This yields an upper bound on the cost incurred due to limited generalisation across real near duplicates: a vocabulary with 40% duplicates (common for LLMs, see Table 1) may reduce data efficiency by up to 10%. We then (3.) investigate the tightness of this upper bound by merging naturally occurring near duplicates in an LM’s vocabulary. We find that deduplicating the vocabulary in this way, in general, hurts performance instead of improving it. This suggests that real near duplicates might be less similar than anticipated, which might impose challenges when trying to leverage their similarity to improve LMs’ performance.

heading: 2 Language Modelling
text: Let Σ be a vocabulary of subwords. A language model p̂ is formally defined as a probability distribution over the set of all finite sequences of subwords w = (w1, w2, ...) ∈ W def= Σ∗:2
p̂(w) =
|w|∏
t=1
p̂(wt | w<t) (1)
where p̂(wt | w<t) is the probability of token wt given the sequence of previous tokens w0, ...wt−1.
In order for p̂ to approximate the true distributions over natural strings p(w), we train this model to minimise its cross-entropy with p:
Ĥ(W) = ∑
w∈W p(w)
|w|∑
t=1
log 1
p̂(wt | w<t) (2)
where W represents a W-valued random variable. Since we do not know p, we approximate this objective using a finite training set Dtrn = {wn}Nn=1 ∼ p(w). This leads to the loss function:
L(p̂(w),Dtrn) = 1
N
∑
w∈Dtrn
|w|∑
t=1
log 1
p̂(wt | w<t) (3)
2We define Σ with a special end-of-sequence symbol (eos). Any string with mid-sequence eos is assigned probability zero.

heading: 3 Subword Duplication
text: Now, let’s assume there exist in our alphabet pairs or groups of subwords which are nearly identical— in both their orthography and semantics. Such near duplicates can arise from various sources, including but not limited to capitalization differences (e.g., now vs. Now), typographical errors (language vs. langauge), the presence or absence of whitespace (e.g., the vs. _the),3 and variations in spelling (e.g., modeling vs. modelling). The question we are concerned with is: how might such subword duplication affect the performance of language models?
To define this question formally, let S be a set of disjoint sets of near duplicate subwords:
S = {
{w 1 , w 2 , w 3 , w 4 }, {w 5 , w 6 }, · · · , {w i , w i+1 }
} (4)
We index these as S i to represent the i’th set of near duplicates. Further, let S = {c i | 0 < i ≤ |S|} be a set of canonical symbols which we will use to represent each of these duplicate sets. To find out how duplication affects LMs, we can create a map S : Σ → Σ which deduplicates subwords, mapping duplicates to the corresponding canonical symbols. This map is defined as:
S(w) = {
w if w /∈ flatten(S) c i if w ∈ S i
(5)
with Σ def= (Σ \ flatten(S)) ∪ S. We are now in a position to define a distribution over deduplicated subword sequences:
p(c) = ∑
w∈W p(w)1{c = S(w)} (6)
where we overload S to operate on sequences w by applying it elementwise on each subword wt ∈ w. Note that p(c) is now a distribution over deduplicated subword sequences c = (c1, c2, ...) ∈ C def= Σ∗. We can further define a hybrid conditional projected distribution as:
pS(ct | w<t) = (7)∑
w∈Σ p(w | w<t)1{ct = S(w)}
In words, the conditional probability pS(ct | w<t) of a deduplicated subword ct is defined as the sum of the conditional probabilities of all subwords which map to it through S(w).
3We denote spaces as ‘_’ for readability.

heading: 3.1 Comparing (De-)Duplicated LMs
text: Typically, LMs are evaluated based on their crossentropy (or perplexity) on a held-out test set. It would, however, be unfair to simply compare the cross-entropies of LMs trained on p(c) and p(w). The cross-entropy is lower bounded by the entropy—with a perfect model’s cross-entropy equalling the entropy. If p(c)’s and p(w)’s entropies are different, they would impose different optima achievable by LMs trained in each setting. We write these distributions’ entropies as:
H(W) = ∑
w
p(w)
|w|∑
t=1
log 1
p(wt | w<t) (8)
and H(C), analogously, where C denotes a Cvalued random variable. Now note that, given their definition in Eq. 6, deduplicated sequences C are deterministic given the original subwords W. The two entropies above are thus related via equation:
H(W) = H(C) + H(W | C) (9) Assuming W cannot be deterministically predicted from C, we have that H(W | C) > 0; this implies that predicting W is strictly harder than C.
To make these settings more easily comparable, we define p(w)’s projected entropy as:
HS(W) = ∑
w∈W p(w)
|w|∑
t=1
log 1
pS(S(wt) | w<t) (10)
This entropy measures the uncertainty in predicting a deduplicated subword ct given the duplicated context w<t. Interestingly, we can show that:
HS(W) = H(C)−MI(W<T;CT | C<T) (11) where MI(W<T;CT | C<T) is the mutual information between a subword context w<t and the next deduplicated token ct conditioned on the previous deduplicated tokens c<t (see App. A for a proof and the precise definition of this mutual information). In both §5.1 and §5.2, we discuss how this value relates to our research question.

heading: 4 Experimental Setup
text: We implement all of our experiments in the codebase of the Languini Kitchen (Stanić et al., 2023). Below, we provide an overview of the models and datasets used here. We refer the reader to Stanić et al.’s (2023) work for more details regarding implementation choices, training setup, and dataset collection.
Model. We use the GPT model from Languini, which is a GPT-2 style transformer decoder (Radford et al., 2019). Unless otherwise noted, we use the “small” configuration with 12 layers, hidden size 768, and 85M non-embedding parameters. We train models with sequence length 512, batch size 128, the Adam optimiser (Kingma and Ba, 2015), and a cosine learning rate schedule from 6e-4 to 6e-6 with 500 warmup steps.
Data. We train on the Languini training data, a filtered version of the books3 subset from the Pile (Gao et al., 2020). For evaluation, we use the heldout Languini test set, which contains 11M tokens. This data is pre-tokenised into a vocabulary of size 16k using a BPE tokeniser (Gage, 1994; Sennrich et al., 2016) trained using SentencePiece (Kudo and Richardson, 2018). Unless otherwise noted, we train our models for 18,265 steps—i.e., the first 1.2B tokens in our dataset—which corresponds to training the small GPT model for 6h on an RTX 3090 GPU; this is Languini’s GPT small 6h setting.
Evaluation. We generally report our model’s perplexity on the test set as our evaluation metric. To ensure sufficient context for all predictions, we use a sliding window with steps of 128: we fill in a 512 tokens context, ignore the model’s outputs on the initial 384, and evaluate it only using the last 128 tokens. For models p̂(c) trained on the deduplicated setting, we simply report their perplexities, defined as the exponentiated cross-entropy evaluated on a held-out test set Deval: PPL(C) = exp (L(p̂(c);Deval)). When evaluating models p̂(w), trained in the duplicated setting, we report their projected perplexity, defined as:
PPLS(W) = exp (LS(p̂(w);Deval)) (12)
= exp  1 N ∑
w∈Deval
|w|∑
t=1
log 1
p̂S(S(wt) | w<t)
 
where p̂S is defined analogously to pS (see Eq. 7). Intuitively, we add up the probabilities of subwords w that are equivalent under S (i.e., which map to the same canonical symbol c), to avoid giving p̂(c) an unfair advantage over p̂(w).

heading: 5 Experiments and Results
text: We evaluate our LM’s ability to generalise over (near) duplicates in two settings: perfect and natural duplication. In the perfect duplication setting, we compare LMs trained using either the default
or a synthetically duplicated vocabulary; this gives us an upper bound for the impact of real near duplicates, as synthetically duplicated subwords are perfectly comparable in terms of their semantics. In the natural duplication setting, we deduplicate the default vocabulary by merging real near duplicates. By comparing performance on the default and the deduplicated vocabulary, we verify whether the effect of natural near duplicates in LMs is comparable to the effect of perfect duplicates.

heading: 5.1 Perfect Duplication
text: In this first set of experiments, we assume an idealised situation where all subwords in a near duplicate set w ∈ S i are perfectly interchangeable with each other. Choosing among these subwords, then, is neither impacted by prior subwords nor impacts future subword choices. In this case, we can relate distributions p(w) and p(c) as:
p(w) =
|w|∏
t=1 p(ct | c<t) p(wt | ct)︸ ︷︷ ︸ duplicate choice
(13)
Further, we can show that in this idealised setting MI(W<T;CT | C<T) = 0; this is because the decomposition above implies conditional independence between wt and any ct′ given ct. We thus have:
HS(W) = H(C) (14)
which creates a perfectly controlled setting to evaluate language models. If we could train a perfect language model on either distribution p(w) or p(c), we would achieve the same performance in both settings. Any difference in language modelling performance between these settings must thus derive from a language model’s lack of ability to generalise from observing near duplicate subwords.

heading: 5.1.1 Empirical Implementation
text: To achieve the perfect duplication described above, we simulate Eq. 13 by duplicating every entry in our subword vocabulary. First, we assume our BPE-generated vocabulary is composed of canonical symbols Σ = {c 1 , c 2 , ...}. This set is composed of 16k subwords. We then duplicate each to get a vocabulary of size 32k:4 Σ = {w 1 , w′1 , w 2 , w′2 , ...}. This gives us the deduplication mapping S(w i ) = S(w′i ) = c i . Given
4Model p̂(w) thus has more embedding parameters than p̂(c); these extra parameters, however, should not yield an unfair advantage (see App. D).
a sequence c from our training set Dtrn, we then create a duplicated sequence w by independently sampling the form of each token c i ∈ c to be either w i or w′i . Unless specified differently, we set all duplicate choice probabilities to be uniform (i.e., p(w i | c i ) = p(w′i | c i ) = 0.5 for all i).

heading: 5.1.2 Raw Performance
text: In this section, we describe our main results using the perfect duplication setting. First, we note that, when training with p(w′i | c i ) = 0.5, each subword w i or w′i is only seen half as often by p̂(w) as its original version c i is seen by p̂(c). However, we achieve significantly better performance with p̂(w) than with p̂(c) trained on only 50% of the dataset (see Table 2). The model thus seems to generalise across duplicates, with data containing w leading to improved performance on w′ and vice versa. Still, duplication significantly hurts performance. Using the duplicated data, the model p̂(w) is only about 85% as data efficient as p̂(c), which is trained on the original data. This suggests our LMs cannot generalise perfectly. Interestingly, this trend stays relatively consistent across different amounts of training data (up to 3x, see Fig. 1) and also applies to a 3x larger GPT-medium model (see Table 2). Further, if we vary the number of subwords we duplicate, interpolating from 0% of the vocabulary (p̂(c) setting) to 100% of the vocabulary (p̂(w) setting), we obtain a roughly linear increase in PPLS (see Fig. 2).
Takeaway 1. The model is capable of generalising across duplicates, yet their presence negatively impacts performance.

heading: 5.1.3 Duplicates’ Alignment
text: One strategy the model could use to generalise across duplicated pairs w i and w′i is to fully align their representations. If these word pairs have a
corresponds to number of training tokens for p(w)number of training tokens for p(c) —is approximately equal to 1 0.85 .
cosine similarity of 1.0, then any change to other model components would affect them similarly. When we analyse our model p̂(w)’s embeddings, we indeed observe a high average cosine similarity of around 0.8 among duplicate pairs. This number is even higher for frequent subwords (see Fig. 3); it appears that a subword’s frequency during training is an underlying driver for alignment. This observation is intuitive: the representations of w i and w′i are both randomly initialised and converge to each other after a certain number of gradient updates.
What causes this alignment of representations
and what is it impacted by? Loosely speaking, the contexts in which the duplicates appear follow the same distribution; this might lead to similar gradient signals throughout training. If this is the case, then this high cosine similarity should not be an exclusive property of transformers, but apply to simpler architectures as well. Interestingly, when training a word2vec model (Mikolov et al., 2013) on the same data, we observe its embeddings exhibit even stronger alignment (details in App. E).
Takeaway 2. Representations of frequent duplicates have high cosine similarity.

heading: 5.1.4 Finetuning Generalisation
text: Presumably, the alignment of duplicated word pairs may cause the model to use the same “circuits” when processing those words, allowing it to gener-
alise what it learns across them (Cammarata et al., 2020; Elhage et al., 2021). To verify this hypothesis, we finetune our p̂(w) model on GLUE (Wang et al., 2019), employing only one subword from each duplicate pair as input (specifically, w i and never w′i ). Intriguingly, when this finetuned model is subsequently evaluated on the unseen subwords (i.e., w′i ) it generalises perfectly, achieving the same accuracy (see Table 3).
Takeaway 3. GLUE performance of p̂(w) generalises across duplicate pairs despite being finetuned with w i and evaluated with w′i .

heading: 5.1.5 Comparing Duplicating or Not a Pair
text: We observed that the alignment of duplicate subword representations seems to drive generalisation. If infrequent subwords have less aligned representations, does this mean that they generalise less? To investigate the effects of duplication in a more controlled manner, we now duplicate only half of our vocabulary and train a model in this setting. This gives us a treatment group of duplicated subwords and a control group of non-duplicated subwords, allowing us to isolate the causal effect of duplication. We will now analyse this effect on the output side (predicting duplicated tokens compared to non-duplicated tokens) and on the input side (predictions based on a context with many duplicated tokens vs few duplicated tokens).
On the output side, we measure the effect of duplication by first taking the difference in
5To reduce noise and ensure readability, we only consider subwords that occur at least 10 times in the test set and bins that contain at least 3 subwords.
surprisal (negative log probability) assigned to each original subword token ct by either model p̂(w) (through p̂S(ct | w<t)) or p̂(c) (through p̂(ct | c<t)). We then average these delta surprisals within either the set of actually duplicated (treatment) subwords, or the ones not duplicated (control). We observe that subword duplication leads to an increase in surprisal which is around 50% higher when predicting duplicated tokens compared to non-duplicated ones (see Table 4). This might seem unexpected, considering that, e.g., a simple n-gram model’s predictions would not be affected by duplicated outputs.6 Yet, in our LM, the output embeddings of duplicated subwords receive only half as many gradient updates, which could explain the lower performance. In line with this, we observe that infrequent subwords are affected the most by this performance loss (see Fig. 4), likely because their output representations are not well aligned after receiving few updates.
To evaluate the impact of duplication on the LM’s input side, we assess how surprisal on subwords changes depending on the number of duplicated (vs non-duplicated) subwords in its context. Here, we observe no clear trend between the number of duplicates in our model’s full context window and its performance loss on observed subwords (see Fig. 5). However, when limited to a more local context of size 16, the number of duplicate tokens seems to have a clear negative impact on prediction performance.
Takeaway 4. Infrequent subwords are predicted worse when duplicated, and duplicated tokens in the LM’s local context tend to hurt its predictions.
6An n-gram’s probabilities are defined by count statistics, and: count(c∥context)
count(context) = count(w∥context) count(context) + count(w′∥context) count(context)
and p̂(c), depending on fraction duplicated subwordsnon-duplicated subwords in context. Fractions are categorised into bins, with average surprisal differences computed for each bin. “Support” shows the number of samples per bin.

heading: 5.2 Natural Duplication
text: After exploring the impact of perfect duplicates in LMs, we now turn our attention to the influence of naturally occurring near duplicates on LM performance. Notably, despite their similarity, near duplicates are seldom perfectly interchangeable. For example, _individual differs from _individuals, the _he in ‘and he writes’ does not convey the same meaning as he in ‘breathe’ (we analyse this in App. C), and a Now at the start of a sentence can subtly vary from a now used mid-sentence. When merging near duplicates, we lose information about such small differences. If this lost information is relevant for predicting a token ct = S(wt), the task should become harder through deduplication. We have MI(W<T;CT | C<T) > 0 and hence
H(C) > HS(W) (15)
which means that even a perfect model would perform worse on the deduplicated data distribution. This subsection examines whether merging naturally occurring near duplicates yields comparable advantages to merging perfect duplicates.

heading: 5.2.1 Empirical Implementation
text: We consider four types of near duplicates, defined by their corresponding mapping S: Sspace ignores leading or trailing whitespace, map-
ping, e.g., _the to the. Around 10% of subwords are merged.
Slower ignores capitalisation, mapping, e.g., Now to now. Around 12% of subwords are merged.
Splural ignores plural markings,7 mapping, e.g., _individuals to _individual. Around 8% of subwords are merged.
7For simplicity, we implement it as an easy rule-based
Sall combines all of the previous mappings, mapping, e.g., _Books to book. Around 29% of subwords are merged.
We train models p̂(c) for each of these deduplication mappings S and compare them to the performance of the regular model p̂(w) when evaluated under the same projection.
The first three mappings all reduce the vocabulary size by around 10%. If the near duplicates were perfectly equivalent, we could expect p̂(c) to perform marginally better than p̂(w), which is what we observe in the synthetic setting with a low duplication rate (see Fig. 2). For the combined Sall, with 29% near duplicates, we would expect a performance boost corresponding to roughly 5% better data efficiency; or, alternatively, a PPLS decrease of roughly 0.2 from p̂(w) to p̂(c) (see the results for 40% duplication rate8 in Fig. 2).
Modern large models show even higher duplication rates than our small vocabulary. Their vocabularies contain around 40% near duplicates under Sall (see Table 1). This translates to duplicating roughly 70% of a canonical vocabulary; in the synthetic case, this corresponds to a potential data efficiency increase of around 10% due to deduplication, assuming the trends in Fig. 2 apply. In the following, we examine whether this is plausible by verifying to what extent these trends transfer from perfect duplicates to near duplicates for our models.
Note that, different to the previous subsection where we duplicated our standard vocabulary, here
mapping: If a subword has a trailing s and at least four characters, not counting whitespace, we ignore the s. While this approach yields false positives, manual inspections suggests the results are generally acceptable.
8A ratio of 29% of duplicates in vocabulary Σ corresponds to duplicating around 40% of the initial Σ ( 0.4
1+0.4 ≈ 0.29).
we deduplicate it; this means the p̂(c) in the previous subsection and p̂(w) in this subsection both refer to a “baseline" over our standard vocabulary.

heading: 5.2.2 Raw Performance & Duplicate Alignment
text: We observe that deduplication hurts performance (see Table 5). Across all considered S, training p̂(c) on deduplicated data yields worse results than the setting p̂(w) in which near duplicates remain untouched. The performance degradation is equivalent to training on 5-10% less data. This is in stark contrast to the trends we observed on perfectly equivalent synthetic duplicates, where deduplication boosts performance. Near duplicates thus seem to be less equivalent than one might expect, noticeably differing in their semantics.
The near duplicates’ learned embeddings, which tend to contain information about the subwords’ semantics, reflect this discrepancy. Near duplicates’ embeddings are not nearly as similar as the embeddings of synthetic perfect duplicates, which had a cosine similarity of 0.8: all three types of natural duplicates exhibit average cosine similarities of around 0.4. This indicates that the model perceives semantic differences between the near duplicates. As described earlier, if we merge them, we will lose information. The associated decline in performance indicates that the information is significant.
Takeaway 5. Near duplicates are not equivalent and merging them hurts performance.

heading: 5.2.3 Comparing Deduplicating or Not a Pair
text: Are predictions worse when more deduplicated tokens are in the context, since more information is “lost”? We investigate this again through a controlled experiment. We run experiments where we deduplicate only half of the near duplicates, obtaining a deduplicated treatment group and an un-
changed control group. This setup is similar to the one in the previous subsection, but treatment and control group do not cover the entire vocabulary; both consist only of subwords that have a near duplicate. To isolate the causal effect of duplication, we investigate differences between these groups.
We first compare the effects of the number of deduplicated subwords and non-deduplicated subwords in the LMs context, i.e., measuring the effect of deduplication on the input side. An increased number of deduplicated subwords seems to generally lead to a steeper increase in surprisal than an increased number of non-deduplicated subwords, at least in the local context (see Fig. 8 in App. F). This suggests that deduplicated subwords in the context do hurt performance, presumably due to the lost information.
Interestingly, deduplicated subwords are also predicted slightly worse on the output side (see Table 6). In this setting, the model is forced to learn only one output embedding for each pair of merged near duplicates. If these merged subwords have different meanings, finding a single embedding to represent both of them might be challenging, as this embedding would need to produce large dot products with the hidden states of contexts from both subwords.
Takeaway 6. The presence of merged near duplicates in the local context tends to reduce an LM’s prediction accuracy.

heading: 5.2.4 Re-adding Information
text: If the missing information in the input causes worse predictions, can we improve performance by reinjecting it? If we provide the model with an extra input that allows it to distinguish between merged near duplicates while still sharing their embeddings, could we obtain the benefits of deduplication without the downside of losing information?
To investigate this, we introduce an extra learnable embedding enon-canonical. This is a single shared vector which is added to the model’s input at every token that corresponds to a non-canonical subword. (A non-canonical subword w is any subword where S(w) ̸= w, e.g., Now when operating under Slower, since Slower(Now) = now ̸= Now.) As Sall combines the three different types of deduplication, we use three different learnable embeddings for the Sall setting, one for each type.
When comparing p̂(c) with and without enon-canonical, such an embedding’s availability significantly improves performance (p ≤ 0.05 in one-sided t-test) in all four deduplication settings (see Table 5). Further, p̂(c) with enon-canonical roughly matches p̂(w) performance for Sspace, Slower, and Splural (differences not statistically signficant with p = 0.22, 0.48, 0.72, respectively, in two-sided t-test). However, we still do not see the same performance improvements due to deduplication as we observed in the synthetic setting; this is especially clear for Sall which has a higher deduplication rate, but clearly fails to improve performance (p̂(c) with enon-canonical is significantly worse than p̂(w) with p ≤ 0.05 in two-sided t-test). Presumably, even if they belong to the same type (Sspace, Slower, or Splural), near duplicate pairs’ semantic differences are diverse and can thus not be fully captured by a single shared embedding vector.
Takeaway 7. Performance losses can be mitigated by accounting for semantic differences of near duplicates via a shared learned embedding. Still, this approach does not achieve the same benefits as observed when merging perfectly equivalent duplicates.

heading: 6 Related Work
text: Our experiments are partly inspired by a number of works on cross-lingual LM generalisation. These works also use duplicated vocabularies, terming the duplicates “fake-english”. Unlike our work, however, they sample entire sequences of either English or “fake-english” tokens (we perform token-wise
i.i.d. sampling; K et al., 2020; Dufter and Schütze, 2020; Schäfer et al., 2024). Relatedly, Huang et al.’s (2023) lex-invariant LMs can be interpreted as an extreme case of token duplication; essentially creating infinite “fake languages”, they show that LMs can learn, to a certain extent, even in the absence of a fixed set of embeddings. More similar to our duplication method is the work of Kharitonov et al. (2021), who use duplicates sampled at the token level to isolate the effect of vocabulary size when investigating the role of BPE tokenisation in a model’s ability to memorise training data.
Another set of related work studies subword regularisation techniques, such as, e.g., BPE dropout (Kudo, 2018; Provilkov et al., 2020). Certain character sequences can be represented by multiple equivalent sequences of subwords. These techniques then non-deterministically choose between these equivalent choices at training time, which can lead to improved LMs. Our perfect duplication experiments, where we train LMs with either of two duplicated tokens, can be seen as similar to these methods. Unlike them, however, our approach duplicates a model’s vocabulary to introduce this ambiguity, which may explain the fact that our results diverge. Relatedly, our projected entropy measure (which sums over subword duplicates) is analogous to the marginalisation proposed by Cao and Rimell (2021), which sums over spurious tokenisations.

heading: 7 Conclusion
text: In this paper, we investigate to what extent (near) duplicate subwords impact language modelling performance, conducting controlled experiments on synthetic perfect duplicates and natural near duplicates. We find that LMs can generalise across duplicated subwords, although this incurs extra cost. When operating on a fully duplicated vocabulary, the LM is about 17% less data efficient. This number depends roughly linearly on the fraction of the vocabulary that is duplicated. Assuming that roughly 40% of subwords are near duplicates in common LM vocabularies, our findings imply that LMs with improved generalisation across duplicates, e.g., by modelling language at the characterlevel, could achieve data efficiency gains of up to 10%. This bound is reached for perfectly equivalent duplicates. However, we find that natural near duplicates are not perfectly equivalent: in practice, the potential for such performance improvements is likely limited.
Limitations
We conduct most of our experiments on models with about 100M parameters, training on roughly 1.2B tokens in English. Although the trends we identify are consistent in up to 3x larger models and datasets, it is uncertain whether they extend to the scale of modern large language models.9 Furthermore, we have not validated that our results transfer to languages beyond English.
When we study the causal effects of (de)duplication on the input and output side, we do not fully isolate the two effects from each other. After a manual inspection, it appears that this does not confound the results; e.g., duplicated subwords are not more frequent in the input/context of duplicated subwords than non-duplicated subwords are. However, to fully exclude the possibility that such patterns affect results, one should run experiments where only the inputs or only the outputs are (de)duplicated.
Finally, while studying natural near duplicates, we only investigate deduplication mappings that can be described by simple heuristics. Alternatively, one could use more involved methods to also deduplicate, e.g., typos or wordform variations beyond plural forms.

heading: Acknowledgements
text: The authors would like to thank Shauli Ravfogel, Andreas Opedal, Marius Mosbach, and the anonymous reviewers for feedback on earlier versions of this manuscript.

heading: A Lemma 1 and Proof
text: Definition 1 (Time-dependent Conditional Entropy and Mutual Information). We define a time-dependent conditional entropy as:
H(W<T | C≤T) def= ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log 1 pS(w<t | c≤t) (16)
with H(W<T | C<T) defined analogously. Note that, while pS(w) stands for the probability of the single event {w}, we write pS(w<t) to represent the probability of the event composed by the union of all sequences with this prefix, i.e., ∪w′∈W{w<t ◦w′}, where ◦ stands for concatenation.
Accordingly, we define a time-dependent mutual information as:
MI(W<T;CT | C<T) = H(W<T | C<T)−H(W<T | C≤T). (17)
Lemma 1. Let S : Σ → Σ be a deterministic function which maps duplicated subwords w ∈ Σ to their deduplicated versions c ∈ Σ. We then have that:
HS(W) = H(C)−MI(W<T;CT | C<T) (18)
Proof. We start with the definition of HS(W) and derive Lemma 1:
HS(W) = ∑
w∈W p(w)
|w|∑
t=1
log 1
pS(S(wt) | w<t) (19a)
= ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log 1 pS(ct | w<t) pS(w, c) = p(w) if S(w) = c else 0 (19b)
= ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log 1 pS(ct | c<t,w<t) pS(c<t | w<t) is deterministic (19c)
= ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log pS(w<t | c<t) pS(ct | c<t) pS(w<t | c≤t) Bayes rule (19d)
= ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 ( log
1
p(ct | c<t) + log pS(w<t | c<t) pS(w<t | c≤t)
) Split log (19e)
= ∑
c∈C p(c)
|c|∑
t=1
log 1
p(ct | c<t) +
∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log pS(w<t | c<t) pS(w<t | c≤t) Split sum (19f)
= H(C) + ∑
w∈W
∑ c∈C pS(w, c) |w|∑ t=1 log pS(w<t | c<t) pS(w<t | c≤t)
Definition of H(C) (19g)
= H(C) + H(W<T | C≤T)−H(W<T | C<T) Definitions of conditional entropies (19h) = H(C)−MI(W<T;CT | C<T) Definition of mutual information (19i)
Note that pS(w, c) > 0 =⇒ S(w) = c =⇒ |w| = |c|, which is required for arriving at Eq. (19f).

heading: B Near Duplicates in Modern LLMs
text: Table 7 shows the near duplicate rates in vocabularies of modern large language models, namely GPT-3.5 and GPT-4 models (Brown et al., 2020; OpenAI et al., 2023), Claude 2.1 (Anthropic, 2023), Llama (Touvron et al., 2023a,b), Mistral 7B and Mixtral 8x7B (Jiang et al., 2023, 2024), and Gemma (Gemma Team et al., 2024).
C Near Duplicates under Sspace Near duplicates under Slower and Splural are generally very close in meaning. Slower pairs often consist of the lowercase and the capitalized version of a subword, where the latter might appear at the beginning of sentences. And (sub)word pairs under Splural tend to refer to the same lemma. For Sspace, however, the contexts in which subword variants with and without a leading space appear might differ much more: in particular, we observe this with shorter subwords that can appear both as their own word (e.g., _he in ‘and he writes‘) and as a substring of a longer unrelated word (e.g., he in ‘breathe‘).
To quantify how frequently Sspace near duplicate pairs show such differences, we manually inspect 100 randomly sampled pairs and note whether the subwords convey comparable meaning in their first occurrences in the training data. We find that 53% of the pairs carry highly similar meaning in both occurrences. Examples include use in compound words (e.g., _writing in isolation vs writing in ‘handwriting’, or _held in isolation vs held in ‘long-held’) or after special characters like newlines, parentheses, or quotation marks (e.g. _wide in isolation vs wide in ‘(wide [...])’). Of the remaining 47%, around half appear in contexts that are clearly different (e.g., he and _he as in example above) while the rest are very short subwords where the conveyed meaning is hard to determine (e.g. _sche in ‘schemed’ vs sche in ‘Nietzsche’).

heading: D Effect of Added Parameters for Duplicates
text: A model p̂(w) over duplicated subwords has more embedding parameters than its deduplicated counterpart p̂(c). In the perfect duplication setting, however, these extra parameters should not yield an unfair advantage. In particular, assume an equiprobable duplicates setting, where p(w i | c i ) = p(w′i | c i ) = 0.5. We can show that, in this setting, if there exists an optimal model p̂(w) (which achieves Ĥ(W) = H(W)), then there also exists an equivalent model p̂(c). Further, the input and output embeddings for duplicates w i and w′i are perfectly interchangeable; the added embeddings thus provide no benefits, as the embedding of any w′i can be replaced with the embedding of the respective w i without affecting performance.
To see this, note that for the optimal p̂(w), we have p̂(w) = p(w) almost everywhere. Consequently, by Bayes, p̂(w | w<t) = p(w | w<t). We thus obtain
p̂(w i | w<t) = p̂(w′i | w<t) (20)
which means that the model makes identical predictions for duplicate subwords. Predictions are thus not affected if we assign all w′i the output embedding of the respective w i (or vice-versa).
An analogous argument holds for input embeddings. Let w<t[w′i → w i ] denote the token sequence obtained when replacing every occurrence of w′i in w<t with w i . For the true distribution, we know that p(w i | w<t) = p(w i | w<t[w′i → w i ]) due to the perfect equivalence, and thus also:
p̂(w i | w<t) = p̂(w i | w<t[w′i → w i ]) (21)
This means that p̂(w) makes identical predictions, whether we replace w′i with w i in the context or not. If we assign w′i the input embedding of w i , model performance is not affected. By applying this argument iteratively for all i, we can show that all duplicates’ input embeddings can be made identical. If the input embeddings of all duplicates can be made identical, then there exists a model p̂(c i | c<t) which, when given as input S(w), outputs the same probability as p̂(w i | w<t) + p̂(w′i | w<t). In particular, let vw be the output embeddings associated with w. We can show that:
p̂(w i | w<t) + p̂(w′i | w<t) = 2 p̂(w i | w<t) (22a)
= 2 exp(vw i h)∑
w∈Σ exp(vw h)
(22b)
= 2 exp(vw i h)
|Σ|∑ j=1 exp(vw j h) + |Σ|∑ j=1 exp(vw′ j h)
(22c)
= 2 exp(vw i h)
2 |Σ|∑ j=1 exp(vw j h)
(22d)
= exp(vc i h)
|Σ|∑ j=1 exp(vc j h)
(22e)
= p̂(c i | c<t) (22f)
So we can construct a p̂(c) model from p̂(w) by simply making the embeddings of c i the same as w i . The identical input embeddings preserve the value of the hidden state h for any context, which, along with the identical output embeddings, ensures identical model outputs.

heading: E Embedding Similarity Results
text: While in Fig. 6, it appears that a balance between duplicated and non-duplicated subwords (i.e. p(w′i | c i ) = 1− p(w i | c i ) close to 0.5) leads to higher similarity of their embeddings, Fig. 7 shows that the underlying factor driving embedding alignment seems to be subword frequency, or more precisely, how often the rarer version of the subword (here always w′ as p(w′i | c i ) < 0.5) occurs.

heading: F Effect of Deduplicated Subwords in the Context
text: An increased number of deduplicated subwords in the context seems to generally lead to a steeper increase in surprisal than an increased number of non-deduplicated subwords, at least in the local context of 16 tokens (see Figure Fig. 8). For the full context, the trend is less clear.

heading: Future Work
text: Choosing among these subwords, then, is neither impacted by prior subwords nor impacts future subword choices. In this case, we can relate distributions p(w) and p(c) as:
p(w) =
|w|∏
t=1 p(ct | c<t) p(wt | ct)︸ ︷︷ ︸ duplicate choice
(13)
Further, we can show that in this idealised setting MI(W<T;CT | C<T) = 0; this is because the decomposition above implies conditional independence between wt and any ct′ given ct. We thus have:
HS(W) = H(C) (14)
which creates a perfectly controlled setting to evaluate language models. If we could train a perfect language model on either distribution p(w) or p(c), we would achieve the same performance in both settings. Any difference in language modelling performance between these settings must thus derive from a language model’s lack of ability to generalise from observing near duplicate subwords