Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
Authors: Woojin Chung
Jeonghoon Kim
Venue: NeurIPS 2025 poster ()
Keywords: Token Frequency Imbalance
Vocabulary Size
Pre-training Language Model
Information Theory
Tokenization
Primary area: deep_learning
TL;DR: Larger vocabulary lowers language modeling difficulty by facilitating models to learn non-i.i.d patterns in text more easily

Abstract: Large language models are trained with tokenizers that map text to a fixed vocabulary, yet the resulting token distribution is highly imbalanced: a few words dominate the stream while most occur rarely.  Recent practice favours ever-larger vocabularies, but it is unclear whether the benefit comes from better word segmentation or from amplifying this frequency skew. To this end, we perform a controlled study that scales the vocabulary of a constant-size Transformer from 24K to 196K symbols while holding data, compute and optimisation unchanged.  Above 24K every common word is already a single token, so further growth only increases imbalance.  Word-level loss decomposition shows that larger vocabularies reduce cross-entropy almost exclusively by lowering uncertainty on the ~$2,500$ most frequent words, even though loss on the rare tail rises. Same frequent words cover roughly $80\%$ of tokens in downstream benchmarks, this training advantage transfers intact.  We further show that enlarging model parameters with a fixed tokenizer yields the same frequent-word benefit, revealing a shared mechanism behind vocabulary and model scaling. Our results recast “bigger vocabularies help” as “sharper frequency imbalance helps,” offering a simple, principled knob for tokenizer–model co-design and clarifying the loss dynamics that govern language-model scaling in pre-training.

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
Woojin Chung" KAIST gartland2228gmail.cc com
Jeonghoon Kim NAVER Cloud & KAIST
j www.:oomaaammmmmmmmm....mm.m.... com
kin/vocab-imbalane
huggsingffcccooooollllloooaaattn wwwp:/riiippo--oooooooe http:///hwuuunnoooooooooooaaaaaa frrouuulryy-rqquuee--mmm immbllanc
Abstract
Large language models are trained with tokenizers, and the resulting token distribu- tion highly imbalanced: few words dominate the stream while most occur rarely. Recent practice favors ever-larger vocabularies, but unclear where the benefit comes from. To this end, perform controlled study that scales the vocabulary the language model from 24K to 196K while holding data, computation, and optimization unchanged. We begin by quantifying the complexity of tokenized text - formalized via Kolmogorov complexity and show that larger vocabularies reduce this complexity. Above 24K, every common word is already tokenized single token, so enlarging vocabulary only deepens the relative token-frequency imbalance. Word-level loss decomposition shows that larger vocabularies reduce cross-entropy loss almost exclusively by lowering uncertainty on the 2, 500 most frequent words, even though loss the rare tail rises. Same frequent words cover roughly 75% of tokens in downstream benchmarks, this training advantage transfers intact. We further show that enlarging model parameters with fixed vocabulary yields the same frequent-wond benefit. Our results recast "bigger vocabularies help" as "lowering complexity of tokenized helps," offering a simple, principled knob for tokenizer model co-design and clarifying the loss dynamics that govern language model scaling pre-training.
1 Introduction
language model incorporates a tokenizer that converts a stream characters into a series of token IDs, each representing a specific substring [22, 47, 12]. Tokenization has re-emerged as an powerful tuning knob for language models, with mounting evidence that simply scaling up vocabulary consistently reduces perplexity and improves downstream accuracy across diverse domains and model scales [58, 65]. Although this trend is consistently observed in practice, the underlying mechanism responsible for it has yet to be thoroughly investigated.
As vocabulary size grows, adding merge candidates segments frequent words in the training data into single token and pushes infrequent ones deeper into the long tail, sharpening the relative token-frequency distribution [52, 35, 20J. In unigram models, this simply lowers the unigram Shannon entropy of training data toward its optimal loss (i.e., entropy rate) [47]. This intuition does not transfer to language models as they condition their next-token prediction on the preceding context,
'Equal contribution
Korea Advanced Institute of Science and Technology (KAIST)
I Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

which contains a mixture of common and rare tokens [2, 8, 42, 7]. Moreover, rare tokens already carry much lower conditional probabilities than marginal ones; mistakes on them incur disproportionately large penalties [45, 391. Yet rare tokens account for small share of the entire dataset, making it unclear how a larger vocabulary reallocates capacity between frequent and rare tokens.
In this study, we explore why enlarging vocabulary size improves the performance of language models by expanding vocabulary from 24K to 196K. Viewing BPE [54] with its pre-tokenization rules as a lossless compressor [13, 33], we assess how much the tokenized text is compressed against raw text by an upper-bound on Kolmogorov complexity and first illustrate that expanding the vocabulary reduces this complexity ($3.3). Above 24K, frequent words are already encoded as single tokens, so the primary shift is a heightened imbalance in relative token- (ekuunnr distribution ($3.4), regardless of dataset quality ($3.5). Furthermore, a word-level loss decomposition shows that larger vocabulary reduces the loss of frequent words, thereby lowering the model's l's overall cross-entropy ($3.6), regardless of dataset and model size.
Analytic experiments trace how enlarging the vocabulary changes both training dynamics and generalization behavior through token-frequecyy imbalance. Our observation suggests that high frequency words in the pre-training corpus largely coincide with those in downstream benchmarks, both in identity and coverage, explaining the close link between training loss and transfer accuracy ($4.1), 1). Scaling the model itself produces the same benefit: it predicts frequent words more accurately, thereby enhancing overall language-movs model performance ($4.2).
Contribution. We identify that larger vocabularies reduce the complexity of tokenized text, thereby facilitating the model to learn non-i.i.d. patterns in the training data more easily. Our experiments further reveal that beyond certain size, vocabulary expansion no longer improves segmentation but instead steepens the skewness of the token-frequency distribution. This sharper imbalance alone lowers global cross-entropy by reducing the top 2,500 frequent words loss despite slight degrada- tion on the rare tail. Through cross cross-dataset overlap analyses, we demonstrate that exploiting, rather than mitigating, token frequency imbalance causally reduces cross-entropy and boosts downstream accuracy. Finally, we show that parameter scaling replicates the same benefit as vocabulary scaling, both primarily reduce uncertainty on the same set of frequent tokens.
2 Motivation
Tokenization - the interface between raw text and the discrete symbols a model actually sees has resurfaced as a powerful, low-cost lever for improving language model quality [22, 47, 12, 20]. A growing body of evidence finds that simply increasing the size of the tokenizer vocabulary yields systematic gains in perplexity and downstream accuracy across domain and model scales [58, 65]. Despite this empirical regularity, the mechanism behind the gain remains underexplored.
The clue may lie in how tokenizers behave as their vocabularies grow. Rajaraman et al. [47] noted that adding merge candidates segments the most frequent words in the corpus into single tokens, making an i.i.d. model over tokens closer approximation to inherently non-i.i.d. data. Moreover, they empirically confirm that larger vocabularies lower unigram cross-entropy. Simultancously, this pushes rare vocabularies further into the long tail, yielding a markedly more skewed -eee-ffry distribution (after the usual whitespace pre-tokenization) [52, 35].
Unfortunately, this explanation does not carry verbatim neural language models. In language models, every prediction is conditioned on a variable-length context. Errors on rare vocabularies disproportionately costly because their conditional probabilities are already orders of magnitude below their marginal frequencies [45, 39], yet they occupy only a tiny fraction of the overall loss. Quantifying how vocabulary growth redistributes modelling capacity between these two regimes, therefore, remains a non-trivial challenge (see the Appendix A for a detailed explanation).
In this work, tackle the problem head-on by pairing large-scale controlled experiments with analytical diagnostics. We ask:
Why does enlarging the tokenizer vocabulary improve Transformer performance, and which component of the loss benefits most?
Clarifying this mechanism is essential for principled tokenizer design and for understanding the true drivers of scaling laws in language modelling.
2

---

Experiments
Guiding Questions. Before diving into settings and metrics, we spell out the concrete questions that steer our empirical study:
1. Tokenized Text Complexity
Does enlarging the vocabulary reduce the Kolmogorov complexity of tokenized text ($3.3)?
2. Skew vs. Segmentation
Does enlarging the vocabulary mainly sharpen the relative token- feeenffreuency distribution, or does it still increase single-token coverage of frequent words ($3.4)?
3. Loss Decomposition
When the complexity decreases and frequency skew increases, how is cross-entropy re- allocated between frequent and rare tokens, and which drives the global loss ($3.6)?
4. Corpus Robustness
Are the above effects stable across different data quality-i.c. do high- high-curation (FineWeb- Edu) and noisier (OpenWeb'Ten) WebText) corpora exhibit the same trends ($3.5)?
The remainder of this section answers these questions in turn.
3.1 Experimental Settings
In these experiments, we train a byte-pair encoding (BPE) tokenizer [54] and estimate token frequen- cies using sample 10 billion GPT-22 tokens from FineWeb-Edu [43] and the entire OpenWebTex WexText [15]. For model pre-training, we use approximately 40 billion characters, about 7.5 billion tokens for Feb-Wee-Edu and billion for Open WebText with a 49K vocabulary. To compute the metrics below, we also drew an additional 5 billion characters that did not overlap with the training corpus. We report word-level average loss to ensure fair comparison across vocabulary sizes. Whenever a smaller-vocabulary tokenizer splits a word into multiple tokens (i.e., subwords),
sum their individ- ual losses. Our model comprises 85 million non cobeedddd parameters with pre-layer normalization (pre-LN) [62]. Training uses AdamW [36] (81 0.9, 82 0.95, 10 10-$) with a learning rate of 6 x 10-4 that follows a cosyne-decay schedule after a 350 million-token warmup, weight decay of 0.1, and gradient clipping 1.0. Every experiment repeated with five seeds (See Appendix J).
3.2 Metrics
Kolmogorov Complexity of Tokenized Text Kolmogorov complexity (X) measures the minimal description length of a bitstring X. thus captures its inherent structure and compressibility without assuming any getta-ennraatig distribution [29, 16]. This is different from Shannon entropy which is the lower bound of an expected code length for a random variable under its probability distribution (e.g., relative token frequency) [56, 57]. Two concepts are highly correlated each other: shannon entropy approximates an average per-toker kolmogorov complexity [18, 40]. We adopt Kolmogorov complexity as a general metric for measuring complexity of tokenized text since different pre- tokenization rules can change token counts dramatically under the same tokenizer settings [35, 52, 48].
Exact Kolmogorov complexity is uncomputable as it reduces to the halting problem [28, 40, 16, 64]. Instead, we calculate computable upper bound on Kolmogorov complexity, which serves as practical metric for measuring the complexity of compressed data [16]. As BPE tokenization [54] is derived from BPE compression algorithm [14], tokenized text can be viewed compressed data. For a BPE-tokenized bitstring X,,, an asymptotic upper bound on K ((XN) is given by
K(X*) S Hy))
Vogg N O(log N),
where N is the total token count of tokenized text, (p) is the unigram Shannon entropy of the token distribution, and the V log, N term accounts for the prefix-free encoding of the token (c.g.. token frequency table). Since each token count is an integer < N requiring [loga(N + 1)] bits, the BPE model is < V Vogg N bits. The first term upper-bounds the tokenized data size, the second bounds the frequency table size, and the third captures minor logarithmic overhead. Since modern language models are trained on massive corpus with billions of tokens, the N H()) component dominates. We use ((X*) R N Hp)), focusing on the complexity of the tokenized sequence itself. Accordingly, we take (p) as the unigram Shannon entropy, despite it being loose upper bound, to enable fair token-count comparisons across tokenizers with different vocabulary sizes.
3

---

Loss Decomposition Metries In these experiments. we calculate three metrics to assess the language model's performance both for individual vocabulary types and overall: (1) Total Loss, (2) Average Per- Word Loss, and (3) Global Cross-Entropy Loss.
For each vocabulary v, accumulate its Total Loss:
14 ΣΣ IEN
Total Loss(u)
10== 1.) 11-(mp((t ttx)),
where N is the set of evaluation documents, [(v = t4) is the indicator that the ith token equals vocabulary v, and In p(t, t.ci) is the negative log-likelihood of that token. Total loss measures the sum of negative log-likelihoods for each vocabulary, reflecting its impact on the model's loss [38].
Average Per- Word Loss for vocabulary is defined
Total Loss(u)
(l,)
(N)
where T. is the total count of occurrences of in the training data. This represents the mean negative log- ligeeiil across all occurrences cach vocabulary e [38].
The Global Cross-Entropy Loss is the weighted average of these per-word losses:
(N)N) T(N) p(l.),
Global Cross Entropy Loss = Σ
where T(N)= = T.(N) is the total token count in the training data. Global Cross-Entropp Loss reflects the model's average uncertainty per prediction [25, 38].
3.3 Increasing vocabulary size reduces complexity of tokenized text
Naturally occurring data contains a shared structure and inher- ently low complexity and models can Iearn effectively from training data, achieving high accuracy across diverse tasks and generalizing to unseen datasets [16]. To quantify how larger vocabulary size helps the tokenizer approximate natural data's 's low intrinsic complexity and produces simpler tokenized se- quences, we measure the upper bound of Kolmogorov com- plexity and the normalized compression ratio (NCR). NCR |C(z)l
Table 1: Upper bound of Kol-
mogorov complexity (K (K(*)) and NCR for the 45.97 billion byte FineWeb-Edu corpus with K ((X*) measured in bytes.
Vocab size K(XN)
NCR
practical compressibility metric NCR(x; C)
24K
10.74B
0.234
[=]
where |a|| is the byte length of data and |C(x) is its size af- ter lossless compression by compressor C; our formulation, K(X*)= ~ [C(z)] N H(p).
49K
10.43B
0.227
98K
10.23B
0.223
196K
10.16B
0.221
Table 1 reports the upper bound on Kolmogorov complexity and NCR for the 45.97 billion byte FineWeb-Edu corpus (10B GPT-2 tokens). The results show that BPE tokenizers with larger vocabularies yield lower complexity and NCR by segmenting frequent non-i.i.d. character sequences in text (e.g., words) as single token. This facilitates models to learn non-i.i.d. statistical patterns natural text more easily by offloading the low-level pattern learning, thereby simplifying language modeling.
3.4 Segmentation already saturates; vocabulary growth mainly sharpens frequency skew
To quantify how each factor evolves with vocabulary growth, we disentangle two factors: (i) relative token-frequenee imbalance (Figure la) and (ii) segmentation efficiency (Figure 1b)-the fraction of frequent words that represented by exactly one token. Frequent words operationally defined as the top 2, 500 word types, chosen because this cutoff already covers 2 70% of corpus tokens in both Feb-WeeBEdh (74.4 %) and OpenWebText (75.5 %; full coverage curves in Appendix B).
In this experiment, relative token-frequency imbalance is quantified by the Bensen-Shannee diver- gence (JSD) from uniform distribution of the same vocabulary size for a fair comparison across

---

FineWeb-BR Edu
Segmentation efficienc
1.004
FineWeb-Edu
Overlap percentage
1.0
0.45
OpenWeb Text
1.03
OpenWebText
0.8
0..40
1.02
0.35
1.01
0.6
1.00
24K 49K 98K 196K
24K
49K
98K 196K
500
1000 1500 2000 2500 Top N Words
Vocabulary size
Vocabulary size
(a) Token-frequency imbalance
(b) Number of tokens per word
(c) Overlap frequent words
Figure 1: Figure la shows that increasing vocabulary size exacerbates relative token token-frequency imbalance. In other words, enlarging the vocabulary size introduces more rare tokens, causing the relative coken-frequency distribution to be further from uniform distribution. Figure 1b reveals that 24K vocabulary size tokenizer already segments 2, 500 frequent words as a single token regardless of dataset quality. This implies that further vocabulary growth offers no added benefit for estimating the probabilities frequent words. Figure Ic shows that the most frequent n words in fine Web-Euu and Open Wex largely overlap, highlighting the universality of frequent vocabulary across different datasets. We report the most frequent 2, 500 words in Win--We-Edh and OpenWebText, which account for approximately 74.4% and 75.5% of each dataset, respectively.
different vocabulary sizes [67, 12, 52]. Segmentation efficiency gauges the average token count per word, that is, how many tokens the tokenizer needs, on average, to encode each of the 2, 500 most frequent words. Figure la shows that JSD grows monotonically with vocabulary size, reflecting greater ooken-frequency imbalance. In contrast, Figure lb indicates that the segmentation efficiency exceeds 95 % by 24K tokens and plateaus later. In other words, beyond 24K, the vocabulary gets larger without providing additional single-token coverage for frequent words. Within the widely used 24K- 196K range [51, 491, enlarging the vocabulary chiefly amplifies relative token-frequeet frequency imbalance rather than improving segmentation efficiency. This finding tempers the common view that "bigger vocabularies help by approximating word-level tokens" [47]: that mechanism appears to saturate once frequent words are already tokenized as a single token.
3.5 Skew-driven effects persist across corpora quality levels
To test whether the effects of increasing vocabulary size hold across datasets of varying quality, conduct experiments both Fine Web-Edu [43] and OpenWebText [15]. Figure la and 1b demonstrate that the impact of increasing vocabulary size has a similar effect on both high-quality datasets (e.g., Fine Edd [43]) and the lower-quality ones (e.g.. Open SpenWebText [15]). Figure Ic further shows that the frequent 2, 500 words coincide with nearly 75% each other, highlighting the universality of frequent vocabulary across different corpora. Overall, the findings indicate that vocabulary size expansion produces the same effects irrespective of corpus quality and that high-frequency words substantially overlap across datasets.
3.6 Larger vocabularies reduce frequent words uncertainty and global loss
The effect of tokenized text complexity and relative token-frent feeeunffr imbalance language models has not yet been explored. Each prediction in a language model depends on previous tokens, and the conditional nature of these probabilities adds significant analytical complexity. However, it is possible to hypothesize that loss of rare token prediction weighs heavily in the overall loss, as conditional probability is often orders of magnitude lower and therefore yields much higher per-token losses (5,39] To address this presumption, we empirically examine how reducing loss on frequent word prediction at the expense of rare word predictions affects overall model performance. Models are pre-trained on 40B bytes and evaluated on separate, non-overlapping 5B byte split of FineWeb-Fao Figure 2a shows that enlarging the vocabulary steadily lowers the average per-word loss for the frequent words, where the gap between 24K and 196K is roughly 0.1 nats for the frequent 2, 500 words. In contrast, the average per-word loss for the rarest 20, 000 words rises with vocabulary size, increasing from roughly 11.183 to 13.399 nats. As shown in Figure 2b, the global cross-entropy declines from about 3.1 to 136 nats, implying that gains on frequent- word prediction outweigh the degradation infrequent words. Frequent tokens dominate the objective in all settings: the top
5

---

Average per word loss
4.08
14
2,500 words coverage
Cross-entropy (hats
3.18
Rare word loss
4.05
13
76
4.03
3.16
4.00
12
74
3.14
3.98
11
24K
49K
98K
196K
24K
49K
98K
196K
Vocabulary size (tokens)
Vocabulary size (tokens)
(a) Frequent vs. rare word loss (85M, 10B)
(b) CE-loss frequentt word coverage (85M, 10B)
Figure 2: Figure 2a illustrates that models with larger vocabulary size reduce loss the most frequent 2, 500 words while increase loss on rarest 20, 000 words. Nevertheless, Figure 2b shows that the global cross-entropy loss declines as vocabulary size increases, demonstrating that the gains from lower loss on frequent words outweigh the losses from poorer infrequent word estimates. further reveals that frequent words account for nearly 75% of the total loss, while loss infrequent words grows with vocabulary size as their conditional probabilities fall due to data sparsity. Models pre-trained 40B bytes and evaluated on disjoint 5B byte split of FineWeb-Edu
2,500 words account for nearly 75% of total loss. Nevertheless, as the vocabulary expands, losses on infrequent tokens grow, reflecting lower conditional probabilities and reduced predictability for those items. These observations highlight a key takeaway: once loss on frequent words declines, the overall objective is governed by their loss contribution, so further skewness the token frequency distribution generally improves performance even though it harms rare-token estimates. This finding consistent with the existing study [8], explaining the benefits increasing vocabulary size. We observe the same qualitative pattern in the OpenWebText (Appendix C).
3.7 Gains from vocabulary expansion hold for larger datasets and models
To assess whether vocabulary size effects persist across data and model scales, train 85M (non-embedding) model on a 30B GPT-2 token subset (184.6B bytes) of FibeWeb-edu and a 450M (non-embedding) model on 10B token subset. On the 30B token split, figure 3a shows frequent- word loss drops from 3.845 to 3.742 nats, while rare-word loss rises from 10.199 to 11.787 nats; figure 3b shows global cross-entropy falls from 2.991 to 2.941 declines from 77.6% to 74.2%, exhibit the same trend reported in Section 3.6. Figure 3c and 3d confirm the same pattern for the 450M models: frequent- word loss decreases from 3.770 to 3.675 nats, rare-word loss increases from 9.694 to 12.762 nats, cross-entropy reduces from 2.989 to 2.888 nats, and frequent word loss coverage from 78.0% to 73.6%. Overall, these findings indicate that vocabulary expansion produces the same effects with larger dataset and model size.
and frequent-word loss coverage
Analysis
Although our results show that minimizing loss on high frequency tokens crucial for reducing global cross-entropy, several questions remain unresolved. To pinpoint the causal chain from tokenizer design to model behavior, we now organize our analysis around additional guiding questions.
Guiding Questions.
1. Transfer Mechanism
How does the heavy overlap of the most frequent 2, 500 words connect pre-training loss drops to downstream accuracy ($4.1)?
2. Parameter Count Vocabulary Scaling Can enlarging model size (with a fixed tokenizer) reproduce the same advantage delivered by larger vocabulary ($4.2)?
The next two subsections answer newly added Q1-Q2 in turn.

---

12
3.00
78
2,500 words coverage
Frequent word loss
3.85
Cross-entropy (nats)
Rare word loss
77
2.98
3.80
11
76
2.96
75
3.75
2.94
74
10
24K
49K
98K
196K
24K
49K
98K
196K
Vocabulary size (tokens)
Vocabulary size (tokens)
(a) Frequent vs. rare word loss (85M, 30B)
(b) CE-loss frrequent word coverage (85M, 30B)
3.80
13
3.00
2,500 words coverage
Frequent word loss
Cross-entropy (nats)
78
Rare word loss
12
3.75
2.95
76
11
3.70
2.90
74
10
24K
49K
98K
196K
24K
49K
98K
196K
Vocabulary size (tokens)
Vocabulary size (tokens)
(c) Frequent vs. rare word loss (450M, 10B)
(d) CF-loss & frequent-word coverage (450M, 10B)
Figure 3: For 85M model trained 30B tokens, larger vocabularies reduce the most frequent 2, 500 word loss while increase the rarest 20, 000 word loss; since frequent words dominate, global cross-entropy drops (figure 3a and 3b). 450M model trained 10B tokens mirrors the pattern (figure 3c and 3d), indicating that these vocabulary-size effects persist across larger datasets and models.
4.1 Frequent-word overlap explains downstream performance transfer
Reducing loss for frequent words is key to achieving lower global cross-entropy loss. But does this effect carry over to downstream task accuracy? Although a strong link between lower loss and better downstream performance has been studied [22, 26], its rationale has not been investigated. To shed light on this phenomenon, we analyze the overlap of frequent words between the pre-training data and evaluation benchmarks and demonstrate that scaling the vocabulary size reduces cross-entropy loss during pre-training and on downstream tasks. Figure 4a shows that the 2, 500 most common words in Webe [43] account for roughly 76-78% of all tokens in ARC [3], HellaSwag [66], and SciQ [61] and about 72% in the PIQA [6] and CC-Main 2003--2003-000 dump [23]. Because lower cross-entropy loss this CC subset correlates closely with stronger reasoning benchmark scores [23], reducing the frequent words loss and driving down global cross-entropy will improve accuracy across downstream tasks. Consistent with that expectation, figure 4b shows that increasing the vocabulary size from 24K to 196K lowers the average per-word loss on the frequent 2, 500 Fine Web-Edu words by about 0.11 nats, translating into roughly 0.07 nats drop in global cross-entropy loss on the CC dataset. Figure 4c further confirms that larger vocabulary also improves average downstream task accuracy of the language model across datasets and model scales. The key insight from this observation is that frequent words highly overlap between the typical training dataset and the downstream benchmark, so the cross-entropy reduction achieved by enlarging the vocabulary size during pre-training naturally translates into better downstream performance.
4.2 Parameter scaling recovers the same frequent-word gains
Thus far, we have shown that enlarging the vocabulary size decreases word-level average per-word loss on frequent words, which translates into lower cross-entropy loss and better downstream task performance. One might ask whether similar gains could be achieved without altering the vocabulary size by adjusting other model hyperparameters. Our experiment illustrates that increasing the model's parameter count replicate the same benefit. To investigate this, the Pythia suite, where

---

2.5 2,500 words coverage
0.80
Average per word loss
4.21
3.30
Cross -etss-entropy (nats)
accuracy cy
60
4.18
3.28
0.75
4.15
3.26
58
0.70
Downstreat
44.12
3.24
56
Base
0.65
4.09
3.22
54
30B
450M
0.60
4.06
3.20
ARC HS PIQASCIQ CC
24K
49K
98K
196K
24K 49K
98K
196K
Downstream tasks
Vocabulary size (tokens)
Vocabulary size (K tokens)
(a) Frequent word coverage
(b) Frequent word loss reduction
(c) Downstream performance
Figure 4: Figure 4a demonstrates that the most frequent 2, 500 words in the Fine Edd comprise
nearly 72 78% of the tokens in other downstream benchmark datasets as well the CC-Main- 2023 40 [23]. ARC refers to ARC-Fasy, and HS refers to HellaSwag. Figure 4b illustrates that larger vocabulary reduces average per-word loss frequent Fine words within the CC dataset, and demonstrates how this translates into lower global cross-entropy loss on CC dataset. Figure 4c confirms that scaling the vocabulary size boosts downstream task performance.
Average per word loss
4.5
2,500 words coverage
Cross-entren entropy (nats)
2,500 words coverage
70
4.
70
4.0
60
60
3
3.5
50
50
2
160M
1B Pythia model size
6.9B
160M
1B Pythia model size
6.9
(a) Frequent word loss in pythia models
(b) Global cross-entropy in pythia models
Figure 5: Figure 5a illustrates that increasing model size reduces loss on high frequency words, and the global cross-entropy loss of larger models is overwhelmingly driven by frequent word losses mirroring the effect of increased vocabulary size. However, unlike the pattern in figure 2a, scaling up model size does not exacerbate errors on infrequent tokens. Figure 5b demonstrates that the global cross-entropy loss declines as model size increases, showing the same tendency of scaling up the vocabulary size (figure 2b).
each checkpoint is trained on the same dataset with identical hyperparameters, differing only in model size [4]. We also examine the OLMo-2 series [41] to determine whether the same pattern persists in contemporary large-scale scale language models (see the Appendix F).
Figure compares Pythia models (160M, 1B, 6.9B parameter count) in terms of word-level average per-word loss, proportion of frequent words losses on total loss, and global cross-entropy loss measured on the Paloma validation dataset [38]. Figure 5a shows that larger models predict frequent words far more accurately: for the most frequent 2, 500 words in FineWeh-Fduu the average per-word loss drops from 4.48 nats (160M) 3.76 nats (1B) and 3.38 nats (6.9B). Furthermore, frequent words loss dominates the total loss as model size grows, which accounts for almost 70% of the total loss in the Pythia 1B and 6.9B model size, compared with roughly 48% in the Pythia 160M. Unlike the trend observed when only the vocabulary size is increased (Figure 2b), scaling model capacity does not inflate loss on rare tokens; their share the loss shrinks. Figure 5b illustrates that the global cross cross-entropy loss on the Paloma validation set drops from 4.32 nats in the Pythia 160M to 2.51 nats and 2.26 nats in the Pythia IB and 6.9B respectively, mirroring the loss reduction seen when expanding the vocabulary (Figure 8b). From this experiment, we can observe that enlarging the model size also lowers loss on frequent words while effectively ignoring rare token losses, so the global cross-entropy reduction is larger than what scaling up vocabulary size alone can deliver.

---

5 Discussion
5.1 Can we reduce the tokenized text complexity without intensifying frequency imbalance?
larger vocabulary reduces the tokenized text com- plexity by decreasing the number of tokens and sharp- ening the token-frequency imbalance. However, Sec- tion 3.6 shows that sharper token-frequency im- balance increases rare-token loss, This raises the question: can we lower tokenized text complexity without enlarging the vocabulary size? SuperBPE [35] is one such example.
Table 2: Upper bound of Kolmogorov com- plexity (K(X*)) and NCR for the 45.97 billion-byte FineWeb-Edu corpus tokenized with various SuperB variants. L and Avg denote the stage-switch vocabulary threshold and an average downstream performance re- ported by [35], respectively. The SuperBPE variant with the highest average performance exhibits the lowest tokenized text complexity.
SuperBPE employs a two-stage BPE algorithm: in the first stage (up threshold vocabulary size t), it operates identically to standard BPE, while in the sec- ond stage, it abandons whitespace pre-tokenization. By permitting merges across whitespace boundaries, Superbpe directs subsequent merges toward frequent tokens, thereby limiting the introduction of new rare tokens, and preventing further token frequency im- balance as the vocabulary expands.
superBPE
K(X*)
NCR
Avg [35]
(B0K(
10.21
0,222
39.8
200K(I 180K)
10.03
0.218
43.8
200K(E 160K)
10.05.
0,219
43.4
200K(1 80K)
10.10/
0.220
42.9
Table 2 reports the upper bound Kolmogorov complexity, NCR of the 45.97 billion-byte FineWeb-Eas corpus tokenized with various SuperBPE variants, along with average downstream performance reported by [35]. The result shows that the superBPE. achieves lower token count and a less skewed token-frequency distribution than BPE at equal vocabulary. Also, the superBPE variant with the highest average performance exhibits the lowest tokenized text complexity. This finding demonstrates that both vocabulary expansion and the superBPE variants leverage the same benefit: they reduce the complexity of the tokenized text by segmenting common character sequences into single tokens, thereby facilitating the model to learn non-i.i.d, patterns in the data more easily.
5.2 Deduplication through the lens of vocabulary frequency imbalance
Deduplication removes exact near-duplicate content from corpus, reducing repetition and leakage so that a larger fraction of the dataset novel [31, 43, 34]. Deduplication can increase the information density of dataset, i.e., the fraction of non-redundant data in the corpus. For non-i.i.d. data, information density can be proxied by the byte-level entropy rate, the theoretical minimum amount of non-redundant information. However, the entropy rate is only tightly approximable with near infinite context compressors, finite window compressors can distort comparisons before and after deduplication. A more practical loose upper bound is the byte-Ievel bits tits-per-byte (BPB): under an i.i.d. assumption it quantifies information density, with unigram (byte) entropy boundib the true entropy rate. Under the byte-level vocabulary, greater information density implies higher BPB, driving entropy toward log, 256 = 8 bits/byte, which corresponds to a more uniform distribution and lower frequency imbalance. Exploring how deduplication alters BPB, information density, and vocabulary frequency imbalance is a promising avenue for future work.
5.3 Why higher token-frequency imbalance degrades machine translation task performance?
Zouhar et al. [67] reported that higher token-frequency imbalance degrades machine translation task performance, contrary to our findings. What explains this contrasting behavior between machine translation and monolingual settings? We presume that this is related to rare word issues in machine translation tasks [55, 53, 27, 37, 19].
In machine translation, the source and target vocabularies overlap minimally: vocabularies that are common in English often occur rarely in French, and vice versa. Conversely, in monolingual setting. the pre-training corpus and downstream benchmarks share not only vocabularies but also many of the same frequent words, resulting extensive text overlap. Consequently, BLEU penalizes every missing target-side n-gram, so mistranslating even a handful of infrequent tokens can sharply lower scores. To address this word challenge, methods like vocabulary trimming deliberately shrink the vocabulary size to reduce the impact of low-frequency tokens translation performance [17, 46, 9].

---

6
Related Work
Impact of tokenization on transformer language models Empirical and theoretical evidence shows that tokenization is a central determinant of both the quality and the speed with which Trans- formers learn. Early byte character-level systems such as CANINE [11] and ByT5 [63] avoid sub-word vocabularies but pay a steep price: sequences become an order of magnitude longer, gra- dients are noisier, and convergence is markedly slower than for sub-word models. Enlarging the BPE dictionary lets even an unigram model approach near-optimal cross-entropy by approximating word-level tokens, whereas the same model without a tokenizer underfits [47]. Furthermore, Con- trolled scaling studies reveal a log linear pattern: exponentially expanding the input vocabulary leads to an almost linear drop in loss across model sizes [58]. Huang et al. [22] push this further with "Dver-Tokenized Transformers," showing that 400M parameter model with 12.8M token encoder matches the loss of 1B model baseline without extra compute.
Beyond the gains from larger vocabularies, a growing body of work shows that permitting merges across word boundaries (i.e., disabling whitespace pretokenization) improves language model perfor- mance. Liu et al. [35], Schmidt et al. [52] turn off white-space pretokenization during BPE tokenizer training once the vocabulary reaches or at Trequeney-driven transition points. Taken together, these results indicate that without tokenization, language models stall at character granularity, but with a tokenizer, they converge faster and generalize better.
Large language model performs a two stage lossless compression Large language models perform two stage lossless compression where tokenizer acts as a pre-compressor and transformer models the tokenized data [13, 333. Lester et al. [33] further explore the role of tokenizer as a pre- compressor by replacing with small transformer model with arithmetic coding, and models trained on neurally compressed text underperforms subword tokenizers but beat byte-level LMs. Delétang al. [13] and Valmeekam et al. [60] has been proposed that transformer language models can be harnessed for compression via arithmetic coding and Heurtel- DDepeiges et al. [21] extends this line universal compressor across text, images, and audio. However, most literatures mixuse the concept of generalization and compression where generalization focuses on predicting unseen data from different datasets by exploiting shared statistical patterns while compression focuses on encoding fixed dataset in as few bits as possible by modeling its distribution and removing redundancy. This work investigates the role and impact of tokenizer on generalization through the lens of data compression and information theory.
by probing the possibility of transformers
BPE tokenizer with inductive biases BPE tokenizers typically adopt pre-tokenization where regular expressions split text into chunks, sometimes called pre-tokens. Recent work shows that removing pre-tokenization degrades downstream performance across diverse tokenizers [12, 50]. Pre-tokenization rules act as an inductive bias grounded in morphology and grammar, helping the tokenizer capture ubiquitous statistical patterns in natural text, which is crucial for transformer language models to generalize unseen data.
7 Conclusion
This work out explain how and why larger vocabularies boost language model performance. Our experiments reveal a single, robust mechanism: enlarging the vocabulary reduces tokenized text complexity, making non-i.i.d. patterns easier learn, better approximating natural data's low intrinsic complexity, and ultimately lowering language modeling difficulty. Once vocabulary reaches roughly 24K size, every common word is already a single token; subsequent growth therefore does not refine segmentation but instead steepens the long-tailed frequency distribution, focusing optimization on the same frequent tokens and driving loss down. Norm- corm-coontraining that erases the frequency signal removes these gains, confirming causality, while enlarging model parameters with a fixed tokenizer reproduces the benefit, pointing to a shared optimization dynamic between vocabulary and model scaling. Recognising this fact turns Kolmogorov complexity of data into principled dial for ookenizer-mode co-design and sharpens our understanding of the scaling forces that govern language-modee model pre-training. We therefore conclude:
Expanding the tokenizer mainly reduces uncertainty for the most common words, with little payoff for the rare tail.
10

---

Acknowledgement
This work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea MOVermmen(MMSIT) Intelligence Graduate School Program(KAIST), and Institute Information communications Technology Planning & Evaluation (IIP)) grant funded by the Korea government(MSIT) RN- 2022-112201 84, 2022-0-00184, Development and Study of Al Technologies to Inexpensively Conform to Evolving Policy Ethics).
Artificial
References
[T] M. Ali, M. Fromm, K. Thellmann, R. Rutmann, M. Lübbering, J. Leveling, K. Klug. J. Ebert, Weber, L. Jurkschat, H. Abdelwahab, C. John, O. Suarez, M. Ostendorff, S. Weinbach, R. Sifa, S. Kesselheim, and N. Flores-Herr. Tokenizer choice for LLM training: Negligible or crucial? K. Duh, H. Aadozo,doroo, and S. Bethard, editors, Findings of the Association for Compulational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 3907-3924, Association for Computational Linguistics, 2024. 120.2..N... doi: maadinngs-aacc... 247. 2024.
N. Doll, J. S. Buschhoff, Jain,
URL https://doo org/10. 18653/v1/
[2] Y. Bengio, R. Ducharme, and P. Vincent. neural probabilistic language model. In K. Leen, T. G. Dietterich, and V. Tresp, editors. Advances in Neural Information Processing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 932 938. MIT Press, 2000. URL https: https::/pprcceedinnggg Awwlsc--bbbttttttttaaaaa html.
ceouripscc/ppppe
[3] S. Bhakthavatsalam, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, and P. Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03311 2021. URL attps://arxiv org/ wttp///.......sme...eeeee...e..t 03315.
[4] S. Biderman, H. Schoelkopf, Q G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolula, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR, 2023. URL https: // //proceedings.om mlr. press/ v222//22/////// v202/biidee biderman23a....
[5] D. Bis, M. Podkorytov, and X. Liu. Too much in common: Shifting of embeddings in trans- former language models and its implications. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Pro- ceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Human Language Technologies, NAACL-HLT 2021, Online, June 6- 2021, pages 5117-5130 Association for Computational Linguistics, 2021. doi: 10.18653/V 1/2022. NAACL-MAIN 403. URL https: :ttttp:://daa //doi org/ 10. 18653/v1/204 www.clacllac.....a....a main 403.
[6] Y. Bisk, R. Zellers, R. L. Bras, Gao, and Y. Choi. PIQA: reasoning about physical common- sense in natural language. In The Thirty- AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence,
EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 1432-7439, AAAI Press, 2020. doi: 10.1609/AAAL. 1341A5.331005.225 URL wttph:ttp:/... //doi org/ 10 1009/669/aaa /000//aaa.... v34105. 6239.
[7] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, Purohit, Reynolds, Tow, B. Wang, and S. Weinbach. Gpt-neox-20b: An open-source autoregressive language model. CoRR, abs/2204. 066455, 2022. doi: 10448550/ARXIV.220..0777 URL. https: //dps:://dd... http://ooo..og....//o 48550/ 18550/arXiv 2204. 2204.06745.

---

[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Vosse, G. Krueger, Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020, URL https: http://pproceee...////// wewpeediiinnnnnniiiine nwwrinu..ncccm http:cccpppee/2//0000/// 1457c0d6bf (157:c066bfc449777444
Abstract. html.
[9] Chizhov, Arett, Korotkova, and I. P. Yamshchikov. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 16587-16604. Association for Computational Linguistics, 2024. URL https: hatpss//accllc ptttts://aclanthll..e acllanhologgy . org 2024. emnlp- -main.
925.
[10] W. Chung, J. Hong, N. M. An, J. Thome, and S. Yun. Stable language model pre-training by reducing embedding variability. In Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FI, USA, November 12-16, 2024, pages 10852-10863. Association for Computational Linguistics, 2024. URL https: / aa//lclaaanttt awo.loonologyy.... org/ 2024.
maim.miinn.......... 606.
[11] H. Clark, Garrette, I. Ture, and J. Wieting. Canine: Pre-training an efficient tokenization- free encoder for language representation. Trans. Assoc. Comput. Linguistics, 10:73-91, 2022. L doi: 10.1162/TACL A) 00448. URL https : //t.///oiooo/// http://oio.org/////// 1162/ (0l16//tacl.. tttpttaccllaa_aaa a_00488..
[12] G. Dagan, G. Synnaeve, and B. Rozière. Getting the most out of your tokenizer for pre-training and domain adaptation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:: httpss//openrevierr net/forum? ZFt....jt...
[13] G. Delétang, A. Ruoss, P. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, I.. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. Language mod- eling is compression. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / //openreview. net. pttt/forummiddjzngggyyo
[14] P. Gage. A new algorithm for data compression. The C Users Journal archive, 12:23-38, 1994. URL. htt.htt/ps//ppp. semanticscholar org//orruut 59804030.
[15] A. Gokaslan, V. Cohen, E. Pavlick, and S. Tellex. Openwebtext corpuss http://Skyllion007. github. hat.ub...///eeeeeeeetttrpppsss 2019.
[16] M. Goldblum, M. Finzi, K. Rowan, and A. G. Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. CoRR, abs/2304.05366, 2023. doi: 1044550/ARXIV2234.005 05366.
URL https://doi org/10. wttrg//..8550///// 48550/arXiv arrpa..ivvvv.... 2304.
[17] T. Gowda and May. Finding the optimal vocabulary size for neural machine translation. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL. pages 3955-3964. Association for Computational Linguistics, 2020. doi: 10. 10.18653/V1/20222 FINDINGS-EMNI 352.
URL httpt::tt///oo http://ai.oog/////// www.oogg//// 18650/11/2/// findings-emnlp
[18) P. Grünwald and P. M. B. Vitányi. Shannon information and kolmogorov complexity. CoRR, cs. 1T/01000000,, 2004. URL http htsbsss..... IT/0410002.
[19] S. Gu, J. Zhang, F. Meng, Y. Feng, W. Xie, J. Zhou, and D. Yu. Token-level adaptive training for neural machine translation. In B. Webher, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020,
12

---

Online, November 16-20. 2020, pages 1035-1046. Association for Computational Linguisties, 2020. doi: 10.18653/V /220000ENNL--MANNN 76. URL https://doi. http:::///oo...111//// 10. 18853/v1/ 2020. emnlp-main. 76.
[20] J. Hayase, A. Liu, Y. Choi, S. Oh, and N. A. Smith. Data mixture inference at- tack: BPE tokenizers reveal training data compositions. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 aapper/222//ang
15. 2024, 2024. URL http: http://papers www..cccpsaaaappeeeee cc/paper_filee htw.:9233444911bbbbrrrrtee htt.://icce--bsttacct-ooooeeeeee html.
[21] D. Heurtel-Depeiges, A. Ruoss, J. Veness, and T. Genewein. Compression via pre-trained
transformers: A study on byte-level multimodal data. CoRR, abs/2410.05078, 2024. doi: 10.48550/ARXI..410000555
URL https://ddo. /ttp////...../....////////////// arxiv.
[22] H. Huang, D. Zhu, B. Wu, Zeng, Y. Wang. Q. Min, and X. Zhou, Over-tokenized transformer: Vocabulary is generally worth scaling. CoRR, abs/2501. 2025. doi: 10.48550/ARXIV.
2501.16975, URL https://doi. org/10. ww8.50//aa....2258 2501 16975.
[23] Y. Huang, J. Zhang, Z. Shan, and J. He. Compression represents intelligence linearly. CoRR, 0bs/2440.099331 2024. doi: 10445500/RRXIV.244..09999 48550/arXiv 2404.09937.
URL https: /ttp:////oooo/10......
[24] H. Inan, K. Khosravi, and R. Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: http:////////vveeeeeeeee eview.revie....mm net. forum?iddraaPssst
(25] F. Jelinek, R. L. Mercer, L. R. Bahl, and J. M. Baker. Perplexity- a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62, 1977. URL https:/ //aapi semanticscholar arg/CorpussDD 121680873.
[26] J. Kim, B. Lee, C. Park, Y. Oh, B. Kim, T. Yoo, S. Shin, D. Han, J. Shin, and K. M. Yoo. Peri-In: Revisiting layer normalization in the transformer architecture. CoRR, abs/2502.00232, 2025. doi: 10.445550/RRXI.2550222
URI. https:::///// //doi. org/10. 0..48855//rat 2502. 2502.02732.
[27] P. Koehn and R. Knowles. Six challenges for neural machine translation. In T. Luong,
A. Birch, G. Neubig, and A. M. Finch, editors, Proceedings of the First Workshop on Neural Machine Translation, NMT@ 2017, Vancouver, Canada, August 4, 2017, pages 28-39, Association for Computational Linguistics, 2017. doi: 10.18653/V1/W11-32200 URL https:
/ //doi. oroiii.og/11..0 11t111111111/1111111111
[28] A. N. Kolmogorov. Three approaches to the quantitative definition of information. International Journal of Computer Mathematics, 2:157-168, 1968. https://: sep...eeatttccshhhllrr
org/Corpus ID: 119745517.
[29] A. N. Kolmogorov. On tables of random numbers (reprinted from "sankhya: The indian journal of statistics", ", series vol. 25 part 4, 1963). Theor. Comput. Sci., 207(2):38773333 1998. doi:
URL wwwhhhttt/////tt /ttp://///..oog/..///////00----0 0010//000000//00049995555559
00075-9.
[30] S. Land and M. Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. In Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 11631-11646, Association for Computational Linguistics, 2024. URL. https: / /al//acllannttt aclanyhlogy org/2024. emnlp-main main..mminn... 649.
[31] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison- Burch, and N. Carlini. Dedupli- cating training data makes language models better. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- ties (Volume I: Long Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 4424-8445, Association for Computational Linguistics, 2022. doi: URL https: /oi. org/ 10. 10w1111611/0000 acl long. 577.
13

---

[32] P. Lesci, C. Meister, T. Hofmann, A. Vlachos, and T. Pimentel. Causal estimation of tokenisation bias, 2025. URL https: wwwprxii....rrooooooo htrg/abs/2500 03149..
[33] B. Lester, J. Lee, A. A. Alemi, J. Pennington, A. Roberts, J. Sohl-Dickstein, and N. Constant. Training Ilms over neurally compressed text. Trans. Mach. Learn. Res., 2024, 2024. URL httpss//openrevile
[34] J. Li, A. Fang, G. Smyrnis, M. Ivgi. M. Jordan, S. Y. Gadre, H. Bansal, E. K. Guha, S. S. Keh, K. Arora, S. Garg, R. Xin, N. Muennighoff, R. Heckel, J. Mercat, M. Chen. S. Gururangan.
M. Wortsman, A. Albalak, Y. Bitton, M. Nezhurina, A. Abbas, C. Hsich, D. Ghosh, J. Gardner,
M. Kilian, H. Zhang, R. Shao, S. M. Pratt, S. Sanyal, G. Ilharco, G. Daras, K. Marathe,
A. Gokaslan, J. Zhang, K. R. Chandu, T. Nguyen, I. Vasiljevic, S. M. Kakade, S. Song,
Lo, A. El-Nouby, H. Pouransari, A. Toshev, S. Wang. D. Groeneveld, L. Soldaini, P. W. Koh, J. Jitsev, T. Kollar, A. Dimakis, Y. Carmon, A. Dave, L. Schmidt, and Shankar. Datacomp-Im: In search of the next generation of training sets for language models. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http: http::::/ppappss..i.... cc cc/paper_aa httppppprfffle/////eeeeeeee /iles/ http://24/aaaa//aaee http://hhhh1eeeeeeeette paper/
S. Sanghavi, F. Faghri, S. Oh, L. Zettlemoyer,
http:///444420000e00000666666669 -tbttbeeee411A-tbbttttoo Datt-atttaaaataaaeee _and_ Benchmarks _
Track. html.
[35] A. Liu, Hayase, V. Hofmann, S. Oh, N. A. Smith, and Y. Choi. Superbpe: Space travel for language models. CoRR, abs/2503.13444 2025. doi: 10.48550/RRXI.2555.BB422 13423.. URL attps://áoi. org/10. 48550/ arviv. 2503. 13423.
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL http:: https:// wtt.ppaarrrraei.eeeeoooo.oaaeaae weet/forrumidd-BggRR...
[37] T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 11-19. The Association for Computer Linguistics, 2015. doi: 10.3115/V1/P15.. 1002. URL https: //ooi. org/10. 3115 3115/v1/p15-1002 1002.
[38] I. Magnusson, A. Bhagia, V. Hofmann, L. Soldaini, A. H. Jha, O. Tafjord, D. Schwenk, E. P. Walsh, Y. Elazar, K. Lo. D. Groeneveld, I. Beltagy. H. Hajishirzi, N. A. Smith, K. Richardson, and J. Dodge. Paloma: A benchmark for evaluating language model fit. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http: http://aappppaeeeee nips. cc/ cc/paper_ppp..c lalppppppllleeeeeelllllllllllple fil.fiilesppeee files/paper happeer2224//haa 7
Abstract----ttttta-aaaa--a----tt Datasets, madd_Benhmarkss Track.
html.
[39] A. Mircea, F. Lohacheva, and I. Rish. Gradient dissent language model training and saturation. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning. 2024. URL https: httpp::/opeenrviiiii
[40] J. X. Morris, C. Sitawarin, C.
Guo, N. Kokhlikyan, G. E. Suh, A. M. Rush, K. Chaudhuri, and S. Mahloujifar. How much do language models memorize? CoRR, abs/2505.24832, 2025. doi: 00.48550/ARXI.25500 248222
24832. URI. /tattts::/doi.rrggg
48550/ 220.......555522.... 24832.
[41] T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, N. Lambert, D. Schwenk, O. Tafjord, Anderson,
Atkinson, Brahman, C. Clark, P. Dasigi, N. Dziri, M. Guerquin. H. Ivison, P. W. Koh, J. Liu, S. Malik, W. Merrill, L. J. V. Miranda, Morrison, Murray, C. Nam, V. Pyatkin, A. Rangapur, M. Schmitz, S. Skjonsberg, D. Wadden, C. Wilhelm, M. Wilson, L. Zettlemoyer, A. Farhadi, N. A. Smith, and H. Hajishirzi. 2 olmo furious. CoRR, abs/2501 00656, 2025. doi: 10.48550/ARRII 00656. URL https: / //toi.dii///// org/10 10. 48550/ arrvv 2501 2501..0666.
14

---

[42] OpenAl. GPT-4 technical report. CORR, abs/2303.08774, 2023. doi: 10.48550/ARXIV22300. 08774. URL https://doi. org/ 10. 48550/ aaw.vv.rrvvv.v. 2303.08777
[43] G. Penedo, H. Kydlícek, L. B. Allal, A. Lozhkov, M. Mitchell, C. A. Raffel, L. von Werra, and T. Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http: / //tp:/:///pppp.appeep nips. cc/ccc..pp... cc/paper.p..p... http/pppr__ieeeee files/paper "ppaprr/000 /eaper/2004///e 2a////2244/hhaaaa 370df50ccf wwteeeedeeeeee0e0991101091111110 Absrrbddd-bbrrrtt....... Datasets _ateeets_dn_____aeeee and Benchmarks, _raack Track. html.
[44] J. Pettty, S. van Steenkiste, I. Dasgupta, F. Sha, D. Garrette, and T. Linzen. The impact of depth and width on transformer language model generalization. CoRR, abs/2310. 19956,,,, 10.48550/ARXIV.231100 19956. URL. ttttps://dos: wwwpei......../
19956, 2023, doi: 48550/arXiv 2310. 19956.
[45] A. Pinto, T. Galanti, and R. Balestriero. The fair language model paradox. CoRR, 1ts/2410..1185, 2024. doi: 10.48550/ARXIV.410000 19....94...11......1.. URL https://doi. sttps://doi.orr/10.. 48550/arXia arriv. 2410. 11985.
[46] I. Provilkov, D. Emelianenko, and E. Voita. Bpe-dropout: Simple and effective subword regularization. In D. Jurafsky. Chai, N. Schluter, and J. R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1882-1 1892. Association for Computational Linguistics, 2020. doi: 10.18653/ VCL-020 170. URL https: //doi org/ 10. 100.11166/v11222 acl-main. 170.
[47] N. Rajaraman, J. Jiao, and K. Ramchandran. An analysis of tokenization: Transformers under markov data. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.. mipppppeeeenpppp....ooeppp.eepse httpe/ppprrffleeaeeeeeeeeeeeeeee files/paper/ /aaash/72240 /24ff 2 -btp::00000dd-tttaaaoa Abst
Conference html.
[48] V. Reddy, W. Schmidt, Y. Pinter, and Tanner. How much is enough? the diminishing returns of tokenization training data. CoRR, abs/2502.20273,, 2025. doi: 1048550/ARXIV2550220077
URL https://doi. org/ 2..0.48500/rraaa arxiv. 2502. 20273.
[49] V. Reddy, C. Schmidt, Y. Pinter, and C. Tanner. How much is enough? the diminishing returns of tokenization training data. CoRR, abs/2502.20073,, 2025. doi: 10.48500ARRXVV3552.207777 URL https:: org/ 10. 48550/ arXiv. 2502 20273.
[50] C. W. Schmidt, V. Reddy, H. Zhang, A. Alameddine, O. Uzan, Y. Pinter, and Tanner. Tokenization is more than compression. In Y. Al-Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 678-702. Association for Computational Linguistics, 2024. doi: 1118855//120224.11. EMNLP-MAIN.40 URL https: //doi. 116533/v1/002
[51] C. W. Schmidt, V. Reddy, H. Zhang, A. Alameddine, O. Uzan, Y. Pinter, and C. Tanner. Tokenization is more than compression. In AI-Onaizan, M. Bansal, and Y. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 678-702. Association for Computational Linguistics, 2024. URL 40.
org 2024.. emnlp-maii main..
[52] C. W. Schmidt, V. Reddy, C. Tanner, and Y. Pinter. Boundless byte pair encoding: Breaking the pre-tokenization barrier, 2025. URL ttttps://axxiii .oaeeiii...////oooooe org/ hasg//bs/25044 .0175000000008.....08
[53] R. Sennrich and B. Zhang. Revisiting low-resource neural machine translation: A case study. In A. Korhonen, D. R. Traum, and L. Marquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2019, Volume 1: Long Papers, pages 21 1-2221 Association for Computational Linguisties, URL https /oi. org/10 10. 18653/v1/p19- 1021.
2019. doi: 10.
15

---

[54] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguisties, 2016. doi: N0.18663/V1/P16-11622 URI. https: / 1/doi.org org/10 11w53//1/p16---. 116111266611.... 1162.
[55] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12. 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguisties, 2016. //doi. www.31866/vsst
doi: 10.18633/V/P16--1622 URL https:
[56] C. E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., (7(3))33794444 423, 1948. doi: 10. 11100//113388 1735..948..T01133.. URL https://dol. org/10. 10. org/10.1002/j 1538-7305. tt01338.01338...
[57] C. E. Shannon. Prediction and entropy of printed english. The Bell System Technical Journal, 30(1):50-64, 1951. doi: 10. 10.1002/j.138-- 1538-7305. 166.166666.6
[58] C. Tao, Q. Liu, L. Dou, N. Muennighoff, Z. Wan, P. Luo, M. Lin, and N. Wong. Scal- ing laws with vocabulary: Larger models deserve larger vocabularies. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, M. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http::/pappeete nips. cc/ cctcccccppp.ee cc/paper_sieeceecceeeeeeeeee (aaa019ee9c11bbbee2233eeeeee
Abstract Conference html.
[59] Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. CoRR, abs/2109.1066 2021. URL atttp://aartie http://axxvvv.///aaa// wwtp:/o////aa//////oe 10686.
[60] C. C. K. Valmeekam, K. Narayanan, D. Kalathil, Chamberland, and S. Shakkottai. Limzip: Lossless text compression using large language models. CoRR, abs/2306.04050, 2023. doi: 10.48550/ARXI.220666
URL https:://dot wto/oo.////ooooo 48550/ aarivv/rrivvvi.a 2306 04050.
[61] J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing multiple choice science questions. In L. Derezynski, W. Xu, A. Ritter, and T. Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@ EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94-106. Association for Computational Linguistics, 2017. doi: 10. 18653/11/W17-4411 URL https:: //wii org/ 10. 18653/v1/w17-44440
[62] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng. C. Xing, H. Zhang, Y. Lan, L. Wang, and Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10524-1000 10533. PMLR, 2020. URL htt.:ht::/// http::///coceeeens.e/eee mlr, www..preessvvve prressvvi99) www1//iiigg/......o html.
[63] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale. A. Roberts, and C. Raffel. Byt5: Towards a token free future with pre-trained byte-to-byte models. Trans. Assoc. Comput. Linguistics, 10:291-306. 2022. 1162/tacl 1162:ttalla_0045556 a_00461.1111
doi: 10. 16. AV 000461. URL https://doi org/10.
[64] O. Yoran, K. Zheng, Gloeckle, J. Gehring, G. Synnaeve, and T. Cohen. The kolmogorov test: Compression by code generation. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL
https://openreviess dtt.://nwww..ttttoordddddodooood (ttp://foommmdddc55eeea C45YgeBDUM.
[65] D. Yu, F. Cohen, B. Ghazi, Y. Huang, P. Kamath, R. Kumar, D. Liu, and C. Zhang. Scaling embedding layers language models. CoRR, abs/2502.016377, 2025, doi: 10.48550/ARXIV. 2502.01637. URL https: / :/dps://///ii /oii org/10. 48550/arXi 2502. 01637.
16

---

[66] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? A. Korhonen, D. R. Traum, and L. Marquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguisties, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pages 4791-4800. Association for Computational Linguistics, 2019. doi: 10. 10.18633/11/119-14722 URL https:/ /doi org/10. 18653/v1/p19-147..
[67] V. Zouhar, C. Meister, L. Gastaldi, L. Du, M. Sachan, and R. Cotterell. Tokenization and the noiseless channel. In A. Rogers, J. L. Boyd- and N. Okazaki, editors, Proceedings of the 6/ Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5184-5207. Association for Computational Linguistics, 2023. doi: //doi.org//i
URL https:
acl-023..cll11 acl-oog............... 284.
17

---

A Influence of token-frequency imbalance on unigram and language models
In this section, we provide detailed explanation of our research question. Natural language exhibits strong contextual dependencies rather than behaving as an i..d. process: each token's 's probability depends on its preceding context. As result, the entropy rate, Ho lim,- (X,, X<t), which captures the optimal per-token uncertainty in text, is strictly lower than the unigram Shannon entropy H1 = - En. (w) log p(w). Although entropy rate and Shannon entropy coincide for truly i.idd data, in natural language, they can differ by several bits per token.
Under pure unigram model, minimum cross-entropy loss exactly equals Shannon entropy, and modifying the vocabulary size of the tokenizer immediately changes the Shannon entropy. Typically, enlarging the vocabulary segments frequent multi-token patterns into single tokens, driving their individual relative frequency up and reducing Shannon entropy. But if the vocabulary size grows too large, the new entries tend to be rare tokens, which lengthen the tail and can actually increase the Shannon entropy as ooken-frequency imbalance rises [67]. However, experimental results show that expanding BPE vocabulary around 80K lowers the Shannon entropy of unigram models, demonstrating that more skewed token feeeenyfyyuency distribution is advantageous at practical vocabulary scales [47]. This pattern extends to n-gram models as well: their conditional Shannon entropy-the the never exceed the unigram entropy, so lowering the
lowest possible n-gram cross-entropy- unigram entropy necessarily lowers the conditional entropy.
Language model loss [(0) minimizes
[()) = Ho + DKL(P ae)) Il Qo(- x<()).
zEV
where denotes the tokenizer's 's vocabulary, p(x) the marginal probability of token x, P( xee) the true next-token distribution given the full history Tet and Qa( x<x)) the model's predicted distribution with parameters A. When the target label is one-hot vector, the language model loss can be written as C(0) = E.ev p(z) log Qo(x x<<)] so it is a average of the model's surprisal - log Qe. By contrast, the Shannon entropy of the unigram model is frequency-weightee average of the self-information log p(z) the dataset. Even though frequent token logits and embedding norms are higher than rare ones (see Appendix D and E), the loss - log Qo depends not only on the underlying relative freuunces but also on training dynamics as well. We therefore cannot derive a closed-form expression to predict how much the loss on frequent tokens shrinks versus how much the rare token losses grow under L = Pfreq Lireg + Prare Lrare. This masking effect makes it substantially harder to measure the influence of token-frequency imbalance language models than in the unigram and n-gram models.
18

---

B Coverage of the most frequent N words
80
80
Coverage
Coverage
70
70
60
60
50
50
500
1500 Top N words in FineWeb-Edu
1000
2000 2500
500
1000
1500 2000 2500
Top N words in OpenWebText
(a) Coverage of frequent words in FineWeb-Edu
(b) Coverage of frequent words Open Webext
Figure 6: Figures 6a and 6b illustrate the cumulative coverage of the 2,500 most frequent words in the Fineweb-edu and Open OpenWeeText datasets, respectively.
In this section, we measure the coverage of the frequent words in Fineweb-edu [43] and Open Text [15]. Both dataset exhibit steep rise in cumulative coverage as we include more high-frequency tokens, but with subtly different baselines and slopes. figure 6a, the most frequent 500 words already cover about 58% of all tokens, climbing to roughly 75% once take the 2, 500 most frequent words. Wexe (Figure 6b) starts marginally higher higher-arouna around 59% at most frequent 500 words, but follows almost identical trajectory, reaching about 76% coverage by the frequent 2, 500 words. This pattern underscores how a relatively small core vocabulary captures the vast majority running text in both corpora, with only modest gains move deeper into the long tail.
C Open Web Text experiments results
Average per word loss
4.50
78
2,500 words coverage
Cross-entropy (nats)
3.4
78
2,500 words coverage
4.40
76
3.3
76
4.30
4.20
3.2
74
4.10
4.00
3.1
24K
49K
98K
196K
24K
49K
98K 196K
Vocabulary size (tokens)
Vocabulary size (tokens)
Frequent word loss on WeenTeeeext
(b) Global cross-entropy on an Open
Figure 7: Figure 7a reveals that expanding the vocabulary from 24K to 196K steadily reduces the
average per-vocabulary loss of high frequency words. Figure 7b indicates that the most frequent 2, 500 tokens still contribute roughly 75 of the total loss, while the rare words losses grow with vocabulary size, similar to 2b. Figure 7b further shows that the global cross-entropy loss falls by about 0.25 nats as the vocabulary grows, demonstrating that the reduction of loss frequent words outweighs the inflation of rare-token losses.
To verify that reducing frequent-word loss is not by-product of dataset quality, repeat the same experiments in section 3.6 on the Open WebText dataset. Figure 7a shows that widening the vocabulary from 24K 196K in Open Web progressively reduces the average loss assigned to high-frequency words. Figure 7b indicates that the most frequent 2, 500 tokens still account for about 75 of the total loss, whereas the loss on infrequent tokens grows with vocabulary size, paralleling the pattern seen in Figure 2b. Figure 7b further demonstrates that the global cross-entropy loss falls from nats at 196K, implying that the reduction loss on frequent words more than offsets the increase in rare-token loss. regardless of dataset quality or type.
3.37 at 24K vocabulary
19

---

D Constraining embedding norms increases cross-entropy
Aierage per word los
5
4.20
Cross-entropy (nats)
3.30
4.15
Base
3.25
4.10
EmbedNorm
Base EmbedNorm
Average
3.20
4.05
4.00
3.15
24K
49K
98K
196K
24K
49K
98K 196K
Vocabulary size (tokens)
Vocabulary size (tokens)
(a) Frequent word loss gap of base and embednorm
(b) Cross-entropy gap of base and embednorm
Figure 8: Constraining the norms both input and output token embeddings to mitigate the impact of token-frequene, imbalance during training increases average per-word loss on 2, 500 frequent words in the LineWeb-Edi as illustrated in figure 8a, which in turn drives up the global cross-entropy loss on the Webe Web-Eddl Edu (figure 8b).
Higher token-frequency imbalance provides plentiful training examples for frequent words but far fewer for rare ones. Section 3.6 demonstrates that this reduces loss on frequent words while inflating it on rare words. We evaluate whether reducing the impact of token- frequuerreyy imbalance during pre-training leads to a performance degradation.
In language models, the token embedding norm of input and output embeddings reflects the token- frequency imbalance in training data. For input embeddings, every time token shows up in training batch, its token embedding receives gradient updates, and tokens that do not appear only undergo weight decay, causing their embedding norms shrink over time [30, 10]. In the output embedding layer, the target token in the output embeddings receives a stronger gradient than non target token embedding during training [39] (see the Appendix E for detail explanation). Thus embedding norm of a frequent token is typically larger in both input and output embedding. Even though language models with pre-LN [62] normalize the final layer hidden states, frequent tokens still develop larger output- trnputtembedding norms and highly align with the final layer hidden states, inflating their logits and predicted probabilities and ultimately reducing their average per-word loss.
To reduce the impact of loken-frequency imbalance during pre-training, we constrain the token embeddings of the input and output embedding layer unit norm, since the token embeddings of our base model already average unit length. Figure 8a demonstrates that constraining the norms of both input and output token embeddings address token-frequency imbalance during training leads to a higher average loss for 2, 500 frequent words. This increases the global cross-entropy on the training dataset and lowers downstream performance as illustrated in figure 8b and figure 4c respectively. This observation demonstrates that placing equal emphasis on rare and frequent tokens undermines performance; ensuring minimal uncertainty when predicting frequent tokens is essential.
"we assume input and output embeddings untied from each other
20

---

E Frequent and rare token norm in output embedding
In this section, we explain why frequent tokens acquire larger oututt-embedding norms and logits by deriving and analysing the gradients of the output embedding. Cross-entropy loss with a vocabulary size [V] and hidden dimension size dmodel, using a one-hot target E RVI, the logit vector is f = Fu where h E Aasss is the final hidden state and Enont [ [ui,..., "yv] is the output-embedding matrix. When input and output embeddings are untied, the gradient with respect to each row of Bout decomposes into
ac E.n.
Oc
(pr - 1) h,
Donn; = Pj (i #t,
(2)
where pt softmax(e), [5, 39). Because pt < the start of training, | Foun, ll2 while each non-target row scales only with Pj ||//|2 (py <1/ Thus, every time token appears, its embedding is pulled almost |//]/2 units along +h, whereas each competing row is nudged by factor of pj <1. As tokens recur in the training data, their token embeddings accumulate gradient updates roughly proportional to their counts, so | ll2 grows in line with token-frequency. Since each logit factorizes & ||A||2 Eout, 2l2 cos 0r and frequent tokens not only acquire the largest norms but also align closely with final hidden-state directions cos 02 R 1, they end up with disproportionately large logits whereas rare tokens suffer both smaller norms and larger angular deviations [5, 30]. Although l2 weight decay can slow this norm inflation, they merely dampen the effect rather than remove the underlying frequency norm logit correlation. This saguencc-proportinna amplification explains the empirical observation that output-embedding norms and softmax logits scale with token- loken-frequency in standard language models.
||4||2
OLMo-2 result
Average per word loss
3.4
74
2,500 words coverage
2.4
74
2,500 words coverage
Cross-entropy (nats)
3.3
73
2.3
73
3.2
72
2.2
72
3.1
71
2.1
71
1B
7B
1B
7B
OLMo-2 model size
OLMo-2 model size
(a) Frequent word loss in OLMo-2 models
(b) Global cross-entropy in OLMo-2 models
Figure 9: Figure 9a illustrates that larger 7B model reduces average per-vocabulary loss from 3.39 nats to 3.07 nats while slightly increasing the proportion of loss covered by frequent words from 71% to 72.5%. Figure 9b further demonstrates that Scaling from 1B to 7B reduces the overall cross-entropy from 2.35 nats to 2.12 nats, confirming the same pattern persists in contemporary large-scale language models.
To identify whether the reduction in frequent words loss with increasing model size holds for contemporary large language models, we perform analogous experiments using the OLMo-2 series [41]. Figure 9a and 9b indicate that the average per vocabulary loss falls from 3.39 nats in the 1B parameter model to 3.07 nats in the 7B variant while slightly increasing the proportion of loss covered by 500 frequent words from 71% to 72.5%. Figure 9h further shows that global cross-entropy loss declines from 2.35 nats for OLMo-2 1B to 2.12 nats for OLMo-2 7B. Notably, OLMo-2 employs a much larger vocabulary (c1100K [42]) than Pythia (50304 tokens [7]) and trains larger corpus [41], which helps drive down the average loss on high-frequency words. These results confirm that the trend holds for modern large-scale language models well.
21

---

G Are the gains mainly driven by parameter increases from vocab expansion?
To probe whether gains come from parameter growth with larger vocabularies, we run controlled study comparing a 24K vocabulary model (122M parameters) to near equal size 49K model (124M parameters) to isolate embedding capacity effects. We reduced the hidden dimension from 768 648 and the feed-forward dimension from 2048 to 1728, while maintaining identical model depth which known to have dominant impact on performance [59, 44]. As shown in the table 3, the 49K model incurs higher cross-entropy loss and delivers inferior results on downstream benchmarks (ARC-E, HellaSWAG, PIQA, and SCIQ) compared to 24K counterpart, indicating that simply expanding the embedding layer contributes far less to model expressivity than is often assumed.
Table 3: A 24K vs, size- matched 49K indicates embedding expansion yields limited gains.
Vocab (Params)
CE Loss
Downstream (%)
24K (122M)
3.179
54.73
49K (124M)
3.563
50.68
49K (161M)
3.171
55.44
H Benefits of larger vocabularies hold in tied embedding models?
Several studies show that tying input and output embeddings change pre-training dynamics [24, 32]. We therefore repeat our study with tied embeddings and report average frequent-word word loss and cross-entropy FineWeb-EBl in the table 4. The tied embedding variant shows broadly similar average frequent word loss and ce-loss to the untied baseline.
Table 4: Average frequent-word loss and cross-entropy of tied and untied embedding models
24K
49K
98K
196K
Avg frequent word loss
4.067
4.029
3.991
3.968
Tied
Cross entropy
3.177
3.160
3.146
3.134
Avg frequent word loss Cross entropy
4.074
4.031
4.001
3.977
Untied
3.179
3.163
3.147
3.136
22

---

I Learning rate exploration
We use fixed learning rate 10 for an 85M sarrembboding-paaraeeeee language model, mirroring the learning rate of GPT-3 with an equivalent non-embedding model size [8]. we further conducted learning-ral exploration with four additional learning rates: 2.4 x 10 3, 1.2 10-3 1.5 x 10- 7.5 x 10-6, Table 5 and 6 report the resulting cross-entrops loss and average word loss of frequent 2500 words in fineweb-edu across different vocabulary sizes and learning rates, using same validation dataset described in Section 3.1.
Table 5: Cross-entropy loss across five learning rates.
Cross-Entropy Loss
2.4e-3
1.2e-3
6.0e-4
1.5e-4
7.5e-5
24K
3.156
3.141
3.179
3,484
3.779
49K
3.135
3.114
3.163
3.461
3.751
98K
3.122
3.099
3.147
3.443
3.736
196K
3.111
3.086
3.136
3.429
3.725
Table 6: Average frequent-word loss across five learning rates.
Frequent Word Loss
2.4e-3
1.2e-3
6.0e-4
1.5e-4
7.5e-5
24K
4.033
4.012
4.074
4.431
4.783
49K
3,990
3,970
4.031
4.399
4.749
98K
3.963
3,939
4.001
4.373
4.725
196K
3,942
3.916
3.977
4.352
4.703
Detailed Experimental Setting
In this section, provide detailed configurations of pretraining to reproduce results. The training setup (Table 7) uses a global batch size of 256, weight decay 0.1, and sequence length 2048. Optimization is Adam with cosine learning-rate schedule, 700-step warmup, and weight- initialization scale of 0.02. The model setup (Table 8) covers two different model size: an 85M model with 12 layers and 12 heads (dmmdel 768, dma 2048, dpead 64) and 450M model with 21 layers and 21 heads (dmodel = 1344, dai 3548, dhead = 21). Together, these tables specify the standardized training hyperparameters and the architectural dimensions for both scales.
Table 7: Training configurations. LR Schedule denotes learning-rate schedule.
Global Batch Size
Weight Decay
Sequence Length
Optimizer
256
0.1
2048
AdamW W
LR Schedule
Warmup
Weight Init.
Cosine
700 steps
0.02
Table 8: Model configurations.
Size
Taaaers
71heads
dmodel
dif
dhead
85M
12
12
768
2048
64
450M
21
21
1344
3548
21
23

---

K
How much increasing the model size lower the frequent and rare word loss?
We conducted experiments to quantify scaling effects frequent and rare words loss and demonstrate that scaling model capacity disproportionately benefits the prediction miigh-frequene, tokens relative to low-frequency tokens. Since BPE vocabulary IDss are not token frequency ordered, we cannot directly distinguish common and rare tokens in the Pythia and OLMo-2 training corpora. proxy frequency, we utilize the Google Web Trillion Word Corpus frequency list, which records counts for 333, 333 unique words appearing at least 10, 000 times in a 1.0249 x 10S2 words. We classify the top 2, 500 entries as frequent words and the bottom 10, 000 rare words. As shown in the table 9, although cross-entropy loss declines for both groups with increasing model size, the reduction for frequent tokens is markedly greater. We therefore conclude that the dominant effect of model scaling the lowering of frequent-tose token loss, which in turn drives the overall cross-entropy reduction.
Table 9: Frequent, rare word loss and cross-entropy loss of Pythia and OLMo-2
Model
Params
Frequent word loss
Rare word loss
CF-loss
IB
3.01
4.99
2.51
Pythia
7B
2.55
4.93
2.26
IB
2.60
4.27
2.22
OLMo-2
7B
2.20
4.19
2.00
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We write the contribution of this paper in section intro and section
motivation
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: report the limitation of our work in section 8.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
25

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not propose any theoretical result assumption in this work
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
See the section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
26

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Please see the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //wppn.cc///////////// htlpppullllllleeeeeepppppppppeee Saaeeeee/ooSeeeiiieeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please see section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
27

---

Justification:
We conduct all our experiments with 5 different seeds (even for pre-training the model). See the section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96 CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the supplemental material
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eerrrpp..ee wwwpeewwww....uuuieeeeeeeeeeeeee hewpulice/thicsuuiiiiie inns?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
28

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please the supplemental material
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please section 3.1. We mentioned all the dataset we have used which open dataset from huggingface.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please section 3.1
Guidelines:
29

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:
[NA]
Justification:
No crowdsourcing experiments and research are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No participants are involved.
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required. this research?
Answer: [NA]
Justification: LLM is used only for editing this work (correct the grammatical error etc..)
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https://neur theeurpps.cc/Cooeeeeeeeeeee ccceec/eeeeeeeees2222225 for what should or should not be described.
31


Reviews:
[{"review_id": "Ki0zZIk2Vq", "reviewer": "Reviewer_8nWv", "summary": "This work studies the  impact of vocabulary  size in the tokenization of large language models. The authors fix all hyperparameters but the vocabulary size, and  train models with varying vocabulary size to study various metrics. They examine total loss, per vocabulary loss, and metrics related to tokenization (overlap, segment efficiency) at various scales.  From these experiments, the authors claim that a larger vocabulary size leads to the most common tokens contributing to a majority of loss, and so if the model can learn to be more confident on those the overall loss will be decreased. The provide further evidence that it is these most common tokens that matter most, the authors train a model where with the input and prediction layers constrained to have  unit norm per token, and show that the benefits provided  to the common tokens no longer appears.", "questions": "- Did the authors try  optimizing any other hyperparaemters when scaling  the vocabulary and the model (as in Section 4.3)? Relating to my comments in the above  section, especially when the model is scaled I doubt using the same learning rate is optimal.\n- Is there a motivation for using the divergence from a uniform distribution to characterize heavy tailed-ness? My intuition is that while  a heavy tailed distribution would have a large divergence, there are likely many other distributions  that do as well which would not  be traditionally described as heavy tailed.\n- Could the authors further comment on the relevance of the section on Renyi  entropy? I’m unsure how well it  connects  to the rest of the paper.", "limitations": "The authors address some limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is looking into an interesting question from  an angle that isn’t so common these days. Typically papers just assume scale is better without looking into why that actually is, so I appreciate the general  investigative direction. The paper is written in a very structured way where  questions are stated and answered  in a clear way.\n\nThe main  weakness I see in this paper is that while it’s  conclusions are possible, I feel that the experimentation lacks the rigour to  be  confident the the authors conclusion holds. Their central  claim is that increasing the vocabulary  size essentially makes the most common token classes more dominant. The mechanism this occurs by is that token merges create new, small, classes, with the number of classes growing faster than the  number of tokens in most dominant classes is diminished by merges. The verification of this hypothesis does have a few issues. First, it isn’t really  possible to  hold the model constant  and  change the vocabulary size, as this impacts the number of the parameters in  the embedding and prediction layer. While this could potentially increase the expressivity of the model, a second issue it creates is changing the  optimization dynamics. In  particular, the  authors never change the learning rate in their experiments, and changing the scale of the model can certainly impact what an optimal learning rate  would be. A very related example of this can be seen in  [1], where it is argued that low frequency and high frequency token classes have very different characteristics during optimization. It could be the case that with an appropriate learning rate, both the loss on  the common  and uncommon  tokens goes down, contrary to this work’s findings. \tIt is also not exactly surprising that adding constraints to the embeddings increases loss, adding any constraint on an unconstrained optimization   problem would likely do so, I suspect the loss also increases for  the  least frequent classes under this constraint. In order to have  a stronger argument for the author's hyothesis, it would likely  be necessary to design more rigorious experiments to  better demonstrate a causal relationship, I feel in this case a causal relationship has been claimed somewhat prematurely.\n\nA more minor comment, at times the the words “word” and  “token” are used interchangeable, which as the authors also point out those are not  the same things. I am also somewhat confused by the inclusion of Section 5, it seems somewhat out of place in the context of the rest of the paper.\n\n[1]\n\"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models\"\nNeurIPS (Spotlight), 2024\nhttps://proceedings.neurips.cc/paper_files/paper/2024/file/350e718ff74062b4bac2c6ffd9e1ac66-Paper-Conference.pdf", "was_revised": true, "final_justification": "In my initial review I expressed some concern about sensitivity to hyperparemters, difficulty making comparsions across vocabulary  sizes,  and the impact of constraints on optimizaiton. The authors added some additional experimentation during the rebuttle period to try to address some of these concerns,  which I found helpful for justifying their  claims. While I still feel the results  could be stronger with some better  desigend experiments, I am comfortable with raising my score to a weak accept.", "rebuttal": "Dear reviewer 8nWv,\n\nWe sincerely appreciate your time and effort. We respond to your comment in what follows:\n\n---\n> *1. \"Altering vocabulary axis invariably changes the embedding and prediction-layer parameters, thus it may enhance model expressivity.\"*\n\nIn response to the reviewer’s comment, we conducted a controlled experiment comparing a 24K vocabulary model (122 M total parameters) with a 49K vocabulary variant of comparable size (124 M parameters) to ascertain whether the benefits of vocabulary enlargement arise primarily from increased embedding capacity. To isolate this effect, we reduced the hidden dimension from 768 to 648 and the feed‑forward dimension from 2048 to 1728, while maintaining identical model depth which known to have a dominant impact on performance [2,3]. As shown in the table below, the 49K model incurs higher cross‑entropy loss and delivers inferior results on downstream benchmarks (ARC‑E, HellaSWAG, PIQA, and SCIQ) compared to its 24K counterpart, indicating that simply expanding the embedding layer contributes far less to model expressivity than is often assumed.\n\n**Table A**\n\n| Vocab (Params) | CE Loss | Downstream (%) |\n|---------------:|--------:|---------------:|\n| 24K (122M)     |   3.179 |          54.73 |\n| 49K (124M)     |   3.563 |          50.68 |\n| 49K (161M)     |   3.171 |          55.44 |\n \n\nAdditionally, we compare ce-loss of base model and tied embedding variants. Were increased embedding capacity the primary driver of loss reduction, the tied‑embedding model would be expected to incur substantially higher loss. However, as shown in the table below, the difference in cross‑entropy loss between these two configurations is negligible.\n\n\n**Table B**\n\n| CE Loss        |  24K  |  49K  |  98K  | 196K  |\n|:--------------:|:-----:|:-----:|:-----:|:-----:|\n| Base           | 3.179 | 3.163 | 3.147 | 3.136 |\n| Tied Embedding | 3.177 | 3.160 | 3.146 | 3.134 |\n\n---\n\n> *2.\"6e-4 is an optimal learning rate for every experiments (24K-196K)?\"*\n\nWe selected a learning rate of 6e-4 because it was employed for GPT‑3 125M models [5]. To address the reviewer’s question, we conducted a learning‑rate sweep using four rates\n(2.4e-3, 1.2e-3, 1.5e-4, 7.5e-5). The table below reports the resulting cross‑entropy losses and word-level average per vocab loss (we’ll rename this as average per word loss) of frequent 2500 words in fineweb-edu across different vocabulary sizes and learning rates, using the same validation dataset described in Section 3.1. Although 1.2e-3 looks more optimal learning rate, same loss curve observed across different vocabulary size. \n\n**Table C**\n\n\n| CE Loss | 2.4e-3 | 1.2e-3 | 6.0e-4 | 1.5e-4 | 7.5e-5 |\n|:-------:|-------:|-------:|-------:|-------:|-------:|\n| 24K     |  3.156 |  3.141 |  3.179 |  3.484 |  3.779 |\n| 49K     |  3.135 |  3.114 |  3.163 |  3.461 |  3.751 |\n| 98K     |  3.122 |  3.099 |  3.147 |  3.443 |  3.736 |\n| 196K    |  3.111 |  3.086 |  3.136 |  3.429 |  3.725 |\n\n\n**Table D**\n\n| Freq. Word Loss | 2.4e-3 | 1.2e-3 | 6.0e-4 | 1.5e-4 | 7.5e-5 |\n|:---------------|-------:|-------:|-------:|-------:|-------:|\n| 24K            |  4.033 |  4.012 |  4.074 |  4.431 |  4.783 |\n| 49K            |  3.990 |  3.970 |  4.031 |  4.399 |  4.749 |\n| 98K            |  3.963 |  3.939 |  4.001 |  4.373 |  4.725 |\n| 196K           |  3.942 |  3.916 |  3.977 |  4.352 |  4.703 |\n\n\n---\n\n> *3.\"Constraining the embeddings would also significantly increase rare token loss.\"*\n\nTo address the reviewer’s concern, we conducted an additional experiment to evaluate the effect of embedding‑norm constraints on cross‑entropy loss for both frequent and infrequent tokens. Specifically, we computed the average per‑token loss for the 2,500 most frequent and 2,500 least frequent tokens using 5B character subset of fineweb-edu (our validation dataset, see section 3.1). Gap quantifies the loss difference between the base models and embedding-normalized models.  While the norm constraint does raise loss on rare tokens, we find that the primary driver of the overall increase in cross‑entropy loss is the elevated loss on frequent tokens. This effect occurs because constraining the embedding norm prevents the model from encoding token‑frequency imbalance during training. We provided a detailed explanation in Appendix D. We are grateful to the reviewer for prompting this clarification.\n\n**Table E**\n\n| Vocab Size | Model     | Avg Freq. Loss | Avg Rare Loss |\n|:-----------|:----------|--------------:|--------------:|\n| 24K        | base      |        4.0298 |        4.5019 |\n| 24K        | embednorm |        4.0690 |        4.5094 |\n| 24K        | Gap       |        0.0392 |        0.0075 |\n\n**Table F**\n\n| Vocab Size | Model     | Avg Freq. Loss | Avg Rare Loss |\n|:-----------|:----------|--------------:|--------------:|\n| 49K        | base      |        3.9942 |        4.9424 |\n| 49K        | embednorm |        4.0353 |        4.9438 |\n| 49K        | Gap       |        0.0411 |        0.0014 |\n\n**Table G**\n\n| Vocab Size | Model     | Avg Freq. Loss | Avg Rare Loss |\n|:-----------|:----------|--------------:|--------------:|\n| 98K        | base      |        3.9532 |        5.4145 |\n| 98K        | embednorm |        3.9988 |        5.4210 |\n| 98K        | Gap       |        0.0456 |        0.0065 |\n\n**Table H**\n\n| Vocab Size | Model     | Avg Freq. Loss | Avg Rare Loss |\n|:-----------|:----------|--------------:|--------------:|\n| 196K       | base      |        3.9369 |        6.2493 |\n| 196K       | embednorm |        3.9817 |        6.2581 |\n| 196K       | Gap       |        0.0448 |        0.0088 |\n\n\n---\n\n> *4.\"“word” and “token” are used interchangeable,which as the authors also point out those are not the same things.\"*\n\nWe apologize for having used “word” and “token” interchangeably throughout the manuscript. In the revised version, we will distinguish and apply these terms consistently. Thank you for your feedback.\n\n---\n\n> *5.\"Why use JSD for measuring heavy-tailedness of distribution?\"*\n\nWe use the Jensen–Shannon divergence from a uniform distribution to quantify token frequency imbalance and thereby evaluate the compressibility (or complexity) of the tokenized text alongside Shannon entropy. This work focuses exclusively on token‑frequency imbalance, not on the distribution’s tail shape; references to “heavy‑tailedness” were unintended. In particular, we will remove the phrase “leading to a heavier long‑tail distribution” from Figure 1’s caption. Thank you for drawing our attention to this issue.\n\n---\n\n> *6.\"How section 5 (discussion) connects with rest of paper\"*\n\nSection 5 addresses the apparent discrepancy with prior work [4], which finds that increased token‑frequency imbalance degrades machine translation performance. We wanted to explain what leads to different conclusion in section 5. In machine translation, source and target vocabularies have minimal overlap—terms frequent in English rarely appear in French and vice versa—whereas in monolingual pre‑training, the training corpus and evaluation benchmarks share extensive lexical overlap, including many common high‑frequency tokens. We wanted to point out that in machine translation, gains from reducing frequent token loss in the source language during training do not necessarily translate into improved performance in the target language and this fundamental contrast accounts for the somewhat different conclusions of [4] and our study.\n\n---\n\n### Concluding Remarks\nYour feedback has been instrumental in this process, and we sincerely extend our gratitude for your invaluable insights. Should you have any inquiries or require clarifications about our rebuttal, please don't hesitate to reach out. We are eager to address any concerns and elucidate potential ambiguities in greater depth.\n\n---\n### Reference\n\n[1] “Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling” ICML 2025\n\n[2] “SCALE EFFICIENTLY: INSIGHTS FROM PRE-TRAINING AND FINE-TUNING TRANSFORMERS” ICLR 2022\n\n[3] “The Impact of Depth and Width on Transformer Language Model Generalization” Arxiv\n\n[4] “Tokenization and the Noiseless Channel” ACL 2023\n\n[5] “Language Models are Few-Shot Learners”"}, {"review_id": "TZFSPDqBq5", "reviewer": "Reviewer_5AqE", "summary": "Motivated by the observation that enlarging tokenizer vocabulary tends to improve performance, this paper examines what mechanism actually underlies this effect. Contrary to conventional wisdom that the mechanism is an increase in the number of common words with single token segmentation, the paper finds that improvements are instead driven by an increase in token frequency skew. They empirically and theoretically show that vocabulary size increase does not improve common word segmentation, but that it does increase token frequency skew. Further they show that increased vocabulary size specifically drives improvements on loss for the most common words and that the most common words drive overall loss. To test the causality of token skew on performance they unit normalize the embedding layers (as previous work show that embedding norms track token frequency). Removing this information about token frequency worsens loss and shows a somewhat less clear improvement with increase in vocabulary size. The paper also shows that that normalizing the embeddings removes the impact of vocabulary size on improved downstream performance and explains this by the overlap in frequent words from training and downstream data. lastly they show that loss on most common words also increasingly drives performance as parameter scale increases.", "questions": "- What artifacts specifically will be released?\n    \n- Section 3.5 claims that the impact of token frequency skew on global loss has not been previously studied. However section 4.2 of [Magnusson et al 2024](https://proceedings.neurips.cc//paper_files/paper/2024/hash/760b2d94398aa61468aa3bc11506d9ea-Abstract-Datasets_and_Benchmarks_Track.html) already reports this finding that global loss is dominated by common tokens across various data distributions. The authors do cite this paper for an unrelated reason (the metrics they use), but they should make it clear that observation about common tokens driving global loss is previously known.\n    \n- The intro is motivating and elegantly written but would benefit from laying out some key understandings like why exactly bigger vocabularies increase certainty on the top tokens.\n    \n- line 34: what is meant by “marginal ones” here? and mistakes on “them” is mistakes on rare tokens or marginal ones?\n    \n- line 56: it’s not immediate clear why norm-constrained ablations should have something to do with the frequency. To make this easier to understand you need to mention up front that models tend to learn longer embedding norms for high frequency tokens.\n    \n- figure 1: It would be great if the caption gave a more specific definition of segmentation efficiency in subplot (b). Also how can segmentation efficiency be above 100%? Does this actually represent avg number of tokens per word not percent of words with single token segmentation? Also it’s not clear to me why subplot (c) should be a line graph over n? What is the significance that overlap increases as you consider a larger slice of the top-n words? Or is the point just that the overlap is always pretty high, in which case the compressed y range is misleading and should be zoomed out.\n    \n- line 105: this word-level averaging is very important. I missed this in my first pass where I just read the intro, equations, and figures and that lead to great confusion. I want to make sure I understand it now: Are you calculating the average loss for all occurrences of some unique word? Or is this something more complex like the average loss per word for all words containing a given subword vocabulary type? I assume it’s the former.\n    \n- figure 2:\n    \n    - I think the y axis is actually the avg of avg per-vocab loss over some set of strings right? Clarify what set that is in the caption.\n        \n    - Also what define frequent words coverage in the caption.\n        \n    - Figure 2b doesn’t directly show anything about the loss on infrequent words but the caption claims it does, maybe just plot the loss on infrequent words also?\n        \n    - What is the shaded area showing here? STD over all strings? Multiple runs? Say this in the caption.\n        \n- Fig 3:\n    \n    - why use a box plot here instead of line with shaded area like in 2(a)? And why is the range of avg per-vocab loss so much wider here. Shouldn’t the base values be the same as fig 2(a)?\n        \n    - is the trend still for the loss to decrease on the highest freq words for the embed norm with increased vocab? Yes the loss is higher than for base but I think the mean could be going down still? It’s really hard to see because of the box plots.\n        \n- Fig 4: I’m not sure I understand why a common crawl dump is being used as a new loss measure here? Isn’t it good enough to show that lower loss on frequent words from the pretraining distribution drives downstream acc since there’s a big token overlap? Why bring in yet another token distribution?\n    \n- Fig 5:\n    \n    - again the caption refers to loss on infrequent tokens but doesn’t show it.\n        \n    - This finding is interesting but I wish there was a bit more of an explanation of what the mechanism is for why larger models should lower loss more on common tokens. Loss going down with increase in params is not at all surprising, but it is interesting that the improvement should increasingly be in the common words.", "limitations": "There is a limitations section but it pretty much just discusses that future work can examine which scaling dimensions specifically improve uncertainty in frequent-word predictions. More relevant limitations might be things like addressing the fact that vocab size does still trend with loss improvement in the embedding normalized model.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- This work provides solid evidence to unseat a common assumption about how to select vocabulary size. This finding will likely have practical impacts for all language model developers.\n    \n- The paper uses an excellent balance of theoretical motivation and carefully targeted empirical investigation. It draws a clear hypothesis and conducts just the right experiments to verify it.\n    \n- The pretraining experiments are conducted at a good scale with a reasonable token to parameter ratio. If these are open sourced they can serve as a very useful artifact for studying controlled differences in vocabulary size.\n    \n\nWeaknesses:\n\n- The paper is brilliant but extremely sloppy in many ways. For instance it neglects to say what tokenizer algorithm it uses (I assume BPE) despite that being the main topic of the paper. Terms are frequently referenced before definition. Line plots frequently have very narrow y ranges making minuscule variations look substantial, even though the paper is trying to argue that the value is essentially staying constant. This doesn’t contradict the correctness of their claims or their importance, but I worry the paper will not be as impactful as it deserves to be without more attention to detail in presentation.\n    \n- Some of the use of terminology in the paper is overlapping in a confusing way: e.g. “vocabulary, v,” seems to be the same as a “word” in many instances but vocabulary size is the number of subwords not words. Ideally I’d like something more consistent and precise. Perhaps something like this: words type (the unique words), word (an occurrence of a word type), vocabulary types (the actual symbols in the tokenizer vocab), tokens (occurrences of vocabulary types). Thus we’d call eq 2 Avg per-word-type an so on.\n    \n- The results in figure 3 are slightly less than convincing as there does seem to still be an improvement with increase in vocabulary size even when using embedding normalization. However, the downstream results in figure 4 do make up for this enough to convince me of the authors hypothesis.", "was_revised": true, "final_justification": "This work provides solid evidence to unseat a common assumption about how to select vocabulary size. This finding will likely have practical impacts for all language model developers.\nThe paper uses an excellent balance of theoretical motivation and carefully targeted empirical investigation. It draws a clear hypothesis and conducts just the right experiments to verify it.\nThe pretraining experiments are conducted at a good scale with a reasonable token to parameter ratio. If these are open sourced they can serve as a very useful artifact for studying controlled differences in vocabulary size.\nLastly the authors have a good plan for improving the clarity of their paper that I believe should be easily achievable for a camera ready.", "rebuttal": "Dear reviewer 5AqE,\n\nWe sincerely thank you for your time and feedback. Below, we address your comments in detail:\n\n---\n\n> *1. \"Missing key methodological details (BPE) and too narrow y range exaggerate trivial fluctuations in figure.\"*\n\nWe only use BPE with GPT-2 pre-tokenization rule. We will increase the y ranges in a revised manuscript. Thank you for your feedback. \n\n---\n\n> *2. \"metric name is confusing + how you measure word-level average per vocab loss?\"*\n\nThank you for the suggestion. We will standardize our terminology as follows: **word type** for unique word forms, **word** for individual occurrences of a word type, **vocabulary type** for subword units in the tokenizer, and **token** for individual occurrences of those vocabulary types. We will also revise Equation 2 as **Average per-word loss** to reflect these definitions. Your interpretation of the word-level per-vocabulary loss calculation is correct: in the updated manuscript, Equation 2—renamed **Average per-word loss**—sums the cross-entropy losses of each word’s constituent tokens, and Figure 2 reports the mean of these per-word losses across the 2,500 most frequent words in FineWeb-Edu.\n\n---\n\n> *3.\"Why embednorm does not reflect token frequency imbalance during training? + please explain this clearly in your manuscript\"*\n\nThroughout training, both input and output embeddings reflect token-frequency imbalances, so frequent tokens develop larger embedding norms and thus produce higher logits. Under standard cross-entropy training, the gradient for the output embedding row of a target token $t$ is $\\frac{\\partial \\mathcal{L}}{\\partial E_{\\mathrm{out},t}} = (p_t -1)h$ and  $\\frac{\\partial \\mathcal{L}}{\\partial E_{\\mathrm{out},j}} = p_j h\\ (j\\neq t)$, respectively. $p_t$ is very small at initialization, the gradient of the target token is approximately $-h$, whereas non-target rows get $p_j h$ closer to 0. Consequently, each occurrence of a frequent token pulls its embedding by roughly $\\lVert h\\rVert$ along the hidden state direction, causing its norm to grow in proportion to its count, while rare tokens accrue only negligible norm increases. Although $\\ell_{2}$ weight decay can slow this norm inflation for frequent tokens, its uniform shrinkage causes rare-token embedding norm, which receives negligible positive gradients, to shrink near zero [6]. By constraining a fixed unit $\\ell_{2}$ norm on every embedding row on every step during training, we prevent embedding norms from encoding token-frequency imbalance during training. We have included this explanation in Appendix D, but will present this more clearly in the rest of the paper. Thank you for your guidance. \n\n---\n\n> *4.\"What artifacts specifically will be released?\"*\n\nWe will release a 24K-196K tokenizer trained on Fineweb-edu (10B gpt2 tokens sample), openwebtext, and tokenized text, respectively. We will also release base and embednorm models trained on fineweb-edu (5 different seeds from 24K-196K vocab size).\n \n---\n\n> *5.\"cite Magnusson et al 2024 in section 3.5 properly. what’s section 3.5 novelty against Magnusson et al 2024?\"*\n\nWe appreciate your correction and will incorporate your feedback. While [1] first identified that global loss is dominated by common tokens, our study further demonstrates that increasing the vocabulary further amplifies this dominance. Our work explains that a larger vocabulary reduces the complexity of the tokenized text and makes it more compressible by merging frequent non-iid character patterns into single tokens and boosting their relative frequency. This increased compressibility enables the model to learn and predict these frequent patterns more easily (even under an unigram assumption), which in turn lowers the cross‑entropy loss.\n\n---\n\n> *6. \"Why exactly bigger vocabularies increase certainty on the top tokens?\"*\n\nThank you for this insightful question. Expanding the vocabulary increases the relative frequency of common tokens as the total token count of the training corpus reduces. Even if frequent character strings are segmented as a single token (Figure 1b), expanding the vocabulary size further decreases the overall token count, which keeps increasing the relative frequency of common tokens. This amplified token‑frequency imbalance facilitates the language model’s ability to learn and predict these frequent tokens, as they occur more often with larger vocabularies. We provided a detailed analysis in Appendix A.\n\n---\n\n> *7. \"line 34: what is meant by “marginal ones” and mistakes on “them”?\"*\n\nThe marginal probabilities represent each token’s relative frequency, and “mistakes on them” indicates mistakes on rare tokens. Because the conditional probability of a rare token is very low, mispredicting such tokens incurs a large penalty.\n\n---\n\n> *8. \"Explain clearly what is the definition of segmentation efficiency. What do you want to show in figure 1c?\"*\n\nWe apologize for the previous ambiguity in our definition of segmentation efficiency. Segmentation efficiency gauges the average token count per word, that is, how many tokens the tokenizer needs, on average, to encode each of the 2,500 most frequent words. In Figure 1c, we aimed to illustrate that the frequent words are highly overlapping across datasets by showing that the overlap between the top N words of fineweb-edu and openwebtext remains consistently high. In response to the reviewer’s suggestion, we will expand the y‑axis range for greater clarity. Thank you for your advice.\n\n---\n\n> *9. \"What frequent words have you used in figure 2?\"*\n\nWe use frequent 2,500 words in the fineweb-edu.\n\n---\n\n> *10. \"define frequent words coverage in the caption.\"*\n\nWe define “frequent‑words coverage” as the proportion of the model’s total loss attributable to the tokens comprising the 2,500 most frequent words in the FineWeb‑Edu corpus. \n\n---\n\n> *11. \"Maybe just plot the loss on infrequent words in figure 2b?\"*\n\nIn Figure 2, we will present the average per-vocab loss for infrequent tokens for each tokenizer instead of rare words, since rare words vary across datasets and most of them are not represented by a single token.\n\n---\n\n> *12. \"What is the shaded area showing here? STD over all strings?\"*\n\nShaded area showing 5 different runs on different random seeds\n\n---\n\n> *13. \"Why use a box plot and why average per vocab loss is so wide here?\"*\n\nWe employ box plots to depict the variability in loss for the 2,500 most frequent words: the shaded box spans one standard deviation above and below the mean, and the central black line corresponds exactly to the average per‑word loss reported in Figure 2a. To enhance interpretability, we will substitute these box plots with a line graph.\n\n---\n\n> *14. \"Frequent word loss of embednorm decreases as vocab scales up? Hard to see in box plots\"*\n\nFor clarity, the following table reports the exact average frequent word loss for both the baseline model and the embedding‑norm variant.\n\n**Table A**\n\n| Model      | 24K   | 49K   | 98K   | 196K  |\n|------------|-------:|-------:|-------:|-------:|\n| Base       | 4.074  | 4.031  | 4.001  | 3.977  |\n| EmbedNorm  | 4.205  | 4.194  | 4.183  | 4.179  |\n\n---\n\n> *15. \"why you use a common crawl dump? Isn’t it enough to show common token overlap between Fineweb-edu and downstream benchmarks?\"*\n\nWe want to verify on an independent validation set that expanding the vocabulary leads to reduced cross‑entropy loss by enabling the model to better capture high‑frequency words. Moreover, empirical findings in [2] report a strong negative Pearson correlation (–0.93) between CC‑Main‑2023‑40 cross‑entropy loss and reasoning benchmark performance, further motivating its use as a predictive validation set.\n\n---\n\n> *16. \"again the caption refers to loss on infrequent tokens but doesn’t show it\"*\n\nWe will revise Figure 5 to match the presentation style of Figure 2. Following your recommendation, Figure 5a will illustrate the average loss for infrequent tokens. \n\n---\n\n> *17. \"Do you have any explanation on why larger models should lower loss more on common tokens? \"*\n\nIt is also an intriguing research direction to understand why larger models lower loss on common tokens. We hypothesize that this effect derives from **simplicity bias** [3,4,5]: the inductive bias of deep networks to favor lower complexity solutions, and the bigger the model, the stronger the bias. [3] demonstrates that larger GPT‑3 models produce output text with lower complexity, which mirrors that of natural data. Moreover, [4] shows that simplicity bias increases with depth, and [5] demonstrates that the representation of models converges as model size grows. Given a training corpus that is both sufficiently large and diverse, simplicity biases enable larger models to more effectively capture the shared, high‑frequency patterns across natural text, thereby achieving lower cross‑entropy loss.\n\n---\n### Concluding Remarks\nThank you again for your thoughtful and detailed feedback. Your comments have helped us clarify key methodological details, tighten our terminology, and strengthen both our analyses and visualizations. Should any further questions arise or additional experiments be desired, we would be delighted to address them. Your insights have greatly improved our paper, and we look forward to submitting a stronger revision.\n\n\n---\n\n### References\n\n[1] “Paloma:  A Benchmark for Evaluating Language Model Fit”  Neurips 2024 (benchmark)\n\n[2] “compression represents intelligence Linearly” COLM 2024 \n\n[3] “The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning” ICML 2024\n\n[4] “The Low-Rank Simplicity Bias in Deep Networks” TMLR 2023\n\n[5] “The Platonic Representation Hypothesis” ICML 2024\n\n[6] “Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models” EMNLP 2024"}, {"review_id": "dvmJze6ceL", "reviewer": "Reviewer_Md1b", "summary": "The paper studies the effect on rare and frequent tokens on LLMs in pretraining. The analysis of three questions related to Skew vs Segmentation, Loss Decomposition, and Corpus Robustness.", "questions": "1. For the Parameter Scaling Recovery: Can you explain if the same reduction is found on the tail tokens as well? I think there are some confounding variables here that are not accounted for?", "limitations": "The paper contains a section on limitations, but I am not sure this is complete. Some key limitations are also the weaknesses I have mentioned earlier.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n - Important and Interesting Area of Study\n - By using the research questions, the paper grounds the work well.\n - Insight from the paper is interesting if it can be carried over to downstream benchmarks.\n\nWeaknesses:\n- Some Assumptions may not be reasonable: (1) the assumption of not tied weights is as ubiquitous as the authors may believe (Qwen3, for example), (2) the authors find that increasing the model size lowers loss on frequent tokens and thus has better performance. However, this might be just cause the model over the whole vocabulary is performing better. \n- No results on downstream benchmarks such as MMLU, etc.\n\n- Writing: \n\n     1. Minor: some of the sentences can be shortened. For example, \"Before diving into settings and metrics, we spell out the concrete questions that steer our empirical study\" can be rewritten to \"We analyze the following concrete questions:\" or something to that effect.\n\n     2. Figure 1: (a) Can you add the line for uniform distribution? From this figure, the caption claim of moving further away from uniform cannot be determined. (b and c) The y-axes are very misleading. It would be good, at least for c, to make them from 0 to 1 or 0.5 to 1.\n\n     3. Notation/Metrics: The notion is a little confusing, why is Global Cross Entropy Loss defined through the Average Per-Vocabulary Loss and Total Loss. Also, $t$ is never defined. I took it to mean the tokens of a document in N. Then, v is defined as a vocabulary, but I think you mean an element in the vocabulary.", "was_revised": true, "final_justification": "Thank you for your response! As my weaknesses have been addressed, I raised my score to a weak accept. The main reason why I have not gone higher is that the conclusion of the paper does not change the current guidance for building tokenizers and models. It does indeed help use explain the current relationship between frequent and rare tokens.", "rebuttal": "Dear reviewer Md1b,\n\nWe are grateful for your time and thoughtful feedback. In the sections that follow, we provide detailed responses to your comments.\n\n---\n\n> *1. \"Why conduct experiments only with an untied embedding model? Provide additional experiments with a tied embedding setting.\"*\n\nWe adopt untied input and output embeddings because contemporary language models typically do not share these parameters, and this design has been employed by several pre-training interpretability studies [1,2,3] to facilitate clearer analysis. Nevertheless, in response to the reviewer’s comment, we conducted additional experiments with a tied embedding architecture. The table below presents the average frequent word loss and cross-entropy loss on Fineweb-edu. Tied embedding shows a similar tendency to untied variants.\n \n\n**Table A**\n\n|                          | 24K    | 49K    | 98K    | 196K   |\n|--------------------------|--------:|--------:|--------:|--------:|\n| AVG frequent word loss   | 4.067  | 4.029  | 3.991  | 3.968  |\n| cross entropy            | 3.177  | 3.160  | 3.146  | 3.134  |\n\n---\n\n> *2. \"No results on downstream benchmarks such as MMLU\"*\n\nThank you for raising this point. We evaluated our models on MMLU, but their scores clustered near the random-guess baseline of 25 %, providing essentially no discriminatory power at our current compute scale. Instead, as shown in the table below, we broadened our evaluation to lighter benchmarks—OpenBookQA, Lambada (openai), WinoGrande, and BLiMP— due to the scarcity of benchmarks that allow fair downstream comparisons for small language models [3]. The table below shows that the larger vocabulary size improves downstream performance on these benchmarks. These suites have proven far more sensitive to the effects under study. If there is a particular additional benchmark you feel would shed further light on our claims, we would be happy to incorporate it in a revised version.\n\n**Table B**\n\n| Benchmark                 | 24K   | 49K   | 98K   | 196K  |\n|--------------------------|-------:|-------:|-------:|-------:|\n| OpenBookQA               | 18.2  | 18.6  | 19.0  | 19.8  |\n| Lambada (openai, acc)     | 23.25 | 23.85 | 23.93 | 24.05 |\n| WinoGrande               | 49.88 | 51.7  | 51.93 | 52.01 |\n| BLiMP                    | 76.82 | 77.7  | 78.28 | 78.42 |\n\n\n---\n\n> *3. \"Write Sentence concisely + add uniform distribution line in figure 1a\"*\n\nWe will shorten our sentences and, in response to your suggestion, add a reference line for the uniform distribution in Figure 1a and expand the y‑axis range to 0.5–1 in both Figures 1b and 1c. Thank you for your feedback.\n\n---\n\n> *4. \"Why is Global Cross-Entropy Loss defined through the Average Per-Vocabulary Loss and Total Loss + definition issue\"*\n\nGlobal cross‑entropy loss is the cross‑entropy computed over the entire training or validation corpus. Because cross‑entropy quantifies the average uncertainty in next‑token prediction, it can be expressed as a weighted sum of each vocabulary’s mean loss, with weights proportional to their relative frequencies. In the revised manuscript, we will clarify that t denotes a sentence drawn from the N documents and v represents a specific vocabulary entry in the tokenizer. Thank you for your feedback.\n\n---\n\n> *5. \"Increasing the model size would lower the whole token loss, including rare tokens?\"*\n\nIn response to the reviewer’s query, we conducted additional experiments demonstrating that scaling model capacity disproportionately benefits the prediction of high‑frequency tokens relative to low‑frequency tokens. Consequently, the reduction in frequent‑token loss emerges as the principal contributor to the overall decrease in cross‑entropy loss.\nBecause BPE vocabulary IDs do not reflect token frequency, we cannot directly distinguish common and rare tokens in the Pythia and OLMo‑2 training corpora. Instead, we utilize the Google Web Trillion Word Corpus frequency list, which records counts for 333,333 unique words appearing at least 10,000 times in a 1.0249 × 10^12 words. We classify the top 2,500 entries as “frequent” words and the bottom 10,000 as “rare” words. As shown in the table below, although cross‑entropy loss declines for both groups with increasing model size, the reduction for frequent tokens is markedly greater. We therefore conclude that the dominant effect of model scaling is the lowering of frequent‑token loss, which in turn drives the overall cross‑entropy reduction.\n\n**Table C**\n\n| Pythia | frequent word loss | rare word loss | CE-loss |\n|:-----:|--------------------------------------:|----------------------------------:|--------------------------:|\n| 1B    |                                 3.01  |                              4.99 |                      2.51 |\n| 7B    |                                 2.55  |                              4.93 |                      2.26 |\n\n**Table D**\n\n| OLMo-2| frequent word loss | rare word loss | CE-loss |\n|:-----:|--------------------------------------:|----------------------------------:|--------------------------:|\n| 1B    |                                 2.60  |                              4.27 |                      2.22 |\n| 7B    |                                 2.20  |                              4.19 |                      2.00 |\n\n---\n\n### Concluding Remarks\n\nThank you for your close reading and constructive criticism. Your comments have prompted several concrete improvements that we believe make the paper clearer and more convincing. To address your main concerns, we will:\n\n1. **Tied-embedding validation** – replicate all core experiments with tied embeddings to verify that the vocabulary-scaling trends hold.\n2. **Broader downstream tests** – add evaluations on OpenBookQA, LAMBADA, WinoGrande, and BLiMP so that readers can observe consistent gains from larger vocabularies, even at our small-model scale.\n3. **Clearer writing and figures** – shorten long sentences, add a uniform-distribution line to Figure 1 for easier comparison, and explain the global-loss formula in plain language.\n4. **Scaling analysis** – extend our study from 1B to 7B parameters (Pythia, OLMo-2) to show that increased model size primarily reduces frequent-token loss, driving the overall cross-entropy drop.\n\nWe believe these revisions directly address the issues you raised and strengthen the paper’s generality and clarity. We remain eager to incorporate any further suggestions that could make the work even more convincing.\n\n\n---\n\n### References\n\n[1] “Paloma:  A Benchmark for Evaluating Language Model Fit”  Neurips 2024\n \n[2] “Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling” ICML 2023\n\n[3] “Polypythias: Stability and Outliers Across Fifty Language Model Pre-Traning Runs” ICLR 2025"}, {"review_id": "QFXCvLt3ob", "reviewer": "Reviewer_5NL9", "summary": "The paper investigates why enlarging the tokenizer vocabulary continues to reduce perplexity even after every common word is already a single token. Fixing other variables, the authors expand the vocabulary and show that the only systematic change is a sharper long-tail token-frequency imbalance - segmentation efficiency saturates by 24k.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper provides controlled study, isolating vocabulary as the only independent variable, avoiding confounds common in scaling/interpretability work.  \n- Loss-decomposition analysis – per-word breakdown pinpoints where perplexity gains come from. Causal ablation study strengthens the argument.\n- Cross-corpus validation confirms identical trends on FineWeb-Edu vs. OpenWebText.\n\nWeaknesses\n- Experiments use an 85M-parameter model. Generality to ≥7B-parameter LLMs is asserted via indirect Pythia/OLMo plots.\n- The paper acknowledges higher loss on tail words but it would make the paper stronger if authors can provides more in-depth error study or harm-reduction strategy for rare-token degradation. \n- Ethical discussion is a bit thin as exploiting frequency imbalance may entrench majority-language bias. Any discussions on potential mitigations could be helpful.", "was_revised": true, "final_justification": "I have carefully reviewed the author's rebuttal responses and their discussions with other reviewers. \n\nTheir responses have answered most of my previous concerns, especially about majority-language bias and higher loss on tail words. I appreciate the computational constraint in completing the experiments with larger models, but glad that the authors confirm that the general trend is the same. I maintain my positive rating for the work. I maintain my positive rating of the paper.", "rebuttal": "Dear reviewer 5NL9,\n\nThank you for your time and insightful feedback. Below, we offer comprehensive responses to each of your comments.\n\n---\n\n> *1. \"Experiments use an 85M-parameter model. Generality to ≥7B-parameter LLMs is asserted via indirect Pythia/OLMo plots.\"*\n\nWe are currently training 450M parameter model variants and based on their training‑loss trajectories and learning dynamics, we confirm that they exhibit trends analogous to those observed in the 85M parameter models. However, due to constraints on time and GPU resources, we were unable to complete the pre-training in time. We will include this experiment in the revised manuscript. \n\n\n---\n\n> *2. \"Exploiting frequency imbalance may entrench majority-language bias. Any discussions on potential mitigations could be helpful.\"*\n\nTo address the majority language bias, one must curate corpora with more uniform language distributions achieved through oversampling underrepresented languages or undersampling majority‑language data. Additionally, developing optimization algorithms that are resilient to frequency skew such as leveraging adaptive methods like Adam in place of vanilla SGD offers a promising strategy for mitigating the effects of token‑frequency imbalance [3].\n\n---\n\n> *3. \"The paper acknowledges higher loss on tail words but it would make the paper stronger if authors can provides more in-depth error study or harm-reduction strategy for rare-token degradation.\"*\n\nExploring harm‑reduction strategies that replicate the benefits of vocabulary expansion is a compelling research direction. A prototypical example is Superbpe [2], which employs a two‑stage BPE algorithm: in the first stage (up to a threshold vocabulary size T), it operates identically to standard BPE, while in the second stage it abandons whitespace pre‑tokenization. By permitting merges across former whitespace boundaries, Superbpe directs subsequent merges toward frequent tokens, thereby preventing the introduction of new rare tokens as the vocabulary expands, prevents further increase of rare token loss. \n\nTo validate whether enlarging vocabulary size and Superbpe variants exploit the same benefits: leading to more compressible, lower complexity tokenized text, we draw on Kolmogorov complexity, which measures the minimal description length of a bitstring and thus captures its inherent compressibility [1]. For a sequence $X^{N}$ tokenized by a BPE, an asymptotic upper bound on its Kolmogorov complexity is given by\n\n$K(X^{N}) \\le NH(p) + V \\log_{2} N + O(\\log N)$\n\nwhere $N$ is the total token count of the entire text, $H(p)$ the shannon entropy of the token distribution, and the $V \\log_{2} N$ accounts for the prefix-free encoding of the token (i.e., token frequency table). Because modern language models are trained on corpora comprising billions of tokens, the $NH(p)$ dominates, yielding the practical approximation. \n\n$K(X^{N}) \\approx NH(p)$\n\nWe adopt this upper bound as a tractable proxy for data complexity, as actual Kolmogorov complexity is uncomputable. The table below reports the estimated Kolmogorov complexities for the 45.7 billion‑byte FineWeb‑Edu corpus and average downstream task performance reported by [2], comparing standard BPE at various vocabulary sizes with Superbpe variants. If no tokenizer is applied and the text is modeled as an i.i.d. source over individual bytes, the Kolmogorov complexity $K(X^{N})$ would be 45.7B bytes. These results demonstrate that both vocabulary expansion and the Superbpe variants leverage the same benefit: they reduce the complexity of the tokenized text by segmenting common character sequences into single tokens, thereby facilitating the model to capture and learn frequent patterns in the training data more easily.\n\n**Table A**\n\n| BPE  (bytes)        | 24K           | 49K            | 98K            | 196K           |\n|---------------|---------------|----------------|----------------|----------------|\n| $K(X^{N})$      | 10.74B | 10.43B | 10.23B | 10.16B |\n\n**Table B**\n\n| SuperBPE  (bytes)    | 20K (bpe)      | 20K (t=180K)    | 20K (t=160K)    | 20K (t=80K)     |\n|---------------|---------------|----------------|----------------|----------------|\n| $K(X^{N})$ | 10.21B | 10.03B | 10.05B | 10.10B |\n| Avg Downstream [2] | 39.8          | 43.8           | 43.4           | 42.9           |\n\n---\n### Concluding Remarks\nThank you for highlighting the importance of scalability, fairness, and rare-token robustness. Your comments prompted us to (i) extend our experiments to 450 M-parameter models now in training, confirming that the key trends observed with 85 M parameters persist at larger scales; (ii) add a focused discussion of corpus-balancing and adaptive-optimizer strategies that can mitigate majority-language bias; and (iii) deepen our error analysis on tail words, contrasting vocabulary-expansion with SuperBPE as complementary harm-reduction approaches. We believe these additions substantially strengthen the paper’s claims and practical relevance.\n\nWe appreciate your insights and remain eager to pursue any further analyses or experiments that would make the study more useful to the community.\n\n---\n\n### References\n\n[1] “The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning” ICML 2024\n\n[2] “SuperBPE: Space Travel for Language Models” COLM 2025\n\n[3] “Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models” Neurips 2024"}]

Author rebuttal:

Meta-review: The paper investigates the role of vocabulary size in LLMs in a controlled setting, and identifies "sharper frequency imbalance" as a key factor behind the benefits of larger vocabularies: even when the most frequent words remain their own tokens, increasing vocab size makes the rarer tokens less frequent in the data, thus making it easier to learn the more frequent tokens more efficiently, leading to overall improvements in learning efficiency. These are interesting findings, and the experiments were carried carefully. Overall, the reviewers were positive about the paper, and I recommend acceptance.

Please make sure to address the presentation issues raised by reviewer 5AqE in the final revision, and please do release the artifacts mentioned.
Decision: Accept (poster)