Title: Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Token Frequency Imbalance

Vocabulary Size

Pre-training Language Model

Information Theory

Tokenization
Abstract: Large language models are trained with tokenizers that map text to a fixed vocabulary, yet the resulting token distribution is highly imbalanced: a few words dominate the stream while most occur rarely.  Recent practice favours ever-larger vocabularies, but it is unclear whether the benefit comes from better word segmentation or from amplifying this frequency skew. To this end, we perform a controlled study that scales the vocabulary of a constant-size Transformer from 24K to 196K symbols while holding data, compute and optimisation unchanged.  Above 24K every common word is already a single token, so further growth only increases imbalance.  Word-level loss decomposition shows that larger vocabularies reduce cross-entropy almost exclusively by lowering uncertainty on the ~$2,500$ most frequent words, even though loss on the rare tail rises. Same frequent words cover roughly $80\%$ of tokens in downstream benchmarks, this training advantage transfers intact.  We further show that enlarging model parameters with a fixed tokenizer yields the same frequent-word benefit, revealing a shared mechanism behind vocabulary and model scaling. Our results recast “bigger vocabularies help” as “sharper frequency imbalance helps,” offering a simple, principled knob for tokenizer–model co-design and clarifying the loss dynamics that govern language-model scaling in pre-training.


Peer reviews:
comments: SUMMARY:
The paper investigates why enlarging the tokenizer vocabulary continues to reduce perplexity even after every common word is already a single token. Fixing other variables, the authors expand the vocabulary and show that the only systematic change is a sharper long-tail token-frequency imbalance - segmentation efficiency saturates by 24k.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper provides controlled study, isolating vocabulary as the only independent variable, avoiding confounds common in scaling/interpretability work.  
- Loss-decomposition analysis – per-word breakdown pinpoints where perplexity gains come from. Causal ablation study strengthens the argument.
- Cross-corpus validation confirms identical trends on FineWeb-Edu vs. OpenWebText.

Weaknesses
- Experiments use an 85M-parameter model. Generality to ≥7B-parameter LLMs is asserted via indirect Pythia/OLMo plots.
- The paper acknowledges higher loss on tail words but it would make the paper stronger if authors can provides more in-depth error study or harm-reduction strategy for rare-token degradation. 
- Ethical discussion is a bit thin as exploiting frequency imbalance may entrench majority-language bias. Any discussions on potential mitigations could be helpful.

QUESTIONS:
Please see above.

LIMITATIONS:
Please see above.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the effect on rare and frequent tokens on LLMs in pretraining. The analysis of three questions related to Skew vs Segmentation, Loss Decomposition, and Corpus Robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - Important and Interesting Area of Study\n - By using the research questions, the paper grounds the work well.\n - Insight from the paper is interesting if it can be carried over to downstream benchmarks.\n\nWeaknesses:\n- Some Assumptions may not be reasonable: (1) the assumption of not tied weights is as ubiquitous as the authors may believe (Qwen3, for example), (2) the authors find that increasing the model size lowers loss on frequent tokens and thus has better performance. However, this might be just cause the model over the whole vocabulary is performing better. \n- No results on downstream benchmarks such as MMLU, etc.\n\n- Writing: \n\n     1. Minor: some of the sentences can be shortened. For example, \"Before diving into settings and metrics, we spell out the concrete questions that steer our empirical study\" can be rewritten to \"We analyze the following concrete questions:\" or something to that effect.\n\n     2. Figure 1: (a) Can you add the line for uniform distribution? From this figure, the caption claim of moving further away from uniform cannot be determined. (b and c) The y-axes are very misleading. It would be good, at least for c, to make them from 0 to 1 or 0.5 to 1.\n\n     3. Notation/Metrics: The notion is a little confusing, why is Global Cross Entropy Loss defined through the Average Per-Vocabulary Loss and Total Loss. Also, $t$ is never defined. I took it to mean the tokens of a document in N. Then, v is defined as a vocabulary, but I think you mean an element in the vocabulary.\n\nQUESTIONS:\n1. For the Parameter Scaling Recovery: Can you explain if the same reduction is found on the tail tokens as well? I think there are some confounding variables here that are not accounted for?\n\nLIMITATIONS:\nThe paper contains a section on limitations, but I am not sure this is complete. Some key limitations are also the weaknesses I have mentioned earlier.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the observation that enlarging tokenizer vocabulary tends to improve performance, this paper examines what mechanism actually underlies this effect. Contrary to conventional wisdom that the mechanism is an increase in the number of common words with single token segmentation, the paper finds that improvements are instead driven by an increase in token frequency skew. They empirically and theoretically show that vocabulary size increase does not improve common word segmentation, but that it does increase token frequency skew. Further they show that increased vocabulary size specifically drives improvements on loss for the most common words and that the most common words drive overall loss. To test the causality of token skew on performance they unit normalize the embedding layers (as previous work show that embedding norms track token frequency). Removing this information about token frequency worsens loss and shows a somewhat less clear improvement with increase in vocabulary size. The paper also shows that that normalizing the embeddings removes the impact of vocabulary size on improved downstream performance and explains this by the overlap in frequent words from training and downstream data. lastly they show that loss on most common words also increasingly drives performance as parameter scale increases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This work provides solid evidence to unseat a common assumption about how to select vocabulary size. This finding will likely have practical impacts for all language model developers.\n    \n- The paper uses an excellent balance of theoretical motivation and carefully targeted empirical investigation. It draws a clear hypothesis and conducts just the right experiments to verify it.\n    \n- The pretraining experiments are conducted at a good scale with a reasonable token to parameter ratio. If these are open sourced they can serve as a very useful artifact for studying controlled differences in vocabulary size.\n    \n\nWeaknesses:\n\n- The paper is brilliant but extremely sloppy in many ways. For instance it neglects to say what tokenizer algorithm it uses (I assume BPE) despite that being the main topic of the paper. Terms are frequently referenced before definition. Line plots frequently have very narrow y ranges making minuscule variations look substantial, even though the paper is trying to argue that the value is essentially staying constant. This doesn’t contradict the correctness of their claims or their importance, but I worry the paper will not be as impactful as it deserves to be without more attention to detail in presentation.\n    \n- Some of the use of terminology in the paper is overlapping in a confusing way: e.g. “vocabulary, v,” seems to be the same as a “word” in many instances but vocabulary size is the number of subwords not words. Ideally I’d like something more consistent and precise. Perhaps something like this: words type (the unique words), word (an occurrence of a word type), vocabulary types (the actual symbols in the tokenizer vocab), tokens (occurrences of vocabulary types). Thus we’d call eq 2 Avg per-word-type an so on.\n    \n- The results in figure 3 are slightly less than convincing as there does seem to still be an improvement with increase in vocabulary size even when using embedding normalization. However, the downstream results in figure 4 do make up for this enough to convince me of the authors hypothesis.\n\nQUESTIONS:\n- What artifacts specifically will be released?\n    \n- Section 3.5 claims that the impact of token frequency skew on global loss has not been previously studied. However section 4.2 of [Magnusson et al 2024](https://proceedings.neurips.cc//paper_files/paper/2024/hash/760b2d94398aa61468aa3bc11506d9ea-Abstract-Datasets_and_Benchmarks_Track.html) already reports this finding that global loss is dominated by common tokens across various data distributions. The authors do cite this paper for an unrelated reason (the metrics they use), but they should make it clear that observation about common tokens driving global loss is previously known.\n    \n- The intro is motivating and elegantly written but would benefit from laying out some key understandings like why exactly bigger vocabularies increase certainty on the top tokens.\n    \n- line 34: what is meant by “marginal ones” here? and mistakes on “them” is mistakes on rare tokens or marginal ones?\n    \n- line 56: it’s not immediate clear why norm-constrained ablations should have something to do with the frequency. To make this easier to understand you need to mention up front that models tend to learn longer embedding norms for high frequency tokens.\n    \n- figure 1: It would be great if the caption gave a more specific definition of segmentation efficiency in subplot (b). Also how can segmentation efficiency be above 100%? Does this actually represent avg number of tokens per word not percent of words with single token segmentation? Also it’s not clear to me why subplot (c) should be a line graph over n? What is the significance that overlap increases as you consider a larger slice of the top-n words? Or is the point just that the overlap is always pretty high, in which case the compressed y range is misleading and should be zoomed out.\n    \n- line 105: this word-level averaging is very important. I missed this in my first pass where I just read the intro, equations, and figures and that lead to great confusion. I want to make sure I understand it now: Are you calculating the average loss for all occurrences of some unique word? Or is this something more complex like the average loss per word for all words containing a given subword vocabulary type? I assume it’s the former.\n    \n- figure 2:\n    \n    - I think the y axis is actually the avg of avg per-vocab loss over some set of strings right? Clarify what set that is in the caption.\n        \n    - Also what define frequent words coverage in the caption.\n        \n    - Figure 2b doesn’t directly show anything about the loss on infrequent words but the caption claims it does, maybe just plot the loss on infrequent words also?\n        \n    - What is the shaded area showing here? STD over all strings? Multiple runs? Say this in the caption.\n        \n- Fig 3:\n    \n    - why use a box plot here instead of line with shaded area like in 2(a)? And why is the range of avg per-vocab loss so much wider here. Shouldn’t the base values be the same as fig 2(a)?\n        \n    - is the trend still for the loss to decrease on the highest freq words for the embed norm with increased vocab? Yes the loss is higher than for base but I think the mean could be going down still? It’s really hard to see because of the box plots.\n        \n- Fig 4: I’m not sure I understand why a common crawl dump is being used as a new loss measure here? Isn’t it good enough to show that lower loss on frequent words from the pretraining distribution drives downstream acc since there’s a big token overlap? Why bring in yet another token distribution?\n    \n- Fig 5:\n    \n    - again the caption refers to loss on infrequent tokens but doesn’t show it.\n        \n    - This finding is interesting but I wish there was a bit more of an explanation of what the mechanism is for why larger models should lower loss more on common tokens. Loss going down with increase in params is not at all surprising, but it is interesting that the improvement should increasingly be in the common words.\n\nLIMITATIONS:\nThere is a limitations section but it pretty much just discusses that future work can examine which scaling dimensions specifically improve uncertainty in frequent-word predictions. More relevant limitations might be things like addressing the fact that vocab size does still trend with loss improvement in the embedding normalized model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the  impact of vocabulary  size in the tokenization of large language models. The authors fix all hyperparameters but the vocabulary size, and  train models with varying vocabulary size to study various metrics. They examine total loss, per vocabulary loss, and metrics related to tokenization (overlap, segment efficiency) at various scales.  From these experiments, the authors claim that a larger vocabulary size leads to the most common tokens contributing to a majority of loss, and so if the model can learn to be more confident on those the overall loss will be decreased. The provide further evidence that it is these most common tokens that matter most, the authors train a model where with the input and prediction layers constrained to have  unit norm per token, and show that the benefits provided  to the common tokens no longer appears.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is looking into an interesting question from  an angle that isn’t so common these days. Typically papers just assume scale is better without looking into why that actually is, so I appreciate the general  investigative direction. The paper is written in a very structured way where  questions are stated and answered  in a clear way.\n\nThe main  weakness I see in this paper is that while it’s  conclusions are possible, I feel that the experimentation lacks the rigour to  be  confident the the authors conclusion holds. Their central  claim is that increasing the vocabulary  size essentially makes the most common token classes more dominant. The mechanism this occurs by is that token merges create new, small, classes, with the number of classes growing faster than the  number of tokens in most dominant classes is diminished by merges. The verification of this hypothesis does have a few issues. First, it isn’t really  possible to  hold the model constant  and  change the vocabulary size, as this impacts the number of the parameters in  the embedding and prediction layer. While this could potentially increase the expressivity of the model, a second issue it creates is changing the  optimization dynamics. In  particular, the  authors never change the learning rate in their experiments, and changing the scale of the model can certainly impact what an optimal learning rate  would be. A very related example of this can be seen in  [1], where it is argued that low frequency and high frequency token classes have very different characteristics during optimization. It could be the case that with an appropriate learning rate, both the loss on  the common  and uncommon  tokens goes down, contrary to this work’s findings. \tIt is also not exactly surprising that adding constraints to the embeddings increases loss, adding any constraint on an unconstrained optimization   problem would likely do so, I suspect the loss also increases for  the  least frequent classes under this constraint. In order to have  a stronger argument for the author's hyothesis, it would likely  be necessary to design more rigorious experiments to  better demonstrate a causal relationship, I feel in this case a causal relationship has been claimed somewhat prematurely.\n\nA more minor comment, at times the the words “word” and  “token” are used interchangeable, which as the authors also point out those are not  the same things. I am also somewhat confused by the inclusion of Section 5, it seems somewhat out of place in the context of the rest of the paper.\n\n[1]\n\"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models\"\nNeurIPS (Spotlight), 2024\nhttps://proceedings.neurips.cc/paper_files/paper/2024/file/350e718ff74062b4bac2c6ffd9e1ac66-Paper-Conference.pdf\n\nQUESTIONS:\n- Did the authors try  optimizing any other hyperparaemters when scaling  the vocabulary and the model (as in Section 4.3)? Relating to my comments in the above  section, especially when the model is scaled I doubt using the same learning rate is optimal.\n- Is there a motivation for using the divergence from a uniform distribution to characterize heavy tailed-ness? My intuition is that while  a heavy tailed distribution would have a large divergence, there are likely many other distributions  that do as well which would not  be traditionally described as heavy tailed.\n- Could the authors further comment on the relevance of the section on Renyi  entropy? I’m unsure how well it  connects  to the rest of the paper.\n\nLIMITATIONS:\nThe authors address some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-08"}