Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Authors: chaofan  tao
qian  liu
longxu  dou
niklas  muennighoff
zhongwei  wan
ping  luo
min  lin
ngai  wong
hong  kong
sea ai lab
contextual  ai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 08:36:26

Abstract: Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs.  Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training.  The code and demo are available at https://github.com/sail-sg/scaling-with-vocab and https://hf.co/spaces/sail/scaling-with-vocab-demo.

Main paper: 
 introduction: 
 Large language models (LLMs) achieve remarkable performance by pre-training on vast text corpora using massive computational resources [47] . Extensive prior work on LLMs has focused on deriving so-called scaling laws: a set of empirical formulas to predict how model performance scales, mainly as computing floating-point operations (FLOPs), model parameters, and quantity of training data change [30, 26, 66, 2, 44, 58] . These works show that power-law fits can effectively predict language modeling loss and by extension downstream performance [23, 55] . However, these scaling laws usually disregard the impact of the vocabulary size. For example, in Kaplan et al. [30] only non-vocabulary parameters are considered in their predictive formula. This negligence has resulted in substantial variability in the vocabulary size of current LLMs. For instance, Llama2-7B employs a vocabulary size of 32K [70] , while Gemma-7B [67] adopts a much larger vocabulary size of 256K despite both having a similar number of total parameters. This variability in vocabulary sizes across LLMs raises the research question: What is the compute-optimal vocabulary size for a LLM? 𝑁 "! Figure 1 : The relationship between non-vocabulary parameters N nv and the corresponding optimal vocabulary parameters N opt v follows a power law, where N opt v should be scaled slower than N nv as γ < 1. Empirical results align with predictions of our proposed approaches, with larger circles indicating higher loss values. Here V refers to the vocabulary size i.e. the number of distinct tokens.The vocabulary size affects performance non-trivially. Intuitively, the optimal vocabulary size should neither be too large nor small. A larger vocabulary size improves tokenization fertility, i.e., splitting sentences into fewer tokens, thereby improving the tokenization efficiency. Additionally, a larger vocabulary enables the model to capture a wider range of concept. However, the risk of under-fitting for rare tokens increases with larger vocabulary sizes, especially in the data-constrained regime [44, 72] . Thus, the optimal vocabulary size needs to be determined by taking the training data and the non-vocabulary parameters into account.In this paper, we show that the effect of vocabulary on scaling laws has been underestimated, and we quantify the effect to derive a prediction for the optimal vocabulary size. We first introduce a normalized loss formulation to ensure a fair comparison across models with varying vocabulary sizes. Utilizing the normalized loss function, we analyze and discuss the underlying rationale behind the existence of an optimal vocabulary size, which depends on the available computational budget.To predict the optimal vocabulary size given a compute budget, we propose three approaches. Approach 1 (Estimating power laws via IsoFLOPs): We pre-train models with non-vocabulary parameters ranging from 33M to 1.13B, with groups of models that share the same FLOPs ("IsoFLOPs") but varying vocabulary configurations. Then, we fit power laws relating FLOPs to non-vocabulary parameters, vocabulary parameters, and training data, respectively. Our analysis reveals that the optimal vocabulary parameters exhibit a power-law growth with respect to the computational budget, however, at a slower rate than non-vocabulary parameters, as shown in Figure 1 . Approach 2 (Derivative-based Estimation): We introduce a derivative-based method that estimates the optimal vocabulary size by using the derivative of FLOPs w.r.t. the vocabulary size and finding the corresponding zero solution. Approach 3 (Parametric Fit of Loss Formula): We modify Chinchilla scaling laws [26] to incorporate vocabulary and fit the resulting formula on our models to predict the normalized loss function based on non-vocabulary parameters, vocabulary parameters, and the amount of training characters jointly. While the prior two approaches are limited to compute-optimal settings, this approach also allows us to determine the optimal vocabulary when the allocation is suboptimal i.e. the model parameters are either trained for too many tokens ("overtrained") or for too few tokens ("undertrained"). Overtraining is very common [23] , such as Llama 2 7B [70] which was trained for 2 trillion tokens, significantly more than the compute-optimal allocation of a 7 billion parameter model of around 150B tokens.As shown in Figure 1 , we observe that the relationship between non-vocabulary parameters N nv and their correspondng optimal vocabulary parameters N opt v follows a power law, according to all of our approaches. Our prediction also suggests that vocabulary parameters should be scaled slower than non-vocabulary parameters, i.e., N opt v ∝ N γ nv where γ ≈ 0.83 < 1. Nevertheless, most of existing LLMs [33, 81, 67, 41, 4, 25, 12, 7, 46, 84] neglect the importance of vocabulary and allocate less vocabulary parameters than the suggestions, shown in Figure 2 . Note that we assume that Figure 2 : Vocabulary parameters of popular LLMs and predicted optimal vocabulary parameters at a compute-optimal number of training tokens. Most current LLMs have suboptimal vocabulary parameters due to vocabulary sizes, which are smaller than the predicted optimal values. Among the current models, StarCoder2-3B, OLMo-7B, InternLM2-20B, and Gemma2-27B have vocabulary sizes that come closest to the optimal allocation for their respective model sizes.the amount of training data for these models is optimally distributed according to Hoffmann et al. [26] . Considering that several LLMs are trained on substantially more data than optimal ones (e.g., Llama2), the optimal vocabulary sizes would likely be larger than currently estimated.Finally, we empirically verify our predictions on models with 3B parameter models. By using our approach to predict the expected vocabulary size in various practical cases when (1) the training data is insufficient ("undertraining"); (2) the training data is equally scaled with the model parameters, following the Chinchilla laws ("compute-optimal training") [26] ; (3) the training data is overly sufficient like in Llama [70] ("overtraining"). The results show that models with our suggested vocabulary sizes steadily outperform baselines adopting commonly used vocabulary configurations under the same FLOPs budget. Our research underscores the overlooked importance of vocabulary and the need to jointly consider the vocabulary size, model parameters, and training data for effective scaling.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please see the abstract and §1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminary: 
 In this section, we first present a general formulation of a commonly used scaling law, and then demonstrate how to modify it to incorporate the vocabulary.Scaling laws consider a computational budget, C, which is measured in FLOPs. The goal is to optimally allocate the compute budget to model parameters N and the number of training tokens D [30, 6, 26, 44] . It can be formulated as:EQUATIONFollowing Radford et al. [51] , the loss function is typically the language modeling loss when evaluating language models, which can be written as:EQUATIONwhere p(w i |w 1:i-1 , V ) is the output probability of word w i given the context w 1:i-1 and the tokenizer with vocabulary size V . Generally, the lower L indicates better performance of the language model. However, due to its dependency on V , L cannot be used to compare language models with different vocabulary sizes. Thus, we propose an adaptation later in §2.2. Fitting scaling laws generally requires various models trained for different configurations [23] . A common approach is to select several compute budgets and train models with varying N and D for each budget to find the best one, i.e. the one with the lowest loss ("IsoFLOPs") [26] . Using fitting techniques we can then estimate a function that maps from the compute budget to the optimal allocation to N and D.As prior work generally assumes the vocabulary size to be fixed, we cannot adopt the attributes in their scaling laws and their evaluation metric directly. Thus, we detail several considerations that allow us to investigate vocabulary scaling laws.Attributes Scaling laws commonly deal with the attributes, model parameters (N ) and number of training tokens (D) [26, 44] . We adapt them for our analysis in the context of vocabulary size. (1) We break down the total model parameters (N ) into non-vocabulary (N nv ) and vocabulary parameters (N v ). To understand the importance of vocabulary parameters, we isolate them from other model parameters, where N = N nv + N v . We use N v = V d to represent both the vocabulary parameters in the output layer 1 . Notably, to change N v we only vary the vocabulary size V and take the embedding dimension d as given based on N nv empirically, see §A.7.2 for details. This is based on the observation by Kaplan et al. [30] that the performance of models with varying depth-to-width ratios converges to a single trend. We also provide further analysis about why we break down the model parameters from the perspective of parameter growing in §A. As detailed above we measure training data in training characters (H). Nonetheless, to connect our findings with existing studies on scaling laws [26, 44] , we need to be able to map from H to D. This mapping is the tokenizer's compression ratio which can be computed via D/H. The more tokens the tokenizer needs to represent H, the larger D, and thus it compresses less. We develop a simple function f (V ) to estimate this ratio solely from the chosen vocabulary size, V . Specifically, we find that a quadratic function on the logarithmic value of V achieves accurate predictions:EQUATIONBy fitting several tokenizers with V ranging from 1K to 1024K, we obtain a = 0.0064, b = -0.1581 and c = 1.2047. We find that our function accurately predicts the compression ratio with a low relative mean square error (RMSE) and a high coefficient of determination (R 2 ). In §A.9, we visualize fitting results and show that our approximation works with different tokenizers and is robust to different V . For all our main experiments, we use the BPE algorithm for tokenization [59] .Vocabulary-insensitive loss To fairly assess models that vary in V , the commonly used language model loss in Equation 2 is inappropriate. Models trained with larger V naturally have a higher loss, as there are more possibilities in the vocabulary to predict. However, this does not mean that the model is worse. Thus, we need to normalize the loss with respect to the vocabulary size. We reformulate the unigram-normalized metric [54] as a loss function. Suppose we have a T -length sequence w 1:T , we design the unigram-normalized language model loss as:EQUATIONwhere p(w i |V ) is the frequency of word w i in the tokenized corpus, given the tokenizer with vocabulary size V . The loss indicates the improvement in probability that a context-aware language model offers over a unigram model without context, allowing us to assess the language model's efficacy. Based on theory from prior work [54] , the normalized loss L u remains consistent for a given model with a fixed non-vocabulary component across different vocabulary sizes. The difference of L u comes from the ability of the language model itself. Compared with L, the value of L u is much smaller and can be negative as L u adds a negative term 1 T T i=1 log p(w i |V ). One may also 1 Vocabulary parameters typically encompass both the word embedding layer and the output layer. In this paper, for clarity and analytical simplicity, we employ V d rather than 2V d to represent the vocabulary parameters. This choice is predicated on empirical observations: the main computational burden, as measured in FLOPs, is associated with the output layer, but not the word embedding layer.Figure 3 : Left: FLOPs curve with various vocabulary sizes, assuming all configurations achieve a fixed loss. There exists an optimal vocabulary size that minimizes FLOPs. Right: Loss curves with various vocabulary sizes given different FLOP budgets. For each budget there exists an optimal vocabulary size that minimizes loss. As the FLOP budget increases this optimal vocabulary size increases (shifts to the right). employ the average bits per character (BPC), a common metric for text compression [27] , as the vocabulary-insensitive loss. The only difference lies in the normalization. BPC represents the raw per-character language model loss over the corpus, while our L u is equivalent to the per-character language model loss normalized by the frequency of each character. In practice, we find that the metric BPC and L u show a significant positive correlation, which experimentally validated our statement, as detailed in the §A.5.3 Analysis: Why the optimal vocabulary size is bounded by compute Analysis 1: The perspective of fixed normalized loss According to Kaplan et al. [30] , the FLOPs (C) of a Transformer model can be estimated as C ≈ 6N D, which can be re-written as:EQUATIONwhere N = N nv + N v and D = Hf (V ) based on §2.2. The reasons why model performance first increases and then decreases as the vocabulary size grows are: (1) At small V , increasing the vocabulary size easily improves tokenization fertility from f (V ). Subsequently, more characters can be learned from the model with a fixed number of tokens, thereby improving model performance. (2) At very large V , the gain from tokenization fertility decreases, while the parameters from expanding the vocabulary cannot be adequately trained with limited data, which leads to a decline in model performance. We present an expanded derivation in §A.1, and show how the corresponding FLOPs change with the vocabulary size in Figure 3 (left).Analysis 2: The perspective of fixed FLOP budget Given a fixed FLOPs budget, we isolate the FLOPs and investigate how the vocabulary influences the loss. For ease, we train models with fixed N nv and different vocabulary sizes for the same steps, and then we use interpolation to predict the loss when FLOPs reaches the budget given the observed FLOPs and loss points. For each budget, we adopt a group of models with similar total parameters and vary vocabulary sizes. In Figure 3 (right) we plot the relationship between the loss w.r.t. the vocabulary size. It reveals that the vocabulary corresponding to the lowest point on the loss curve increases as the FLOPs budget increases. This suggests that with more computational resources, LLMs can harness larger vocabularies to reduce loss. However, merely expanding the vocabulary does not always lower the loss. For a fixed FLOPs budget, the loss initially decreases with the increase in vocabulary and then starts to rise, indicating that an optimal point exists for the vocabulary.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Appendix B.Guidelines:• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
estimating the optimal vocabulary size: 
 In this section, we describe three complementary approaches to estimate the optimal vocabulary size.We define 6 groups of models with N nv ranging from 33M to 1.13B. Within each group, we solely vary the vocabulary size V from 4K to 96K, and evaluate different models under the same FLOPs budget. We evaluate the normalized loss L u on a held-out validation dataset. This approach allows us to directly answer the question: For a given FLOPs budget, what is the optimal allocation to non-vocabulary parameters, vocabulary parameters, and training data? Setup Given a certain N v , the embedding dimension d is fixed, thus N v increases as V increases. For all experiments, we uniformly sample the training data from different domains in the SlimPajama dataset [61] . All other hyperparameters are fixed with more details in §A.7.Fitting We select data points with the minimum L u for each FLOP budget, with all runs visualized in Figure 4 . These points are the compute-optimal allocation to (N nv , N v , H). Following Kaplan et al. [30] and Hoffmann et al. [26] , we hypothesize that the optimal vocabulary parameters N v meet a power law w.r.t. the FLOPs C, just like the non-vocabulary parameters and the amount of training data. Specifically,N nv = k 1 C α1 ,N v = k 2 C α2 and H = k 3 C α3 .As model size and training data should be scaled equally for compute-optimal training [26] , we set α 1 = α 3 . As our new attribute V significantly increases the number of possible experimental configurations, we employ interpolation across data points to obtain more configurations cheaply. The details of the fitting are in §A.7.4.Figure 5 : Fitting results of the Approach 1. Blue stars denote the selected data points where the combination (N nv , N v , H) reaches the lowest loss given various FLOPs budgets. We find power law fits with respect to the optimal non-vocabulary parameters, vocabulary parameters, and the number of training characters, respectively.Results and Usage In Figure 5 , we display the fitted power laws: N nv = 0.08 * C 0.50 , N v = 0.20 * C 0.42 and H = 6.42 * C 0.50 , where C is the FLOPs budget.The low RMSE and high R 2 values indicate the strength of our fit. Given a certain FLOPs budget, we can utilize the aforementioned relationships to obtain the optimal allocation (N nv , N v , H). We also draw the following conclusions:(1) LLMs are data-hungry. Compared to the non-vocabulary parameters N nv , practitioners should allocate more compute to the training data [80, 44] . (2) Vocabulary parameters scale in a powerlaw relation with FLOPs (N v ∝ C 0.42 ). As models become more computationally intensive, a larger vocabulary enhances the model's ability to understand a more diverse array of text, and thus the vocabulary size is critical to scaling. (3) Vocabulary parameters N v should be scaled slower than non-vocabulary parameters N nv . This difference can be seen in their power law exponents, i.e. γ = 0.42/0.50 = 0.84 < 1. We hypothesize the reason is that: once a sufficiently rich embedding space is present via a large vocabulary, it is more critical to scale non-vocabulary parameters to learn the intricate syntactic and semantic structures of language via Transformer blocks.We propose an alternative approach leveraging insights from the estimation of the FLOPs itself. Prior work [26, 30] usually considers a fixed compute budget in FLOPs and then aims to minimize loss by finding the optimal allocation to model parameters N and training tokens D. Here we flip this recipe on its head following recent work [57] . We aim to find the minimum FLOPs to achieve a certain loss L u (N nv , V, H) = ℓ through optimal allocation of the vocabulary size V :EQUATIONBy computing the minimum point of FLOPs C with respect to V via derivative:EQUATIONwe can estimate the optimal V under the assumption that it can achieve a certain lossL u (N nv , V, H) = ℓ.The parameters a, b and c can be easily obtained from building f (V ) ( §2.2). In theory, as long as the non-vocabulary parameters N nv are provided, V can be numerically searched via the solution of ∂C ∂V = 0. More details are in §A.1. Usage When the compute allocation is near optimal, the loss exhibits a power-law relationship with respect to the FLOPs budget, as described by the scaling law [30] . This relationship allows us to use FLOPs as a reliable proxy for observing the scaling behavior of the optimal vocabulary parameters. In practice, we can first determine an empirically optimal vocabulary size in a low-cost setting (e.g., finding the compute-optimal vocabulary parameters on a small model). Then, we can scale the optimal vocabulary parameters proportionally based on γ. Specifically, we obtain a set of derivative-optimal vocabulary parameters N v for different non-vocabulary parameters N nv , represented as(N i nv , N i v )|i = 1, • • • , n .We then fit the relationship between N nv and N v using the power-law function N v ∝ N γ nv . This results in the scaling equation:N v /N 0 v = (N nv /N 0 nv ) γ where N 0nv is a small model (e.g., 33M), and N 0 v is the searched optimal vocabulary parameter. By combining the γ from the derivative and the empirical solution on a small model, we can estimate the optimal vocabulary by:N opt v = N 0 v * ( N nv N 0 nv ) γ ,where the scaling proportion γ = 0.83 after our fitting. Consistent with the observation in Approach 1, we find that non-vocabulary parameters should be scaled faster than vocabulary parameters to achieve an optimal allocation.Finally, we directly predict the loss given the non-vocabulary parameter, vocabulary parameter and the amount of training characters. Then, the optimal vocabulary configuration can be predicted by finding the minimum point of loss with respect to the vocabulary. Following a classical risk decomposition used in Hoffmann et al. [26] , we design the vocabulary-dependent loss formula as:EQUATIONwhere D = Hf (V ). The first term captures the normalized loss for an ideal generative process. The subsequent terms reflect the effect of the non-vocabulary parameters, vocabulary parameters, and the number of training data on the loss, respectively. E, A 1 , A 2 , B, α 1 , α 2 , β are learned parameters.Fitting We use the points (N nv , N v , H) collected for experiments in §4.1. Note that we do not only consider the points with the lowest loss for each FLOP budget as we want to predict loss for any combination of (N nv , N v , H). We add the constraint α 1 = β following Muennighoff et al. [44] . We also filter out points with very small FLOPs following Hoffmann et al. [26] . Fitting yieldsA 1 = 1.831, A 2 = 0.196, B = 2.124, E = 5.533, α 1 = β = 0.447, α 2 = 0.671.The detailed fitting process is written in §A.7.4.Usage After fitting the parameters in Equation 8, the optimal vocabulary size can be obtained by finding the lowest loss w.r.t the vocabulary size, with a constraint of FLOPs budget. For example, As illustrated, the vocabulary parameters remain predominantly underestimated. With the exception of Gemma2-9B, all models allocate a smaller vocabulary parameter count than our prediction.given N nv and FLOPs budget C , by replacing [Hf (V )] with C/(6(N nv + N v )) and finding the solution of ∂Lu ∂V = 0 via numerical search, we can get the prediction. The details of ∂Lu ∂V is written in §A.2. Note that all of the proposed approaches can be used in optimally allocating (N nv , N v , H) altogether, while Approach 3 is more flexible in predicting the locally optimal N v when (N nv , H) are not following the Chinchilla's law [26] , i.e. equally-scaled law. The reason is that the loss formula in Approach 3 does not only considers the combinations (N nv , N v , H) which reach the optimal given a certain training budget. By fixing N nv and varying C in Approach 3, we can predict the locally optimal vocabulary size with different amount of training characters. This property makes Approach 3 more valuable, since modern LLMs [70, 67, 3, 4, 7] usually leverage overly sufficient training data to build powerful models with relatively low inference costs.In Figure 6 , we remove the assumption [26] for the practical reason that the parameters and training data are not equally scaled. Then, we predict the locally optimal vocabulary parameters. It can be observed that the allocation of vocabulary parameters are typically under-estimated. 
discussion: 
 Predicting allocations for larger models Table 1 reports the predicted optimal vocabulary parameters and sizes based on the proposed three approaches, where the amount of training data is optimally Table 1 : We report the predicted optimal vocabulary parameters N v and the vocabulary size V by the proposed three approaches given N nv . We assume the training FLOPs are optimally allocated i.e. that the non-vocabulary parameters and training data are scaled equally. "App" denotes the approach. allocated, i.e. equally scaled with the non-vocabulary parameters [26] . Aligned with the trend shown in Figure 1 , the predictions from all proposed approaches align closely. N nv should be scaled faster than N v . Notably, mainstream LLMs typically assign fewer parameters to vocabulary than what is optimal. However, the community is starting to shift to larger vocabularies, such as with Llama3 [41] having a 128K vocabulary size up from 32K of Llama2 [70] . However, scaling data is still the most critical part, and solving data scarcity issues should be a focus of future work [72] .N nv N opt v -App1 N opt v -App2 N opt v -App3 Dim. V opt -App1 V opt -App2 V opt -App3To empirically verify our prediction, we train models with N nv = 2.87B under a compute-optimal training FLOPs budget and evaluate them using lighteval 2 . For the baseline model we use the common vocabulary size of V = 32K. The other model uses V opt as predicted by Approach 3. In Table 2 , we show that the model allocated according to our vocabulary predictions yields better performance across multiple downstream tasks. This verifies that our predictions hold at scale.where training compute budget is the main constraint and we seek to allocate it optimally to parameters and training data. This is the typical setting in scaling law studies [30, 26, 52] . However, in the real world, we often deal with scarce data ("data-constrained [44] ") forcing us to train sub-optimally or would like to make use of excessive data to train a smaller model that is cheaper to use [84] . To verify that our Approach 3 can handle these practical scenarios, we compare the model with V = 32K and the model with the vocabulary size V opt predicted by Approach 3. As shown in Table 3 , our prediction enables a better model by only adjusting the vocabulary size in different FLOPs budgets.In Figure 7 , we further study the trend about how does the optimal vocabulary size shift with different number of training data. We only vary the amount of data but keep the non-vocabulary parameters fixed. The choices of vocabulary size are 8K, 10K, 16K, 24K, 32K and 48K. Taking N nv = 302M as an example, when available data is the bottleneck, the optimal vocabulary size decreases empirically, i.e. 16K → 10K. This is a mechanism to prevent over-fitting. Conversely, when training on excessive amounts of data, e.g., Llama3-8B uses much more training tokens than what would be compute-optimal for its budget, the optimal vocabulary size increases, i.e. 16K → 24K. Note that here we focus solely on training compute-optimal. It is also important to note that expanding the vocabulary size also increases the computational demands during inference. Therefore, we recommend using the optimal vocabulary size corresponding to a given N nv , assuming optimal allocation of training data, even in scenarios where overtraining may occur. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Please see the supplemental material.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Language models The Transformer [71] has proven to be a scalable architecture for language models, especially large language model (LLMs) [11, 14, 52, 47, 20, 29, 53, 70, 73, 41, 8, 4, 38, 25, 62, 67, 7, 39, 32, 88] . These models typically acquire a deep understanding of language enabling them to perform multiple tasks after a pre-training period and an optional fine-tuning period. Their capabilities include code generation [33, 3, 43, 87, 86] , mathematical reasoning [76, 5] , question answering [48, 45] among others. Given the expensive deployment costs required by the language models, various techniques can be adopted for efficient inference [64, 65, 78, 74, 36] . In our work, we pre-train large language models from scratch on English corpora and focus on their validation loss and downstream performance after training.Scaling laws Scaling laws aim to develop a predictive framework to find the best allocation of compute resources to maximize model performance. Besides language models, they have been studied in other domains [40, 68, 13] . For language models, Kaplan et al. [30] show that performance improves as a power law with more compute allocated to both parameters or data. Hoffmann et al. [26] show that the compute allocation of parameters and data should be scaled equally. Other work considers various cases such as downstream performance [23, 28, 55] , inference time [57] or data constraints [44, 80] . However, the effect of vocabulary size has generally been ignored previously. 
conclusion: 
 We investigate the impact of the vocabulary size in language models. We analyze and verify that there exists an optimal vocabulary size for a given FLOPs budget. Subsequently, we develop 3 approaches to predict the optimal vocabulary size. Our first approach uses empirical training runs across different IsoFLOPs regimes to fit a scaling law. The second approach investigates the FLOPs w.r.t. the vocabulary size and estimates the vocabulary size with derivatives. The third approach consists of a parametric function to predict the impact of different attributes on loss. Across all approaches, we find that while vocabulary parameters should be scaled slower than other parameters, they are still critical for performance and we can accurately predict their optimal allocation. We make predictions for larger models and empirically verify our approaches on up to 3B parameters and on varying amounts of training data. We show that models trained with an optimal vocabulary size as predicted by our approaches outperform models with a conventional vocabulary size under the same FLOPs budget.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We provide standard deviation for downstream performance in Table 3 and confidence intervals for our experiments in §A.10. For our pre-training experiments, we do not provide error bars due to their computational cost. Specifically, our variables are "The number of considered models" × "The number of considered vocabulary size" × "The number of training characters" × "The number of repeated experiments". Training multiple models across all these axes for more statistical significance would be prohibitively expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please see §A.7.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Please see Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]  
 
 Appendix: A.1 The derivation of FLOPs w.r.t the vocabulary size for the Approach 2Here we provide the detailed process of how we compute the extreme point of FLOPs C with respect to V . From Kaplan et al. [30] , we know that:EQUATIONWe then compute the derivative ∂C ∂V as follows:∂C ∂V = ∂ ∂V [6(N nv + dV )H (f (V ))] = ∂ ∂V 6(N nv + dV )H a(log(V )) 2 + b log(V ) + c = 6H (N nv + dV ) d dV a(log(V )) 2 + b log(V ) + c + a(log(V )) 2 + b log(V ) + c d dV (N nv + dV ) = 6H (N nv + V d) 2a log(V ) + b V + a(log(V )) 2 + b log(V ) + c d .The solution of ∂C ∂V = 0 corresponds to the minimum point of the FLOPs. Since the variable V in this equation is not separated conveniently, we use a numerical search method, specifically scipy.optimize.fsolve, to find the solution.Example demonstration Figure 8 illustrates the relationship between the derivative of FLOPs with respect to the vocabulary size V and V itself. Setting V as the solution to ∂C ∂V = 0, we find the point at which FLOPs are minimized. As depicted in Figure 8 (right), the FLOPs budget is fixed, and we observe how the training character varies with V . Notably, at the optimal vocabulary size V , the model expends the maximum number of training characters for a given budget. This observation provides insight into why an optimal vocabulary size exists for a given FLOPs budget. Here we provide how we derive the loss w.r.t the vocabulary size given a FLOPs budget C in Approach 3. After substituting the [Hf (V )] with the C/(6(N nv + N v ) based on Equation 9:EQUATION)The loss is solely dependent on the N v = V d, given a N nv . The derivative w.r.t. V is:∂L u ∂V = ∂ ∂V A 2 (V d) α2 + ∂ ∂V    B C 6(Nnv+V d) β    = -α 2 A 2 d (V d) α2+1 + β B Cd 6(Nnv+V d) 2 F 6(Nnv+V d) β+1 .The solution of ∂Lu ∂V = 0 corresponds to the optimal V . Similar with Approach 2, we use scipy.optimize.fsolve to find the solution.A.3 More visualizations for the analyses: Why the optimal vocabulary size is bounded by the compute Word embeddings in a large vocabulary are hard to learn when FLOPs are constrained Previous studies have shown embeddings suffer from representation degradation, where low-frequency word embeddings cluster together due to limited parameter updating [24] . In Figure 9 , we visualize how the word embeddings distribute using different vocabulary sizes. We use the average Euclidean distance among all the embeddings, D avg , to quantify the degree of clustering, which is 1.067, 1.011, and 0.952 for V = 4K, V = 16K and V = 64K, respectively. Larger vocabularies (64K) lead to more clustering of embeddings, especially for infrequent words. This clustering suggests that they have been insufficiently trained. Conversely, a small-sized vocabulary (4K) and middle-sized vocabulary (16K) display a more dispersed distribution of embeddings. These observations suggest that there exists an optimal vocabulary size that balances lexicon coverage and sufficient updating of word embedding. Language models with large vocabulary sizes may have better lexicon coverage, but on the other hand, hinder the model's ability to sufficiently update the word embeddings, especially for low-frequency words.Because of computational resource constraints, the vocabulary sizes we used to fit the scaling laws are in the range of 4K to 96K. This range is sufficient to fit, because the optimal vocabulary sizes for all the training configurations we used fall in this range.To further verify that there is always an optimal vocabulary size holds for a larger range of vocabulary lists, we increase the range of vocabulary sizes from 0.5K to 512K, with the N nv fixed as 33M.As depicted in the Figure 10 , the model's performance declines consistently as the vocabulary size increases beyond the optimal configuration. This figure shows loss curves for vocabulary sizes up to 512K, given a specific FLOPs budget. The data indicates a consistent degradation in model performance with the vocabulary size away from the optimal one. It suggests that there is a critical point beyond which the model's efficiency in handling the vocabulary diminishes. This exploration underscores the importance of carefully selecting the vocabulary size to maintain optimal model performance within the constraints of a given computational budget.A.5 The Vocabulary-insensitive Metric: L u and BPC BPC reflects the ability to compress external text corpora [27] , while the unigram-normalized loss reflects the model's ability to predict tokens normalized by the token frequency. Figure 11 shows the relationship between the Unigram-normalized Loss L u and Bits Per Character (BPC) with a linear fit. We select the models of the final training steps for each N nv and each vocabulary size. The high correlation coefficient (ρ = 0.9888) and low error (e = 0.0683) indicate a strong linear relationship between these two metrics generally. However, it exists slight different trends due to different normalizations.Traditionally, scaling up model parameters in language models has been approached in two ways: increasing depth (i.e., the number of layers) or width (i.e., the hidden size). While extensive research has been conducted on these methods, current empirical practices often involve expanding both simultaneously [66] . This approach, however, may overlook crucial distinctions in how different parameters benefit from these expansions.Non-vocabulary parameters can benefit from increases in both depth and width, allowing for more complex hierarchical representations and broader feature capture. In contrast, vocabulary parameters, associated with word embeddings and language model heads, are generally confined to a single layer, limiting their ability to benefit from increases in the model depth. Their primary avenue for expansion is through increasing the width. This disparity in growth potential between non-vocabulary and vocabulary parameters suggests that to maintain a balanced growth rate, it may be necessary to expand the vocabulary size along with the depth. This would allow the vocabulary parameters to keep pace with the growth of non-vocabulary parameters.We list the architectures of the models and the corresponding number of training characters in Table 4 .For each model family, we fix the non-vocabulary parameters N nv and vary the vocabulary size. We adopt the Llama architecture [69] , except for the vocabulary size. For the vocabulary size, we use numbers divisible by 128 for compatibility with NVIDIA's tensor core to accelerate matrix multiplication 3 According to the observation in Kaplan et al. [30] , the depth-width ratio has a relatively small effect on performance given the total non-vocabulary parameters. Thus, to ease the modeling of our scaling laws taking vocabulary size into account, we take the width (i.e. embedding dimension) as given following prior work [30, 26, 44, 70, 84] . The relationship between the non-vocabulary parameters N nv and embedding dimension d used in our experiments are in Table 5 .The maximum learning rate is set to 4e-4 and decays to 10% i.e. 4e-5 similar to prior scaling work [26, 44] . We use AdamW [37] as our optimizer and accelerate training with bfloat16 mixed EQUATIONprecision training. For models with N nv < 1130M , we use a single node with 8 GPUs for training. Otherwise, we adopt the Megatron-LM framework [60] for multi-node training with 8 GPUs on each node. For our experiments with N nv = 2870M , it takes about 120 hours to train on over 500B training characters with 64 total GPUs. We use a global batch size of 512 for all runs and run all experiments on 40GB Nvidia-A100 GPUs.Approach 1 To avoid numerical underflow and overflow of the fitting parameters, we fit the data in a logarithmic form inspired by Hoffmann et al. [26] . Taking N nv as an example, we learn the parameters k 1 , α 1 by minimizing:EQUATIONwhere K 1 = log(k 1 ) and Huber δ denotes the Huber loss with delta value δ (δ is 0.001 in our paper).We use the LBFGS algorithm to find the local minima of the function. The later Approach 2 and 3 use the same optimization algorithm. We initialize all attributes from the same uniform grid where K ∈ [-20, 15] and α ∈ [0, 1] with 20 initial guesses respectively. The fitting takes less than half of one minute.To cheaply obtain more experimental data points, we perform interpolation of (N nv , N v , H) triplets in the logarithmic scale and predict the validation loss based on real data points. Then, we compute the required FLOPs for each data point using Equation 5.Approach 2 By using different N nv and obtaining the corresponding optimal N v based on Equation 7, we have a set of {(N nvi , N vi )|i = 1, ..., n}. Denoting D nvi = N nvi /N nv0 and D vi = N vi /N v0 , we learn the scaling proportion γ by minimizing:EQUATIONThe initial guess of γ is uniformly sampled from [0, 1].We recast the designed vocabulary-dependent loss formula here:EQUATION)where β = α 1 . In practice, we try to minimize: min a1,a2,b,e,α1,α2Huber δ ( -exp(e) + exp(a1 -α 1 * log(N nv ) + exp(a 2 -α 2 * log(N v ) + exp(b -β * log([Hf (V )])), L u ),where A 1 = exp(a 1 ), A 2 = exp(a 2 ), B = exp(b), E = exp(e). We initialize all attributes from the same uniform grid wherea 1 ∈ [0, 5], a 2 ∈ [0, 5], b ∈ [0, 5], e ∈ [0, 2], α 1 ∈ [0, 1] and α 2 ∈ [0, 1]with 3 initial guesses respectively. Given the prior that the scaling factor is typically ranged between 0 and 1 [26] , we add a constraint 0.1 < α 1 , α 2 < 1 during fitting. The fitting also takes less than half of one minute.A.8 Details of fitting tokens-character relationship function f (V )We train 25 Later, we apply the trained tokenizers on the validation set of the Slimpajama dataset and collect the number of tokens D for each tokenizer with vocabulary size V . We use scipy.optimize.curve_fit to fit the parameters a, b, c in f (V ) ( §2.2).A.9 Robustness of the tokens-characters relationship function f (V )Robustness to the type of tokenizers Besides the widely adopted BPE tokenizer used in our experiment, we also consider the unigram tokenizer and the word-based tokenizer. We visualize their tokens-characters ratio and corresponding predictive function in Figure 12 . We find that our proposed formula of f (V ) is a good predictor for the tokens-character ratio, regardless of which tokenizer is used. This verifies the effectiveness of our proposed formula. The tokenization fertility of the unigram tokenizer is close to that of the BPE tokenizer as seen in their similar y-axis values, since they both employ subword-based tokenization. Meanwhile, the tokenization fertility of word-based tokenization is poor, thus requiring more tokens on average to compress characters. Robustness to the range of the vocabulary size The quadratic function on the logarithmic value of vocabulary size that we propose can precisely predict the tokens-characters ratio with an RMSE of 1.5e-6 and R 2 of 0.99. However, as a quadratic function is single-peaked, increasing V will increase the output value of f (V ) = a log 2 (V ) + b log V + c when V is very large, e.g. V > exp(-b/2a) ≈ 218K in our case.Fortunately, when V is sufficiently large, the tokenization fertility improvement of the tokenizer decays sharply, which results in almost no change to the value of f (V ). This is because the words in the training corpus can already be effectively covered by the vocabulary list when the vocabulary size is sufficiently large. In this extreme, the tokenization fertility of the corresponding tokenizer is approaching saturation, thus further increasing the vocabulary size will hardly improve the tokenization fertility.As an example, there are about 2300M characters in the validation set of the Slimpajama corpus. A tokenizer using a vocabulary size of 2K would yield 140M fewer tokens than a 1K counterpart, but the number of tokens only decreases by 0.7M when going from a vocabulary size of 256K to 257K. Therefore, we add min(V, 200K) before calculating f (V ) to ensure its decreasing nature. According to our prediction, a model with 300B parameters has an optimal vocabulary size of no more than 400K with a sufficient amount of training data. If we need to consider extremely large V in the future, we can train tokenizers with larger V in the process of fitting f (V ) to arrive at more precise predictions.A.10 Experimental verification on the fairness of the unigram-normalized language modeling loss (a) Relationship between downstream task performance and the commonly-used language modeling loss.(b) Relationship between downstream task performance and the unigram-normalized language modeling loss.Figure 13 : Empirical examination of the fairness of our unigram-normalized loss, L u . Dots correspond to trained models with varying vocabulary size. We plot their losses (y-axis) and performance on 7 downstream tasks (x-axis): WG [56] , PIQA [9] , OBQA [42] , Hellaswag [83] , BoolQ [16] , ARC-E [17] and ARC-C [17] . The straight line reflects the results of the regression fit with the shade indicating the confidence interval.In §2.2, we have explained that we use a unigram-normalized loss, L u , to fairly evaluate models that vary in vocabulary size. Here we empirically verify this choice. We train models with a fixed number of non-vocabulary parameters N nv and embedding dimension d but varying vocabulary sizes V . Thus, their vocabulary parameters N v also vary. We plot the final language model loss and unigram-normalized loss of these models compared to downstream performance in Figure 13 . The language modeling loss exhibits a positive correlation with downstream performance: Models with a higher language modeling loss have better downstream performance. This is because our models with larger vocabularies naturally have a higher loss due to the objective function, yet they can be actually better models with better downstream performance. Our unigram-normalized loss solves this problem and exhibits the expected negative correlation between loss and downstream performance: a lower loss comes with better downstream performance. This empirically justifies our use of L u throughout this work.While our primary experiments focus on the Llama2 vocabulary size, we also extend our conclusions to Llama3, predicting its optimal vocabulary under optimal compute allocation. As shown in Figure 14 , we provide detailed predictions for various sizes of Llama3 models. Although Llama3 significantly increases its vocabulary size from 32K to 128K, our research suggests that this may still be insufficient for the larger model sizes of 70B and 400B.A.12 More Related WorkVocabulary in language models The vocabulary of a language model influences its performance significantly [63, 75, 79] . A larger vocabulary size helps cover more words thus reducing the likelihood of out-of-vocabulary (OOV) cases [21] . Takahashi and Tanaka-Ishii [63] find that larger𝑁 ! "#$ ∝ 𝑁 %! & 𝑁 ! 𝑁 "!Figure 14 : The replication of Figure 1 for Llama3. As shown, the predicted optimal vocabulary size for the Llama3-400B model is as high as 487K.vocabularies are better at capturing the true statistical distribution of language. Similarly, expanding vocabulary in multilingual models [75, 15, 85, 34] improves performance, especially for low-resource languages. However, large vocabularies [31] increase the computational overhead during both training and generation phases. For example, Liao et al. [35] demonstrate that low-frequency words often lack sufficient examples to develop robust representations when vocabularies are excessively large. Dou et al. [19] reveal that expanding vocabularies during continual pre-training can lead to significant performance degradation for low-resource languages. More recently, Dagan et al. [18] explored the trade-offs associated with vocabulary size, proposing optimal vocabulary sizes for both memory efficiency and inference speed in code generation tasks. Our work complements these efforts by focusing on the broader impact of vocabulary size on downstream performance across various tasks. Specifically, we address a critical, under-explored question: How can we optimally allocate vocabulary size to maximize the downstream performance with the same compute budget?Byte-level language models Recent work has explored byte-level language models [82, 77] , which offer advantages in decoding efficiency and noise robustness compared to token-level models. However, typically limited to parameters under 1B, these models have not been effectively scaled up.Our scaling laws suggest that the limited vocabulary (i.e., 256 in byte-level language models) may constrain their performance, especially for larger models. The insight provides a potential explanation for the challenges in scaling byte-level models and implies that successful scaling of language models may require proportional increases in vocabulary size.The Approach 1 provides a broader solution by predicting the allocation of computational resources across non-vocabulary parameters, vocabulary parameters, and training data based on experimental data points. This method's strength lies in its holistic view, allowing for a balanced resource distribution that potentially enhances model efficiency and performance. However, this approach is constrained by the granularity and range of the experimental data points available, which can introduce errors in the fitting process. The requirement for substantial computational resources to perform these fittings may also limit its accessibility and scalability. Despite these challenges, when experimental data is ample and computational resources are sufficient, the Approach 1 can significantly refine the precision of resource allocation decisions in the development of large-scale language models.Approach 2 By calculating the derivative of FLOPs with respect to the vocabulary size and solving for zero, this approach fundamentally relies on the precision of the FLOPs equation and our tokenscharacters relationship function. Further, this method does not allow us to independently determine the optimal allocation of non-vocabulary parameters and training data size. Therefore, it necessitates information about the relationships between these attributes and the FLOPs budget from the experimentally fitted scaling laws, making this approach less useful in practice. Despite these limitations, the derivative-based approach offers notable advantages, including closely matched predictions with the scaling laws derived from actual experimental data in the Approach 2. Furthermore, its reliance on numerical solutions rather than exhaustive deep learning experiments makes it rapid and broadly applicable across various tokenizers, highlighting its utility in preliminary model configuration stages where quick estimates are key.Approach 3 Similar with the Approach 1, the proposed Approach 3 requires multiple experimental runs across different non-vocabulary parameters, vocabulary sizes and number of training data. Therefore, the approach is constrained by the granularity and range of the experimental data points available to some extent. However, the proposed Approach 3 is flexible that it considers the fact that the non-vocabulary parameters and the number of training data are not always following the compute-optimal scaling laws [26] , i.e., equal scaling, in real-world applications.We have shown that our predictions hold for models with up to three billion parameters ( §5). However, LLMs are often orders of magnitude larger, such as the 400-billion parameter Llama-3 model [41] . Further, we have decided to focus on dense transformer language models, as they are most commonly used for LLMs. However, many non-transformer models have been proposed and scaled up to billions of parameters [49, 50] . Exploring to what extent our findings hold in even larger models and with different architectures is a promising direction for future work.Researchers [26, 44] consider modeling the language modeling loss with parametric functions in the form of L = P 1 + P 2 /N α + P 3 /D β , where {P 1 , P 2 , P 3 , α, β} are learnable variables. The first term of loss represents the minimum achievable loss, and the second and third terms represent the contribution to the loss from the model size N and number of training tokens D. The parametric function allows predicting the loss L given N and D even if (N ,D) are not optimally allocated.In prior work, this loss formula accounts for changes in model size and training data but does not explicitly address the complexities introduced by varying vocabulary sizes. Incorporating vocabulary size into the loss predictor is challenging: Vocabulary size affects the model directly as well as the number of training tokens and the quality of tokenization by the tokenizer. A tokenizer with a large vocabulary size makes it easier to capture semantic information in raw text and reduces the frequency of out-of-vocabulary words. For instance, a large vocabulary size may cover common phrases, common subwords, and specialized terminology. Therefore, even if the same number of tokens are trained, the performance of the model trained on tokens with different qualities will be different.Future work in this area could explore various parametric non-linear loss functions to predict the interactions between vocabulary size, model size, and training data with different compute allocations, not just the case of optimal compute allocation. Additionally, empirical studies on different datasets could help in understanding how vocabulary size impacts loss under varied data conditions, guiding the development of more adaptive loss prediction models.Future work could extend the proposed approaches to encompass multilingual and multimodal scenarios. Multilingual models require a nuanced understanding of vocabulary due to linguistic diversity, which may affect the optimal vocabulary size and the computation of FLOPs differently across languages. Adapting these methods to consider linguistic features and tokenization variations could lead to more tailored and efficient resource allocations for multilingual models. Different languages compete with each other for the model's ability to allocate to that language [10] , which makes it necessary to take into account the relationship between different languages when setting the size of word lists for different languages in a multilingual scenario.For multimodal models that integrate text with other data types such as images or video, the optimal vocabulary size might interact uniquely with non-linguistic parameters. Recent work [1, 68] models visual concepts in an autoregressive manner with tokenization like the processing of text data. It is interesting to explore the size of visual vocabulary size, i.e., the codebook size [22] , in the visual tasks and vision-language tasks. How to set the vocabulary size and the compute resource efficiently for different modalities remains an open issue.The positive potential social impact of this research on vocabulary size in language model scaling is substantial. By optimizing large language models with the consideration of the vocabulary size and other attributes jointly, the paper provides a foundational understanding that can lead to more lightweight and cost-effective pre-trained large language models. This efficiency can democratize access to advanced language processing technologies, making it feasible for smaller organizations and the general public to benefit from powerful AI tools. Such advancements can benefit various domains, for example, improve accessibility features for individuals with disabilities, where efficient language models can be used to analyze medical records and assist in diagnostics. Furthermore, the reduction in computational requirements for training these models can lead to a decrease in energy usage, contributing positively to environmental sustainability efforts.On the other hand, the misuse of pretrained language models may pose risks, including the creation of highly realistic deepfakes that can spread disinformation and undermine trust in media and institutions. These models can generate misleading content, automate cyberattacks through convincing phishing schemes, and produce large-scale spam, degrading online communication. Additionally, they can be used to generate harmful or abusive content, such as hate speech, which perpetuates discrimination and harms vulnerable populations. To mitigate these risks, it is crucial to develop trustworthy language models, implement robust monitoring systems, and foster collaboration among researchers, policymakers, and users.NeurIPS Paper Checklisthttps://github.com/huggingface/lightevalhttps://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/ index.html 


Decision: This paper studies the important aspect of vocabulary size in the context of scaling laws for LLMs. Vocabulary size has not dealt with in detail in prior work and this paper fill that gap. The study is rather limited in scale, but is perhaps understandable considering the very high computational demands that pretraining creates. It would be great if the study could be scaled up a little more, so that there is more confidence in the generalization.

Reviews: review: paper_summary: paper_summary: This paper investigates the impact of vocabulary size on the efficiency of large language models (LLMs). Using models with 33 million to 3 billion parameters, it finds that optimal vocabulary size is limited by computational resources. The study introduces two methods to determine the best vocabulary size, showing that vocabulary parameters should scale slower than other parameters. Results highlight the significant yet underestimated role of vocabulary in scaling LLMs effectively, suggesting that larger vocabularies can improve model performance.
strength_weakness: strength_weakness: This paper conducted experiments on language models of various parameter sizes, but the largest model tested was only 3 billion parameters. It would be better if we could further verify models with more than 7 billion parameters. I believe both the industrial and academic communities are eager to know whether the scaling law for vocabulary can generalize to larger models.
questions: questions: NA
limitations: limitations: This paper discusses the limitations in Appendix B.
paper_summary: paper_summary: This study primarily explores the role of vocabulary size in scaling large language models (LLMs). Traditional research has focused on model parameters and training data size, often overlooking the impact of vocabulary size. While intuitively larger vocabularies can enable more efficient tokenization by representing sentences with fewer tokens, they also increase the risk of under-fitting representations for rare tokens. By training models ranging from 33M to 3B parameters on up to 510B characters with various vocabulary configurations, we discovered that the optimal vocabulary size is constrained by the computational budget. We propose two methods to determine the optimal vocabulary size: an empirical IsoFLOPs approach and a fast derivative-based approach. Both methods indicate that vocabulary parameters should be scaled slower than non-vocabulary parameters. Nonetheless, vocabulary parameters are critical for performance and are under-allocated in current LLMs. By increasing the vocabulary size beyond the conventional 32K, we trained a better 3B parameter model despite using fewer training tokens. Our work reveals the underestimated role of vocabulary and the necessity of jointly considering vocabulary size, model parameters, and training data for efficient scaling.
strength_weakness: strength_weakness: 1. Lacks performance on large-scale models, such as whether increasing the vocabulary size to a greater extent performs better than existing models in the market. Table 2's experiments look a little bit less.
2. IsoFLOPs method is very sensitive, the experiments also looks not enough.
questions: questions: In Use Case section, the authors mentioned that  (2) One has already conducted scaling law experiments following the Chinchilla laws with a fixed vocabulary size (e.g., 244 32K) and aims to estimate the optimal vocabulary size for a given model parameter. In determining the scaling law for the relationship between non-embedding model size and data (such as the Chinchilla law), why is it assumed that the vocabulary size is independent of these two factors
limitations: limitations: Yes, the authors addressed limitations.
paper_summary: paper_summary: The paper presents empirical scaling laws for the size of the vocabulary for LLMs. The findings in the paper are:

- Empirically, the vocabulary size minimizing the loss increases when FLOPs are increased (Fig 2, right, Fig 3)
- Through mathematical derivations from scaling laws, the optimal vocabulary size decreases when the embedding size increases. (Fig 4).
- With a given FLOPs budget, 43k vocav size beats 32k on academic benchmarks like Boolq (Table 2.)
strength_weakness: strength_weakness: - The results are probably mostly applicable to a small number of well-funded labs.
questions: questions: 1. The abstract states that “beyond the conventional 32K” – is this really the convention? See e.g. GPT4o.
2. How does Table 2 look if you train on the same number of tokens instead of using the same FLOPs budget?
limitations: limitations: na

Comments: title: Response to reviewer Y8zo
Any New Comments Would be Greatly Appreciated
Any New Comments Would be Greatly Appreciated
Any New Comments Would be Greatly Appreciated
Response to reviewer Y8zo's second part
comment: Dear Reviewer Y8zo,

Thank you for your timely feedback. We're pleased that you found our new results compelling and appreciate your insightful comments on areas for improvement, particularly regarding Figure 3 (left). We would like to offer the following clarifications:

Our study expands on traditional scaling law approaches by incorporating additional parameters: Nnv (non-vocabulary parameters), Nv (vocabulary parameters), and H (training characters). This contrasts with previous work that primarily considered N (total model parameters) and D (training tokens). The inclusion of these additional factors adds complexity to the scaling law fitting process. Given the high computational costs associated with scaling law experiments, we concentrated our efforts on a limited set of non-vocabulary parameters while exploring a broader range of vocabulary parameters. This focus aligns with the primary objective of our research: investigating the impact of vocabulary size on model performance.

To achieve this, we deliberately selected 10 groups with varying vocabulary sizes for each of the 6 fixed groups of non-vocabulary parameters. While this design choice accounts for the uneven distribution seen in Figure 3 (left), it also allows for a more thorough exploration of vocabulary size effects, as demonstrated by the diverse data points in Figure 3 (middle). Our experimental design and results robustly support our main conclusion: larger models benefit from larger vocabularies.

We believe these clarifications provide a more comprehensive understanding of our research methodology and findings. We appreciate your careful review and hope these explanations enhance your confidence in our paper. We look forward to any further feedback you may have.

Best regards,

The Authors
Thank you for your thorough response to my questions. I am pleased to see the new results. However, there is room for improvement in this paper, such as Figure 3 (left), where the distribution of model sizes is very uneven. For example, for models below 100M, the basic points of model size are concentrated at 33M and ~90M, which will seriously affect the exponential factor of the fitted curve.
Dear Reviewer EsaU,

We are truly appreciative of the time and effort you have dedicated to reviewing our paper. Your thoughtful feedback and constructive suggestions are valuable to us. We have carefully addressed your comments in our rebuttal to enhance the quality of our work.  As we approach the final days of the Author-Review Discussion period, we would like to ensure that all your concerns have been comprehensively addressed. Should there be any remaining questions or issues, we are willing to provide further clarification or additional revisions.Thank you once again for your insightful contributions to our work.

Best regards,

The Authors
Dear Reviewer zt86,

We are deeply grateful for your detailed review and the insightful suggestions you provided for our paper. We have carefully considered and responded to each of your comments in our rebuttal.

As the Author-Review Discussion period is coming to an end, we want to ensure that all your concerns have been thoroughly addressed. If there are any remaining questions or issues, we would be glad to offer further clarification or make any necessary revisions.Thank you once again for your valuable feedback.

Best regards,

The Authors
Dear Reviewer Y8zo

We sincerely appreciate your thorough review and the valuable suggestions and comments you provided for our paper. We have carefully considered each of your points and have addressed them in detail in our rebuttal.

With the Author-Review Discussion period nearing its conclusion, we want to ensure that all your concerns have been fully addressed. If there are any questions or unresolved issues, we are eager to provide further clarification or make any necessary revisions.

Thank you again for your thoughtful feedback. 

Best regards,

The Authors
### W2.2: The experiments also looks not enough.
It is noteworthy that we conduct extensive experiments on **1200 models pre-trained from scratch** (6 non-vocabulary parameters settings x 10 vocabulary sizes settings x 20 training data settings) for the fitting of our vocabulary scaling law. The key contributions in this paper is the several findings about how the vocabulary affects the model performance and how much compute should be allocated on the vocabulary based on the proposed 2 approaches. 

Following the previous study [1,2,3], we mainly use the held-out validation loss value for the evaluation of the trained 1200 models. It is a better metric than the downstream tasks performance as the held-out loss  provides an unbiased measure of the model’s ability to generalize to new data, but also enjoys high computing efficiency. Instead, the performance of downstream tasks has a great variety across different tasks, which is not suitable as the main evaluation metric.

The evaluation of downstream tasks is part of the ways to verify our prediction, therefore we do not take too much content to discuss it in our main paper. For downstream tasks, we conduct more experiments in the answer of your #Q1. The new results will be added in our polished version.

### Q1: In determining the scaling law for the relationship between non-embedding model size and data (such as the Chinchilla law), why is it assumed that the vocabulary size is independent of these two factors.

Thanks for your question! We do not assume that the vocabulary size is independent of parameters and data. Instead, we make some adjustments in the Section of Preliminary: 1) We break down the total parameters into non-vocabulary parameters and vocabulary parameters; 2) We measure data not in tokens but in training characters.

By doing so, the vocabulary size $V$ is independent with the non-vocabulary parameters $N_{nv}$ and the number of training characters $H$. In an experimental configuration, the developers can vary the vocabulary size without affecting non-vocabulary parameters or training characters.

Then, we details our motivation why we separate the vocabulary parameter and  non-vocabulary parameter below:

Traditionally, scaling up model parameters in language models has been approached in two ways: increasing depth (i.e., the number of layers) or width (i.e., the hidden size). Current empirical practices often involve expanding both simultaneously [4]. This approach overlook crucial distinctions in how different parameters benefit from parameters expansions. Non-vocabulary parameters can benefit from increases in both depth and width, allowing for more complex hierarchical representations and broader feature capture. In contrast, vocabulary parameters, associated with word embeddings and language model heads, are generally confined to a single layer, limiting their ability to benefit from increases in the model depth. This disparity in growth potential between non-vocabulary and vocabulary parameters suggests that to maintain a balanced growth rate, it is better to separate the vocabulary parameter  and  non-vocabulary parameter into consideration. 


[4] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. 2023. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12342–12364, Singapore. Association for Computational Linguistic.