Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Authors: Chaofan Tao
Qian Liu
Longxu Dou
Niklas Muennighoff
Zhongwei Wan
Ping Luo
Min Lin
Ngai Wong
Venue: NeurIPS 2024 poster (2024)
Keywords: Natural Language Processing
Scaling Laws
Efficient Neural Networks
Large Language Models
Primary area: natural_language_processing
TL;DR: This paper introduces a novel framework to study models with different vocabularies, substantiating a scaling law that optimizes computational resources with the consideration of vocabulary and other attributes jointly.

Abstract: Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies. Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2-70B should have been at least 216K, 7 times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs.  Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training.  The code and demo are available at https://github.com/sail-sg/scaling-with-vocab and https://hf.co/spaces/sail/scaling-with-vocab-demo.

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Chaofan Tao 1.2
Qian Liu
Longxu Dou
Niklas Muennighoff 3.4 Ngai Wong" if
Zhongwei Wan
Ping Luo'
Min Lin?
'The University of Hong Kong
Sea Al Lab 'The Ohio State University
Contexttal AI
*Stanford University
Abstract
Research on scaling large language models (LLMs) has primarily focused on model parameters and training data size, overlooking the role of vocabulary size. We investigate how vocabulary size impacts LLM scaling laws by training models ranging from 33M 3B parameters on up 500B characters with various vocabulary configurations. We propose three complementary approaches for predicting the compute-optimal vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit of the loss function. Our approaches converge on the conclusion that the optimal vocabulary size depends on the compute budget, with larger models requiring larger vocabularies, Most LLMs, however, use insufficient vocabulary sizes. For example, we predict that the optimal vocabulary size of Llama2- 70B should have been at least 216K, times larger than its vocabulary of 32K. We validate our predictions empirically by training models with 3B parameters across different FLOPs budgets. Adopting our predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes. By increasing the vocabulary size from the conventional 32K to 43K, we improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21 FLOPs. Our work highlights the importance of jointly considering tokenization and model scaling for efficient pre-training. The code and demo are available at https://gg thub. com/ stsssaalllaa sggilllgg/eee htgp//clllngwwiiiiieo titg-wwit-voccall and https://htt cta.f.co/spppee htspspaacs////e satl//sall/aaaa /es//saalnng ing- with th vocab-demo.
Introduction
Large language models (LLMs)) achieve remarkable performance by pre-training on vast text corpora using massive computational resources [47]. Extensive prior work on LLMS has focused on deriving so-called scaling laws: a set of empirical formulas to predict how model performance scales, mainly computing floating-poin operations (FLOPs), model parameters, and quantity of training data change [30, 26, 66, 2, 44, 58]. These works show that power-law fits can effectively predict language modeling loss and by extension downstream performance [23, 55). However, these scaling laws usually disregard the impact of the vocabulary size. For example, in Kaplan et al. [30] only non-vocabulary parameters are considered in their predictive formula. This negligence has resulted in substantial variability in the vocabulary size of current LLMs. For instance, Llama2-7B employs a vocabulary size of 32K [70], while Gemma-7B [67] adopts much larger vocabulary size of 256K despite both having a similar number of total parameters. This variability in vocabulary sizes across L.LSs raises the research question: What is the compute-optimat vocabulary size for L.M?
"Corresponding authors. The project was done during Chaofan Tao's 's internship at Sea AI Lab, For more information, please contact cftao@connect. hku, hk and liuqian. sea0gmail. com.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1010
Vocabulary Parameters Ng
Approach 1, y=0.83
7B
Approach 2. y=0.84
Laama2-138
Optimal
Approach 3, y-0.86
Liama2 708
109
Empirical Results
Optimal
Optimal V. 57K
10a
Oooo
10²
O0
0,00
Ny x (N
10
10a
1010
10
Non-Vocabulary Parameters Nnv
Figure I: The relationship between non-vocabulary parameters Nuv and the corresponding optimal vocabulary parameters Nopt follows power law, where Nopt should be scaled slower than Nuv as y < 1. Empirical results align with predictions of our proposed approaches, with larger circles indicating higher loss values. Here V refers to the vocabulary size i.c. the number of distinct tokens.
The vocabulary size affects performance non-trivially. Intuitively, the optimal vocabulary size should neither be too large nor small. A larger vocabulary size improves tokenization fertility, i.e., splitting sentences into fewer tokens, thereby improving the tokenization efficiency. Additionally, larger vocabulary enables the model to capture wider range of concept. However, the risk of under-fitting for rare tokens increases with larger vocabulary sizes, especially in the data-constrained regime [44, 72). Thus, the optimal vocabulary size needs to be determined by taking the training data and the non-vocabulary parameters into account.
In this paper, we show that the effect of vocabulary on scaling laws has been underestimated, and we quantify the effect derive prediction for the optimal vocabulary size. We first introduce normalized loss formulation to ensure a fair comparison across models with varying vocabulary sizes. Utilizing the normalized loss function, analyze and discuss the underlying rationale behind the existence of an optimal vocabulary size, which depends on the available computational budget.
To predict the optimal vocabulary size given compute budget, we propose three approaches. Approach (Estimating power laws via IsoFLOss): We pre-train models with non-vocabulary pa- rameters ranging from 33M to 1.13B, with groups of models that share the same FLOPs ("Is)FLOPPS") but varying vocabulary configurations. Then, we fit power laws relating FLOPs to non-vocabulary parameters, vocabulary parameters, and training data, respectively. Our analysis reveals that the optimal vocabulary parameters exhibit a power-law growth with respect to the computational budget, however, slower rate than non-vocabulary parameters, as shown in Figure Approach IDerivative-basel Estimation): We introduce a derivative-based method that estimates the optimal vocabulary size by using the derivative of FLOPs w.r.t. the vocabulary size and finding the corre- sponding zero solution. Approach 3 (Parametric Fit of Loss Formula): We modify Chinchilla scaling laws [26] to incorporate vocabulary and fit the resulting formula on our models to predict the normalized loss function based on non-vocabulary parameters, vocabulary parameters, and the amount of training characters jointly. While the prior two approaches are limited to compute-optimal settings, this approach also allows us to determine the optimal vocabulary when the allocation is suboptimal i.c. the model parameters are either trained for too many tokens ("overtrained") or for too few tokens ("undertrained"". Overtraining is very common [23], such Llama 2 7B [70] which was trained for 2 trillion tokens, significantly more than the compute-optimal allocation a 7 billion parameter model of around 150B tokens.
As shown in Figure 1, we observe that the relationship between vocabbuayy parameters Nuy and their correspondng optimal vocabulary parameters Nopt follows a power law, according to all of our approaches. Our prediction also suggests that vocabulary parameters should be scaled slower than non-vocabulary parameters, i.e., Noopt x NY. where 7 0.83 < 1. Nevertheless, most of existing L.LMs [33, 81, 67, 41, 4, 25, 12, 7, 46, 84) neglect the importance of vocabulary and allocate less vocabulary parameters than the suggestions, shown in Figure 2. Note that we assume that

---

Optimal Vocabulary Parameters
Appreach
10
Approach
Approach
Nomotron4- 3408
Public LLMs
2-BB2-72B
10
Gemma2-98
Gemma2-22
Uam
Qwen2-78
-lama3-ap ap
IntemLMM-220
DDepSeekk 678
Llama2-708
OLMO-78
Daam22 340
StarCader2-336
Llama2-78
10
Finybhmaa-1.18
109
10 Non-Vocabulary Parameters
10
1012
Figure 2: Vocabulary parameters of popular LLMs and predicted optimal vocabulary parameters at compute-optimal number of training tokens. Most current LLMs have suboptimal vocabulary parameters due to vocabulary sizes, which are smaller than the predicted optimal values. Among the current models, StarCoder2-33 BB,, OLMo-7B, InternLM2-200 and Gemma2 B-BBB have vocabulary sizes that come closest to the optimal allocation for their respective model sizes.
the amount of training data for these models is optimally distributed according to Hoffmann al. [26]. Considering that several LLMs are trained on substantially more data than optimal ones (e.g.. Llama2), the optimal vocabulary sizes would likely be larger than currently estimated.
Finally, we empirically verify our predictions on models with 3B parameter models. By using our approach to predict the expected vocabulary size in various practical cases when (1) the training data insufficient ("undertraining"" "): (2) the training data is equally scaled with the model parameters, fol- lowing the Chinchilla laws ("compute-optimal training"" [26]: (3) the training data is overly sufficient like in Llama [70] ("overtraining") The results show that models with our suggested vocabulary sizes steadily outperform baselines adopting commonly used vocabulary configurations under the same FLOPs budget. Our research underscores the overlooked importance vocabulary and the need to jointly consider the vocabulary size, model parameters, and training data for effective scaling.
2
Preliminary
In this section, we first present a general formulation of commonly used scaling law, and then demonstrate how modify incorporate the vocabulary.
2.1 Scaling law
Scaling laws consider a computational budget, C, which is measured FLOPs. The goal to optimally allocate the compute budget to model parameters N and the number of training tokens D [30, 6, 26, 44]. It can be formulated as:
(Nops Dopt) arg min C(N, N.D
FLOPS D) C,
Following Radford et al. [51], the loss function is typically the language modeling loss when evaluating language models, which can be written as:
1 E log pog
L
where p(w:/w1s-1 V) is the output probability of word w; given the context W1s-1 and the tokenizer with vocabulary size V. Generally, the lower L indicates better performance of the language model. However, due to its dependency on V, L cannot be used to compare language models with different vocabulary sizes. Thus, we propose an adaptation later in $2.2. Fitting scaling laws generally requires various models trained for different configurations [23]. A common approach is select several

---

compute budgets and train models with varying N and D for each budget to find the best one, i.e. the one with the lowest loss ("IsoFLOPs") [26]. Using fitting techniques we can then estimate function that maps from the compute budget to the optimal allocation to N and D.
2.2 Scaling law with vocabulary
As prior work generally assumes the vocabulary size be fixed, we cannot adopt the attributes in their scaling laws and their evaluation metric directly. Thus, we detail several considerations that allow us to investigate vocabulary scaling laws.
Attributes Scaling laws commonly deal with the attributes, model parameters (N) and number of training tokens (D) [26, 44]. We adapt them for analysis in the context of vocabulary size. (1) We break down the total model parameters (N) into non-vocabulary (Nav) and vocabulary parameters (N,). To understand the importance of vocabulary parameters, we isolate them from other model parameters, where N = Nuy + N₁. We use N,- = Vd to represent both the vocabulary parameters in the output layer ! Notably, to change N, only vary the vocabulary size V and take the embedding dimension d as given based on Nny empirically, see $A.7.2 for details. This is based on the observation by Kaplan et al. [30] that the performance of models with varying depth-to-we widdh ratios converges to a single trend. We also provide further analysis about why we break down the model parameters from the perspective of parameter growing in $A.66 (2) We measure data not in tokens (D) but in training characters (H). The number of tokens depends on the vocabulary of the tokenizer. By studying training characters, we can better see how the data volume affects the performance regardless of different vocabulary sizes.
Mapping from training characters (H) to tokens (D) As detailed above we measure training data in training characters (H). Nonetheless, to connect our findings with existing studies on scaling laws [26, 44], we need be able map from H to D. This mapping is the tokenizer's is compression ratio which can be computed via D/ H. The more tokens the tokenizer needs to represent H, the larger D, and thus it compresses less. We develop a simple function /(V) to estimate this ratio solely from the chosen vocabulary size, V. Specifically, we find that a quadratic function on the logarithmic value of V achieves accurate predictions:
f(V) log log)(V) + blog(V) +
By fitting several tokenizers with V ranging from 1K to 1024K, we obtain a = 0.0064, b = -0.1581 and e 1.2047. We find that our function accurately predicts the compression ratio with low relative mean square error (RMSE) and a high coefficient of determination (R²). In $A.9, we visualize fitting results and show that our approximation works with different tokenizers and is robust to different V. For all our main experiments, we use the BPE algorithm for tokenization [59].
Vocabulary loss To fairly assess models that vary in V, the commonly used language model loss in Equation 2 is inappropriate. Models trained with larger V naturally have a higher loss, there are more possibilities in the vocabulary to predict. However, this does not mean that the model is worse. Thus, we need to normalize the loss with respect to the vocabulary size. We reformulate the sequence WI:7, we design the unigram-normalizee normalized language model loss as:
metric [54] loss function. Suppose we have a T-length
1 p( M/w//ww:- (erllwss-,, V) Lu = T
log
V(w;VV)
where p(w; V) is the frequency of word w; in the tokenized corpus, given the tokenizer with vocabulary size V. The loss indicates the improvement in probability that a context-aware language model offers over a unigram model without context, allowing us to assess the language model's efficacy. Based on theory from prior work [54], the normalized loss L. remains consistent for a given model with fixed nocvbucay component across different vocabulary sizes. The difference of C, comes from the ability of the language model itself. Compared with L, the value of L, is much smaller and can be negative as C. adds negative term 1 > log p( V). One may also
'Vocabulary parameters typically encompass both the word embedding layer and the output layer. In this paper, for clarity and analytical simplicity, we employ V d rather than 2V to represent the vocabulary parameters. This choice is predicated on empirical observations: the main computational burden, as measured in FLOPs, is associated with the output layer. but not the word embedding layer.

---

3.4 1e19
-3.0
The solution of #FLO%s
FLOPs=2.5e+17
FLOPS=166e188
-3.5
FLOPs-4.9e+18
FLOPS=2.3c+19
-4.0
FLOPs=1.1e+20
FLOPs=3.7e+20 7e+20
2.5
-5.0
2.4
10*
10a
104
10a
Vocabulary
Vocabulary Size (V)
Figure 3: Left: FLOPs curve with various vocabulary sizes, assuming all configurations achieve fixed loss. There exists an optimal vocabulary size that minimizes FLOPs. Right: Loss curves with various vocabulary sizes given different FLOP budgets. For each budget there exists an optimal vocabulary size that minimizes loss. As the FLOP budget increases this optimal vocabulary size increases (shifts to the right).
employ the average bits per character (BPC), a common metric for text compression [27], as the vocabulary-insensitiee loss, The only difference lies in the normalization. BPC represents the raw per-character language model loss over the corpus, while our C, is equivalent the per-character language model loss normalized by the frequency of each character. In practice. we find that the metric BPC and L, show a significant positive correlation, which experimentally validated our statement, as detailed in the $A.5.
Analysis: Why the optimal vocabulary size is bounded by compute
Analysis 1: The perspective of fixed normalized loss (C) of Transformer model be estimated as C 6ND, which be re-written
According to Kaplan et al. [30], the FLOPs
C nJ 6ND Ro 6(N + Vd)Hf(V),
(5)
where N = NNv N. and D HV(() based $2.2. The reasons why model performance first increases and then decreases as the vocabulary size grows are: (1) At small V, increasing the vocabulary size easily improves tokenization fertility from f(V). Subsequently, more characters be learned from the model with a fixed number of tokens, thereby improving model performance. (2) At very large V. the gain from tokenization fertility decreases, while the parameters from expanding the vocabulary cannot be adequately trained with limited data, which leads to a decline in model performance. We present expanded derivation in $A.1, and show how the corresponding FLOPs change with the vocabulary size in Figure (left).
Analysis 2:
The perspective of fixed FLOP budget Given fixed FLOPs budget, isolate the FLOPs and investigate how the vocabulary influences the loss. For ease, we train models with fixed Naw and different vocabulary sizes for the same steps, and then use interpolation to predict the loss when FLOPs reaches the budget given the observed FLOPs and loss points. For each budget, we adopt group of models with similar total parameters and vary vocabulary sizes. In Figure 3 (right) we plot the relationship between the loss w.r.t. the vocabulary size. reveals that the vocabulary corresponding to the lowest point the loss curve increases the FLOPs budget increases. This suggests that with more computational resources, LLMs can harness larger vocabularie to reduce loss. However, merely expanding the vocabulary does not always lower the loss. For fixed FLOPs budget, the loss initially decreases with the increase in vocabulary and then starts to rise, indicating that optimal point exists for the vocabulary.
4 Estimating the optimal vocabulary size
In this section, describe three complementary approaches to estimate the optimal vocabulary size.
5

---

4.1 Approach 1: Estimating power laws via IsoFLOPs
We define 6 groups of models with Nav ranging from 33M to I.13B. Within each group, we solely vary the vocabulary size V from 4K to 96X, and evaluate different models under the same FLOPs budget. We evaluate the normalized loss Ca on a held-out validation dataset. This approach allows us to directly answer the question: For a given FI.OPs budget, what is the optimal allocation to non-vocabulary parameters, vocabulary parameters, and training data?
Setup Given certain Ny. the embedding di- mension is fixed, thus Ny increases V in- creases. For all experiments, we uniformly sam- ple the training data from different domains in the SlimPajama dataset [61]. All other hyperpa- rameters are fixed with more details in $A.7.
100000
80000
Vocabulary Size
-2
60000
-3
Fitting We select data points with the mini- mum Lu for each FL.OP budget, with all runs visualized in Figure 4. These points are the compute-optimal allocation to (Nuv. Nv. H). Following Kaplan et al. [30] and Hoffmann et al. [26]. we hypothesize that the optimal vocabu- lary parameters N. meet a power law w.r.t. the FLOPs just like the non vocabbauyy parame- ters and the amount of training data. Specifically, Nuv k2 C ,N, = k2 and H = Cy As model size and training data should be scaled equally for compute-optima training [26], we set (r) Qz., As our new attribute V signifi- cantly increases the number of possible experimental configurations, we employ interpolation across data points obtain more configurations cheaply. The details of the fitting are in $A.7.4.
40000
20000
-5
0
1036 10 1018 10 10
FLOPs
Figure 4: Training curves of the experiments used in Approach ($4.1) and Approach 3 (84.3). We train models with the -oncobbuulrry parameters fixed and vocabulary sizes varying from 4K to 96K.
Non-Vocabulan Parameters
RMSE-0.000 0.0090
RMSE-0 00799
10a
R² -0-0.52
0.08Fleps5..5
Vocabulary Parameters
M, 0.20F
R'-0.87
Training Characters
10
6 1-6642
R'
107
1010
10a
10a
10
10t6
103H
1000
10"7
10:4
10
1000
10"
103H
10
102H
FLOPs
FLOPs
FLOPs
Figure 5: Fitting results of the Approach 1. Blue stars denote the selected data points where the combination (Nnv. Ny, H) reaches the lowest loss given various FLOPSs budgets. We find power law fits with respect to the optimal non-vocabulary parameters, vocabulary parameters, and the number of training characters, respectively.
Results and Usage In Figure 5, we display the fitted power laws: N.v = 0.08 * 5 Nr 0.20 C and H 6.42 C0.50 where C is the FLOPSs budget. The low RMSE and high R² values indicate the strength of our fit. Given a certain FLOPS budget, we can utilize the aforementioned relationships obtain the optimal allocation (N H). We also draw the following conclusions: (1) LLMS are data- hata-hungry. Compared to the non-vocabulary parameters Nuv. practitioners should allocate more compute to the training data [80, 44]. (2) Vocabulary parameters scale in power- law relation with FLOPs (N, x CO.42). As models become more computationally intensive, larger vocabulary enhances the model's 's ability to understand a more diverse array of text, and thus the vocabulary size is critical to scaling. (3) Vocabulary parameters N,, should be scaled slower than non-vocabulary parameters Nuv. This difference can be seen in their power law exponents, i.e. 0.42/0.50 = 0.84 < 1. We hypothesize the reason is that: once sufficiently rich embedding space present via large vocabulary, is more critical to scale non-vocabulary parameters to learn the intricate syntactic and semantic structures of language via Transformer blocks.

---

4.2 Approach 2:
Derivative-baset fast estimation
We propose an alternative approach leveraging insights from the estimation of the FLOPs itself. Prior work [26, 30] usually considers a fixed compute budget in FLOPs and then aims to minimize loss by finding the optimal allocation to model parameters N and training tokens D. Here we flip this recipe on its head following recent work [57]. We aim to find the minimum FLOPSs to achieve certain loss f. Nnv. V, H) through optimal allocation of the vocabulary size V:
arg min (N,,
C(Nnv, H).
(6)
By computing the minimum point FLOPs C' with respect to V via derivative:
ac OV = 6H (Nnv + Vd) 2a Bog
VV + + [a(log(V)) + (llg())++bb)))))) + c)
V
we can estimate the optimal V. under the assumption that it can achieve certain loss C., (Nnw, V, H) The parameters a, b and e can be easily obtained from building S(V) ($2.2). In theory, long the non-vocabulary parameters Nuv are provided, V can be numerically searched via the solution of 8C = O. More details are in $A.1.
BV
Usage When the compute allocation is optimal, the loss exhibits power-law relationship with respect to the FLOPs budget, as described by the scaling law [30]. This relationship allows us use FLOPs parameters. In practice, we can first determine an empirically optimal vocabulary size in a low-cost setting (e-g., finding the compute-optimal vocabulary parameters on small model). Then, can scale the optimal vocabulary parameters proportionally based on Specifically, we obtain a set of derivative-optimal vocabulary parameters N₂ for different non-vocabulary parameters Nu represented as { N2 j = 1, ,n). We then fit the relationship between Nav and N, using the power-law function Nv x N This results in the scaling equation: N,/N. (Nnv/Ni.) y where No a small model (e.g., 33M), and N is the searched optimal vocabulary parameter. By combining the from the derivative and the empirical solution on a small model, we can estimate the optimal vocabulary by:
reliable proxy for observing the scaling behavior of the optimal vocabulary
Nopt No
NO0
where the scaling proportion 7 = 0.83 after our fitting. Consistent with the observation in Approach 1, we find that non-vocabulary parameters should be scaled faster than vocabulary parameters to achieve an optimal allocation.
4.3 Approach 3: Parametric fit of loss formula
Finally, we directly predict the loss given the non-vocabulary parameter, vocabulary parameter and the amount of training characters. Then, the optimal vocabulary configuration can be predicted by finding the minimum point of loss with respect to the vocabulary. Following classical risk decomposition used in Hoffmann al. [26], we design the vocabulary-deepedenn loss formula
A
A2
B D
Cy --
(
where D (VV). The first term captures the normalized loss for ideal generative process. The subsequent terms reflect the effect of the non-vocabulary parameters, vocabulary parameters, and the number of training data on the loss, respectively. E, A1, A2, B,0,, 02, are learned parameters.
Fitting We use the points (Nav. Nv. H) collected for experiments in $4.1. Note that we do not only consider the points with the lowest loss for each FLOP budget any combination of (Nav: Ny, H). We add the constraint a1 = 3 following Muennighoff et al. [44]. We also filter out points with very small FLOPs following Hoffmann et al. [26]. Fitting yields
want to predict loss for
A = 1.831, A₂ = 0.196, B = 2.124, E = 5.533, a = 8 = 0.447, a2 = 0.671. The detailed fitting process written in $A.7.4.
Usage After fitting the parameters in Equation 8, the optimal vocabulary size can be obtained by finding the lowest loss w.r.t the vocabulary size, with constraint of FLOPs budget. For example,
7

---

Current Optimal Vocabulary Parameters
Carert
StarCoder2-3B
B22.8OM
Cyrim
OLMo-7B
408.00M
Llama2-7B
131
Llama22133
113.0M
633,00M
Gemma2-98
12,05M
Gemma2-27B
Llama3-8B
135h 3466
Llama3-70B
1.058
2:216
M2-2
DeepSeek-67B
77211
Qwen2-72B
1.228
Falcon-180B
005.480 48M
Nernotron4-3408
7138
10
10
10
Vocabulary Parameters
Figure 6: Vocabulary parameters of popular LLMs and predicted optimal vocabulary parameters their reported number of training tokens, as determined by our Approach 3 ($4.3). Here consider the practical scenarios where parameters and training data are not necessarily equally scaled. As illustrated, the vocabulary parameters remain predominantly underestimated. With the exception of Gemma2-95 2-B,, all models allocate smaller vocabulary parameter count than our prediction.
given Nav and FLOPs budget C. by replacing [VV) with C/(6(NNvv + N.)) and finding the solution of dc 0 via numerical search, we can get the prediction. The details of 8C.. is written in $A.2. Note that all of the proposed approaches can be used in optimally allocating (N H) altogether, while Approach 3 is more flexible in predicting the locally optimal N, when (Nuv H) not following the Chinchilla's law [26], i.e. equally-scales law. The reason is that the loss formula in Approach 3 does not only considers the combinations (Nnv: Nv, H) which reach the optimal given a certain training budget. By fixing Nav and varying C' in Approach 3, we can predict the locally optimal vocabulary size with different amount of training characters. This property makes Approach 3 more valuable, since modern LLMs [70, 67, 3, 4, 7] usually leverage overly sufficient training data build powerful models with relatively low inference costs.
In Figure 6, we remove the assumption [26] for the practical reason that the parameters and training data not equally scaled. Then, predict the locally optimal vocabulary parameters. It can be observed that the allocation of vocabulary parameters are typically enterrestimated.
5 Discussion
Predicting allocations for larger models Table reports the predicted optimal vocabulary parame- ters and sizes based on the proposed three approaches, where the amount of training data optimally
Table 1: We report the predicted optimal vocabulary parameters N, and the vocabulary size V by the proposed three approaches given Nav. We assume the training FLOPs are optimally allocated i.c. that the non-voca con-ooabullary parameters and training data are scaled equally. "App" denotes the approach.
N.. Nopt-Appl
Nopt App2
Appp3
Dim.
popt
yopt.App2
App3
FLOPS Budget
3B
0.1B
0.1B
0.1B
3200
39K
43K
37K
1.3:21
7B
0.3B
0.3B
0.28
4096
62K
67K
60K
7.1e21
13B
0.4B
0.5B
0.4B
5120
83K
91K
81K
2.4e22
30B
0.9B
0.9B
6048
142K
154K
142K
1.323
70B
1.7B
1.9B
8192
212K
231K
218K
7.1623
130B
2.9B
3.2B
3.0B
12888
237K
258K
248K
2.4244
300B
5.8B
6.4B
6.3B
16384
356K
389K
383K
1.325

---

Table 2: Zero-shot performance of models with Nuo 2.87B comparing the commonly used V = 32K with our predicted optimal vocabulary Vogt. We consider the scenario where the number of training data equally scaled with the non-vocabulary parameters. We report accuracy and standard deviation in percentages. Accuracy is normalized: The predicted likelihoods are divided by the length of each choice for multiple choices to eliminate the effect of text length on predictions.
N D H ARC-C ARC-E Hellaswag OBQA
WG
PIQA
BoolQ
Average
FLOPs Budget 1.2e21 (Optimalyy-llocane) Training Data)
V=32K 0.10B 67.3B 266.6B 28.5 49.2.10
17.5
31.6121
50.4 71,4111 56.4.09
47.9
v35'= 0.IIB 67.IB 268.2B 1931 50.6
18.1
31.6121 51.9114 1A 71.4111 10.
48.5
Table 3: Zero-shot performance of models with Nuv = 2.87B comparing the commonly used
V = 32K with our predicted optimal vocabulary /opt when undertraining or overtraining.
N
D
ARC-C ARC-E Hellaswag OBQA
WG
PIQA
BoolQ
Average
FLOPs Budget 2. 2.8e20 (Insufficient Training Data, "Undertruining")
V=32K
0,10B
15.7B
62.2B
23.611.2 40.8
34,4
29.0 49.711. 64.911.1 19.8
43.2
=24K
0.08B
15.8B
60.8B
24.2
42.2
36.0
28.6120 50.0 64.9 61.5
43.9
FLOPs Budget
2.3e21
(Overly Sufficient Training Data, "Ovenvaining")
V=32K
0,10B
128.5B
509.1B
11, 53.5+10
53.0=as
33.0+21 52.0+14 72.0 59.5+0v
50.3
1 #33K 0.14B 127.0B 517.5 32.0 54.7
54.1105
33.0121 52.8114 72.611. 61.9109
51.6
allocated, i.e. equally scaled with the non-vocabulary parameters [26]. Aligned with the trend shown in Figure 1, the predictions from all proposed approaches align closely. Nnv should be scaled faster than Ny. Notably, mainstream LLMs typically assign fewer parameters vocabulary than what is optimal. However, the community is starting to shift to larger vocabularies, such as with Llama3 [41] having 128K vocabulary size up from 32K of Llama2 [70]. However, scaling data is still the most critical part, and solving data scarcity issues should be a focus of future work [72].
To empirically verify our prediction, training FLOPs budget and evaluate them using lighteval 2. For the baseline model we use the common vocabulary size of V 32K. The other model uses /ropt as predicted by Approach 3. In Table 2, we show that the model allocated according to our vocabulary predictions yields better performance across multiple downstream tasks. This verifies that our predictions hold at scale.
train models with N.v 2.87 under compute-optimal
Experiments with scarce and excessive training data Our prior experiments focus on the setting where training compute budget is the main constraint and we seek to allocate it optimally to parameters and training data. This the typical setting in scaling law studies [30, 26, 52]. However, in the real world, we often deal with scarce data ("data-constrained [44]") forcing us to train sub-optimally or would like to make use of excessive data train a smaller model that cheaper to use [84]. To verify that our Approach 3 can handle these practical scenarios, we compare the model with V 32K and the model with the vocabulary size Vops predicted by Approach 3. As shown in Table 3, our prediction enables better model by only adjusting the vocabulary size in different FLOPs budgets.
In Figure 7, we further study the trend about how does the optimal vocabulary size shift with different number of training data. We only vary the amount of data but keep the non-vocabulary parameters fixed. The choices of vocabulary size are 8K. 10K, 16K, 24K, 32K and 48K. Taking Nn. 302M as example, when available data is the bottleneck, the optimal vocabulary size decreases empirically, iee. 16K - 10K. This a mechanism to prevent over-fitting. Conversely, when training on excessive amounts of data, e.g., Llamm3-8B uses much more training tokens than what would be compute-optimal for its budget, the optimal vocabulary size increases, i.e. 16K + 24K. Note that here we focus solely on training compute-opimmal It is also important to note that expanding the vocabulary size also increases the computational demands during inference. Therefore, we recommend using the optimal vocabulary size corresponding to given Nav. assuming optimal allocation of training data, even in scenarios where overtraining may occur.

---

40K
Insufficient Training Data
V=48K
Optimally alloccted Treining
Overly sufficient Training Data
Vocabulary Size
32K
-4.7
V-24K
Training Tokens
V-16K
24<
V-8K
10k
4.99
10C
1
OK
-5.1
L.att Poot Lo Se Loar Soer 5.0++ S.
5.3e+11
FLOPs Budget
FLOPs Budget
The heatmap illustrates how the best vocabulary size among all choices of vocabularies shifts with the training data. The non-vocabulas vocabulayy parameter is fixed (N = 302M). Each cell in the heatmap represents the loss given certain FLOPs budget for fair evaluation, with the color intensity indicating the loss value. The black line with markers denotes the best vocabulary size for each FL.OPs budget, which basically increases as the number of training data increases. Right: The number training tokens are slightly varying for different vocabulary sizes given a certain FLOPs budget. To keep FLOPs consistent, models with larger vocabulary sizes trained fewer tokens.
Figure 7: Left:
6 Related work
Language models The Transformer [71] has proven to be scalable architecture for language
models, especially large language model (LLMs) [11, 14, 52, 47, 20, 29, 53, 70, 73, 41, 8, 4, 38, 25, 62, 67, 7, 39, 32, 88). These models typically acquire deep understanding of language enabling them to perform multiple tasks after a pre- pre-ttrinnin period and an optional fine-tuning period. Their capabilities include code generation [33, 3, 43, 87, 86], mathematical reasoning [76, 5]. question answering [48, 45] among others. Given the expensive deployment costs required by the language models, various techniques can be adopted for efficient inference [64, 65, 78, 74, 36). In work, we pre-train large language models from scratch on English corpora and focus on their validation loss and downstream performance after training.
Scaling laws Scaling laws aim to develop predictive framework to find the best allocation of compute resources to maximize model performance. Besides language models, they have been studied in other domains[40, 68, 13]. For language models, Kaplan al. [30] show that performance improves as a power law with more compute allocated to both parameters or data. Hoffmann et al. [26] show that the compute allocation of parameters and data should be scaled equally. Other work considers various cases such as downstream performance [23, 28, 55], inference time [57] or data constraints [44, 80]. However, the effect of vocabulary size has generally been ignored previously.
7 Conclusion
We investigate the impact of the vocabulary size in language models. We analyze and verify that there exists an optimal vocabulary size for given FLOPs budget. Subsequently, we develop approaches to predict the optimal vocabulary size. Our first approach uses empirical training runs across different IsoFLOPs regimes fit scaling law. The second approach investigates the FLOPs w.r.t. the vocabulary size and estimates the vocabulary size with derivatives. The third approach consists of parametric function to predict the impact of different attributes on loss. Across all approaches, we find that while vocabulary parameters should be scaled slower than other parameters, they are still critical for performance and we can accurately predict their optimal allocation. We make predictions for larger models and empirically verify our approaches on up to 3B parameters and on varying amounts of training data. We show that models trained with an optimal vocabulary size as predicted by our approaches outperform models with conventional vocabulary size under the same FLOPs budget.
10

---

Acknowledgements
This work supported by in part by the Theme-based Research Scheme (TRS) project T45-701/22- of the Research Grants Council (RGC), Hong Kong SAR.
References
[1] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. 2022. Cm3: A causal masked multimodal model of the internet. arXiv preprint (.Xi:22201.07220
[2] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023, Scaling laws for generative mixed-modal language models. In International Conference Machine Learning, pages 265-279. PMLR.
[3] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. SantaCoder: don't reach for the stars! arXiv preprint arXiv:2301.0394 033988.
[4] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malar- tic, et al. 2023. The falcon series of open language models. arXiv preprint 1xXXx:2311..6800
[5] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. arXiv preprint 2rxiv:2330..000
[6] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701.
[7] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng. Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Alex X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie. Tian Pei. Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha. Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang. Peiyi Wang. Shiyu Wang, Yaohui Wang, Yongji Wang. Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie. Yiliang Xiong. Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. CORR, has/2401.02954
[8] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR.
[9] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa:
Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439.
[10] Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A Smith, and Luke Zettlemoyer. 2024. Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models. arXiv preprint arXiv:2401.10444.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901.

---

[12] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang. Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv. Haijun Lv, Kai Lv. Li Ma, Runyuan Ma. Zerun Ma, Wenchang Ning, Linke Ouyang. Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang. Bin Wang. Guoteng Wang, Jiaqi Wang. Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang. Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang. Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. 2024. I2ternLM2 Technical Report. arxiv.
[13] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-82110 28-9.2..
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113.
[15] Hyung Won Chung. Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving Multilin- gual Models with Language- -laustrr- Vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536-4546, Online. Association for Computational Linguistics.
[16] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905
[17] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 10330:1803.05457.
[18] Gautier Dagan, Gabriel Synnaeve, and Baptiste Rozière. 2024. Getting the most out of your tokenizer for pre-training and domain adaptation. CoRR, abs/2402.010355.
[19] Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. 2024. Sailor: Open Language Models for South-Fast Asia. CORR, abs/2404.03608.
[20] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2 10360.
[21] Seppo Enarvi, Peter Smit, Sami Virpioja, and Mikko Kurimo. 2017. Automatic speech recogni- tion with very large conversational finnish and estonian vocabularies. IEEE/ACM Transactions on audio, speech, and language processing, 25(1 1):0085-0207
[22] Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming transformers for high- resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12883-12883.
[23] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. 2024. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.088: 08550..
[24] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- YieYann Liu. 2019. Representation degener- ation problem in training natural language generation models. arXiv preprint arXiv: 1907.190900
12

---

Singapore. Association for [25] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. Olmo: Acceler- ating the science of language models. arXiv preprint arXiv:2402.008S
[26] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:
[27] Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. 2024. Compression Represents Intelligence Linearly. CoRR, 0bs/2440.00937
[28] Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling Laws for Downstream Task Performance of Large Language Models. arXiv preprint arXiv:2402.04177.
[29] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825
[30] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown. Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint S.tiv:2001.0836:1
[31] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2023, Bloom: 176b-parameter open-access multilingual language model.
[32] Jeffrey Li, Alex Fang, Georgios Smymiss Maor Iygi. Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang. Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Iharco,, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer. Kyle Lo, Alaacldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexan- dros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. 2024. DataComp-LM: In search of the next generation of training sets for language models.
[33] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. 2023. Starcoder: may the source be with you! arXiv preprint 2rXiv:2305.0616.
[34] Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages Computational Linguistics.
[35] Xianwen Liao, Yongzhong Huang, Changfu Wei, Chenhao Zhang, Yongqing Deng, and Ke Yi. 2021. Efficient estimate of low-frequency words' embeddings based on the dictionary: a case study on Chinese. Applied Sciences, 11 11(22):11018.
[36] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. 2024. DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. arXiv preprint arXiv:2406.0177
[37] Ilya Loshchilov and Frank Hutter. 2017, Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
[38] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint 2917732402.19173.
13

---

[39] Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna- Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. 2023. Fingpt: Large generative models for small language. arXiv preprint urXiv:231.00566
[40] Kangfu Mei, Zhengzhong Tu, Mauricio Delbracio, Hossein Talebi, Vishal M Patel, and Peyman Milanfar. 2024. Bigger is not Always Better: Scaling Properties of Latent Diffusion Models. arXiv preprini arXiv:2404.01367.
[41] Meta AI. 2024. Meta LLaMA-3: The most capable openly available LLM to date. https: //ai. .eeta. cww com/blog/metal
[42] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of ar- mor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:180.002789.
[43] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07131
[44] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36.
[45] Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual Generalization through Multitask Finetuning.
[46] Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar. Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng. Jimmy Zhang, Jing Zhang. Vivienne Zhang, Yian Zhang, and Chen Zhu. 2024. Nemotron-4 340B Technical Report. arxiv.
[47] OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744.
[49] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, al. 2023. Rwkv: Reinventing mas for the transformer era. arXiv preprint arXiv: 13048.
[50] Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Fu- gene Cheah, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, et al. 2024. Bagle and Finch: RWKV with matrix- states and dynamic recurrence. arXiv preprint 2404::2204.0
14

---

[51] Alec Radford, Karthik Narasimban, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
[52] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 /2. 1446.
[53] Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang. Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. 2023. Pangu-( (iggma]: Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint arXiv:2303. 10845.
[54] Jihyeon Roh, Sang-Hoon Oh, and Yoo-Young Lee. 2020. Unigram-normalizer perplexity as a language model performance measure with different vocabulary sizes. arXiv preprint arXiv:2011.13222
[55] Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Observational Scaling Laws and the Predictability of Language Model Performance.
[56) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande:
An adversarial winograd schema challenge at seale. Communications of the ACM, 64(9):99-106.
[57] Nikhil Sardana and Jonathan Frankle. 2023. Beyond chinchilla-opimmal:: Accounting for inference language model scaling laws. arXiv preprint arXiv:2401.0044/
[58] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022. What language model to train if you have one million gpu hours? arXiv preprint arXiv:22 154244
[59] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
[60] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-Im: Im: Training multi-billion parameter language models using model parallelism. arXiv preprint wwX.vv/909.000
[61] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.
[62] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et 2024. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, arXiv preprint arXiv:2402.00115.
[63] Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2017. Do neural nets learn statistical laws behind natural language? PloS one, 12(12)e0189336.
[64] Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023. Structured pruning for efficient generative pre-trained language models. Findings of the Association for Computational Linguistics: ACL 2023, pages 10880-10895.
[65] Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203. 10705
[66] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. 2023. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? In Findings of the Association for Computational Linguisties: EMNLP 2023, pages 12342-12364, Singapore. Association for Computational Linguistics.
15

---

[67] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint 24Xi::4433.02295
[68] Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng, and Liwei Wang. 2024. Visual Autore- gressive Modeling: Scalable Image Generation via Next-Scale Prediction. arXiv preprint wnXiv:240.002055.
[69] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023, Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
[70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar arXiv:2307.09288.
[71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
[72] Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? an analysis of the limits of scaling datasets in machine learning, arXiv preprint arXiv:2211.0444
[73] Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, et al. 2023. Efficient large language models: A survey. arXiv preprint arXiv:2312.03866 1.
[74] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang. Siqi Luo, Jing Xiong, and Mi Zhang. 2024. D20: Dynamic Discriminative Operations for Efficient Generative Inference of Large Language Models. arXiv preprint arXiv:2406. 13035.
[75] Hai Wang. Dian Yu, Kai Sun, Janshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual models with vocabulary expansion. arXiv preprint 1rXiv:1909.12444
[76] Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujum Li. Lin Li, et al. 2023. Dt-solver: Automated theorem proving with dynamic-tree sampling guided by proof-level value function. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12632-12646.
[77] Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. 2024. Mam- babyte: Token-free selective state space model. arXiv preprint arXiv:2401.13660.
[78] Taiqiang Wu, Chaofan Tao, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. arXiv preprint arXiv:2404.02657.
[79] Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2020. Vocabulary learning via optimal transport for neural machine translation. arXiv preprint arXiv:2012.15671.
[80] Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. 2024. To repeat or not to repeat: Insights from scaling Ilm under token-crisis. Advances in Neural Information Processing Systems, 36.
[81] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024. Qwen2 Technical Report. arxiv.
16

---

[82] Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2024. Megabyte: Predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems, 36.
[83]
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:190S.0788.
[84] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.
[85] Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre- Training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages -215, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
[86]
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong. Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. 2024. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions.
[87] Terry Yue Zhuo Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. 2024. Astraios: Paramette-Efffcieet Instruction Tuning Code Large Language Models.
[88] Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong. Wei- Yin Ko, Daniel D Dosuzaa Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadace, Julia Kreutzer, and Sara Hooker. 2024. Aya Model: An Instruction Finetuned Open- Access Multilingual Language Model.
17

---

A
Appendix
A.1 The derivation of FLOPs w.r.t the vocabulary size for the Approach
Here we provide the detailed process of how we compute the extreme point of FLOPs C' with respect V. From Kaplan al. [30], we know that:
C C = 6(N, + Vd)
(9)
We then compute the derivative 8C
DV as follows:
ac
oV 16( (NN + H (S(V))
aV
[6(N + dV) (a(log(V))) + blog(V) + c)]
0v
(Nnv + dV) d dV (a(log(V)) + blog(V) +c)
6//
+ (allog(V))² + blog(V) c) dv (Nnv + dV)
2a Laglo((V V) +
6H (Nnv + Vd)
V + (a(log(V))) + blog(V) + +c) d
The solution of 8C 0 corresponds to the minimum point of the FLOPS. Since the variable V in this equation not separated conveniently, scipy. www.mizetmmee...m feolime..e..leee......eo fsolve, to find the solution.
numerical search method, specifically
Example demonstration Figure 8 illustrates the relationship between the derivative of FLOPs with ac respect the vocabulary size V and V itself. Setting V as the solution to ØV O, find the point at which FLOPs are minimized. As depicted in Figure (right), the FLOPs budget is fixed, and we observe how the training character varies with V. Notably, the optimal vocabulary size V, the model expends the maximum number of training characters for given budget. This observation provides insight into why optimal vocabulary size exists for given FLOPs budget.
1e14
3.4 1e19
1e10
The solution of
#FLOPs
Derivative of FLOPs
3.2
Training Characters
4.0
-2
3.6
2.6
The solution
3.2
The solution of .f.0s
-6
2.4
10
10a
10*
10a
10*
10a
Vocabulary size (V)
Vocabulary size (V)
Vocabulary size
Figure 8: Left: The curve of the derivative FLOPs with respect vocabulary size V. The curve of 8C increases V increases, and the FLOPs reach minima at the solution of #C 0. Middle: The 2V
curve of FLOPs with respect to vocabulary size V. where reaches its optimal point V. Right: The curve of training characters with given FLOPSs budget. Take N.. 302M and H 43B as example. The FLOPPs budget is decided by the Nnv. and the predicted V.
A.2 The derivation of loss w.r.t the vocabulary size in Approach 3
Here we provide how we derive the loss w.r.t the vocabulary size given FLOPs budget C in Approach 3. After substituting the [V)) with the C/(6(NVv NN)) based Equation 9:
18

---

A2
C.
(10)
Nuv
N
CC/(6(N + N.)1
The loss is solely dependent on the No
Vd., given a Nav. The derivative w.r.t V is:
ac 0
A2
a
OV
2V
(Vdjoz)
3
((Nav+Vat)
A2d
B
Cd VVdz
02 (Vdjoz+1 + B
B+I
6(N
The solution of ac2 0 corresponds to the optimal V. Similar with Approach 2, we use ay scipy. opt.mize........... fsolvv fsolve to find the solution.
A.3 More visualizations for the analyses: Why the optimal vocabulary size bounded by the compute
Normalizee aenilleeed logifrequency!
0.12
1.0
Norma Feeeveenyyyyy)
0.12
Norma lized (egiFeequency
0.8
0.8
0.8
0.6
0,6
0.8
0.00
0.00
0.4
0.4
0.
0.2
0.
0.2
-0.12
0.0
-0.12
-0.12
-0.1
0.0
0.1
-0.1
0.0
0.1
0.1
0.0
0.1
Figure 9: The SVD plots of the learned word embedding for V=4K (left), V=16K (middle) and V=64K (right) for a model with Nuv 85M.. Different colors represent different log frequencies.
Word embeddings in large vocabulary are hard to learn when FLOPs are constrained Previous studies have shown embeddings suffer from representation degradation, where low-frequency word embeddings cluster together due to limited parameter updating [24]. Figure 9, we visualize how the word embeddings distribute using different vocabulary sizes. We use the average Euclidean distance among all the embeddings, Daug. to quantify the degree of clustering, which is 1.067, 1.01 1, and 0.952 for V 4K, V 16K and V 64K, respectively. Larger vocabularies (64K) lead to more clustering of embeddings, especially for infrequent words. This clustering suggests that they have been insufficiently trained. Conversely, small-sized vocabulary (4K) and middle-sized vocabulary (16K) display a more dispersed distribution of embeddings. These observations suggest that there exists optimal vocabulary size that balances lexicon coverage and sufficient updating of word embedding. Language models with large vocabulary sizes may have better lexicon coverage, but on the other hand, hinder the model's ability sufficiently update the word embeddings, especially for low-frequency words.
A.4 Exploration of Larger Range Vocabulary Sizes
Because of computational resource constraints, the vocabulary sizes used to fit the scaling laws are the range of 4K to 96K. This range is sufficient fit, because the optimal vocabulary sizes for all the training configurations we used fall in this range.
To further verify that there always an optimal vocabulary size holds for a larger range of vocabulary lists, we increase the range of vocabulary sizes from 0.5K to 512K, with the Nay fixed as 33M. As depicted in the Figure 10, the model's performance declines consistently as vocabulary size increases beyond the optimal configuration. This figure shows loss curves for vocabulary sizes up to 512K, given specific FL.OPs budget. The data indicates a consistent degradation in model performance with the vocabulary size away from the optimal one. suggests that there is critical
19

---

-15
11075=2.5e+17
-2.0
-2.
-3.0
-3.5
-4.0
103
10a
103
BPC reflects the ability to compress external text corpora [27], while the unigram-normalized normalized loss reflects the model's ability predict tokens normalized by the token frequency. Figure shows the relationship between the -neeraai--o Loss L, and Bits Per Character (BPC) with linear fit. We select models of final training steps for each No. and each vocabulary size. Vocabulary Size (V)
Figure 10: Loss curves with larger range of vocabulary sizes (from [4K, 96K] to [0.5K, 512K)), given certain FLOPs budget. The model performance degrades consistently when the vocabulary size goes beyond the optimal configuration.
point beyond which the model's efficiency in handling the vocabulary diminishes. This exploration underscores the importance of carefully selecting the vocabulary size maintain optimal model performance within the constraints of given computational budget.
A.5 The Vocabulary-insensitty Metrie: C., and BPC
-3.8
Ihigram-normalized Loss Co
p=0.9888
-..0
e=0.0683
4.2
-4.4
-4.6
-4.8
-5.0
-5.2
-5.4
0.8
0.9
1.0
1.1
1.2
1.3
1.4
Bits Per Character (BPC)
Figure 11: Correlation between the unigram-norman loss Lu and BPC, where p and denote the Pearson correlation coefficient and the root mean square error of the linear fit, respectively.
The high correlation coefficient (p 0.9888) and low error (e 0.0683) indicate a strong linear relationship between these two metrics generally. However, it exists slight different trends due to different normalizations.
20

---

A.6 More Explanations about Why We Separate Vocabulary Parameters from the total Model Parameters
Traditionally, scaling up model parameters in language models has been approached in two ways: increasing depth (i.e., the number of layers) or width (i.e., the hidden size). While extensive research has been conducted on these methods, current empirical practices often involve expanding both simultaneously [66]. This approach, however, may overlook crucial distinctions how different parameters benefit from these expansions.
Von-vocabulary parameters can benefit from increases in both depth and width, allowing for more complex hierarchical representations and broader feature capture. In contrast, vocabulary parameters, associated with word embeddings and language model heads, are generally confined to single layer, limiting their ability to benefit from increases in the model depth. Their primary avenue for expansion is through increasing the width. This disparity in growth potential between -on-vocabulary and vocabulary parameters suggests that to maintain balanced growth rate, it may be necessary to expand the vocabulary size along with the depth. This would allow the vocabulary parameters keep pace with the growth of con-oocabulary parameters.
A.7 Implementation details
A.7.1 Setting of model architecture, vocabulary size and training characters
We list the architectures of the models and the corresponding number of training characters in Table For each model family, we fix the non-vocabulary parameters Nav and vary the vocabulary size. adopt the Llama architecture [69], except for the vocabulary size. For the vocabulary size, we use numbers divisible by 128 for compatibility with NVIDIA's tensor core to accelerate matrix multiplication 3. Specifically, the vocabulary sizes we adopt for each model family are 4096, 6144, 8192, 10240, 16384, 24576. 32768, 48128, 64512 and 96256. The expected number of training tokens D and characters И vary slightly given a fixed number of non vocabulary parameters and a FLOP budget. We use the middle-sized of 16384 determine the number of training characters and the corresponding FLOPs budget, except for Nuv 2870M we use 32K.
Table 4: The architectures of the models and the corresponding number of training characters adopted in our experiments.
#Sequence Length
#Embedding Dim.
#Intermediate Size
Training Characters (B)
Nav (M)
#Layers
#Heads
2048
512
2048
4.3
85
2048
12
12
768
2048
11.1
151
2048
16
12
768
3072
19.6
302
2048
1024
4096
43.0
631
2048
20
24
1536
4800
101.6
1130
2048
22
32
2048
5632
201.3
2870
2048
32
3200
8192
509.3
A.7.2
The relationship between non-vocabulary parameters and embedding dimension
According to the observation Kaplan et al. [30], the depth-widtt width ratio has relatively small effect on performance given the total non- vocabulary parameters. Thus. to case the modeling of our scaling laws taking vocabulary size into account, we take the width (i.e. embedding dimension) as given following prior work [30, 26, 44, 70, 84]. The relationship between the non-vocabulary parameters Nuv and embedding dimension used in our experiments are Table
A.7.3 Training details
The maximum learning rate is set to 4e-4 and decays to 10% i.c. 4e-5 similar to prior scaling work [26, 44]. use AdamW [37] as our optimizer and accelerate training with 6 mixed
'nttps://docs nvidia. htt/////eeeeeeeee///oo/ooeaeoeee
materm-triiic i mitippicationn
21

---

Table The relationship between the non-vocabulary parameters Nay and the embedding dimension used in our experiments.
Non-vocabulary Parameters Nuv
#Embedding Dim.
N S 50M
512
50M N S 200M
768
200M < Nnv S 500M
1024
500 < NN Σ 1B
1536
1B < Nnv S 2B
2048
2B < Nnv 2 5B
3200
5B < Nav S 10B
4096
10B < Nuv Σ 20B
5120
20B < NNw I 50B
6048
50B < Nav _ 100B
8192
100B Nnv S 200B
12288
200B < NnV I 500B
16384
500B Nav 1000B
20480
precision training. For models with Nav < 1130M we use a single node with 8 GPUs for training. Otherwise, adopt the Megatron- LM framework [60] for multi-node training with 8 GPUs on each node. For our experiments with Nnv = 2870M, it takes about 120 hours to train on over 500B training characters with 64 total GPUs. We global batch size of 512 for all runs and run all experiments on 40GB Nvidia-A GPUs.
A.7.4 Fitting techniques
To avoid numerical underflow and overflow of the fitting parameters, we fit the data in a logarithmic form inspired by Hoffmann et al. [26]. Taking Nav as an example, we learn the parameters ki, on by minimizing:
Approach 1
min Huber, Ki + on log(C), log((n.))). K1,01
(11)
where K₁ (k1) and Huber; denotes the Huber loss with delta value 8 0.001 in our paper). We use the LBFGS algorithm to find the local minima of the function. The later Approach and use the same optimization algorithm. We initialize all attributes from the same uniform grid where K E [-00, 15) and E [0, with 20 initial guesses respectively. The fitting takes less than half of one minute.
To cheaply obtain more experimental data points, we perform interpolation of (Nav- No, H) triplets in the logarithmic scale and predict the validation loss based on real data points. Then, we compute the required FLOPs for each data point using Equation 5.
Approach By using different Nov and obtaining the corresponding optimal N,, based Equation 7, we have a set of {(Nor; No:) Vi 1, n.. Denoting Div. Nnv,/N /NNep and D Nv, NNo. we learn the scaling proportion y by minimizing:
min Hubers (r * log(D"".) log (D")),
(12)
The initial guess of y uniformly sampled from [0, 1].
Approach 3 We recast the designed vocabulary loss formula here:
A
A2
B
C. E
+
(13)
No1
Noa
[Hf(V)}
where 3 = aj. In practice we try to minimize:
min at br,,,,,,
Hubers( exp(e) + exp(a1 an * log((Nw) + cxp(a2 ag * lo,(NN,)
+ cxp(( loog((H((V))) L.).
22

---

where A₁ = exp(a1). = exp(az), B = exp(b), E exp(e). We initialize all attributes from the same uniform grid where o₁ E [0,5]. d2 [0,5], b E [0. 5]. E [0, 2|. a E [0. 1] and a2 E [0. 1] with 3 initial guesses respectively. Given the prior that the scaling factor is typically ranged between and I [26]. we add a constraint 0.1 < on, 02 < during fitting. The fitting also takes less than half of one minute.
A.8 Details of fitting tokens-character relationship function f(V)
We train 25 tokenizers with the following vocabulary sizes: 1024, 2048, 3072, 4096, 5120, 6144, 7168. 8192, 9216, 10240, 12288, 16384, 20480, 24576, 28672, 32768, 48128, 64512, 78848, 96256, 128000, 256000, 512000, 1024000. Then, we train the tokenizers on a uniformly sampled version of the Slimpajama dataset.
Later, we apply the trained tokenizers on the validation of the Slimpajama dataset and collect the number of tokens D for each tokenizer with vocabulary size V. We use scipy.optinn optimize.............. curve_fit to fit the parameters b, in f(V) ($2.2).
A.9 Robustness of the tokens-characters relationship function f(V)
Robustness to the type of tokenizers Besides the widely adopted BPE tokenizer used in our experiment, we also consider the unigram tokenizer and the word-based tokenizer. We visualize their tokens-characters ratio and corresponding predictive function in Figure 12. We find that our proposed formula of f(V) is good predictor for the tokens-character ratio, regardless of which tokenizer used. This verifies the effectiveness of our proposed formula. The tokenization fertility of the unigram tokenizer is close to that of the BPE tokenizer seen in their similar y-axis values, since they both employ subword- based tokenization. Meanwhile, the tokenization fertility of based tokenization is poor, thus requiring more tokens average compress characters.
Tokens-Cheractery retoo (DM)
Real
Thkrens www.Chhaartrrsse ratio (D/H)
0.45
Real
Tokeens, Characters ratio (DH)
Real
0.40
Prediction
Prediction
0.8
0.40
0.7
0.35
0.35
0.6
0.30
0.30
0.5
0.25
0.25
10
10
10a
10a
10
10a
10
10a
10'
Vocabulary size (V)
Vocabulary size (V)
Vocabulary size (V)
(a) BPF tokenizer
(b) Unigram tokenizer
(c) Word-based tokenizer
RM =0.99
RMM SLMSE=5.2e. R==0.98
R. L.M.6-3.6.55. =0=09
Figure 12: The modeling of function f(V) with different tokenizers. relative mean square error and coefficient of determination, respectively.
RMSE and R² denote the
Robustness to the range of the vocabulary size The quadratic function on the logarithmic value of vocabulary size that we propose can precisely predict the tokens-characted ratio with an RMSE of 1.5e-6 and R² of 0.99. However,
quadratic function single-peaked, increasing V will increase the output value of ʃ(V) a (V) + blog V when V is very large, e.g. V >>>exx exp(-b/2a) Rv 218K in our case.
Fortunately, when V is sufficiently large, the tokenization fertility improvement of the tokenizer decays sharply, which results in almost no change to the value of f(V). This is because the words in the training corpus can already be effectively covered by the vocabulary list when the vocabulary size sufficiently large. In this extreme, the tokenization fertility of the corresponding tokenizer is approaching saturation, thus further increasing the vocabulary size will hardly improve the tokenization fertility.
As an example, there are about 2300M characters in the validation set of the Slimpajama corpus. A tokenizer using vocabulary size of 2K would yield 140M fewer tokens than a IK counterpart, but the number of tokens only decreases by 0.7M when going from vocabulary size of 256/ to
23

---

257K. Therefore, we add min(V, 200 before calculating f(V) to ensure its decreasing nature. According to our prediction. model with 300B parameters has an optimal vocabulary size of no more than 400K with a sufficient amount of training data. If we need to consider extremely large V in the future, we can train tokenizers with larger V in the process of fitring f(V) to arrive more precise predictions.
A.10 Experimental verification on the fairness of the loss
language modeling
4.75
2.8
Language Modeling Loss
corrmmmloommlizedd Loss
-4.0
2.6
-4.8
2.4
-4.9
2.2
-4.95
-5.00
0.406
0.412
0.418
0.424
0.406
0.412
0.418
0.424
Average Normalized Accuracy
Average Normalized Accuracy
(a) Relationship between downstream task per- formance and the commonly-used language modeling loss.
(h) Relationship between downstream task per- formance and unigram-norr normalized language modeling loss.
Figure 13: Empirical examination of the fairness of our noremaizedd loss, Cy. Dots corre- spond to trained models with varying vocabulary size. We plot their losses (y-axis) and performance on downstream tasks (x-axis): WG [56], PIQA [9], OBQA [42], Hellaswag [83], BoolQ [16], ARC-E [17] and ARC-C [17]. The straight line reflects the results of the regression fit with the shade indicating the confidence interval.
In $2.2, we have explained that we use unigram-normalizze loss, C.. to fairly evaluate models that vary in vocabulary size. Here we empirically verify this choice. We train models with fixed number of non-vocabllary parameters N and embedding dimension d but varying vocabulary sizes V. Thus, their vocabulary parameters N₂ also vary. We plot the final language model loss and mnigram-normalizee loss of these models compared to downstream performance in Figure 13. The language modeling loss exhibits a positive correlation with downstream performance: Models with a higher language modeling loss have better downstream performance. This is because our models with larger vocabularies naturally have a higher loss due to the objective function, yet they can be actually better models with better downstream performance. Our unigram-normen problem and exhibits the expected negative correlation between loss and downstream performance: lower loss comes with better downstream performance. This empirically justifies of L., throughout this work.
loss solves this
A.11 Prediction for Llama3
While our primary experiments focus on the Llama2 vocabulary size, we also extend conclusions to Llama3, predicting its optimal vocabulary under optimal compute allocation. As shown in Figure 14, we provide detailed predictions for various sizes of Llama3 models. Although Llama3 significantly increases its vocabulary size from 32K to 128K, our research suggests that this may still be insufficient for the larger model sizes of 70B and 400B.
A.12 More Related Work
Vocabulary in language models The vocabulary of language model influences its performance
significantly [63, 75, 79]. A larger vocabulary size helps cover more words thus reducing likelihood of out-of vocabulryy (OOV) cases [21]. Takahashi and Tanaka-Ishi [63] find that larger

---

10
Vocabulary Parameters Nv
Approach 1, y=
Liama3 8B
Approach 2, 0.88
Uama3-708
1010
Approach y=0.86
Uama3-4008
Optimal Vocab: 487K
Liama3 Vocab: 128K
Empirical Results
Optimal Vocab: 212K
109
108
0000
000
Optimal Vocab:
000
000
opt o (Nnv))
107
0000
1000
000
108
109 Non-Vocabulary Parameters Nnv
1010
10
10
Figure 14: The replication of Figure for Llama3. As shown, the predicted optimal vocabulary size for the Llama3-400B model is high 487K.
vocabularies are better at capturing the true statistical distribution of language. Similarly, expanding vocabulary in multilingual models [75, 15, 85, 34) improves performance, especially for low-resource languages. However, large vocabularies [31] increase the computational overhead during both training and generation phases. For example, Liao et [35] demonstrate that low-frequency words often lack sufficient examples develop robust representations when vocabularies are excessively large. Dou al. [19] reveal that expanding vocabularies during continual pre-training can lead to significant performance degradation for low-resource languages. More recently, Dagan et al. [18] explored the trade-offs associated with vocabulary size, proposing optimal vocabulary sizes for both memory efficiency and inference speed in code generation tasks. Our work complements these efforts by focusing on the broader impact of vocabulary size on downstream performance across various tasks. Specifically, address critical, under- under-explored question: How can optimally allocate vocabulary size to maximize the downstream performance with the same compute budget?
Byte-level language models Recent work has explored byte-level language models [82, 77]. which offer advantages in decoding efficiency and noise robustness compared to token-level models. However, typically limited to parameters under IB, these models have not been effectively scaled up. Our scaling laws suggest that the limited vocabulary (i.e., 256 in byte-level language models) may constrain their performance, especially for larger models. The insight provides potential explanation for the challenges in scaling byte-level models and implies that successful scaling of language models may require proportional increases in vocabulary size.
B Limitation and future work
B.1 Limitations of our proposed approaches
Approach The Approach provides a broader solution by predicting the allocation of computa- tional resources across non-vocabulary parameters, vocabulary parameters, and training data based on experimental data points. This method's strength lies its holistic view, allowing for balanced resource distribution that potentially enhances model efficiency and performance. However, this approach is constrained by the granularity and range of the experimental data points available, which can introduce errors in the fitting process. The requirement for substantial computational resources to perform these fittings may also limit its accessibility and scalability. Despite these challenges, when experimental data is ample and computational resources are sufficient, the Approach I can significantly refine the precision of resource allocation decisions in the development of large-scale language models.
25

---

By calculating the derivative of FLOPs with respect to the vocabulary size and solving for zero, this approach fundamentally relies on the precision of the FLOPs equation and our tokens- characters relationship function. Further, this method does not allow us to independently determine the optimal allocation of non-vocabulary parameters and training data size. Therefore, it necessitates information about the relationships between these attributes and the FLOPs budget from the experi- mentally fitted scaling laws, making this approach less useful in practice. Despite these limitations, the derivative-based approach offers notable advantages, including closely matched predictions with the scaling laws derived from actual experimental data in the Approach 2. Furthermore, its reliance on numerical solutions rather than exhaustive deep learning experiments makes it rapid and broadly applicable across various tokenizers, highlighting its utility in preliminary model configuration stages where quick estimates are key.
Approach 2
Similar with the Approach I, the proposed Approach 3 requires multiple experimental runs across different non-vocabulary parameters, vocabulary sizes and number of training data. Therefore, the approach is constrained by the granularity and range of the experimental data points available to some extent. However, the proposed Approach 3 is flexible that it considers the fact that the vocabuuuyy parameters and the number of training data are not always following the compute-optimal scaling laws [26], i.e., equal scaling, in real-world applications.
Approach 3
B.2 Larger models and different architectures
We have shown that our predictions hold for models with up to three billion parameters ($5). However, LLMs are often orders of magnitude larger, such as the 400-billion parameter Llama-3 model [41]. Further, we have decided to focus on dense transformer language models, as they are most commonly used for LLMs. However, many transfsnffrrrr models have been proposed and scaled up to billions of parameters [49, 50J. Exploring to what extent our findings hold in even larger models and with different architectures a promising direction for future work.
B.3 Parametric function for the loss when considering the vocabulary
Researchers [26, 44] consider modeling the language modeling loss with parametric functions in the form of L = Pi + P2/Na + Pa/D2, where (Pi, P3, a, 3} are learnable variables. The first term of loss represents the minimum achievable loss, and the second and third terms represent the contribution to the loss from the model size N and number of training tokens D. The parametric function allows predicting the loss L given N and D even if (N, are not optimally allocated. In prior work, this loss formula accounts for changes in model size and training data but does not explicitly address the complexities introduced by varying vocabulary sizes. Incorporating vocabulary size into the loss predictor is challenging: Vocabulary size affects the model directly well as the number of training tokens and the quality of tokenization by the tokenizer. A tokenizer with a large vocabulary size makes it easier to capture semantic information in raw text and reduces the frequency of vooabul-yca words. For instance, large vocabulary size may cover common phrases, common subwords, and specialized terminology. Therefore, even if the same number of tokens are trained, the performance of the model trained on tokens with different qualities will be different.
Future work in this area could explore various parametric non-linear loss functions to predict the interactions between vocabulary size, model size, and training data with different compute allocations, not just the case of optimal compute allocation. Additionally, empirical studies on different datasets could help in understanding how vocabulary size impacts loss under varied data conditions, guiding the development of more adaptive loss prediction models.
B.4 Extensions to multilingual and multimodal scenarios
Future work could extend the proposed approaches to encompass multilingual and multimodal scenarios. Multilingual models require a nuanced understanding of vocabulary due to linguistic diversity, which may affect the optimal vocabulary size and the computation of FLOPs differently across languages. Adapting these methods to consider linguistic features and tokenization variations could lead to more tailored and efficient resource allocations for multilingual models. Different languages compete with each other for the model's ability to allocate to that language [10], which
26

---

makes it necessary to take into account the relationship between different languages when setting the size of word lists for different languages a multilingual scenario.
For multimodal models that integrate text with other data types such as images or video, the optimal vocabulary size might interact uniquely with non-linguistic parameters. Recent work [1, 68] models visual concepts in an autoregressive manner with tokenization like the processing of text data. It is interesting explore the size of visual vocabulary size, i.e., the codebook size [22], in the visual tasks and vision-language tasks. How to set the vocabulary size and the compute resource efficiently for different modalities remains an open issue.
C Potential social impact
The positive potential social impact of this research on vocabulary size in language model scaling is substantial. By optimizing large language models with the consideration of the vocabulary size and other attributes jointly, the paper provides foundational understanding that can lead more lightweight and cost-effective pre- tre-trained large language models. This efficiency can democratize access to advanced language processing technologies, making it feasible for smaller organizations and the general public to benefit from powerful AI tools. Such advancements can benefit various domains, for example, improve accessibility features for individuals with disabilities, where efficient language models can be used to analyze medical records and assist in diagnostics. Furthermore, the reduction in computational requirements for training these models can lead to a decrease in energy usage, contributing positively environmental sustainability efforts.
On the other hand, the misuse of pretrained language models may pose risks, including the creation of highly realistic deepfakes that can spread disinformation and undermine trust in media and institutions. These models can generate misleading content, automate cyberattacks through convincing phishing schemes, and produce large-scale spam, degrading online communication. Additionally, they can be used to generate harmful or abusive content, such as hate speech, which perpetuates discrimination and harms vulnerable populations. To mitigate these risks, it is crucial to develop trustworthy language models, implement robust monitoring systems, and foster collaboration among researchers, policymakers, and users.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the abstract and $1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix B.
Guidelines:
27

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please see $A.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please $A.7.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
28

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Please the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Aelllii/ggiess//e http///ooSumiisiiiooo..oo Policy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: // //nips.co cct/sccccpppo wwtpc///pppc//eeeeeeeeeeeeeeeele wwwpc//gggeeeeeeeeeeeeeeeeeeeeee httggddeeeeoeeedbiiiieeeee www.///Summssssioooooolloooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
29

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please see $4 and $A.7.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide standard deviation for downstream performance in Table 3 and confidence intervals for our experiments in $A.10. For pre-training experiments, do not provide error bars due to their computational cost. Specifically, our variables are "The number of considered models" "The number of considered vocabulary size" x "The number of training characters" x "The number repeated experiments". Training multiple models across all these axes for more statistical significance would be prohibitively expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see $A.7.3.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps//parrpp...cc/bbl////////
ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
31

---

Justification:
Please see Appendix C.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see $4.
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thooee. com datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
No new assets are introduced.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
32

---

Answer:
[NA]
Justification:
No crowdsourcing experiments and research are involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No participants are involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "jRgdSmp58v", "reviewer": "Reviewer_EsaU", "summary": "This paper investigates the impact of vocabulary size on the efficiency of large language models (LLMs). Using models with 33 million to 3 billion parameters, it finds that optimal vocabulary size is limited by computational resources. The study introduces two methods to determine the best vocabulary size, showing that vocabulary parameters should scale slower than other parameters. Results highlight the significant yet underestimated role of vocabulary in scaling LLMs effectively, suggesting that larger vocabularies can improve model performance.", "questions": "NA", "limitations": "This paper discusses the limitations in Appendix B.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a unique aspect of language model scaling by investigating the impact of vocabulary size on model performance, a dimension that is often overlooked in LLM research.\n\nThe introduction of two novel methods to determine the optimal vocabulary size—empirical IsoFLOPs and a derivative-based approach—provides practical tools for optimizing LLM training and deployment.\n\nThe analysis in this paper is quite in-depth, and some conclusions can provide references for subsequent LLM training efforts.", "weaknesses": "This paper conducted experiments on language models of various parameter sizes, but the largest model tested was only 3 billion parameters. It would be better if we could further verify models with more than 7 billion parameters. I believe both the industrial and academic communities are eager to know whether the scaling law for vocabulary can generalize to larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: This paper conducted experiments on language models of various parameter sizes, but the largest model tested was only 3 billion parameters.\nWe acknowledge the importance of evaluating our approach on larger models to establish its scalability. Increasing the model size necessitates pre-training on a larger corpus, which in turn demands more computational resources. For instance, conducting pre-training experiments on 7B models would require an immense computational budget exceeding 10^22 FLOPs, translating to approximately 6 weeks of training time on a cluster with 64 A100 GPUs. However, such a substantial level of computational resources is currently beyond our reach during the rebuttal period. Despite our desire to explore larger model sizes, we are constrained by the practical limitations of our available resources.\n\nNonetheless, the significance of the scaling law lies in investigating it through experiments on a relatively small scale to help us reasonably allocate computational resources when training a large model, thus avoiding wasted computational power. Our experiments with 1200 pre-trained models have demonstrated the existence of an optimal vocabulary size under FLOPs constraints, and the predictions from the theoretical analysis (derivative-based approach) and experimental fitting (IsoFLOPs-based approach) agree with each other. In fact, we have also made predictions for the pre-training of a 300B model, and the two approaches align well, and so we believe it should work for larger models.\n\nFurthermore, the change in vocabulary size from 32K in Llama-2 to 128K in Llama-3, which resulted in performance improvement, can be also seen as a verification of our conclusion regarding increasing the vocabulary size when there are more computational budgets.\n\nIn conclusion, we appreciate your suggestions and we will try our best to conduct more experiments on larger models. We hope you can understand our computational limitations. We will also discuss this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJ4WFsEfmo", "reviewer": "Reviewer_Y8zo", "summary": "This study primarily explores the role of vocabulary size in scaling large language models (LLMs). Traditional research has focused on model parameters and training data size, often overlooking the impact of vocabulary size. While intuitively larger vocabularies can enable more efficient tokenization by representing sentences with fewer tokens, they also increase the risk of under-fitting representations for rare tokens. By training models ranging from 33M to 3B parameters on up to 510B characters with various vocabulary configurations, we discovered that the optimal vocabulary size is constrained by the computational budget. We propose two methods to determine the optimal vocabulary size: an empirical IsoFLOPs approach and a fast derivative-based approach. Both methods indicate that vocabulary parameters should be scaled slower than non-vocabulary parameters. Nonetheless, vocabulary parameters are critical for performance and are under-allocated in current LLMs. By increasing the vocabulary size beyond the conventional 32K, we trained a better 3B parameter model despite using fewer training tokens. Our work reveals the underestimated role of vocabulary and the necessity of jointly considering vocabulary size, model parameters, and training data for efficient scaling.", "questions": "In Use Case section, the authors mentioned that  (2) One has already conducted scaling law experiments following the Chinchilla laws with a fixed vocabulary size (e.g., 244 32K) and aims to estimate the optimal vocabulary size for a given model parameter. In determining the scaling law for the relationship between non-embedding model size and data (such as the Chinchilla law), why is it assumed that the vocabulary size is independent of these two factors", "limitations": "Yes, the authors addressed limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The study takes a holistic approach by examining the role of vocabulary size in the scaling of large language models (LLMs), addressing a gap in traditional research that often overlooks this aspect.\n2. The introduction of two novel methods—an empirical IsoFLOPs approach and a fast derivative-based approach—for determining the optimal vocabulary size showcases the study's innovation and practical contributions.\n3. Demonstrating that increasing vocabulary size beyond the conventional 32K can lead to better model performance with fewer training tokens highlights the practical implications and potential for efficiency gains.", "weaknesses": "1. Lacks performance on large-scale models, such as whether increasing the vocabulary size to a greater extent performs better than existing models in the market. Table 2's experiments look a little bit less.\n2. IsoFLOPs method is very sensitive, the experiments also looks not enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: Lacks performance on large-scale models, such as whether increasing the vocabulary size to a greater extent performs better than existing models in the market. Table 2's experiments look a little bit less.\n\n\nThank you for raising this concern. We share the intention to compete with existing powerful models in the market, such as Llama-2-7B. However, training a 7B model on 2 trillion tokens from scratch is far beyond our current computational resources. Nevertheless, we have evaluated our 3B models on more benchmarks to alleviate your concern.\n\nSpecifically, we have added new experimental results on the following benchmarks:\n- **MMLU**:Massive Multitask Language Understanding benchmark for broad domain language evaluation. \n- **CommonsenseQA**: A multiple-choice QA dataset for measuring commonsense knowledge.\n- **CoQA**: Conversational question answering tasks to test dialog understanding.\n- **TruthfulQA**: A QA task aimed at evaluating the truthfulness and factual accuracy of model responses.\n- **Lambada**: Tasks designed to predict the endings of text passages, testing language prediction skills.\n\nThe following table combines the original Table 2 in the paper and the new experimental results. As shown, our prediction enables better model performance by adjusting the vocabulary size within different FLOPs budgets. The 3B model with a 43K vocabulary size outperforms the 32K counterpart on 11 out of 12 tasks using the same FLOPs budget. For example, we improve performance on ARC-C from 29.1 to 31.5. In conclusion, the model using our suggested vocabulary size (i.e., 43K) consistently outperforms its counterpart (i.e., 32K) by a clear margin.\n\n\n| Tasks         | Metric              | $V$=32K (Baseline)   | $V^{opt}$=43K (Ours) |\n|---------------|---------------------|-----------|---------------|\n| Winogrande    | Normalized Accuracy | 55.7±1.4  | **58.7**±1.4      |\n| PIQA          | Normalized Accuracy | 72.6±1.0  | **72.7**±1.0      |\n| OBQA          | Normalized Accuracy | **34.4**±2.1  | 33.0±2.1      |\n| Hellaswag     | Normalized Accuracy | 55.1±0.5  | **55.7**±0.5      |\n| BoolQ         | Normalized Accuracy | 60.1±0.9  | **62.3**±0.8      |\n| ARC-E         | Normalized Accuracy | 53.4±1.0  | **55.0**±1.0      |\n| ARC-C         | Normalized Accuracy | 29.1±1.3  | **31.5**±1.4  |\n| MMLU          | Normalized Accuracy | 25.0±0.4  | **25.5**±0.4  |\n| CommonsenseQA | Normalized Accuracy | 20.2±1.2  | **21.0**±1.1  |\n| CoQA          | Exact Match         | 32.3± 2.0 | **37.4**± 2.0 |\n| TruthfulQA    | BLEU                | 30.4±1.6  | **31.3**±1.6  |\n| Lambada       | Normalized Accuracy | 43.0±0.7  | **44.9**±0.7  |\n\nAnother feasible way to compete with models in the market would be continual pre-training with a larger vocabulary size. We believe this is a good topic for discussion, but it involves several non-trivial research challenges not strongly related to the main contributions of this paper, i.e., exploring the optimal compute allocation considering vocabulary sizes. Therefore, we will discuss it in the revised version and leave it as an important future work. The challenges we will discuss include:\n- Expanding the vocabulary necessitates changes in the tokenization process, which can lead to inconsistencies in token segmentation.\n- Ensuring that these new embeddings are compatible and effectively integrate with the pre-trained embeddings is non-trivial.\n- Catastrophic forgetting of old word embeddings when learning new word embeddings.\n\nWe will discuss all the above in the revised version. Thank you again for your valuable comments.\n\n\n\n### W2.1: IsoFLOPs method is very sensitive. \n\nThank you for your insightful question. You raise a valid point – the IsoFLOPs-based approach can be sensitive to some extent, depending on the granularity, range, and quality of the fitting data. Since the pioneering work on scaling laws by Kaplan et al. 2020 [1] and Hoffmann et al. 2022 [2], the IsoFLOPs-based approach has become a widely-used tool to study the trend of model performance [3]. We have discussed it in our Appendix B.1, and we will add more details on how to reduce sensitivity, such as outlier data removal and repeated experiments, in our polished version.\n\nTo evaluate the goodness of fit, we use relative mean square error (rMSE) and the coefficient of determination (R^2). As shown in the table below (also in Figure 3), the results indicate a good fit, with rMSE < 0.001 and $R^2$ >= 0.89 for all the considered attributes: non-vocabulary parameters ($N_{nv}$), vocabulary parameters ($N_v$), and training characters ($H$). This suggests that these attributes follow a power law with respect to the FLOPs budget.\n\n|  | $N_{nv}$ | $N_v$ | $H$ |\n|--------|----------|------|-----|\n| rMSE   | 0.00026  | 0.00051 | 0.00017 |\n| $R^2$  | 0.93  | 0.89  | 0.96  |\n\nFurthermore, the optimal vocabulary predictions (Table 1) from the IsoFLOPs-based method and the derivative-based method are aligned across small-scale and large-scale models. This independent verification by the derivative-based method validates the predictions from the IsoFLOPs-based method. Therefore, we believe that the IsoFLOPs-based method works well in our case.\n\n[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.\n\n[2] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556\n\n[3] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L5XnTsntRb", "reviewer": "Reviewer_zt86", "summary": "The paper presents empirical scaling laws for the size of the vocabulary for LLMs. The findings in the paper are:\n\n- Empirically, the vocabulary size minimizing the loss increases when FLOPs are increased (Fig 2, right, Fig 3)\n- Through mathematical derivations from scaling laws, the optimal vocabulary size decreases when the embedding size increases. (Fig 4).\n- With a given FLOPs budget, 43k vocav size beats 32k on academic benchmarks like Boolq (Table 2.)", "questions": "1. The abstract states that “beyond the conventional 32K” – is this really the convention? See e.g. GPT4o.\n2. How does Table 2 look if you train on the same number of tokens instead of using the same FLOPs budget?", "limitations": "na", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Vocabulary scaling laws are an interesting and novel research direction.\n- The paper is well written.", "weaknesses": "- The results are probably mostly applicable to a small number of well-funded labs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1: The results are probably mostly applicable to a small number of well-funded labs.\n\nThanks for pointing this out! We want to clarify that we are not a well-funded lab either. Due to our limited computing resources, we can only afford to train models with up to 3B parameters in our experiments, as the cost of validating scaling law experiments is indeed very high.\n\nHowever, we believe that our conclusions are beneficial to the general research community, especially for small labs. Our scaling laws with vocabulary provides a compute-optimal allocation suggestion, enabling small labs to train high-performance models without repeatedly trying different vocabulary configurations, thereby saving computing resources.\n \nEven for teams who want to conduct scaling law experiments themselves, our derivative-based method offers a simple and feasible approach based on theoretical derivation. Researchers do not need to run a large number of scaling law experiments to obtain a good vocabulary configuration. This is particularly advantageous for small labs. We will also make all our scaling law experimental results public so that more people can benefit from our work.\n\n### Q1: The abstract states that “beyond the conventional 32K” – is this really the convention? See e.g. GPT4o.\nThank you for your insightful question! We acknowledge that there is no single \"conventional\" vocabulary size for language models, as it can vary based on the pre-training corpus and the intended use case. A vocabulary size of 32K is widely regarded as a common choice, particularly for models trained on English-centric corpora, such as Llama-1, Llama-2, and Mistral. Since our work primarily utilizes the English-centric SlimPajama corpus for pre-training, we have adopted the 32K vocabulary setting employed by these models as a \"conventional\" vocabulary size. We will modify the statement in the abstract accordingly to reflect this clarification.\n \nAs for GPT4o, we think its vocabulary size is relatively larger because it is designed to handle multiple languages (e.g., Chinese). This also highlights an important consideration for future research: determining the optimal vocabulary size for multilingual models, which we have discussed in Appendix B.4.\n \nOur broader goal is to draw attention to the importance of vocabulary size in training language models and to encourage the appropriate allocation of computational resources for this aspect. Recently, there has been a shift in the industry, with major companies recognizing that their previous allocations for vocabulary were insufficient. For example, Llama has increased its vocabulary size from 32K to 128K, reflecting this evolving understanding. We hope this clarification helps, and will add the discussion in the revised version.\n\n\n### Q2: How does Table 2 look if you train on the same number of tokens instead of using the same FLOPs budget?\n\nThanks for your prompting question! As you suggested, we also trained the model using the same number of tokens, i.e., 129B tokens, beyond the same FLOPs budget setting. As shown in the following table, the performance of the model with the suggested vocabulary size of 43K improves further compared to the 32K vocabulary size when using the same number of training tokens. We will add the results in the revised version.\n\n| **$V$** | **$N_v$** | **$D$** | **Winogrande** | **PIQA** | **OBQA** | **Hellaswag** | **BoolQ** | **ARC-E** | **ARC-C** | **Average** |\n|---------|-----------|---------|----------------|----------|----------|---------------|-----------|-----------|-----------|-------------|\n| 32K (Baseline)     | 0.20B     | 129B    | 55.7 ± 1.4     | 72.6 ± 1.0 | **34.4** ± 2.1 | 55.1 ± 0.5  | 60.1 ± 0.9  | 53.4 ± 1.0  | 29.1 ± 1.3  | 51.5        |\n| 43K (Ours with same FLOPs)    | 0.27B     | 125B    | **58.7**±1.4 |**72.7**±1.0 |33.0±2.1 |55.7±0.5 |62.3±0.8  | 55.0±1.0  | 31.5±1.4 |  52.7\n | 43K (Ours with same Tokens)    | 0.27B     | 129B    | 58.6±1.4 |  **72.7**±1.0  | 33.6±2.1  | **55.8**±0.5  | **62.4**±0.9  | **55.5**±1.0  | **31.5**±1.4 |  **52.9**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General Response

We are grateful for the reviewers' efforts and the recognition of our contributions:
- **Novel Research Topic:** The paper explores the unique impact of vocabulary size on language model performance, an aspect often overlooked in LLM research [EsaU,Y8zo,zt86].
- **Analyses:** We provide in-depth analyses why it exists a  optimal vocabulary size with the FLOPs and the optimal vocabulary size increases with more FLOPs, with theoretical anayses in Appendix A.1 and demostration experiments in Sec 3 and Apppendix A.2. [EsaU,zt86]
- **Experiments:** The paper includes two effective methods (IsoFLOPs and a derivative-based approach) to predict the optimal vocabulary setting. Extensive experiments on **1200 pre-trained models** (6 non-vocabulary parameters settings x 10 vocabulary sizes settings x 20 number of training data settings) are conducted. [EsaU,zt86]
- **Applications:** The study's findings offer two practical tools for optimizing the compute allocaiton of LLMs training, with the consideration of the vocaulary size. [EsaU,Y8zo,zt86]

In response to the feedback of reviewers, we have performed additional analyses and experiments to address the raised concerns. Below, we summarize our responses and the improvements made to our paper:
- We evaluate the 3B pre-trained models with different vocabulary sizes on more downstream tasks. The model that uses our suggested vocabulary size outperforms its counterpart by a clear margin.
- We supplement the paper with a new perspective from parameter growing to demontrate that the vocabulary parameters need to be separately considered from the total parameters, and larger models deserve larger vocabularies.
- We compare the models with the same training tokens instead of the same training FLOPs as asked.

All of the suggestions will be considered in our polished version.

Meta-review: This paper studies the important aspect of vocabulary size in the context of scaling laws for LLMs. Vocabulary size has not dealt with in detail in prior work and this paper fill that gap. The study is rather limited in scale, but is perhaps understandable considering the very high computational demands that pretraining creates. It would be great if the study could be scaled up a little more, so that there is more confidence in the generalization.
Decision: Accept (poster)