Enhancing Large Language Models through Adaptive Tokenizers
Authors: Mengyu Zheng
Hanting Chen
Tianyu Guo
Chong Zhu
Binfan Zheng
Chang Xu
Yunhe Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Tokenizer
Primary area: natural_language_processing

Abstract: Tokenizers serve as crucial interfaces between models and linguistic data, substantially influencing the efficacy and precision of large language models (LLMs). Traditional tokenization methods often rely on static frequency-based statistics and are not inherently synchronized with LLM architectures, which may limit model performance. In this study, we propose a simple but effective method to learn tokenizers specifically engineered for seamless integration with LLMs. Initiating with a broad initial vocabulary, we refine our tokenizer by monitoring changes in the model’s perplexity during training, allowing for the selection of a tokenizer that is closely aligned with the model’s evolving dynamics. Through iterative refinement, we develop an optimized tokenizer. Our empirical evaluations demonstrate that this adaptive approach significantly enhances accuracy compared to conventional methods, maintaining comparable vocabulary sizes and affirming its potential to improve LLM functionality.

Enhancing Large Language Models through Adaptive Tokenizers
Mengyu Zheng
Hanting Chen
The University of Sydney
Huawei Noah's Ark Lab
Huawei Noah's Ark Lab mzhe4259@uni. sydney edu. au
chenhantinn@huuwei.. com
Tianyu Guo
Chong Zhu Huawei Noah's Ark Lab
Binfan Zheng
Huawei Noah's Ark Lab
Huawei GTS AI Computing LAB
gunyuuggu@ com
2huchong40huawei. com
zhengbinfan @huaaeei.com
Chang Xu
Yunhe Wang"
The University of Sydney c.xussydney. edu.
Huawei Noah's Ark Lab yunhe. wang@huawei com
Abstract
Tokenizers serve crucial interfaces between models and linguistic data, sub- stantially influencing the efficacy and precision of large language models (LLMs). Traditional tokenization methods often rely on static frequency-based statisties and are not inherently synchronized with LLM architectures, which may limit model performance, this study, we propose simple but effective method to learn tok- enizers specifically engineered for scamless integration with LLMs. Initiating with a broad initial vocabulary, we refine our tokenizer by monitoring changes in the model's perplexity during training, allowing for the selection of tokenizer that is closely aligned with the model's evolving dynamics. Through iterative refinement, develop optimized tokenizer. Our empirical evaluations demonstrate that this adaptive approach significantly enhances accuracy compared to conventional methods, maintaining comparable vocabulary sizes and affirming its potential improve L.LM functionality.
Introduction
In recent years, large language models (LLMs) have emerged as foundational tools across a spectrum of applications in natural language processing 3 24 From generating human-like text enabling complex question-anse answering systems (28). LLMs have proven to be exceptionally versatile and capable. At the of these models lies the tokenizer, critical component that dictates how natural language is transformed into format amenable to computational processing. The effectiveness of a tokenizer directly influences the model's ability understand and generate language, thus playing pivotal role in the overall performance of the L.M.. Recognizing this integral relationship. it becomes essential to develop tokenizers that not only effective but also dynamically adaptable to the evolving architectures of contemporary LLMs.
Current tokenization methods for large language models (LLMs) primarily include Byte Pair Encoding (BPE) 32 WordPiece 43 and Unigram [9) each serving to enhance text preprocessing by splitting
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

it into manageable subwords. BPE focuses on reducing the dataset size through a greedy merging strategy based on character or subword frequency, effectively addressing the issue of infrequent words by splitting them into more common subunits. WordPiece, similar to BPF, starts with a base vocabulary and iteratively refines by merging the most frequent pairs but incorporates a likelihood maximization step, which makes it slightly more context-aware than BPE. Unigram tokenization operates somewhat inversely, beginning with a large vocabulary and iteratively pruning it down based on token utility calculated through negative log likelihood, aiming to optimize the vocabulary against corpus loss metrics. Despite their efficiency in handling large vocabularies and improving computational feasibillity, these tokenization methods typically fixed once developed and not designed to adapt or learn from the model's 's evolving understanding of language during training.
While traditional tokenization methods have been instrumental in enhancing the efficiency and effectiveness of large language models (LL.Ms), they are typically decoupled from the model's s learning mechanisms. This means they do not adapt evolve based on the model's performance or the specific requirements of the tasks being addressed. Instead, these methods prioritize compressing the vocabulary size, which can sometimes lead to suboptimal performance in complex language tasks where adaptability and contextual understanding are crucial. Recent advances in end-to-end learnable tokenization 17 16 38) aim to address these deficiencies by more closely integrating tokenization with the model's 's learning processes. However, these systems, while innovative, introduce significant computational overhead (e.g., gradient- based tokenization and pooling modules) and lack the flexibility of traditional tokenization methods, which can be casily transferred across different models.
In this study, we address the limitations of traditional tokenization methods by creating a system where the tokenizer's ss development is coupled with the performance of the LLM itself. Specifically, we introduce an adaptive tokenizer that begins with a comprehensive initial vocabulary. As training progresses, we fine-tune this tokenizer by closely monitoring the model's perplexity. This ongoing adjustment allows the tokenizer to evolve in tandem with the LLM, ensuring that the tokenization process remains optimally aligned with the model's dynamic learning patterns. Our empirical results confirm that this adaptive approach markedly improves accuracy over traditional methods, demonstrating its potential to significantly enhance LLM functionality.
2 Related Works
2.1 Subword tokenizer
Subword tokenizers are widely applied in many large language models (LLMs) 32 19
42
35
20
such as GPT-3 77 BERT [2], and T5 [30]. This because subword tokenizers do not face the ouc-ol-vocabulary issues that word-level tokenizers do. Unlike character-level tokenizers 36 27) they do not require processing longer sequences at the character level, which significantly increases the complexity of the models quadratically
(21) Specifically, BPE (Byte Pair Encoding) 332 applies compression algorithm [13] to the task of word segmentation. Unlike BPE, which builds a vocabulary from smaller to larger units, Unigram [19 starts by preparing a large seed vocabulary. The vocabulary is pruned until it reaches the specified size. Similar to the Unigram framework and its assumptions, BytePiece (55 trims the vocabulary based on token frequency to the required size. In its initial text encoding phase, BytePiece converts the corpus into bytes, enhancing the training speed of the tokenizer and achieving a higher compression rate. Obviously, these tokenizers are data-driven [5). with the generated vocabulary built based on the frequencies of word fragments (6. However, they cannot directly interact with LLMs to enhance model capabilities (
2.2
Learnable tokenizer
Unlike traditional tokenizers that offer a predefined and fixed vocabulary, learnable tokenizers 17 16
38
37 34. g can be integrated with large language models into an end-to-end learning framework,
resulting task-specific tokenization to enhance the performance of LLMs. MANTa [16] introduces a gradient- based tokenization and pooling module that can be jointly learned with an encoder-decoder
LLM (2. RETVee [8] embeds words into a high-dimensional vector with a pre-trained model to be robust against adversarial attacks. Neural [17 adapts the tokenization behavior to the downstream task after pre-training the tokenization by distilling from a language-specific subword tokenizer. However, such tokenizers have high requirements for the quantity and quality of the training data. If

---

Traditional tokenizer
Setpl. Obtain Initial Tokenizer
Traditional Subword Igorithm
Initial/Updated Tokenizer
Data
Partial Data
Score,
Ty
Scorex
Setp2. Training and Inference LLM with Current Tokemizer
Tokenizer
Randomly Initialized LLM Model
7
Loss
Loss
Loss
Lossy
N X
Setp3. Pruning Tokens from Current Vocabulary
Model
7
Score,
Loss,
Score;
Loss;
Tg
Scoreg
Loss
x
T
F(Score, Loss;
7
F(Scorey,
F F(Score Lossk
A daptive
Tokenizer
A daptive Tokenizer
b) LLM eahanced tokenizer
c) Pipeline of the proposed ADAT ADAT
Illustration of the proposed ADAT pipeline. (a) The traditional tokenizer algorithm that directly extracts vocabulary from data. (b) The framework of the LLM-enhanced tokenizer, iteratively refining vocabulary based on model feedback. (c) Overview of ADAT, encompassing initial tokenizer acquisition, training and inference to derive token losses, token pruning based on scores and losses.
Figure 1:
the data distribution is unbalanced or contains too much noisy data, it can lead to poor generalization of the tokenizer and negatively affect the performance LLMs. For instance, Neural [[7] requires pre-training dataset curated with space-separated tokens and two carefully crafted heuristics to improve the ground label of the dataset. Therefore, stringent requirements for the quality of training data limit their widespread application.
In addition, some existing works have applied the concept of adaptive tokenizers in several fields, such as neural machine translation (26). domain adaptation [1], and text generation [23]. These task-adaptive tokenizers integrate tokenizers generated from different data distributions, focusing on how to combine the task-specifir tokenizer with the other one. In contrast, our proposed model, ADAT, designed to learn general tokenizer. Therefore, the purpose of ADAT is different from that of the aforementioned adaptive tokenizers.
Adaptive Tokenizers
For Large Language Models (LLMs), two critical aspects are accuracy and inference speed, both of which are deeply intertwined with the design of the tokenizer. Specifically, an optimal vocabulary V with maximum size N in an objective dataset D can be described by the following optimization problem:
min Length(Dnr V) AAce(D,M, V
(I)
where M denotes the trained LLM. Length and Acc denotes the sequence length and accuracy (or the performance) using vocabulary V in dataset D, and the À is a hyper-parameter to balance the two terms. This problem incorporates two primary objectives: given a fixed vocabulary size, the first is to maximize inference speed given a fixed vocabulary size, and the second is to maximize model accuracy. However, existing tokenizer schemes typically focus on one aspect over the other; for instance, traditional frequency- schemes emphasize speed, while end-to-end approaches prioritize accuracy. address this, we propose a method that optimizes both aspects. Therefore, we introduce improved approach based traditional frequency statistics, termed "adaptive tokenizers." " This method aims to refine the balance between vocabulary efficiency and performance, thereby enhancing both the speed and accuracy of the model.

---

3.1 Unigram Model
Traditional tokenization methods generally fall into two categories: one approach, exemplified by Byte Pair Encoding (BPE) and WordPiece, starts with small set of symbols and incrementally builds larger vocabulary by merging the most frequent adjacent pairs. The other approach, typified by the Unigram method, begins with large initial vocabulary which is progressively pruned based on token utility, a method found to generally offer superior performance due to its probabilistic foundation.
The Unigram model operates on the principle that the probability of sentence is determined by the individual probabilities of its tokens. Here we briefly review the Unigram model. Initially, large vocabulary V is established. This extensive initial set includes potentially every unique word or subword unit observed in the training corpus, ensuring that the vocabulary can cover all possible textual inputs. The process of refining the vocabulary involves several key steps repeated in cycles: 1. Probability Estimation: For each token 2; in the current vocabulary, we calculate its probability p(z,) based on its frequency of occurrence in the corpus. 2. Loss Calculation: We then compute the loss for each token, which is determined by how much the overall loss of the model would decrease if that token were removed. The loss function is calculated as:
M
[
P Σ XES((X(())
CP( log(p(X(())
Σ log(
(x)),
where S(X) is set of segmentation candidates built from the input sentence X, and x (ZI, Tx) subword sequence that P(x)= I p(z;). The loss for each token x; is then formulated as
(x.)) (VV)) - -P - z.). 3. Token Pruning: Tokens are ranked according to their calculated
loss. Finally, a proportion of tokens contributing the most to increasing the overall loss is pruned from the vocabulary.
3.2 LLM-Enhanced Tokenization
To enhance the integration of Large Language Models (LLMs) with our tokenization process, we have developed simple but effective refined method for calculating the loss associated with each token, incorporating insights directly from the LLM's performance metrics. This approach aims to optimize the tokenizer's vocabulary to better align with the LLM's understanding and generation of text. The framework is illustrated in Figure
Our revised loss calculation method integrates the traditional Unigram model's frequency- -based loss with performance-driven loss derived from LLM. Specifically, we first train an LLM M in the vocabulary D using a training dataset T. This model is designed to capture the linguistic nuances relevant to the tasks is trained for, providing a robust framework for assessing token utility. For each token in the vocabulary, we measure its individual contribution to the model's error using a cross-entropy loss function. The loss for each token is calculated as:
CM(x:)) Σ CE(MM(--1))
ZEET
Here, C'E represents the cross-entropy function, M (I;-1) is the LLM's output given the previous token 2;-1, and z; is the actual next token. This formula assesses how well the LLM predicts each token following its predecessor, providing a direct measure of each token's 's impact on model performance. Finally, The cross-entropy loss for each token is then combined with the traditional Unigram frequency-base their frequency of occurrence but also their actual contribution the LLM's task performance. The final loss for pruning the vocabulary is given by:
loss. This combined loss ensures that tokens are evaluated not only on
((z,) = F(Cp(z:),
where F(...) is a function to balance the importance of frequency- based loss and LLM--divee loss, which will be discussed in experiments. Using this enhanced loss metric, we iteratively refine the vocabulary by pruning tokens that contribute the least to the combined loss, thus optimizing the vocabulary for both general language understanding and specific task performance. This process continues until the vocabulary compact enough to manage while still being comprehensive enough to support the LLM effectively.

---

Random sampling. In the training of Large Language Models (LLMs), ensuring that each token within the set vocabulary receives equal and substantial training is crucial prevent loss bias due to uneven training. While iterating over all possible tokenizations of the training corpus would ideally provide the most comprehensive learning experience, this approach is computationally prohibitive due to the immense variety of potential segmentations. To address this, we adopt the classic Viterbi algorithm [40] to perform randomized tokenization of the training data. This method allows for diverse and balanced exposure of all tokens within the vocabulary to the learning process. The Viterbi algorithm efficiently determines the most probable tokenization paths through a probabilistic model of token occurrence, which significantly reduces the computational overhead compared to exhaustive methods. By leveraging this approach, our LLM can learn each token in the vocabulary more uniformly, enhancing the overall robustness and performance of the model.
Loss momentum. In the iterative process of training Large Language Models, maintaining the stability of the vocabulary is crucial to ensure consistent learning outcomes. To achieve this, we propose a momentum- based improvement for calculating the loss during each iteration. Specifically, the loss for iteration j of token I,, denoted as L (2:), is not solely computed based on the current data but is also weighted by the loss from the previous iteration Li-1, This approach allows for a smoother convergence and mitigates fluctuations in training dynamics. The formula for updating the loss at each iteration is given by:
Caseenma (I:) 3 (z:) + (x;),
(5)
where / is the momentum coefficient that controls the extent to which the previous loss influences the current loss. This methodology not only stabilizes the vocabulary updates across iterations but also enhances the model's ability to generalize from the training data by reducing the variability in loss across successive training epochs.
Experiments
In this section, we outline the comprehensive experimental framework designed to assess the ef- fectiveness of our proposed tokenizer, Adaptive Tokenizer (ADAT), in comparison to established methods such as Byte Pair Encoding (BPE) [32] and the Unigram model [19]. These evaluations
utilize the Pythia [3] suite of models at various scales, leveraging substantial corpus to ensure robust
and generalizable results.
4.1 Experimental Setup
Model Framework We deploy the Pythia framework (3 for its lightweight design and adaptability across different computational setups. Pythia's 's flexibility facilitates reproducibility and consistent assessment of performance, making it an ideal choice for evaluating the scalability and efficiency of various tokenization strategies across model sizes of 70M, 160M, and 410M parameters.
Data Corpus The study utilizes substantial corpus extracted from The Pile 14] consisting of 56GB of raw data across 91 files. We specifically excluded subsets from DM_Mathematics and Github to ensure the relevance and quality of the data. The remaining data, approximately 16 billion tokens after a random shuffle, was tokenized using a Unigram [19] tokenizer with a vocabulary size of 50,000 tokens. A detailed enumeration of the data files used is available Supp. A.8
Baseline Methods Our investigation compares four tokenization methods: Bytepiece [35], Byte Pair Encoding (BPE) (32), Unigram [19], and our proposed ADAT, These tokenizers were selected based their established efficacy in handling large corpora and their theoretical implications for processing complex linguistic data.
The effectiveness of each tokenization strategy rigorously evaluated using several metrics. These include Perplexity (PPL), which measures the model's predictive accuracy, and Compression Rate(refer to A. assessing how efficiently the tokenization process reduces vocabulary size while preserving linguistic diversity. We calculate PPL. for all models on PG19 [29] dataset. Specifically, we use its test set and the first 2048 tokens for each book. Furthermore, we use [15] to run five-shot evaluations eight common language
Evaluation Metries
the Language Model Evaluation Harness
5

---

Table I: Performance Comparison of Different Tokenization Methods.
Metric
BPE
BytePiece
+ADAT(Ours)
Unigram
+ADsS(Ourr)
PPL
22.31
71.5
67.19(-4.31)
16.52
6.97(-9.55)
ARC-C
17.32 1.11
18.69 = 1.14
18,94 1.15
19.54=1.16
18.46 + 1.12
ARC-E
37.58 + 0.99
33.80 + 0.97
33.71
0.97
37.04+0.99
40.57++0.9
Boolq
61.28 + 0.85
42.12 + 0.87
62.20+0.85
53.06 + 0.87
61.19 0.85
Lambda
10.89 0.43
8.80 0.39
13.55 0.48
17.27 + 0.53
17.97+0.52
LogiQA
23.04 1.65
20.28 = 1.58
22.27 1.63
23.20 1.66
24.22111.70
PIQA
59.25 1 1.15
57.83 + 1.15
56.96 1.16
6..50
59.93 + 1.14
SciQ
66.60 1 1.49
54.01 + 1.58
51.90 1.58
68.10 + 1.47
72.40111.44
SST-2
51.26 + 1.69
49.08 = 1.69
50.23 1.69
49.77 1.69
54.24+1.69
Winogrande
49.96 1.41
50.31 = 1.41
49.41 1.41
51.46 + 1.40
51.62++140
Avg. (%)
41.91
37.21
39.91
42.22
44.51 2.29)
modeling benchmarks: Lambada (OpenAl) 25
PIQA 4
WinoGrande [. ARC-Fasy [0] ARC-
Challenge [10), SciQ [18), LogiQA [22] and SST-2 each method's capabilities.
33 41
provide comprehensive insight into
By analyzing the impact of tokenization model scalability and the influence of vocabulary size variations, this study aims to enhance our understanding of how tokenization strategies can optimize language models for efficiency and linguistic performance. The findings are expected to contribute significantly to the development of more robust and adaptable language processing tools, catering to wide array of NLP applications. The Pythia models are trained using corpus of 15B tokens, where training the 70M model consumes approximately 48 GPU hours with FlashAttention [1] The models used for loss calculation require additional 2 GPU hours.
4.2 Tokenization Methods Evaluation
In this section, we examine the effects of different tokenization strategies on the training effectiveness. The core objective is to explore how variations in the vocabulary. induced by different tokenization methods, affect model training and performance.
The Pythia-70M model is selected due to its moderate size and efficiency, which help mitigate the complexities associated with larger model architectures. It is initialized with random weights and undergoes a single training epoch using pre-training data. This data is processed with vocabularies generated from 1/10th of the training corpus (approximately 1.5 billion tokens), each containing 50,000 tokens size consistent with the Pythia 3 setup.
Baseline tokenization methods including BPE, Unigram, and BytePiece, generate vocabularies consisting of 50,000 tokens directly from initial data that approximately one-tenth of the training corpus (about 1.5 billion tokens). In contrast, for the proposed ADAT method, initial vocabularies are generated using either BytePiece or Unigram with 150,000 tokens. These are then methodically refined down to 50,000 tokens over 5 iterative steps, matching the baseline vocabulary size. At each step, a randomly initialized model is trained on approximately 0.3 billion tokens from the initial dataset. Subsequently, the model performs inference on subset of 0.1 billion tokens, during which token loss is calculated. This loss data, when combined with token frequency using the formula 1) guides the vocabulary pruning process. An ablation study on the combination methods will be discussed in Dection[4.6.4
llg(bb-))
As illustrated in Table | our method achieves its best performance when initialized with the Unigram vocabulary, recording score of 44.51. This score represents a considerable improvement of 2.29 points over the standard Unigram model and surpasses the BPE model by 2.6 points. Additionally, our approach shows a notable enhancement of 2.7 points when utilizing BytePiece as the initial vocabulary. Although the BytePiece vocabulary generally exhibits inferior baseline results, our method effectively elevates its performance, indicating robustness across both high-quality (Unigram) and lower-quality (BytePiece) vocabularies. These results not only affirm the efficacy of our method also demonstrate its adaptability to different initial conditions, thereby validating potential for broad adeptness diverse vocab initialization.

---

Table Evaluation on Different Scale Model Size.
Metric
70M
160M
410M
Unigram
ADAT
Unigram
ADAT
Unigram
ADAT
PPL
16.52
6.97(-
13.97
6.19(-7.78)
10.92
5.78(-5.14)
ARC-C
19.54+1.16
18.46 + 1.12
18.69 + 1.14
18.94#1.15
20.82+1.199
19.29 # 1.21
ARC-E
37.04+0.99
40.57=0.99
39.52 + 1.00
42.87=1.01
44.65 + 1.02
46.69 1.03
Boolq
53.06 + 0.87
61.19+0.85
58.56+00.86
57.68 + 0.86
54.80 + 0.87
60.81+0.87
Lambda
17.27 +: 0.53
17.97=0.52
19.06 + 0.55
25 25.02+00.60
27.81 + 0.62
28.94
LogiQA
23.20 +: 1.66
2..22+1.70
25.65-t1.71
25.04 + 1.65
23.20 + 1.66
24.32 1..22:1.60 1.68
PIQA
60.5011.14
59.93 + 1.14
60.83 + 1.14
61. 11.14
63.38 + 1.12
64.61+1.1
SciQ
68.10 1.47
12.40
72.10
+ 1.42
79.60
80.70 + 1.25
83.5011.14
SST-2
49.77 + 1.69
54.24-1.69
52.06
+ 1.69
52.78-1.69
50.69
1.69
14.7111.66
Winogrande
51.46 + 1.40
51. 61.62-1.440
49.88 + 1.41
50.69#1.41
52.41 + 1.40
52.93
Avg
42.22
44.51
44.04
46.05
46.50
48.42
4.3 Scalability
We examine the scalability of proposed tokenization method that tailors the vocabulary to model size, unlike the static Unigram method which maintains consistent vocabulary across various model capacities. The scalability the tokenization methods is tested using the Pythia framework configured at three different levels of computational complexity: 70M, 160M, and 410M parameters. For each model size, our method generates an optimized vocabulary specific to that configuration, allowing to analyze how adjustments in vocabulary affect performance model size increases. In contrast, the Unigram method employs uniform 50,000-word vocabulary across all sizes, serving baseline. We gauge performance using Perplexity (PPL) and scores from benchmark datasets designed to assess the linguistic capabilities of each model under various conditions, providing insights into the efficiency and adaptability of the tokenization methods scale. For training larger models, the same volume of data will lead to insufficient warm-up, potentially resulting in slight decline in the accuracy of loss computations used for determining token priority. a result, we increase the data volume for training the loss calculation model according to the size of model.
The results of this experimental framework, as presented in Table indicate substantial performance variations across different model sizes employing varied tokenization strategies. Specifically, average performance scores across all evaluated metrics demonstrate consistent improvements with increases in model sizes: ADAT achieves a score of 44.51 in the 70M model, significantly surpassing Unigram's 's 42.22: 46.05 compared to 44.04 in the 160M model; and 48.32 versus 46.50 in the 410M model. These findings highlight the superior efficacy of ADAT in managing diverse model volumes compared to the more static approach of Unigram, which exhibits limited scalability with increasing model size. Remarkably, the performance of the 70M model using ADAT exceeded that of the Unigram on the 160M model by nearly 5%, illustrating the substantial enhancement and ability of our method to bridge a parameter gap of over double. Furthermore, the performance of our 160M model approaches that of the 410M model, emphasizing the robust adaptability of the ADAT method across varying computational scales.
Table 3: Cross-Model Adaptability of Vocabularies.
Model Size
Unigram
70M-Model Vocabulary
410M-Model Vocabulary
70M
42.22
44.51
42.62
160M
44.04
45.03
45.83
410M
46.50
47.66
48.42
4.4 Cross-Model Adaptability
This experiment evaluates the adaptability of vocabularies generated by our proposed tokenization method across various configurations of the Pythia model, particularly assessing whether vocabularies optimized for one model size can effectively scale to others. We initially create vocabularies using the 70M and 4 410M configurations. These are then used to train models both scales evaluate performance in downstream tasks, allowing to assess how vocabularies designed for specific
7

---

Table Impact of Vocabulary Size on Model Performance Across Different Model Sizes.
Model Size
Vocabulary Size
Tokenization Method
Accuracy
Perplexity (PPL)
Unigram
42.22
16.52
50,000
ADAT
44.51(+2.29)
6.97
70M
Unigram
40.93
32.53
30,000
ADAT
43.33(+2.40)
7.38
Unigram
44.04
13.97
50,000
ADAT
46.05(+2.01)
6.19
160M
Unigram
43.08
15.21
30,000
ADAT
45.26(+2.14)
6.11
size perform when applied to both smaller and larger models, thus examining their cross-model adaptability.
Table iilustrates the cross-model adaptability of vocabularies across different model sizes. By apply- ing vocabularies derived from different model sizes to various models, we observe that vocabularies generated by the 70M and 410M models surpass the performance the standard Unigram model. This indicates the adaptability of the ADAT vocabularies across different model sizes. Furthermore, we note that the vocabulary from the 410M model achieves only marginal improvement of 0.4 when applied to the 70M model, significantly less than the 2.29 increase afforded by the 70M model's vocabulary. This suggests that the vocabularies selected by ADAT possess strong capacity for targeted optimization, enabling the selection of tokenization strategies that are specifically tailored to the characteristics of different models.
4.5
Model and Vocabulary Size
The experiment aims to assess the impact of different tokenizer strategies on model performance across two vocabulary sizes, comparing standard 50, 50,000-token set with reduced 50,000-token set. We utilize two configurations the Pythia model-70M and 160M-to explore how vocabulary size influences model efficiency. Each model is tested using both the standard Unigram and our proposed tokenization method. This setup allows us to directly observe the effects of reduced vocabulary sizes the performance dynamics, providing insights into how smaller vocabularies impact the computational efficiency and efficacy of language models.
The experimental results, as presented in Table 4 support the hypothesis that changes in vocabulary size can significantly affect model performance, with different impacts observed across varying model sizes. For large language models, it is common for models of vastly different sizes to utilize vocabularies of similar or identical sizes [3]. This practice can lead to issues of performance or efficiency. Our method offers more effective solution by tailoring tokenization strategies to the specific sizes of models, thereby mitigating these challenges. For the 70M model, ADAT achieved a notable improvement in accuracy from 42.22 44.51 (+2.29) and substantial reduction in perplexity from 16.52 to 6.97 when using a 50,000-word vocabulary. Even with a reduced vocabulary of 30,000, ADAT enhances accuracy to 43.33 (+2.40) and decreases perplexity to 7.38, suggesting robustness against vocabulary size reduction. In contrast, the 160M model, which has a greater parameter capacity, also shows improvements with ADAT: accuracy increases from 44.04 46.05 (+2.01), and perplexity drops sharply from 13.97 to 6.19 for the 50,000 vocabulary size. With 30,000 vocabulary, accuracy still increases 45.26 (+2.14), and perplexity remains low at 6.11, underscoring that larger models not only handle vocabulary reductions well but also benefit significantly in terms of computational efficiency and model quality.
Ablation Study
This ablation study structured into three distinct parts to explore how variations in the inference corpus size used for calculating token loss, initial vocabulary sizes, momentum strategy, and balance function F(a,) b) influence the efficacy of our proposed tokenization method on 70M parameter model. More ablation results can be referred to in the supplementary.

---

Table Ablation Studies Results.
Infer Data Volume Tokens
Initial Vocabulary Size Init Size
Momentum
Balance F(a, b)
Acc.
Acc.
Methods
Acc.
Methods
Acc.
IM
43.13
75k
43,42
Unigram
42.20
Ab
42.70
10M
43.74
100k
43.78
ADAT+By
44.51
log(a) Ab
43.23
100M
44.51
150k
44.51
-w/o Mnt.
43.16
lb
44.51
4.6.1
Corpus Size used in Loss Calculation
We conducted experimental study to investigate the effects of varying corpus sizes the accuracy of token loss calculations. The experiment assessed the performance of models trained on different sizes of inference data, specifically I million (IM), 10 million (10M), and 100 million (100M) tokens. The results are summarized in the table
These results indicate direct correlation between the volume of the corpus used during the loss calculation phase and the overall accuracy token loss estimates. When smaller corpora are used, significant number of tokens are absent, resulting in numerous instances where loss values cannot be computed. Furthermore, the precision of token loss estimations tends to decrease with smaller data sets. Even with just IM tokens, there was noticeable improvement over the baseline unigram vocabulary accuracy 42.22. This enhancement became more pronounced with larger data volumes, reaching an increase of 44.51 in accuracy with 100M tokens.
4.6.2 Initial Vocabulary Size
This segment of our study assesses the effect of different initial vocabulary sizes on model perfor- mance. Adjusting the vocabulary from baseline of 150,000 tokens either 100,000 or 75,000 tokens, we explore the influence of vocabulary scale on training outcomes. The results, detailed in Table illustrate the trade-off's associated with varying vocabulary sizes.
From the experiment, it is evident that models equipped with a larger initial vocabulary of 150,000 tokens tend achieve lower Perplexity and higher Accuracy, indicating robust ability to capture diverse linguistic nuances that significantly enhance performance. In contrast, reducing the vocabulary size to 75,000 tokens results in increased perplexity and decreased accuracy, highlighting potential compromise in linguistic detail that adversely affects model functionality, especially in complex linguistic scenarios.
4.6.3 Momentum Strategy
This experiment evaluates the impact of incorporating momentum strategy into our tokenization algorithm's vocabulary pruning process. The performance of vocabularies pruned under both the momentum and non-momentum conditions is directly compared in Table
The results in Table demonstrate that the momentum approach significantly enhances model accuracy, with notable improvement from 43. 16% to 44.51% in the ADAT method with Unigram initialization vocabulary when momentum is applied. Similarly, the Unigram method shows baseline performance of 42.20% accuracy. These results confirm that integrating momentum allows for more refined pruning process by effectively utilizing historical performance data to make more informed decisions, thereby preserving valuable linguistic features.
4.6.4 Balance Strategy
In this ablation study, we investigate various functions to balance token frequency and loss value in tokenization algorithm. The primary objective is adhere to the principle that tokens with higher frequency and lower loss should be assigned higher priority. Given the significant difference their magnitudes, explored subtraction and division methods. We evaluated three functions, detailed in 3 Here, À is a scaling factor introduced to adjust the balance between frequency and loss, and set 1 in practice.
The results demonstrate that the subtraction methods a Ab and a log(b) yielded accuracies of 42.70 and 43.23 respectively. These results indicate relatively poor performance, even with

---

the logarithmic transformation applied to the score log(a) \b. This underperformance is likely attributable the significant disparity in the magnitudes of frequeney and loss values, which the subtraction methods struggle to reconcile effectively. In contrast, the division method Xlog(b-Tj significantly outperformed the subtraction approaches with accuracy of 44.51. This superior performance suggests that the division method more naturally balances the influence of frequency and loss by scaling the loss logarithmically before the division, thereby mitigating the impact of numerical range discrepancies. This method's ability to integrate frequency and loss without requiring additional adjustments for scale disparities results in more stable and effective prioritization of tokens.
4.7 Analysis of the Compute Costs
To prove that the proposed method is feasible in practice. We analyzed the empirical runtime introduced by ADAT. To measure runtime, we used NVIDIA A100 GPUs, an Intel 8378A CPU, and Py PyTorch 2.1.2 with CUDA 12.1. The tokenizer optimization involves epochs, where each epoch consists of training the LLM O.3B corpus, followed by inference concludes with a vocabulary pruning step (90 seconds for a 100K tokens vocabulary). Therefore, the total computational cost of the tokenizer optimization process is calculated as:
O.1B corpus, and
5 x (0.3B training + 0.1. inference + pruning time) 1.5B training + 0.5B inference + 450s.
ADAT introduces an additional training cost of 1.5B tokens and an inference cost of 0.5B tokens, along with minimal vocabulary pruming time. Compared to the hundreds of billions or even trillions of tokens required for LLM training, these computational costs are negligible. As shown in Table 6 the full-scale training of the LLM incurs significantly higher computational costs. For instance, when training models with a 16B and 60B corpus, the tokenizer optimization accounts for only 4.17% and 1.04% of the total training time, respectively. The Pythia-700, model takes 510 GPU hours train with the full Pile dataset [3], and exceeds the tokenizer optimization' 's computational cost by over 255 times. Therefore, the additional computational cost introduced by our method is minimal, making feasible in practice.
Table 6: Runtime of ADAT optimization and training models.
Tokenizer Optimization
Training on 16B Training on 60B
Pythia Report
Runtime
2 GPU hours
48 GPU hours
192 GPU hours
510 GPU hours
5 Limitations
The adaptive nature of our proposed tokenizer method introduces variations in tokenizers across different model sizes, leading to inconsistent vocabularies. This inconsistency complicates tasks such knowledge distillation and speculative decoding, which rely on the assumption of uniform vocabulary across both smaller and larger models.
Conclusion
In this paper, we have presented novel approach to tokenizer design that integrates key aspects of both accuracy and inference speed, addressing the inherent limitations found in existing tokenizer schemes. By innovating beyond the traditional frequency-based and end-to-end methodologies, our adaptive tokenizer framework strategically optimizes vocabulary construction, ensuring both rapid processing and high precision in language modeling tasks. Our results demonstrate that the adaptive tokenizer significantly enhances the performance of Large Language Models (LLMs) across various benchmarks, providing balanced solution that does not sacrifice speed for accuracy or vice versa. Future work will focus on refining these adaptive tokenization techniques, exploring further integration with neural network architectures, and expanding their applicability to a broader range of languages and complex linguistic tasks.
10

---

Acknowledgements
This work was supported by the Australian Research Council under Projects DP240101 848 and FT230100549.
References
Winogrande: An adversarial winograd schema challenge scale. 2019.
[2] I. Beitagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 055500 2020.
[3] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, F. Hallahan, M. A. Khan, S. Purohit, Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[4] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Poery-hFuutth AAAI Conference on Artificial Intelligence, 2020.
[5] B. Boccking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Nau- mann, A. Nori, J. Alvarcz-Valle, al. Making the most of text semanties to improve biomedical vision-language processing. In European conference computer vision, pages 1-21. Springer, 2022.
[6] K. Bostrom and G. Durrett. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint wxXiv2004.037700 2020.
T. Brown, B. Mann, Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1873: 13:719711 2020.
[8] E. Bursztein, M. Zhang. O. Vallis, X. Jia, and A. Kurakin. Retvec: Resilient and efficient text vectorizer. Advances in Newral Information Processing Systems, 36, 2023.
[9] J. H. Clark, D. Garrette, I. Ture, and J. Wieting, Canine: Pre-training an efficient tokenization-fre encoder for language representation. Transactions the Association for Computational Linguistics, 10:73-9 91, 2022.
[10] P. Clark, Cowhey, O. Etzioni, T. Khot, A. Sahharwal, Schoenick, and O. Tafjord. Think you have solved question answering? try are, ai2 reasoning challenge. urXiv:1803005457v 2018.
[11] T. Dao, with oo-awareness. Advances Neural Information Processing Systems, 35:16344- 16359, 2022.
Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention
[12] Devlin, M.-W. Chang, K. Lec, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1800001810 2018.
[13] P. Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
[14] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pilc: An 800gb datasct of diverse text for language modeling. arXiv preprint 1oXiv:2101.0002 2020.
[15] I. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac"h, h, H. Li, K. McDonell, N. Muennighoff, Ociepa, Phang, L. Reynolds, H. Schoelkopf, A. Skowron, I.. Sutawika, F. Tang, Thite, B. Wang, Wang, and A. Zou. A framework few-shot language model evaluation, 12 2023.
[16] N. Godey, R. Castagné, É. de la Clergerie, and B. Sagot. Manta: Efficient gradient-basse tokenization for robust end-to-end language modeling. arXiv preprint arXiv:2212.07288 2022.
[17] M. M. Islam, G. Aguilar, P. Ponnusamy, C. S. Mathialagan, C. Ma, and C. Guo. A vocabulary-free multilingual neural tokenizer for end-to-end task learning. arXiv preprint arXiv:2204 10815, 2022.
[18] M. G. Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.
[19] T. Kudo, Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint 18X1v:1804.10959, 2018.

---

[20] T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint urXiv:1808.06226, 2018.
[21] W. Ling, I. Trancoso, C. Dyer, and A. W. Black. Character-based neural machine translation. arXiv preprint / 15::::5511.05555 1,0456,, 2015.
[22] J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiga: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXi:2000..88124 2020.
[23] S. Liu, N. Deng, S. Sabour, Y. Jia, M. Huang, and R. Mihalcea. Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15264-15281, 2023.
[24] P. Nawrot, J. Chorowski, A. Lańcucki, and E. M. Ponti. Efficient transformers with dynamic token pooling. arXiv preprint arXiv:2211 1..0296111099,, 2022.
[25] D. Paperno, G. Kruszewski, A. Lazaridou, Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The lambada dataset, Aug 2016.
[26] C. Park, Eo, H. Moon, and H.-S. Lim. Should we find another model?: Improving neural machine translation performance with one-piece tokenization method without model modification. In Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies: Industry papers, pages 97-104, 2021.
[27] A. Radford, R. Jozefowicz, and I. Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint 1..iv:/704.01444 2017.
[28] A. Radford, K. Narasimhan, Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[29] W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019.
[30] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and J. Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of machine learning research. 21(140y:1 67, 2020.
[31] V. Sachidananda, J. S. Kessler, and Y-A. Lai. Efficient domain adaptation of language models via adaptive tokenization. arXiv preprint (:Xiv:2109.07460 2021.
[32] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. arXiv preprint arxiv: 200.000009999 2015.
[33] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-16422 Seattle, Washington, USA, Oct. 2013. Association for Computational Linguistics.
(34] M. N. Sreedhar, X. Wan, Y. Cheng, and Hu. Local byte fusion for neural machine translation. arXiv preprint 220s::2005. 1490, 2022.
[35] J. Su. Bytepiece: A more pure and effective tokenizer. huttg:://ggtaab.bccc///ooo/////s 2023.
[36] I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th international conference on machine learning (ICMI-/1), pages 1017-1024, 2011.
[37] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler. Charformer: Fast character transforeers via grasee subword tokenization. arXiv preprint a0Xiv:2106. 12672, 2021.
[38] A. Thawani, S. Ghanekar. X. Zhu, and Pujara. Learn your tokens: Word-pooled tokenization for language modeling, arXiv preprint arXiv:2310.1162. 2023.
[39] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13971, 2023.
[40] A. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions Information Theory, (3(2):260-263 1967.
12

---

[41] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy. and S. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In T. Linzen, G. Chrupala, and A. Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics.
[42] X. Wang, Ruder, and G. Neubig. Multi-view subword regularization. arXiv preprint arXiv:2103.0843 2021.
[43] Y. Wu, Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, Krikun, Cao, Q. Gao, K. Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 1609,004400 2016.
13

---

A Appendix
A.1
Compression Rate results.
The compression rate of the proposed model and baselines across Pythia model 70M models are shown in Table
Table 7: Performance comparison of different tokenization methods.
Metric
BPE
BytePiece
ADAT+By
Unigram
ADAT+U
Compression Rate
4.38
4.81
4.98
3.96
2.91
A.2 Results of ADAT in the 1B model.
To further demonstrate the scalability of our proposed ADAT method, we expanded our experimental results larger models and larger corpus. Specifically, add results(shown in Table ( from training a IB model on 60B corpus. The results demonstrate that on larger models and with more data, ADAT continues show substantial improvements over the baseline, indicating that ADAT has strong scalability.
Table 8: Results in the 1B model.
Metrie
Unigram
ADAT
Avg
49.11
51.20
A.3 Analysis of Differences Between Vocabularies
To illustrate the differences between the vocabulary results obtained by ADAT and those from unigram, calculated the overlap ratio of the two token sets as follows:
Ratio
|Avocab n Bvocabl/ Baieeabb///eccce Beocabl-
(6)
Where n and denote the intersection and union of two sets, respectively, we present the overlap ratios between the tokenizers obtained using ADAT and Unigram on different models under the same vocabulary size setting, as well as overlap ratios between these tokenizers themselves Table 9 As shown in Table 1, there are significant differences between the vocabulary generated by ADAT and that generated by unigram. This disparity arises because ADAT not entirely data-driven in its vocabulary generation process. By incorporating the loss from the L.M,, ADAT can simultaneously focus on enhancing the performance of the LLM. Additionally.the overlap ratio between ADAT- 160M and ADAT-410M is higher than that between ADAT- 160M and ADAT-70M. This also indirectly explains why, as shown in Table 3 the tokenizer generated by ADAT-410M more suitable for Pythia-160M compared to the tokenizer generated by ADAT-70M.
Table 9: Comparison between Vocabulary of ADAT and Unigram.
Unigram
ADAT
Model size
Ratio
Model size
Ratio
ADAT-70M
0.18
70M 160M
0.71
ADAT-160M.
0.19
70M 410M
0.69
ADAT-410M
0.21
160M I 410M
0.84
To compare the vocabulary obtained by ADAT with the initial vocabulary, we sorted the 100k initial vocabulary by score in descending order and calculated the percentage of tokens, pruned by ADAT,
with Apache-2.. 2.0 license
14

---

that fall into each score interval. As shown in Table ADAT-70M-50K, tokens are most densely distributed not the 0-25% interval, indicating that ADAT relies not only on token frequency but also on the prediction difficulty of tokens in the LLM during training.
Table 10: Comparison with Initial Vocabulary with vocabulary size 100k.
Vocabulary 0-25 %
25-50% %
50-75%
75-100% %
Unigram-50k ADAT-70M-50k
54.75% 21.43%
23.95%
8.11% 32.72%
13.19%
31.60%
14.24%
A.4 Impact of Training Epochs
We investigate the effects of varying the number of training epochs for developing the vocabulary. Specifically, the model is trained using vocabularies that have been developed over 3, 5, and 7 epochs. The model initialized with random weights for each training session to evaluate the immediate impact of the epoch variation.
As shown in Table III the results from varying the number of training epochs suggest a clear relationship between training duration and vocabulary efficacy. Before the epoch reaches a certain value (5 epochs), increasing the number of epochs benefits accuracy, indicating a more refined and effective vocabulary. However, accuracy does not significantly improve with further increases in the number of epochs. This suggests that, given the specified vocabulary size, ADAT can quickly and efficiently learn appropriate vocabulary without the need for prolonged training over many epochs. Therefore, in our experiments, the default number of training epochs set to 5.
Table 11: Impact of Training Epochs on Model Performance.
Training Epochs
Accuracy
3
43.67
5
44.51
7
44.58
A.5 Expanded Analysis on Infer Data Volume Tokens and Initial Vocabulary Size
have expanded the analysis of 'infer data volume tokens' and 'initial vocabulary size'-bott variables are explored within and beyond settings in Table The expanded results are displayed in the table We observe that an inference data volume of 100M tokens sufficient, with larger volumes yielding only marginal improvements. Regarding the initial vocabulary size, increasing it to 150K important enhance performance. However, when it increases 200K. the score shows almost no improvement, indicating that the 150K vocabulary likely already includes most of the potential final candidate tokens. Therefore, further increasing the initial vocabulary size will not bring additional benefits
Table 12: Results for different Infer Data Volume Tokens and Initial Vocabulary Sizes.
Infer Data Volume Tokens
75K
100K
150K
200K
IM
42.89
43.07
43.13
43.20
10M
43.19
43.39
43.74
43.77
100M
43.42
43.78
44.51
44.56
1000M
43.45
43.83
44.53
44.57
A.6
Details of Model Parameters
The detailed parameters of the different model sizes applied in our experiment are shown in Table 13
15

---

Table 13: Specifications of different LLMs used in the paper.
Model Size
Layers
Model Dim
Heads
Learning Rate
Batch Size
70M
6
512
8
10.0 x 10-4
32
160M
12
768
12
2.5 x 10-4
16
410M
24
1024
16
2.5 x 10-1
16
A.7 Societal Impact
The adaptive tokenizer we propose is an important component of large language models. The proposed tokenizer is fine-tuned by closely monitoring the model's perplexity, enabling the language model to perform well on various tasks, such as machine translation and question answering. However, it may also face the same issues as existing subword tokenizers [19 433. such as privacy leakage. For instance, the tokenizer might segment sensitive information, like names, addresses, identity identifiers, into results tokens that could be recognized in text generation. Therefore, we recommend that the training corpus undergo preprocessing to remove private information. In addition, in real-world applications, the tokenizer should be employed in conjunction with privacy-preservine technologies and specially configured filtering rules. In specific scenarios, such as medical diagnostics and legal consultations, human experts should be involved to review the tokenization results.
A.8
List of Training Dataset
The specific corpus used for training tokenizers from The Pile and the detailed list of files is shown below.
pile ArXiv 025.json
pile ArXiv 069.json
pile ArXiv _00iv_oo
Dell__rXiv Siiiv_099.jjon
pile_ArXiv_Laeeeeeee SiKiv_098.json
pile Arr.AriXi3._j__
pile_ArXiv gile_Arriv_124.jjoo
pile_ArXiv_133.jss
pile_ArXiv_133.jse
pile ArXiv_
pile Books3
Sooks3_015_ppp 015.json
pile Books3 016.json
[rooks3_052.json 052.json
Dill_Bookks3 3rs$33305__oooo
pile Bookk_Booo33BB1
20___1111_ 071.json
pile Bacil__ooo33__1 083.json
Sile_Bookk3 0s4.json
pile Bas___Book3_093___p_
Bai_BBook3_1111
pile Books3 134.json
pile Books3 173.json
pile Books3_lokk json
pile Books3 203. 203.json
pile Books3 235.json
pile Books3 242.json
pile Books3 221_222.,2. 247.json
Bition__mmillsOOOe
pile FreeLaw_104.
PG-1990444jjon
"URL.hhtps:/plieeccechheeaa
16

---

pile Gutenberg PG-19 PG-19-049.jso
pile OpenSubtitles 008.json
pile 10peSuubritles_03111
pile OpenSubtitles, 037.json
Selleeppeeeebeex22222 __1 10joo
pile OeweeOeeeeeeeee0S55e0 006o.soonn
pile Setee_eeeeeeeee2000..ppp
pile 110
pile 10peeweebecxt2_155__.1
pile Site__ppeneebeex216622222
pile Setee_OOeeWeeeex222111111
pile Sese-OOpeWeeTTeee 2_22__222nnn
2all__ppeneebext1222222 240.__245.jnnn
pile_Open 25te___eeeeeeeee2256ppppppp
pilc_Pile-CC_0011jjoo
pile Pile-CC Pile-CC_024.json
pile Pile-CC Pilc-CCC069.json
pile Pile-CC 176-Coo
pile_Pile-CCC_122.jssm
pple_Pile-CCC1133.jjm
pile 188s
pile_Pile-CC_2209.jonn
pile_Pile-CC_2111.jsm
pile Bile_Pile-CC_233_jsola
pile Pile-CC Pile-CC__54.json
pile Pile-CC 259.json
pile PubMed Abstrrcts
Sottracts_037.jjso 037.json
BertMedd_Abstacee _otottss__9jjooo
pile halle_PuMedd_Pe Abstracct
Baaiicts_05__jjsee 054.json
pile PPPubMed_Ceetrai 028.json
pple__PbbMed_Cettte 053.json
pile_ Pane-uuuMe__Cetraa 067.json
pile Laa___uMMed_eetteee 0699.sson
pile_ Lil-PuubMe_eeerrrr 085.json
pile PubMed Central 123.json
pile PubMed Central 125.json
pile PubMed_ Central 132.json
pile_PPuullee PalMed_ PubMeddCCetrrallll 149.json
pile (eroMMdd_Cenraal
Dill_PubMeed PPuuMed_Ceetralll _ecntral_l73jjssllll
Biiie_PuMeed_CCeee 215.json
Balle_PubMed_Cmmee
pile_USPTO 1arrro__aakground__0000 0012.json
pile _SPTO Backgrounds Bacckrrundss_027jjjoom
pile USPTO Backgrounds 03 03 1json
CullC Backgrounds__ 051 151..sonn
pile pile_Lbuntu Lia___uuuuu_____O__111 _____IRRCCoooo
Sallttubuntuu_ B_t_ttuuuRCOO11_______ JRC_017.jsoo
pile_ "iikipedia_ee_0066jjp
pile_ Siie__iiiipdi__e__0999
pile_ Eikipedia__en043.jjee
Wikipediia_en_0533
pile Wikipedia 070.json
17

---

pile YoutubeSSuttitle__0G0 000.0008.nnn
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Limitations section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No theoretical results are included in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information needed to reproduce the experimental results included in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer:
[No]
Justification: The list of the dataset is included in the appendix. We will release codes after completing the necessary preparations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ htwps///oooeeuubsssssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwppbblccgguiecuceccccceooooope wetgguddddeeoeeubbinnnnn htt.SSbbbiiiiooooiiiiiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are included in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the experiment results with std error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The sufficient information on the computer resources is included in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:://hhaaaaesccccppcc//tcaaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both potential positive societal impacts and negative societal impacts are discussed Societal Impact section of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Included all citations.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:Thii paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "nc6HuVNBj2", "reviewer": "Reviewer_t7me", "summary": "Language model vocabularies are typically learned using frequency-based objectives. This objective does not entirely align with the language modeling objective–the task for which these vocabularies are ultimately used–and may therefore cause a bottleneck in language model performance. This proposes a new tokenization method that addresses this issue. Specifically, the vocabulary is optimized using an objective that incorporates both frequency-based and language-modeling losses. The method is relatively simple: use the Unigram LM tokenization method albeit with an altered loss function (i.e., not just unigram negative log-likelihood but also the cross entropy under a language model trained using the vocabulary under consideration). The two loss terms can be balanced using different functional forms and loss terms from previous iterations can be incorporated. Empirically, the authors find that their approach leads to better model performance across a variety of tasks. They perform ablations on various design choices.", "questions": "* In equation 1, the notation suggests that you’re optimizing the dataset. I’m guessing this is not the case though. Could you clarify?\n* Was switching between V and D to denote the vocabulary and D and T to denote the dataset an intentional choice?\n* How much more computation does ADAT take? Unclear… computational complexity should be discussed\n* Do you have Insights about why 410M vocab doesnt lead to big improvements for the smaller model?\n* The jump from 70M to 410M parameters isn’t huge. Do you think a 70M model could be used in ADAT for finding the vocabulary of a much larger model? This is perhaps the only way I see the algorithm being scalable.\n* Smaller points: The relationship between inference speed and sequence length (discussed first in section 3) should be made explicit; I believe lines 121-22 are a misstatement. Should either be least -> most or increasing -> decreasing", "limitations": "Limitations are generally not discussed. The most important points that the authors need to address are the changes to the computational complexity of the tokenization step and the fact that only English is explored empirically.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This work takes a step towards the more end-to-end learning of language models, exploring perhaps one of the last remaining components in language modeling that has not been well optimized yet.\n* The proposed method is a simple extension of a widely used algorithm, which could make its adoption easier. \n* Empirical results show how sensitive models are to tokenization, which is something that is perhaps underestimated and is an important point that the community should be aware of. \n* Even though the method is rather computationally expensive, it appears that using smaller models in the computation of the CE component of the tokenization loss is also effective. Training these smaller LMs can be considered a drop in the bucket compared to the total amount of computation used for training larger language models, although its unclear from the experiments how much smaller the tokenization LM can be compared to the final LM while still leading to good performance", "weaknesses": "* Ultimately, the algorithm may not be practically feasible, given that it requires training an LM at each iteration of vocabulary pruning. This is a very important aspect that the authors do not discuss, either in terms of runtime analysis or empirical runtimes.\n* The writing is fairly poor and imprecise. Comments such as \"maintaining the stability of the vocabulary is crucial to ensure consistent learning outcomes\" (line 159) are very vague. What is the \"stability of the vocabulary\"? What does it mean to have \"consistent learning outcomes\"? There are numerous other examples such as this and collectively, they will leave readers confused or perhaps even misinformed. \n* Stylistic points: There is a lot of redundancy between the introduction and section 2.1 that can be eliminated. \n* There are a huge number of spelling errors. Please run the manuscript through a spell checker!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our paper and valuable comments. \n\n**Q1. The algorithm's repeated LM training and unaddressed runtime issues may limit practicality.**\n\nOur algorithm is feasible in practice, because the optimization phase of the tokenizer only requires a minimal amount of data (0.3B), compared to the full training of the Large Language Model (LLM), resulting in only slight computational overhead.\n\nWe analyzed the empirical runtime introduced by ADAT. To measure runtime, we used 8 NVIDIA A100 GPUs, an Intel 8378A CPU, and PyTorch 2.1.2 with CUDA 12.1. The tokenizer optimization involves 5 epochs, where each epoch consists of training the LLM on a 0.3B corpus, followed by inference on a 0.1B corpus, and concludes with a vocabulary pruning step (90 seconds for a 100K tokens vocabulary). Therefore, the total computational cost of the tokenizer optimization process is calculated as,\n$$5 \\times (0.3B \\text{ training} + 0.1B \\text{ inference}+\\text{ pruning time }) = 1.5B \\text{ training} + 0.5B \\text{ inference} + 450s$$\n\nIn contrast, the full-scale training of the LLM incurs significantly higher computational costs. For instance, when training models with a 16B and 60B corpus, the tokenizer optimization accounts for only 4.17% and 1.04% of the total training time, respectively. The Pythia-70M model takes 510 GPU hours to train with the full Pile dataset [1], and exceeds the tokenizer optimization's computational cost by over **255 times**. Therefore, the additional computational cost introduced by our method is minimal, making it feasible in practice.\n\n|         | Tokenizer Optimization | Training on 16B | Training on 60B | Pythia Report |\n| ------- | ---------------------- | --------------- | --------------- | ------------- |\n| RunTime | 2 GPUh                 | 48 GPUh         | 192 GPUh        | 510 GPUh      |\n\n[1] Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. S Biderman.\n\n**Q2. Writing lacks clarity.**\n\nWe appreciate your feedback regarding the readability of our paper. In the revised manuscript, we will ensure more thorough proofreading and polishing to improve the overall quality and precision of the writing. Specifically, we clarify as follows,\n\n\\- **Vocabulary stability** refers to the consistency of each token's score relative to the previous round during each iteration of vocabulary pruning. To mitigate significant fluctuations, we introduced a momentum mechanism to smooth these changes and ensure score stability.\n\\- **Consistent Learning Outcomes** refers to achieving stable and reliable resulting vocabulary with same data and strategies, ensuring that the outcomes do not significantly vary due to different conditions or random factors.\n\n**Q3. Redundancy between the introduction and section 2.1.**\n\nThank you for your suggestions. We will revise Section 2.1 of the related work to eliminate redundancy.\n\n**Q4. Why 410M vocab doesnt lead to big improvements for the smaller model?**\n\nThis phenomenon may be due to larger models with more parameters are better equipped to capture complex token relationships. Thus, vocabularies from larger models have higher model capabilities, leading to smaller performance improvements when applied to smaller models. In contrast, vocabularies generated by smaller models are less dependent on extensive model capabilities, and thus, when applied to larger models, they are able to sustain better performance gains.\n\n**Q5. Do you think a 70M model could be used in ADAT for finding the vocabulary of a much larger model?**\n\nThank you for this insightful question. Yes, I believe it is a feasible way to enhance scalability.  We add experimental results of 70M-ADAT on a larger model (1B) in the table below. As the table shows, the 70M-ADAT also improves performance compared to Unigram. This demonstrates that the vocabulary found by 70M-ADAT exhibits good generalizability across different model sizes.\n\nBased on our analysis of compute costs in response to Q1, directly scaling ADAT to a large model is within an acceptable cost range compared to training a large model.\n\n| 1B model with Unigram | 1B model with 70M-ADAT |\n| --------------------- | ---------------------- |\n| 48.63                 | 49.68                  |\n\n**Q6. Only English is explored empirically.**\n\nTo verify the effectiveness of ADAT in other languages, we further assess our method using a **mixed corpus of Chinese and English**. We evaluate the performance on the same English benchmarks as in the manuscript and the Language Model Evaluation Opencompass for the Chinese benchmark FewCLUE [2]. The results shown in the table below indicate that ADAT shows a 2.13 increase on the Chinese benchmark and a 2.28 for English benchmarks, demonstrating its effectiveness in other languages.\n\n| benchmark | English | Chinese |\n| --------- | ------- | ------- |\n| Unigram   | 37.75   | 44.75   |\n| ADAT      | 40.03   | 46.88   |\n\n[2] FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark\n\n**Q7.  Smaller points and misstatements.**\n\nWe appreciate your attention to detail, and we will carefully examine the manuscript and correct all spelling errors and misstatements.\n\n- Equation 1:  The vocabulary $V$ will be optimized using Equaton 1, which will be rewritten as $ \\min_V \\textbf{Length}(D_o,V)- \\lambda \\textbf{Acc}(D,M,V) , \\vert V\\vert \\leq N $ in the final version.\n\n- Notation of vocabulary and dataset:  We will use consistent notation.\n\n- Inference Speed and Sequence Length: For a given text, longer token sequences extend inference time and reduce inference speed due to increased computational demands. We will include clearer expressions in the final version.\n\n- Line 121: We will change \"least\" to \"most\".\n\n**Q8. Discussion of Limitations.**\n\nPlease refer to General Response.\n\nWe hope we have adequately addressed your concerns. If there is still anything unclear, please feel free to let us know during the rebuttal window.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ctZ1mMznpm", "reviewer": "Reviewer_aWgE", "summary": "This paper proposes a method to learn the tokenizer of a language model as part of language model training. The method works by combining a compression loss (which is also used by classical tokenization algorithms) with a language modeling loss, and iteratively removing tokens that contribute the least to the combined loss. The authors show that this method leads to improved performance on several downstream tasks compared to commonly-used tokenizers and also conduct an extensive quantitative analysis of various design choices (e.g., vocabulary size, corpus size, initial vocabulary size).", "questions": "These are mostly comments and smaller points:\n\n- Citation [8] should be NeurIPS 2023, not 2024.\n- An important paper not cited is [Nawrot et al. (2023)](https://aclanthology.org/2023.acl-long.353/).\n- 84: \"Neurall\" -> \"Neural\"\n- 86: \"Stringent\" -> \"stringent\"\n- Figure 1: \"Troditional\" -> \"Traditional\"\n- 194: \"We\" -> \"we\"\n- 226: How did you test for significance? Please mention the statistical test and exact results or else rephrase (e.g., \"substantial\").\n- 195: How exactly do you conduct these evaluations? Are they zero-shot? Please provide more details.\n- 235: \"examines\" -> \"examine\"\n- 244: \"trianing\" -> \"training\"\n- 4.6.2: Why did you not test above 150K tokens?", "limitations": "The authors say that limitations are discussed in the \"Experimental Setup\" subsection, but I could not find any discussion of the limitations there.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors take a fresh look at tokenization and come up with a novel method that is simpler than many alternative approaches while still resulting in clear performance improvements. The presentation of the method is clear and easy to follow. The experiments are quite extensive and include analyses into the different components of the method. The results show systematic performance gains compared to the standard tokenizers. Overall, I think that this is a valuable contribution to the emerging field of tokenization research.", "weaknesses": "There are a few places where decisions of the authors seemed questionable to me and/or where I would have liked to see more details:\n\n- The authors make the assumption that an individual token $x$ contributes to the overall language modeling loss only via the cross-entropy loss in places where $x$ is the to-be-predicted token. However, this ignores the impact that $x$ has on the language modeling loss as part of the left context (i.e., when $x$ is among the tokens processed before predicting the next token). I understand that this might be a necessary simplification to make the method feasible, but I would have expected a more in-depth discussion of this limitation and its potential repercussions.\n- Based on the details provided in the paper, the setup of evaluating performance in terms of perplexity seems to be flawed: if I understand correctly, the adaptive tokenizer is trained using the token losses on the Pile, and the Pile is then again used to evaluate the different methods in terms of perplexity. I think this gives an unfair advantage to the adaptive method. The authors should either (i) evaluate perplexity on a different dataset not used for training or (ii) refrain from using perplexity as an evaluation measure.\n- I would have liked to see a more in-depth analysis of the compute costs of the adaptive tokenizer compared to the standard tokenizers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work and your insightful comments.\n\n**Q1. This ignores the impact that token $x$ has on the language modeling loss as part of the left context. I would have expected a more in-depth discussion.**\n\nThanks for your insightful feedback. In response to your comments, we conducted additional experiments to examine the impact of token $x$ when it appears as part of the left context in the prediction target. We allocated the predicted token's loss to its left-hand side tokens based on attention weights. To manage computational costs effectively, we implemented a lookback window that only considers the immediate token to the left for loss updates.\n\nThese experiments were carried out on a model with 70M parameters, and the results are detailed in the table below. The findings indicate that incorporating this \"lookback loss\" has a minimal impact on model performance. This suggests that while the contextual influence of $x$ is indeed present, its effect is sufficiently captured under our current modeling framework without necessitating significant computational overhead.\n\n|  | Unigram | ADAT | ADAT+Win |\n| ----- | ---- | ------ | ------ |\n| Avg. | 42.22 | 44.51 | 44.38 |\n\n**Q2. PPL evaluation method may unfairly advantage adaptive tokenizer.**\n\nThank you for your valuable comments regarding our evaluation methodology. I would like to provide clarifications on two key points you raised:\n\nFirstly, as stated in line 193 of our manuscript, the perplexity (ppl) evaluation metric was conducted on the  **PG19 dataset, not the Pile dataset**. This distinction ensures that our training and testing were performed on separate datasets, effectively mitigating any potential issues of data leakage.\n\nSecondly, to comprehensively assess the effectiveness of our method, we reported the accuracy (ACC) of our model across nine different datasets. These results robustly validate the performance of our approach.\n\nIn the final version of our paper, we will clarify these evaluation details more explicitly to prevent any possible misunderstandings.\n\n**Q3. I would have liked to see a more in-depth analysis of the compute costs.**\n\nIn the table below, we outline the computational costs for standard tokenizers, the proposed ADAT, and the training phases of an LLM. Although ADAT introduces additional computational expenses, these costs are minimal and marginal compared to the significant resources required for LLM training.\n\nWe analyzed the empirical runtime introduced by ADAT. The tokenizer optimization involves 5 epochs, where each epoch consists of training the LLM on a 0.3B corpus, followed by inference on a 0.1B corpus, and concludes with a vocabulary pruning step (90 seconds for a 100K tokens vocabulary). Therefore, the total computational cost of the tokenizer optimization process is calculated as,\n$$5 \\times (0.3B \\text{ training} + 0.1B \\text{ inference}+\\text{ pruning time }) = 1.5B \\text{ training} + 0.5B \\text{ inference} + 450s$$\n\nADAT introduces an additional training cost of 1.5B tokens and an inference cost of 0.5B tokens, along with minimal vocabulary pruning time. Compared to the hundreds of billions or even trillions of tokens required for LLM training, these computational costs are negligible.\n\nTo measure runtime, we used 8 NVIDIA A100 GPUs, an Intel 8378A CPU, and PyTorch 2.1.2 with CUDA 12.1. As illustrated in the table below, the Unigram tokenizer consumes 0.33 CPU hours, whereas ADAT requires around 2 GPU hours. The LLM training phase demands significantly more resources, exceeding 500 GPU hours [1]. This comparison underscores that the additional computational expense introduced by ADAT is relatively insignificant, further justifying its use given its potential benefits.\n\n|                |Unigram| ADAT-70M|Pythia-70M|\n| ------------ | ------ | -------- | ---------- |\n| Runtime | 0.33 CPUh | 2 GPUh  | 510 GPUh   |\n\n[1] Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. S Biderman.\n\n**Q4. References.**\n\nWe will include and discuss the reference [2] in the final version.\n\n[2] Nawrot P, Chorowski J, Lancucki A, et al. Efficient Transformers with Dynamic Token Pooling\n\n**Q5. Lin226. How did you test for significance? Please mention the statistical test and exact results or else rephrase (e.g., \"substantial\").**\n\nSorry for the misleading. We use the word \"significant\" merely to describe the degree of improvement. We will replace it with \"substantial\" or \"considerable.\"\n\n**Q6. How exactly do you conduct these evaluations?**\n\nWe apply five-shot evaluations in experiments, we will explicitly clarify this in the final version.\n\n**Q7. 4.6.2 Why did you not test above 150K tokens.?**\n\nAs shown in Table 2 of the manuscript, increasing the initial vocabulary size up to 150K markedly enhances performance. However, when it increases to **200K**, the score is **44.56**, which is comparable to the score of 44.51 for 150K, indicating that the 150K vocabulary likely already includes most of the potential final candidate tokens. Therefore, further increasing the initial vocabulary size will not bring additional benefits.\n\n**Q8 Typos.**\n\nThanks for your suggestions and corrections. We will correct all typos and revise the year information regarding citation [8] in the manuscript.\n\n**Q9. Discussion of Limitations.**\n\nPlease refer to General Response.\n\nIf there is still anything unclear, Please feel free to let us know during the rebuttal window.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rNi21bmtFm", "reviewer": "Reviewer_GBKH", "summary": "This study introduces an adaptive tokenizer whose development is integrated with the performance of the LLM. The tokenizer has the particularity that it is fine-tuned based on the model’s perplexity during training. Empirical results show that this approach improves accuracy compared to “traditional” tokenization methods.", "questions": "The main suggestion that I have for this work would be to compare it with other tokenizers that might not be mainstream but that have been shown to perform better. Comparing it with other Task-Adaptive Tokenizers would be a start.", "limitations": "See weaknesses.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well motivate and easy to understand\n- Experiments are very comprehensive\n- Insightful ablation study\n- Answer most of the questions that one may have for a tokenizer: impact on perplexity, accuracy in downstream tasks, and performance depending on the vocabulary size,", "weaknesses": "- There are only comparisons with very common tokenizers: BPE, BytePiece, and Unigram. This is enough to guide engineers but for scientific work, I would expect comparisons with related work, even if they are not widely adopted. For instance, how does it compare to other adaptive tokenizers such as the one proposed by “Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond” (not cited)?\n- The limitations of this work are not very well discussed. When does ADAT completely fail? For which scenario shouldn’t we use it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts to enhance the quality of our manuscript. We appreciate the issues you identified, and we believe we have thoroughly clarified and addressed them as follows.\n\n**Q1. There are only comparisons with very common tokenizers: BPE, BytePiece, and Unigram. This is enough to guide engineers but for scientific work, I would expect comparisons with related work, even if they are not widely adopted. For instance, how does it compare to other adaptive tokenizers such as the one proposed by “Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond” (not cited)?**\n\nThanks for your feedback regarding the comparison method in our experiments. Task-adaptive tokenizers represent a promising direction, yet they **serve a different purpose from the methodology outlined in our paper**. Like BPE and Unigram tokenizers, our proposed model ADAT is designed to learn a general tokenizer. In contrast, the task-adaptive tokenizer[1] you mentioned focuses on integrating tokenizers derived from varying data distributions.\n\nIn the current experimental setup, we incorporated the task-adaptive tokenizer[1] by utilizing data from 'the pile' as Distribution A, while employing the training dataset from test set ARC-E, PIQA and SciQ as Distribution B. This approach facilitated the generation of a vocabulary through task-adaptive tokenization. The outcomes, presented in the table below, illustrate that the task-adaptive method exhibits robust performance on tasks ARC-E, PIQA and SciQ. However, on the other 6 tasks, its performance is inferior to ADAT, and on 5 of those tasks, it falls below the baseline, attributable to its merge strategy that favours task-specific tokens, consequently neglecting universal tokens.\n\nGiven that the task-adaptive methodology accesses an expanded dataset from Distribution B (the training set of the test set ARC-E, PIQA and SciQ), the findings suggest its limitations as a universally applicable tokenization strategy. Detailed discussions and comparisons with Reference[1] are planned for the final version of our paper to thoroughly investigate these observations.\n\nThese findings are significant as they demonstrate the robustness of our method, even when compared to more specialized, task-adaptive tokenization strategies. We hope this additional analysis addresses your concerns and further validates the effectiveness of our proposed method.\n\n|                         | ARC-C | ARC-E | Boolq | Lambda | LogiQA | PIQA  | SciQ  | SST-2 | Winogrande | Avg.  |\n| ----------------------- | ----- | ----- | ----- | ------ | ------ | ----- | ----- | ----- | ---------- | ----- |\n| Unigram                 | 19.54 | 37.04 | 53.06 | 17.27  | 23.20  | 60.50 | 68.10 | 49.77 | 51.46      | 42.22 |\n| ADAT(Ours)              | 18.46 | 40.57 | 61.19 | 17.97  | 24.22  | 59.93 | 72.40 | 54.24 | 51.62      | 44.51 |\n| Task-specific tokenizer | 17.15 | 37.82 | 55.63 | 14.36  | 22.73  | 59.58 | 69.20 | 49.08 | 50.36      | 41.77 |\n\n[1] Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond. EMNLP2023\n\n**Q2. Discussion of Limitations.**\n\nWe apologize for not clearly and separately discussing the limitations of our method in the manuscript. We will add the following discussion of limitations to the beginning of the Conclusion section as a separate part in the final version.\n\nThe adaptive nature of our proposed tokenizer method introduces variations in tokenizers across different model sizes, leading to inconsistent vocabularies. This inconsistency complicates tasks such as knowledge distillation and speculative decoding, which rely on the assumption of a uniform vocabulary across both smaller and larger models.\n\n\n\nWe hope that our answer has addressed your concerns. Please feel free to let us know during the rebuttal window if there is still anything unclear. We appreciate your suggestions and comments! Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cyLxrgRS9", "reviewer": "Reviewer_L8Xj", "summary": "The presented paper proposes an adaptive tokenization scheme that is learned jointly with an LLM and assesses its performance on downstream tasks for smaller model sizes and token budgets.", "questions": "N/A", "limitations": "Might need a separate section to address limitations.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Interesting idea to include both the loss of the current iteration as well as the loss of the previous iteration via a momentum term to stabilize the training procedure\n- Interesting results on Cross-Model Adaptability of vocabularies for different model sizes", "weaknesses": "### Weaknesses\n---\n\n\n- **[medium]**: both the model size (up to 410M parameters) and the corpus size (16B tokens) seem to be on the smaller side and it is unclear if these findings would generalize to the billion level parameter level and trillion level token budget which is more representative of the scales for current state-of-the-art LLMs.\n- **[medium]**: The fertility of a tokenizer is essential for comparing the average sequence length as well as performance across all supported languages i.e. a fertility of 1 would indicate that every single word is contained in the vocabulary **[1]**. I believe those values should be compared across all methods to give a better understanding.\n- **[medium]**: For ARC-C, LogiQA, PIQA, and Winogrande (4/9) the scores are actually lower than the Unigram baseline or within the standard deviation range which calls the significance of the results into question. Additionally, the model size scaling experiments indicate a reduction in gains +2.29 (70M), +2.01 (160M), +1.92 (410M) over the baselines and the gap will likely continue to narrow as we scale up the model size, potentially losing its effectiveness for production-scale models. Ideally, we'd want some sort of significance tests to verify the impact.\n- **[medium]**: the loss calculation in Equation 2 seems expensive for a large vocabulary $V$ as well as a large corpus. How big is the impact on the runtime and/or is this done on mini-batches? How does this scale with a bigger corpus i.e. within + beyond what is considered in Table 5? Naturally, from the ablation we would just scale both, when does this become too expensive or flatten out in terms of performance gains? Some scaling plots for both variables would be helpful for practitioners.\n\n\n\n\n### Minor Comments & Typos\n---\n\n- please sort references such that [7,2] is ordered as [2,7]\n- the fraction balance method in Table 5 needs brackets to reflect the correct fraction from l. 349\n\n\n### References\n---\n\n- **[1]**: [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://aclanthology.org/2021.acl-long.243) (Rust et al., ACL-IJCNLP 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work and your thoughtful review. \n\n**Q1. Both the model size (up to 410M) and the corpus size (16B) seem to be on the smaller side and it is unclear if these findings would generalize to the billion level parameter level and trillion level token budget.**\n\nTo demonstrate the scalability of our proposed ADAT method, we expanded our experimental results on larger models and larger corpus. Specifically, we add results(table below) from training a **1B model** on **60B corpus**. The results demonstrate that on larger models and with more data, ADAT continues to show substantial improvements over the baseline, indicating that ADAT has strong scalability.\n\n|      | Unigram | ADAT          |\n| ---- | ------- | ------------- |\n| Avg. | 49.11   | 51.20 (+2.09) |\n\n**Q2. The fertility of a tokenizer is essential for comparing the average sequence length as well as performance across all supported languages[1]. I believe those values should be compared across all methods to give a better understanding.**\n\nRegarding the fertility of tokenizers, we have included a comparative analysis of the fertility rates for all methods. We apply the four English treebanks used in Reference [1] and present the average values in the table below.\n\n|           | BPE  | BytePiece | BytePiece+ADAT | Unigram | Unigram+ADAT |\n| --------- | ---- | --------- | -------------- | ------- | ------------ |\n| Fertility | 1.25 | 1.07      | 1.08           | 1.32    | 1.49         |\n\nIt's crucial to recognize that the fertility metric does not directly correlate with a model's final performance. For example, a fertility value of 1 indicates a vocabulary that covers all corpus words, similar to a bag of words model, which fails to capture intrinsic semantic connections like word roots or affixes. At the other extreme, high fertility could lead to character-level or byte-level tokenization, resulting in over-segmentation and a loss of semantic priors.\n\n[1] How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (Rust et al., ACL-IJCNLP 2021)\n\n**Q3.** **The significance of the results and  losing its effectiveness for production-scale models. we'd want some sort of significance tests to verify the impact.**\n\nWhile ADAT scores are lower than Unigram on very few tasks, it outperforms the baseline on 7 out of 9 tasks for the 70M model and on 8 out of 9 tasks for the 410M model, with an overall improvement of approximately 2 points. This may be due to different datasets requiring different sub-optimal tokenization strategies. ADAT maintains accuracy on these datasets while improving performance on others.\n\nWe used **t-tests** to compare the results of the ADAT method with the Unigram baseline to statistically validate the performance differences. For both the 70M and 410M models, the p-values comparing the ADAT method to the baseline were 0.002 and 0.0009, respectively, indicating that **ADAT achieved statistically significant performance gains over the baseline**. (p-value < 0.05).\n\nTo address the concern regarding the impact of model size scaling and the potential reduction in gains, we conducted statistical tests and additional experiments as follows,\n\n- To assess the effect of the ADAT algorithm across different model sizes, we performed **t-tests** on **the performance gains provided by ADAT relative to the baseline** for the 70M and 410M models. The resulting **p-value of 0.48** indicates that the performance improvement of the ADAT method **does not significantly differ between the 70M and 410M** models, showing **consistent gains across various model sizes**.\n\n- Furthermore, As shown in the table of the response to Q1, the performance gain(2.09) is higher than that of the 410M model (1.92) and similar to that of 160M model (2.01), demonstrating the effectiveness of scalability in model size for ADAT. \n\nThese results collectively confirm that the ADAT method is effective across various model sizes.\n\n**Q4. Equation 2 seems expensive. How big is the impact on the runtime and/or is this done on mini-batches? How does this scale with a bigger corpus i.e. within + beyond what is considered in Table 5?**\n\nEquation 2 is employed for generating the initial vocabulary, which is expensive for large datasets. For this, **a 5GB subset of** the corpus was used. Furthermore, during the actual calculations, an approximation of the loss is applied. The empirical runtime to produce an initial vocabulary of 150k is 914 seconds.\n\nFurthermore, we have expanded the analysis of 'infer data volume tokens' and 'initial vocabulary size'—both variables are explored within and beyond typical settings. The expanded results are displayed in the table below. We observe that an inference data volume of 100M tokens is sufficient, with larger volumes yielding only marginal improvements. Regarding the initial vocabulary size, increasing it to 150K is important to enhance performance. However, when it increases to 200K, the score shows almost no improvement, indicating that the 150K vocabulary likely already includes most of the potential final candidate tokens. Therefore, further increasing the initial vocabulary size will not bring additional benefits.\n\n|       | 75K   | 100K  | 150K  | 200K  |\n| ----- | ----- | ----- | ----- | ----- |\n| 1M    | 42.89 | 43.07 | 43.13 | 43.20 |\n| 10M   | 43.19 | 43.39 | 43.74 | 43.77 |\n| 100M  | 43.42 | 43.78 | 44.51 | 44.56 |\n| 1000M | 43.45 | 43.83 | 44.53 | 44.57 |\n\n**Q5.Typos.**\n\nThanks for the corrections regarding the typos in sort references and Table 5. We will carefully examine the manuscript and rectify typos.\n\n**Q6. Discussion of Limitations.**\n\nPlease refer to General Response.\n\nWe hope we have adequately addressed your concerns. If there is still anything unclear, Please feel free to let us know during the rebuttal window.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' time and the valuable feedback they have provided for our paper. These constructive comments have been instrumental in enhancing the quality of our work. Here is one common concern we would like to address:

**General Response: Discussion of Limitations.**

We apologize for not clearly and separately discussing the limitations of our method in the manuscript. We will add the following discussion of limitations to the beginning of the Conclusion section as a separate part in the final version.

The adaptive nature of our proposed tokenizer method introduces variations in tokenizers across different model sizes, leading to inconsistent vocabularies. This inconsistency complicates tasks such as knowledge distillation and speculative decoding, which rely on the assumption of a uniform vocabulary across both smaller and larger models.

Meta-review: This paper presents an interesting study of how different tokenizers can affect the quality of an LLM. This is a less studied problem in existing literature, and the authors do a good job at motivating the problem and providing analyses of results. The authors have also presented numbers that show that their method can work not only for smaller models nut also of models upto 1B parameters in size. The paper can benefit from better writing and from expanded set of experiments as noted by the reviewers.
Decision: Accept (poster)