Enhancing Large Language Models through Adaptive Tokenizers

Authors: mengyu  zheng
hanting  chen
tianyu  guo
chong  zhu
binfan  zheng
chang  xu
yunhe  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 06:12:28

Abstract: Tokenizers serve as crucial interfaces between models and linguistic data, substantially influencing the efficacy and precision of large language models (LLMs). Traditional tokenization methods often rely on static frequency-based statistics and are not inherently synchronized with LLM architectures, which may limit model performance. In this study, we propose a simple but effective method to learn tokenizers specifically engineered for seamless integration with LLMs. Initiating with a broad initial vocabulary, we refine our tokenizer by monitoring changes in the model’s perplexity during training, allowing for the selection of a tokenizer that is closely aligned with the model’s evolving dynamics. Through iterative refinement, we develop an optimized tokenizer. Our empirical evaluations demonstrate that this adaptive approach significantly enhances accuracy compared to conventional methods, maintaining comparable vocabulary sizes and affirming its potential to improve LLM functionality.

Main paper: 
 introduction: 
 In recent years, large language models (LLMs) have emerged as foundational tools across a spectrum of applications in natural language processing [3, 7, 24] . From generating human-like text to enabling complex question-answering systems [28] , LLMs have proven to be exceptionally versatile and capable. At the core of these models lies the tokenizer, a critical component that dictates how natural language is transformed into a format amenable to computational processing. The effectiveness of a tokenizer directly influences the model's ability to understand and generate language, thus playing a pivotal role in the overall performance of the LLM. Recognizing this integral relationship, it becomes essential to develop tokenizers that are not only effective but also dynamically adaptable to the evolving architectures of contemporary LLMs.Current tokenization methods for large language models (LLMs) primarily include Byte Pair Encoding (BPE) [32] , WordPiece [43] , and Unigram [19] , each serving to enhance text preprocessing by splitting it into manageable subwords. BPE focuses on reducing the dataset size through a greedy merging strategy based on character or subword frequency, effectively addressing the issue of infrequent words by splitting them into more common subunits. WordPiece, similar to BPE, starts with a base vocabulary and iteratively refines it by merging the most frequent pairs but incorporates a likelihood maximization step, which makes it slightly more context-aware than BPE. Unigram tokenization operates somewhat inversely, beginning with a large vocabulary and iteratively pruning it down based on token utility calculated through negative log likelihood, aiming to optimize the vocabulary against corpus loss metrics. Despite their efficiency in handling large vocabularies and improving computational feasibility, these tokenization methods are typically fixed once developed and are not designed to adapt or learn from the model's evolving understanding of language during training.While traditional tokenization methods have been instrumental in enhancing the efficiency and effectiveness of large language models (LLMs), they are typically decoupled from the model's learning mechanisms. This means they do not adapt or evolve based on the model's performance or the specific requirements of the tasks being addressed. Instead, these methods prioritize compressing the vocabulary size, which can sometimes lead to suboptimal performance in complex language tasks where adaptability and contextual understanding are crucial. Recent advances in end-to-end learnable tokenization [17, 16, 38] aim to address these deficiencies by more closely integrating tokenization with the model's learning processes. However, these systems, while innovative, introduce significant computational overhead (e.g., gradient-based tokenization and pooling modules) and lack the flexibility of traditional tokenization methods, which can be easily transferred across different models.In this study, we address the limitations of traditional tokenization methods by creating a system where the tokenizer's development is coupled with the performance of the LLM itself. Specifically, we introduce an adaptive tokenizer that begins with a comprehensive initial vocabulary. As training progresses, we fine-tune this tokenizer by closely monitoring the model's perplexity. This ongoing adjustment allows the tokenizer to evolve in tandem with the LLM, ensuring that the tokenization process remains optimally aligned with the model's dynamic learning patterns. Our empirical results confirm that this adaptive approach markedly improves accuracy over traditional methods, demonstrating its potential to significantly enhance LLM functionality. 
subword tokenizer: 
 Subword tokenizers are widely applied in many large language models (LLMs) [32, 19, 42, 35, 20] , such as GPT-3 [7] , BERT [12] , and T5 [30] . This is because subword tokenizers do not face the out-of-vocabulary issues that word-level tokenizers do. Unlike character-level tokenizers [36, 27] , they do not require processing longer sequences at the character level, which significantly increases the complexity of the models quadratically [21] . Specifically, BPE (Byte Pair Encoding) [32] applies a compression algorithm [13] to the task of word segmentation. Unlike BPE, which builds a vocabulary from smaller to larger units, Unigram [19] starts by preparing a large seed vocabulary. The vocabulary is pruned until it reaches the specified size. Similar to the Unigram framework and its assumptions, BytePiece [35] trims the vocabulary based on token frequency to the required size. In its initial text encoding phase, BytePiece converts the corpus into bytes, enhancing the training speed of the tokenizer and achieving a higher compression rate. Obviously, these tokenizers are data-driven [5] , with the generated vocabulary built based on the frequencies of word fragments [6] . However, they cannot directly interact with LLMs to enhance model capabilities [9] .Unlike traditional tokenizers that offer a predefined and fixed vocabulary, learnable tokenizers [17, 16, 38, 37, 34, 8 ] can be integrated with large language models into an end-to-end learning framework, resulting in task-specific tokenization to enhance the performance of LLMs. MANTa [16] introduces a gradient-based tokenization and pooling module that can be jointly learned with an encoder-decoder LLM [2] . RETVec [8] embeds words into a high-dimensional vector with a pre-trained model to be robust against adversarial attacks. Neural [17] adapts the tokenization behavior to the downstream task after pre-training the tokenization by distilling from a language-specific subword tokenizer. However, such tokenizers have high requirements for the quantity and quality of the training data. If the data distribution is unbalanced or contains too much noisy data, it can lead to poor generalization of the tokenizer and negatively affect the performance of LLMs. For instance, Neural [17] requires a pre-training dataset curated with space-separated tokens and two carefully crafted heuristics to improve the ground label of the dataset. Therefore, stringent requirements for the quality of training data limit their widespread application.In addition, some existing works have applied the concept of adaptive tokenizers in several fields, such as neural machine translation [26] , domain adaptation [31] , and text generation [23] . These task-adaptive tokenizers integrate tokenizers generated from different data distributions, focusing on how to combine the task-specific tokenizer with the other one. In contrast, our proposed model, ADAT, is designed to learn a general tokenizer. Therefore, the purpose of ADAT is different from that of the aforementioned adaptive tokenizers. 
adaptive tokenizers: 
 For Large Language Models (LLMs), two critical aspects are accuracy and inference speed, both of which are deeply intertwined with the design of the tokenizer. Specifically, an optimal vocabulary V with maximum size N in an objective dataset D can be described by the following optimization problem:EQUATIONwhere M denotes the trained LLM, Length and Acc denotes the sequence length and accuracy (or the performance) using vocabulary V in dataset D, and the λ is a hyper-parameter to balance the two terms. This problem incorporates two primary objectives: given a fixed vocabulary size, the first is to maximize inference speed given a fixed vocabulary size, and the second is to maximize model accuracy. However, existing tokenizer schemes typically focus on one aspect over the other; for instance, traditional frequency-based schemes emphasize speed, while end-to-end approaches prioritize accuracy. To address this, we propose a method that optimizes both aspects. Therefore, we introduce an improved approach based on traditional frequency statistics, termed "adaptive tokenizers." This method aims to refine the balance between vocabulary efficiency and performance, thereby enhancing both the speed and accuracy of the model.Traditional tokenization methods generally fall into two categories: one approach, exemplified by Byte Pair Encoding (BPE) and WordPiece, starts with a small set of symbols and incrementally builds a larger vocabulary by merging the most frequent adjacent pairs. The other approach, typified by the Unigram method, begins with a large initial vocabulary which is progressively pruned based on token utility, a method found to generally offer superior performance due to its probabilistic foundation.The Unigram model operates on the principle that the probability of a sentence is determined by the individual probabilities of its tokens. Here we briefly review the Unigram model. Initially, a large vocabulary V is established. This extensive initial set includes potentially every unique word or subword unit observed in the training corpus, ensuring that the vocabulary can cover all possible textual inputs. The process of refining the vocabulary involves several key steps repeated in cycles: 1. Probability Estimation: For each token x i in the current vocabulary, we calculate its probability p(x i ) based on its frequency of occurrence in the corpus. 2. Loss Calculation: We then compute the loss for each token, which is determined by how much the overall loss of the model would decrease if that token were removed. The loss function is calculated as:EQUATIONwhere S(X) is a set of segmentation candidates built from the input sentence X, and x = (x 1 , ..., x K ) is a subword sequence that P (x) = K i=1 p(x i ). The loss for each token x i is then formulated as L P (x i ) = L P (V )-L P (V -x i ). 3. Token Pruning: Tokens are ranked according to their calculated loss. Finally, a proportion of tokens contributing the most to increasing the overall loss is pruned from the vocabulary.To enhance the integration of Large Language Models (LLMs) with our tokenization process, we have developed a simple but effective refined method for calculating the loss associated with each token, incorporating insights directly from the LLM's performance metrics. This approach aims to optimize the tokenizer's vocabulary to better align with the LLM's understanding and generation of text. The framework is illustrated in Figure 1 .Our revised loss calculation method integrates the traditional Unigram model's frequency-based loss with a performance-driven loss derived from an LLM. Specifically, we first train an LLM M in the vocabulary D using a training dataset T . This model is designed to capture the linguistic nuances relevant to the tasks it is trained for, providing a robust framework for assessing token utility. For each token x i in the vocabulary, we measure its individual contribution to the model's error using a cross-entropy loss function. The loss for each token is calculated as:EQUATIONHere, CE represents the cross-entropy function, M (x i-1 ) is the LLM's output given the previous token x i-1 , and x i is the actual next token. This formula assesses how well the LLM predicts each token following its predecessor, providing a direct measure of each token's impact on model performance. Finally, The cross-entropy loss for each token is then combined with the traditional Unigram frequency-based loss. This combined loss ensures that tokens are evaluated not only on their frequency of occurrence but also on their actual contribution to the LLM's task performance.The final loss for pruning the vocabulary is given by:EQUATIONwhere F (•, •) is a function to balance the importance of frequency-based loss and LLM-driven loss, which will be discussed in experiments. Using this enhanced loss metric, we iteratively refine the vocabulary by pruning tokens that contribute the least to the combined loss, thus optimizing the vocabulary for both general language understanding and specific task performance. This process continues until the vocabulary is compact enough to manage while still being comprehensive enough to support the LLM effectively.Random sampling. In the training of Large Language Models (LLMs), ensuring that each token within the set vocabulary receives equal and substantial training is crucial to prevent loss bias due to uneven training. While iterating over all possible tokenizations of the training corpus would ideally provide the most comprehensive learning experience, this approach is computationally prohibitive due to the immense variety of potential segmentations. To address this, we adopt the classic Viterbi algorithm [40] to perform randomized tokenization of the training data. This method allows for a diverse and balanced exposure of all tokens within the vocabulary to the learning process. The Viterbi algorithm efficiently determines the most probable tokenization paths through a probabilistic model of token occurrence, which significantly reduces the computational overhead compared to exhaustive methods. By leveraging this approach, our LLM can learn each token in the vocabulary more uniformly, enhancing the overall robustness and performance of the model.Loss momentum. In the iterative process of training Large Language Models, maintaining the stability of the vocabulary is crucial to ensure consistent learning outcomes. To achieve this, we propose a momentum-based improvement for calculating the loss during each iteration. Specifically, the loss for iteration j of token x i , denoted as L j (x i ), is not solely computed based on the current data but is also weighted by the loss from the previous iteration L j-1 . This approach allows for a smoother convergence and mitigates fluctuations in training dynamics. The formula for updating the loss at each iteration is given by:EQUATIONwhere β is the momentum coefficient that controls the extent to which the previous loss influences the current loss. This methodology not only stabilizes the vocabulary updates across iterations but also enhances the model's ability to generalize from the training data by reducing the variability in loss across successive training epochs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we outline the comprehensive experimental framework designed to assess the effectiveness of our proposed tokenizer, Adaptive Tokenizer (ADAT), in comparison to established methods such as Byte Pair Encoding (BPE) [32] and the Unigram model [19] . These evaluations utilize the Pythia [3] suite of models at various scales, leveraging a substantial corpus to ensure robust and generalizable results.Model Framework We deploy the Pythia framework [3] for its lightweight design and adaptability across different computational setups. Pythia's flexibility facilitates reproducibility and consistent assessment of performance, making it an ideal choice for evaluating the scalability and efficiency of various tokenization strategies across model sizes of 70M, 160M, and 410M parameters.The study utilizes a substantial corpus extracted from The Pile [14] , consisting of 56GB of raw data across 91 files. We specifically excluded subsets from DM_Mathematics and Github to ensure the relevance and quality of the data. The remaining data, approximately 16 billion tokens after a random shuffle, was tokenized using a Unigram [19] tokenizer with a vocabulary size of 50,000 tokens. A detailed enumeration of the data files used is available in Supp. A.8.Baseline Methods Our investigation compares four tokenization methods: Bytepiece [35] , Byte Pair Encoding (BPE) [32] , Unigram [19] , and our proposed ADAT. These tokenizers were selected based on their established efficacy in handling large corpora and their theoretical implications for processing complex linguistic data.The effectiveness of each tokenization strategy is rigorously evaluated using several metrics. These include Perplexity (PPL), which measures the model's predictive accuracy, and Compression Rate(refer to A.1), assessing how efficiently the tokenization process reduces vocabulary size while preserving linguistic diversity. We calculate PPL for all models on PG19 [29] dataset. Specifically, we use its test set and the first 2048 tokens for each book. Furthermore, we use the Language Model Evaluation Harness [15] to run five-shot evaluations on eight common language modeling benchmarks: Lambada (OpenAI) [25] , PIQA [4] , WinoGrande [1] , ARC-Easy [10] , ARC-Challenge [10] , SciQ [18] , LogiQA [22] , and SST-2 [33, 41] , to provide a comprehensive insight into each method's capabilities.By analyzing the impact of tokenization on model scalability and the influence of vocabulary size variations, this study aims to enhance our understanding of how tokenization strategies can optimize language models for efficiency and linguistic performance. The findings are expected to contribute significantly to the development of more robust and adaptable language processing tools, catering to a wide array of NLP applications. The Pythia models are trained using a corpus of 15B tokens, where training the 70M model consumes approximately 48 GPU hours with FlashAttention [11] . The models used for loss calculation require additional 2 GPU hours.In this section, we examine the effects of different tokenization strategies on the training effectiveness. The core objective is to explore how variations in the vocabulary, induced by different tokenization methods, affect model training and performance.The Pythia-70M model is selected due to its moderate size and efficiency, which help mitigate the complexities associated with larger model architectures. It is initialized with random weights and undergoes a single training epoch using pre-training data. This data is processed with vocabularies generated from 1/10th of the training corpus (approximately 1.5 billion tokens), each containing 50,000 tokens-a size consistent with the Pythia [3] setup.Baseline tokenization methods including BPE, Unigram, and BytePiece, generate vocabularies consisting of 50,000 tokens directly from initial data that approximately one-tenth of the training corpus (about 1.5 billion tokens). In contrast, for the proposed ADAT method, initial vocabularies are generated using either BytePiece or Unigram with 150,000 tokens. These are then methodically refined down to 50,000 tokens over 5 iterative steps, matching the baseline vocabulary size. At each step, a randomly initialized model is trained on approximately 0.3 billion tokens from the initial dataset. Subsequently, the model performs inference on a subset of 0.1 billion tokens, during which token loss is calculated. This loss data, when combined with token frequency using the formula a λ log(b+1) , guides the vocabulary pruning process. An ablation study on the combination methods will be discussed in Section 4.6.4. We examine the scalability of a proposed tokenization method that tailors the vocabulary to model size, unlike the static Unigram method which maintains a consistent vocabulary across various model capacities. The scalability of the tokenization methods is tested using the Pythia framework configured at three different levels of computational complexity: 70M, 160M, and 410M parameters.For each model size, our method generates an optimized vocabulary specific to that configuration, allowing us to analyze how adjustments in vocabulary affect performance as model size increases. In contrast, the Unigram method employs a uniform 50,000-word vocabulary across all sizes, serving as a baseline. We gauge performance using Perplexity (PPL) and scores from benchmark datasets designed to assess the linguistic capabilities of each model under various conditions, providing insights into the efficiency and adaptability of the tokenization methods at scale. For training larger models, the same volume of data will lead to insufficient warm-up, potentially resulting in a slight decline in the accuracy of loss computations used for determining token priority. As a result, we increase the data volume for training the loss calculation model according to the size of model.The results of this experimental framework, as presented in This experiment evaluates the adaptability of vocabularies generated by our proposed tokenization method across various configurations of the Pythia model, particularly assessing whether vocabularies optimized for one model size can effectively scale to others. We initially create vocabularies using the 70M and 410M configurations. These are then used to train models at both scales to evaluate performance in downstream tasks, allowing us to assess how vocabularies designed for a specific size perform when applied to both smaller and larger models, thus examining their cross-model adaptability. The experiment aims to assess the impact of different tokenizer strategies on model performance across two vocabulary sizes, comparing a standard 50,000-token set with a reduced 30,000-token set.We utilize two configurations of the Pythia model-70M and 160M-to explore how vocabulary size influences model efficiency. Each model is tested using both the standard Unigram and our proposed tokenization method. This setup allows us to directly observe the effects of reduced vocabulary sizes on the performance dynamics, providing insights into how smaller vocabularies impact the computational efficiency and efficacy of language models.The experimental results, as presented in Table 4 , support the hypothesis that changes in vocabulary size can significantly affect model performance, with different impacts observed across varying model sizes. For large language models, it is common for models of vastly different sizes to utilize vocabularies of similar or identical sizes [3, 39] . This practice can lead to issues of performance or efficiency. Our method offers a more effective solution by tailoring tokenization strategies to the specific sizes of models, thereby mitigating these challenges. For the 70M model, ADAT achieved a notable improvement in accuracy from 42.22 to 44.51 (+2.29) and a substantial reduction in perplexity from 16.52 to 6.97 when using a 50,000-word vocabulary. Even with a reduced vocabulary of 30,000, ADAT enhances accuracy to 43.33 (+2.40) and decreases perplexity to 7.38, suggesting robustness against vocabulary size reduction. In contrast, the 160M model, which has a greater parameter capacity, also shows improvements with ADAT: accuracy increases from 44.04 to 46.05 (+2.01), and perplexity drops sharply from 13.97 to 6.19 for the 50,000 vocabulary size. With a 30,000 vocabulary, accuracy still increases to 45.26 (+2.14), and perplexity remains low at 6.11, underscoring that larger models not only handle vocabulary reductions well but also benefit significantly in terms of computational efficiency and model quality.This ablation study is structured into three distinct parts to explore how variations in the inference corpus size used for calculating token loss, initial vocabulary sizes, momentum strategy, and balance function F (a, b) influence the efficacy of our proposed tokenization method on a 70M parameter model. More ablation results can be referred to in the supplementary. We conducted an experimental study to investigate the effects of varying corpus sizes on the accuracy of token loss calculations. The experiment assessed the performance of models trained on different sizes of inference data, specifically 1 million (1M), 10 million (10M), and 100 million (100M) tokens.The results are summarized in the table 5 .These results indicate a direct correlation between the volume of the corpus used during the loss calculation phase and the overall accuracy of token loss estimates. When smaller corpora are used, a significant number of tokens are absent, resulting in numerous instances where loss values cannot be computed. Furthermore, the precision of token loss estimations tends to decrease with smaller data sets. Even with just 1M tokens, there was a noticeable improvement over the baseline unigram vocabulary accuracy of 42.22. This enhancement became more pronounced with larger data volumes, reaching an increase of 44.51 in accuracy with 100M tokens.This segment of our study assesses the effect of different initial vocabulary sizes on model performance. Adjusting the vocabulary from a baseline of 150,000 tokens to either 100,000 or 75,000 tokens, we explore the influence of vocabulary scale on training outcomes. The results, detailed in Table 5 , illustrate the trade-offs associated with varying vocabulary sizes.From the experiment, it is evident that models equipped with a larger initial vocabulary of 150,000 tokens tend to achieve lower Perplexity and higher Accuracy, indicating a robust ability to capture diverse linguistic nuances that significantly enhance performance. In contrast, reducing the vocabulary size to 75,000 tokens results in increased perplexity and decreased accuracy, highlighting a potential compromise in linguistic detail that adversely affects model functionality, especially in complex linguistic scenarios.This experiment evaluates the impact of incorporating a momentum strategy into our tokenization algorithm's vocabulary pruning process. The performance of vocabularies pruned under both the momentum and non-momentum conditions is directly compared in Table 5 .The results in Table 5 demonstrate that the momentum approach significantly enhances model accuracy, with a notable improvement from 43.16% to 44.51% in the ADAT method with Unigram initialization vocabulary when momentum is applied. Similarly, the Unigram method shows a baseline performance of 42.20% accuracy. These results confirm that integrating momentum allows for a more refined pruning process by effectively utilizing historical performance data to make more informed decisions, thereby preserving valuable linguistic features.In this ablation study, we investigate various functions to balance token frequency and loss value in our tokenization algorithm. The primary objective is to adhere to the principle that tokens with higher frequency and lower loss should be assigned higher priority. Given the significant difference in their magnitudes, we explored subtraction and division methods. We evaluated three functions, detailed in Table 5 . Here, λ is a scaling factor introduced to adjust the balance between frequency and loss, and we set it as 1 in practice.The results demonstrate that the subtraction methods a -λb and a -λ log(b) yielded accuracies of 42.70 and 43.23 respectively. These results indicate relatively poor performance, even with the logarithmic transformation applied to the score log(a) -λb. This underperformance is likely attributable to the significant disparity in the magnitudes of frequency and loss values, which the subtraction methods struggle to reconcile effectively. In contrast, the division method a λ log(b+1) significantly outperformed the subtraction approaches with an accuracy of 44.51. This superior performance suggests that the division method more naturally balances the influence of frequency and loss by scaling the loss logarithmically before the division, thereby mitigating the impact of numerical range discrepancies. This method's ability to integrate frequency and loss without requiring additional adjustments for scale disparities results in more stable and effective prioritization of tokens.To prove that the proposed method is feasible in practice. We analyzed the empirical runtime introduced by ADAT. To measure runtime, we used 8 NVIDIA A100 GPUs, an Intel 8378A CPU, and PyTorch 2.1.2 with CUDA 12.1. The tokenizer optimization involves 5 epochs, where each epoch consists of training the LLM on a 0.3B corpus, followed by inference on a 0.1B corpus, and concludes with a vocabulary pruning step (90 seconds for a 100K tokens vocabulary). Therefore, the total computational cost of the tokenizer optimization process is calculated as:5 × (0.3B training + 0.1B inference + pruning time) = 1.5B training + 0.5B inference + 450s.ADAT introduces an additional training cost of 1.5B tokens and an inference cost of 0.5B tokens, along with minimal vocabulary pruning time. Compared to the hundreds of billions or even trillions of tokens required for LLM training, these computational costs are negligible. As shown in Table 6 , the full-scale training of the LLM incurs significantly higher computational costs. For instance, when training models with a 16B and 60B corpus, the tokenizer optimization accounts for only 4.17% and 1.04% of the total training time, respectively. The Pythia-70M model takes 510 GPU hours to train with the full Pile dataset [3] , and exceeds the tokenizer optimization's computational cost by over 255 times. Therefore, the additional computational cost introduced by our method is minimal, making it feasible in practice.  
limitations: 
 The adaptive nature of our proposed tokenizer method introduces variations in tokenizers across different model sizes, leading to inconsistent vocabularies. This inconsistency complicates tasks such as knowledge distillation and speculative decoding, which rely on the assumption of a uniform vocabulary across both smaller and larger models. 
conclusion: 
 In this paper, we have presented a novel approach to tokenizer design that integrates key aspects of both accuracy and inference speed, addressing the inherent limitations found in existing tokenizer schemes. By innovating beyond the traditional frequency-based and end-to-end methodologies, our adaptive tokenizer framework strategically optimizes vocabulary construction, ensuring both rapid processing and high precision in language modeling tasks. Our results demonstrate that the adaptive tokenizer significantly enhances the performance of Large Language Models (LLMs) across various benchmarks, providing a balanced solution that does not sacrifice speed for accuracy or vice versa. Future work will focus on refining these adaptive tokenization techniques, exploring further integration with neural network architectures, and expanding their applicability to a broader range of languages and complex linguistic tasks. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Included all citations.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Compression Rate results.The compression rate of the proposed model and baselines across Pythia model2 -70M models are shown in Table 7 . To further demonstrate the scalability of our proposed ADAT method, we expanded our experimental results on larger models and larger corpus. Specifically, we add results(shown in Table 8 ) from training a 1B model on 60B corpus. The results demonstrate that on larger models and with more data, ADAT continues to show substantial improvements over the baseline, indicating that ADAT has strong scalability. To illustrate the differences between the vocabulary results obtained by ADAT and those from unigram, we calculated the overlap ratio of the two token sets as follows:EQUATIONWhere ∩ and ∪ denote the intersection and union of two sets, respectively, we present the overlap ratios between the tokenizers obtained using ADAT and Unigram on different models under the same vocabulary size setting, as well as the overlap ratios between these tokenizers themselves in Table 9 .As shown in Table 1 , there are significant differences between the vocabulary generated by ADAT and that generated by unigram. This disparity arises because ADAT is not entirely data-driven in its vocabulary generation process. By incorporating the loss from the LLM, ADAT can simultaneously focus on enhancing the performance of the LLM. Additionally,the overlap ratio between ADAT-160M and ADAT-410M is higher than that between ADAT-160M and ADAT-70M. This also indirectly explains why, as shown in Table 3 , the tokenizer generated by ADAT-410M is more suitable for Pythia-160M compared to the tokenizer generated by ADAT-70M. To compare the vocabulary obtained by ADAT with the initial vocabulary, we sorted the 100k initial vocabulary by score in descending order and calculated the percentage of tokens, pruned by ADAT, that fall into each score interval. As shown in Table 10 , ADAT-70M-50K tokens are most densely distributed not in the 0-25% interval, indicating that ADAT relies not only on token frequency but also on the prediction difficulty of tokens in the LLM during training. We investigate the effects of varying the number of training epochs for the vocabulary. Specifically, the model is trained using vocabularies that have been developed over 3, 5, and 7 epochs.The model is initialized with random weights for each training session to evaluate the immediate impact of the epoch variation.As shown in Table 11 , the results from varying the number of training epochs suggest a clear relationship between training duration and vocabulary efficacy. Before the epoch reaches a certain value (5 epochs), increasing the number of epochs benefits accuracy, indicating a more refined and effective vocabulary. However, accuracy does not significantly improve with further increases in the number of epochs. This suggests that, given the specified vocabulary size, ADAT can quickly and efficiently learn an appropriate vocabulary without the need for prolonged training over many epochs. Therefore, in our experiments, the default number of training epochs is set to 5. we have expanded the analysis of 'infer data volume tokens' and 'initial vocabulary size'-both variables are explored within and beyond settings in Table 5 . The expanded results are displayed in the table 12 . We observe that an inference data volume of 100M tokens is sufficient, with larger volumes yielding only marginal improvements. Regarding the initial vocabulary size, increasing it to 150K is important to enhance performance. However, when it increases to 200K, the score shows almost no improvement, indicating that the 150K vocabulary likely already includes most of the potential final candidate tokens. Therefore, further increasing the initial vocabulary size will not bring additional benefits. The detailed parameters of the 3 different model sizes applied in our experiment are shown in Table 13 . The adaptive tokenizer we propose is an important component of large language models. The proposed tokenizer is fine-tuned by closely monitoring the model's perplexity, enabling the language model to perform well on various tasks, such as machine translation and question answering. However, it may also face the same issues as existing subword tokenizers [19, 43] , such as privacy leakage. For instance, the tokenizer might segment sensitive information, like names, addresses, or identity identifiers, into results of tokens that could be recognized in text generation. Therefore, we recommend that the training corpus undergo preprocessing to remove private information. In addition, in real-world applications, the tokenizer should be employed in conjunction with privacy-preserving technologies and specially configured filtering rules. In specific scenarios, such as medical diagnostics and legal consultations, human experts should be involved to review the tokenization results.The specific corpus used for training tokenizers is from The Pile3 , and the detailed list of files is shown below.• pile_ArXiv_025. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.with Apache-2.0 licenseURL:https://pile.eleuther.ai/ 


Decision: This paper presents an interesting study of how different tokenizers can affect the quality of an LLM. This is a less studied problem in existing literature, and the authors do a good job at motivating the problem and providing analyses of results. The authors have also presented numbers that show that their method can work not only for smaller models nut also of models upto 1B parameters in size. The paper can benefit from better writing and from expanded set of experiments as noted by the reviewers.

Reviews: review: paper_summary: paper_summary: Language model vocabularies are typically learned using frequency-based objectives. This objective does not entirely align with the language modeling objective–the task for which these vocabularies are ultimately used–and may therefore cause a bottleneck in language model performance. This proposes a new tokenization method that addresses this issue. Specifically, the vocabulary is optimized using an objective that incorporates both frequency-based and language-modeling losses. The method is relatively simple: use the Unigram LM tokenization method albeit with an altered loss function (i.e., not just unigram negative log-likelihood but also the cross entropy under a language model trained using the vocabulary under consideration). The two loss terms can be balanced using different functional forms and loss terms from previous iterations can be incorporated. Empirically, the authors find that their approach leads to better model performance across a variety of tasks. They perform ablations on various design choices.
strength_weakness: strength_weakness: * Ultimately, the algorithm may not be practically feasible, given that it requires training an LM at each iteration of vocabulary pruning. This is a very important aspect that the authors do not discuss, either in terms of runtime analysis or empirical runtimes.
* The writing is fairly poor and imprecise. Comments such as "maintaining the stability of the vocabulary is crucial to ensure consistent learning outcomes" (line 159) are very vague. What is the "stability of the vocabulary"? What does it mean to have "consistent learning outcomes"? There are numerous other examples such as this and collectively, they will leave readers confused or perhaps even misinformed. 
* Stylistic points: There is a lot of redundancy between the introduction and section 2.1 that can be eliminated. 
* There are a huge number of spelling errors. Please run the manuscript through a spell checker!
questions: questions: * In equation 1, the notation suggests that you’re optimizing the dataset. I’m guessing this is not the case though. Could you clarify?
* Was switching between V and D to denote the vocabulary and D and T to denote the dataset an intentional choice?
* How much more computation does ADAT take? Unclear… computational complexity should be discussed
* Do you have Insights about why 410M vocab doesnt lead to big improvements for the smaller model?
* The jump from 70M to 410M parameters isn’t huge. Do you think a 70M model could be used in ADAT for finding the vocabulary of a much larger model? This is perhaps the only way I see the algorithm being scalable.
* Smaller points: The relationship between inference speed and sequence length (discussed first in section 3) should be made explicit; I believe lines 121-22 are a misstatement. Should either be least -> most or increasing -> decreasing
limitations: limitations: Limitations are generally not discussed. The most important points that the authors need to address are the changes to the computational complexity of the tokenization step and the fact that only English is explored empirically.
paper_summary: paper_summary: This paper proposes a method to learn the tokenizer of a language model as part of language model training. The method works by combining a compression loss (which is also used by classical tokenization algorithms) with a language modeling loss, and iteratively removing tokens that contribute the least to the combined loss. The authors show that this method leads to improved performance on several downstream tasks compared to commonly-used tokenizers and also conduct an extensive quantitative analysis of various design choices (e.g., vocabulary size, corpus size, initial vocabulary size).
strength_weakness: strength_weakness: There are a few places where decisions of the authors seemed questionable to me and/or where I would have liked to see more details:

- The authors make the assumption that an individual token $x$ contributes to the overall language modeling loss only via the cross-entropy loss in places where $x$ is the to-be-predicted token. However, this ignores the impact that $x$ has on the language modeling loss as part of the left context (i.e., when $x$ is among the tokens processed before predicting the next token). I understand that this might be a necessary simplification to make the method feasible, but I would have expected a more in-depth discussion of this limitation and its potential repercussions.
- Based on the details provided in the paper, the setup of evaluating performance in terms of perplexity seems to be flawed: if I understand correctly, the adaptive tokenizer is trained using the token losses on the Pile, and the Pile is then again used to evaluate the different methods in terms of perplexity. I think this gives an unfair advantage to the adaptive method. The authors should either (i) evaluate perplexity on a different dataset not used for training or (ii) refrain from using perplexity as an evaluation measure.
- I would have liked to see a more in-depth analysis of the compute costs of the adaptive tokenizer compared to the standard tokenizers.
questions: questions: These are mostly comments and smaller points:

- Citation [8] should be NeurIPS 2023, not 2024.
- An important paper not cited is [Nawrot et al. (2023)](https://aclanthology.org/2023.acl-long.353/).
- 84: "Neurall" -> "Neural"
- 86: "Stringent" -> "stringent"
- Figure 1: "Troditional" -> "Traditional"
- 194: "We" -> "we"
- 226: How did you test for significance? Please mention the statistical test and exact results or else rephrase (e.g., "substantial").
- 195: How exactly do you conduct these evaluations? Are they zero-shot? Please provide more details.
- 235: "examines" -> "examine"
- 244: "trianing" -> "training"
- 4.6.2: Why did you not test above 150K tokens?
limitations: limitations: The authors say that limitations are discussed in the "Experimental Setup" subsection, but I could not find any discussion of the limitations there.
paper_summary: paper_summary: This study introduces an adaptive tokenizer whose development is integrated with the performance of the LLM. The tokenizer has the particularity that it is fine-tuned based on the model’s perplexity during training. Empirical results show that this approach improves accuracy compared to “traditional” tokenization methods.
strength_weakness: strength_weakness: - There are only comparisons with very common tokenizers: BPE, BytePiece, and Unigram. This is enough to guide engineers but for scientific work, I would expect comparisons with related work, even if they are not widely adopted. For instance, how does it compare to other adaptive tokenizers such as the one proposed by “Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond” (not cited)?
- The limitations of this work are not very well discussed. When does ADAT completely fail? For which scenario shouldn’t we use it?
questions: questions: The main suggestion that I have for this work would be to compare it with other tokenizers that might not be mainstream but that have been shown to perform better. Comparing it with other Task-Adaptive Tokenizers would be a start.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: The presented paper proposes an adaptive tokenization scheme that is learned jointly with an LLM and assesses its performance on downstream tasks for smaller model sizes and token budgets.
strength_weakness: strength_weakness: ### Weaknesses
---


- **[medium]**: both the model size (up to 410M parameters) and the corpus size (16B tokens) seem to be on the smaller side and it is unclear if these findings would generalize to the billion level parameter level and trillion level token budget which is more representative of the scales for current state-of-the-art LLMs.
- **[medium]**: The fertility of a tokenizer is essential for comparing the average sequence length as well as performance across all supported languages i.e. a fertility of 1 would indicate that every single word is contained in the vocabulary **[1]**. I believe those values should be compared across all methods to give a better understanding.
- **[medium]**: For ARC-C, LogiQA, PIQA, and Winogrande (4/9) the scores are actually lower than the Unigram baseline or within the standard deviation range which calls the significance of the results into question. Additionally, the model size scaling experiments indicate a reduction in gains +2.29 (70M), +2.01 (160M), +1.92 (410M) over the baselines and the gap will likely continue to narrow as we scale up the model size, potentially losing its effectiveness for production-scale models. Ideally, we'd want some sort of significance tests to verify the impact.
- **[medium]**: the loss calculation in Equation 2 seems expensive for a large vocabulary $V$ as well as a large corpus. How big is the impact on the runtime and/or is this done on mini-batches? How does this scale with a bigger corpus i.e. within + beyond what is considered in Table 5? Naturally, from the ablation we would just scale both, when does this become too expensive or flatten out in terms of performance gains? Some scaling plots for both variables would be helpful for practitioners.




### Minor Comments & Typos
---

- please sort references such that [7,2] is ordered as [2,7]
- the fraction balance method in Table 5 needs brackets to reflect the correct fraction from l. 349


### References
---

- **[1]**: [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://aclanthology.org/2021.acl-long.243) (Rust et al., ACL-IJCNLP 2021)
questions: questions: N/A
limitations: limitations: Might need a separate section to address limitations.

Comments: title: Thank you
Response to the Authors' Rebuttal
Thank you
Thank you
Response to rebuttal
comment: We are grateful for your recognition and for increasing your score. Thank you so much!
I thank the authors for these helpful explanations. I have raised my score.
We truly appreciate your revised evaluation and the increase in score, your support is valuable to our work. Thank you very much. 

We have conducted further research on the existing work of adaptive tokenizers, including in the field of neural machine translation (NMT). For example,

ONE-Piece[1] proposed a subword tokenization using morphological segmentation and vocabulary communication to address OOV problem. This method leverages the correspondence between two languages to create the tokenizer, specifically tailored for translation tasks. Moreover, it differs from our approach by not adapting to the model. AT[2] adapts the tokenizer of the pretrained model to transfer the pretrained language model to new domains. It defines adaptive tokenization as augmenting the existing tokenizer and fixed subword embeddings with new entries from a novel corpus. But it does not create a general tokenizer. Our proposed method, which aims to build a general tokenizer, differs in purpose from the mentioned approaches. Considering the differences in setting—such as ONE-Piece[1] being designed for NMT tasks that use two corresponding languages to generate a tokenizer, and AT[2] focusing on expanding an existing vocabulary to new domains—these methods are not directly comparable to ours. 

We will further discuss the mentioned references in the final version.

Thank you once again for your time and efforts in reviewing our paper and for raising your score.



[1] Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification. (Park et al., NAACL 2021)

[2] Efficient domain adaptation of language models via adaptive tokenization[J]. arXiv preprint arXiv:2109.07460, 2021.
Thank you for the additional experiments.
I increased my score but also significantly decreased my confidence score. I was quite sure that your work was not the first to propose an adaptive tokenizer for this use case. 
Especially in neural machine translation, adaptive tokenization has been studied a lot. But I have to admit that I can't find the papers that I'm thinking of, hence my late reply.
We sincerely appreciate your recognition and thank you very much for raising your score!
Thank you for your recognition and for raising your score! Your support is greatly appreciated.
Thank you for the clarifications. I am raising my score slightly
Thanks for the additional details and experiments. I've raised my score accordingly.
Dear Reviewer GBKH,

Thank you once again for the time you spent reviewing our paper and for your efforts to enhance the quality of our manuscript. We hope that our response can fully address your concerns.

Given that the discussion period concludes on August 13th, we would appreciate any further questions you might have about our paper,  and we are glad to have a discussion with you over the coming days. If all the concerns you have raised have been addressed by our response, would you mind considering re-evaluating our work based on our response?
Dear Reviewers,

Many thanks to all reviewers for your insightful and valuable feedback. We hope we have addressed your concerns. Please let us know if you have any remaining concerns and questions.