Title: A Partition Cover Approach to Tokenization
Venue: NeurIPS 2025
Accepted: yes
Keywords: tokenization

large language models
Abstract: Tokenization is the process of encoding strings into tokens of a fixed vocabulary size, and is widely utilized in Natural Language Processing applications. The leading tokenization algorithm today is Byte Pair Encoding (BPE), which formulates the tokenization problem as a compression problem and tackles it by performing sequences of merges. In this work, we formulate tokenization as an optimization objective, show that it is NP-hard via a simple reduction from vertex cover, and propose a polynomial-time greedy algorithm GreedTok. Our formulation naturally relaxes to the well-studied weighted maximum coverage problem which has a simple $(1 - 1/e)$-approximation algorithm GreedWMC. Through empirical evaluations on real-world corpora, we show that GreedTok outperforms BPE and Unigram on compression and achieves a covering score comparable to GreedWMC. Finally, our extensive pre-training for two transformer-based language models with 1 billion parameters, comparing the choices of BPE and GreedTok as the tokenizer, shows that GreedTok achieves a lower bit per byte even when we control for either the total dataset proportion or total training tokens.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the computational complexity of the tokenization problem and proves that it is NP-hard. The authors establish this result by reducing the original token search problem to a tokenization decision problem, which they show to be NP-hard via a reduction from the Vertex Cover problem. Constrainet optimization-based formulation is also introduced.\n\nTo address the intractability, the paper proposes a new approximation algorithm designed to improve compression quality while enjoying a polynomial time complexity. Experimental results on real-world datasets demonstrate that the proposed method outperforms existing approaches such as BPE and Unigram in terms of compression/perplexity performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper provides an alternative algorithmic viewpoint on the complexity of tokenization.\n- The proposed tokenization scheme outperforms the standard baseline\n\nWeakness:\n- [Major] The writing is confusing (see limitations). \n- [Minor] No approximation guarantee for GreedTok.\n\nQUESTIONS:\nIt would be helpful to clarify whether there is any meaningful distinction between the problem formulation presented in Problem 1 and Definition 2 in [WBP24], especially since this work aims to establish analogous results. A direct comparison or discussion would strengthen the exposition and help the reader appreciate the novelty or alignment of the proposed formulation with prior work.\n\nLine 261 notes that preprocessing is required to extract words from the corpus. Could the authors comment on whether their algorithm can be adapted to work without such preprocessing? This would align more closely with the setting considered by SuperBPE [1], which, to my understanding, also reflects the original BPE algorithm without additional preprocessing steps. Addressing this would help clarify the scope and flexibility of the proposed approach.\n\nLine 237-243, is it fair to say that GreedTok is an extension of BPE but without pairwise merge constraints, i.e., we are allowed to merge any $n$ tokens to create a new one? How optimal is the proposed algorithm compares to other recent tokenization scheme such as [SRZ+24]?\n\nIs there any insights why there is little difference in terms of performance in Figure 2b? For tokens/words $t$ that occurs in both vocabularies, is there any significant different in terms of the empirical next-token probability $P(t | prompt')$?\n\n[1] Liu, Alisa, et al. \"Superbpe: Space travel for language models.\" arXiv preprint arXiv:2503.13423 (2025).\n\nLIMITATIONS:\nThe paper currently lacks clarity in several foundational definitions, making it difficult to follow. For example, in Section 2, the content of the alphabet $\\Sigma$ is not explicitly defined. Additionally, the notation $\\Sigma^+$ is used without explanation—what does it represent, and how does it relate to $\\Sigma$?\n\nIt is also unclear why $\\Sigma^+$ is assumed to contain \\emph{words}, and this assumption is neither justified nor motivated. Further confusion arises from inconsistent usage: $\\Sigma^+$ is described as containing \\emph{words} on line 89 and \\emph{tokens} on line 97. Are \\emph{tokens} and \\emph{words} considered equivalent in this context, or is there a meaningful distinction between them? Clarifying these points would significantly improve the precision and readability of the paper. In fact, I have to jump back and forth between Section 2,3,4 to better understand the notations.\n\nEmpirical/Theoretical runtime complexity should be included in the paper as a table of reference for practitioner.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for tokenization. The authors first show that the tokenization problem can be reduced from the vertex cover problem, showing that it is an NP-hard optimization problem. They then propose a greedy optimization algorithm to address it. Since the relaxed version of the problem corresponds to the weighted maximum coverage problem, greedy algorithms are expected to yield good solutions. Through experiments on multiple datasets, the authors demonstrate that their method achieves higher compression rates than BPE and Unigram. Furthermore, when used for language model pre-training, the proposed method outperforms BPE on various downstream benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- This paper reveals a connection between tokenization and the vertex cover problem and presents a novel tokenization method from a fresh perspective, distinct from existing approaches. The proposed method is original and also appealing in its simplicity.\n\n- The paper is clearly written and includes many illustrative examples, making it easy to follow.\n\n- Experimental results show the advantage of the proposed method over standard tokenization baselines.\n\n- While no theoretical guarantee is provided for the solution quality, the weighted maximum coverage problem is well known to admit good approximations with greedy algorithms. The proposed method empirically achieves strong performance in this regard.\n\n# Weaknesses\n- As a method for selecting an optimal token dictionary, VOLT [1] is known to be effective. VOLT has been reported to outperform SentencePiece and BPE on machine translation tasks. Although I appreciate the simplicity of the proposed method, it would be preferable to include a comparison with VOLT or a discussion of the advantages of the proposed method relative to it.\n\n[1] Xu et al. Vocabulary Learning via Optimal Transport for Neural Machine Translation, ACL-IJCNLP 2021. https://aclanthology.org/2021.acl-long.571.pdf\n\nQUESTIONS:\n- While lazy evaluation strategies are well known to be fast in practice, I think their exact cost is unknown. How do you drive  $\\Theta(|T|\\sum_{W}|W|)$ in line 350?  If you aim to speed up the greedy algorithm, applying the stochastic greedy algorithm [2] may offer further speedups.\n\n- Why is the proposed method not compared against UNIGRAM in the experiments presented in Section 5.2?\n\n\n[2] Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. 2015. Lazier than lazy greedy. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI'15). AAAI Press, 1812–1818.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new optimization-based formulation for the tokenization problem in NLP, generalizing beyond the standard merge-based algorithms like Byte-Pair Encoding (BPE). The authors formulate tokenization in a more general form as a partition cover optimization problem and argue that optimal tokenization is NP-hard providing a clearer intuituve arguments. Then they propose GREEDTOK, a polynomial-time greedy algorithm for token selection, and empirically show it outperforms BPE and Unigram in terms of compression on real-world corpora.\n\nThe authors also try to demonstrate practical downstream impact but that part of the paper is less convincing to me. Based on the reported experiments, GREEDTOK achieves better compression and equal or improved language modeling performance on several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has several good aspectes that clearly speak in its favour:\n\n1. The partition cover formulation is an interesting conceptual advance\n\n2 I found the NP-hardness discussion vetu elegant. The connection to weighted maximum coverage is both theoretically and practically valuable.\n\n3 The proposed GREEDTOK consistently achieves better compression than BPE and Unigram\n\n4 The paper provides detailed pseudocode, experimental protocols, and ablation studies, facilitating reproducibility.\n\nSince tokenization is a foundational problem for large language models improvements here have broad implications for efficiency and model capability.\n\nThere are also some of the issues I have with the presented paper.\n\nFirst and formost, error bars or statistical significance are not provided for downstream tasks, though this is understandable given the resource requirements, in my opinion it significantly undermines the practical importance of the proposed algorithm. This is further enhanced due to the absence of some comparison with various optimised versions of BPE, such as BPE-dropout (https://aclanthology.org/2020.acl-main.170/), PickyBPE (https://aclanthology.org/2024.emnlp-main.925/) or Boundless BPE (https://arxiv.org/abs/2504.00178). On top of that, while compression is shown to improve, the analysis of token quality (e.g., semantic coherence, impact on multilingual or morphologically rich languages) is less extensive than some recent studies.\n\nQUESTIONS:\nCould you provide some futher analysis of token quality?\n\nCould you elaborate on why you only compare the proposed method with two relatively old ones (the inefficiencies and flaws of BPE and Unigram are well documented in the literature, after all)?\n\nLIMITATIONS:\nI generally like the paper, but I am very concered about statistical significance of the proposed results (especially for downstream tasks). I think that authors could have addressed the limitation quality themselves in a more comprehensive manner.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce GreedTok, a polynomial-time greedy tokenization algorithm which draws on theory from the weighted maximum coverage problem. They show GreedTok outperforms byte pair encoding on compression while more closely resembling Unigram-like tokens, and run pretraining experiments that show models using GreedTok outperforms BPE accuracy when configured to use the same number of tokens as BPE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Clear writing and well structured paper\n- Experimental results comparing performance when using GreedTok vs BPE are compelling and interesting\n- The method is practically viable for full-scale modern NLP workloads\n- Appendix contains many important details for replicability as well as further interesting findings and analysis\n\nWeaknesses\n- While I am not an expert in theory and algorithms, the theoretical contribution appears sound to me and is backed up by empirical experiments\n- The algorithm is more computationally complex than BPE, although this is not too bad given it amounts to a one-time cost upfront that does not affect future model training runs\n\nQUESTIONS:\n- There is some discussion of GreedTok having a more closely aligned tokenization to Unigram. Unigram is left out of the Table 4 evaluation on accuracy across popular benchmarks, presumably for computational reasons. However, it would be interesting to see how Unigram compares, even if on a smaller scale, to both BPE and GreedTok. This may further validate your hypothesis that performance gains are because GreedTok “may inherit some of UNIGRAM’s favorable token characteristics”\n- Any evidence of how this tokenization would do in terms of robustness to adversarial attacks? BPE is a strong defense against text perturbation attacks which attempt to produce out-of-vocabulary tokens. Presumably GreedTok achieves this as well due to its high compression rate, but it is never discussed explicitly.\n\nLIMITATIONS:\n- Limited to 1B parameter models, though due to computational costs this is reasonable\n- No exploration of low-resource languages", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}