Package: bytepiece
[中文|English]BytePieceBytePiece is a Byte-based Unigram tokenizer, implemented purely in Python, making it more readable and expandable. Due to the use of a new training algorithm, its compression rate is often higher than existing Tokenizers, and it also supports multiprocessing acceleration for training. Moreover, it directly operates on the UTF-8 Bytes of the text, with almost no preprocessing, making it more pure and language-independent.Blog:https://kexue.fm/archives/9752https://kexue.fm/archives/9768CharacteristicsAn ideal Tokenizer and its training algorithm should have the following characteristics:Lossless reconstructionHigh compression rateLanguage-independentData-drivenTraining-friendlyThe mainstreamSentencePiecebasically has the above characteristics, but there are still some problems. For example, it supports both BPE and Unigram algorithms. BPE often has a higher compression rate, but the training is extremely slow and consumes a lot of memory. Moreover, it does conduct some language-related preprocessing on the text, so it's not purely "language-independent". Besides, it's written in C++, which is a black box for most users, thus not conducive to research and modification.BytePiece has conceived a new training method based onByte-based N-gram Language Model (BNLM), which can obtain a higher compression rate vocabulary table, support multiprocessing training, and significantly accelerate compared to SentencePiece's BPE training under the same corpus. The code is pure Python, easy for everyone to read and modify. In addition, BytePiece also provides a more efficient random segmentation algorithm thanSubword Regularization.PrincipleBytePiece is not simply a rewrite of the existing Unigram model based on Byte-based and multiprocessor, but a new training method designed for Unigram, which is one of the key reasons for its higher compression rate.The new training method is based on the new word discovery algorithm of the N-gram language model, first proposed in the author's blog 7 years ago《【Chinese Word Segmentation Series】 5. Unsupervised Word Segmentation Based on Language Model》. Please visit the blog for details.For the new random segmentation algorithm, you can refer to"A Brief Exploration of Random Segmentation: From Viterbi Decoding to Viterbi Sampling"and"Further Exploration of Random Segmentation: From Viterbi Sampling to Perfect Sampling Algorithm".InstallationBytePiece can only run on Python3 and usespyahocorasickto accelerate the training process. Since BytePiece is Byte-based, and the pyahocorasick on PyPi is Unicode-based, it cannot be used directly. Please follow the instructions below to install the Byte-based version of pyahocorasick:# If already installed, please uninstall firstpipuninstallpyahocorasick# Compile and install directly from git, note to pass the environment variable AHOCORASICK_BYTESAHOCORASICK_BYTES=1pipinstallgit+https://github.com/WojciechMula/pyahocorasick.gitThen install Cython:pipinstallCythonAfter that, you can install BytePiece via pip:pipinstallbytepiece==0.6.3UsageAll source code of BytePiece is actually in a single file, includingTrainerandTokenizertwo classes, corresponding to training and tokenization respectively.TrainingTo train Tokenizer, you just need to import theTrainerclass:frombytepieceimportTrainerThen prepare the training corpus. BytePiece supports not reading all corpora into memory at once, but since BytePiece training needs to go through the data twice, it does not support Generator input, but needs to be written in the form of Iterator, for example:importjsonclasscorpus:def__iter__(self):f='data_sample.json'withopen(f)asf:forlinf:yieldjson.loads(l)['text']# Return a Unicode each timeThen you can start the actual training:trainer=Trainer(order=6,max_vocab_size=100000,min_count=32)trainer.train(corpus(),workers=64,batch_size=1000)trainer.save('bytepiece.model')Here,orderis the order of the n-gram language model, it is recommended to keep the defaultorder=6;max_vocab_sizeis the maximum size of the vocabulary, note that due to redundancy removal, the final vocabulary may not precisely equal max_vocab_size, it might be slightly less;min_countis the minimum occurrence frequency of tokens, when the data volume is large, it can be appropriately increased, it generally doesn't significantly affect the training results;workersis the number of parallel training processes, which can utilize all cores of the machine;batch_sizeis the batch size, it won't affect the training results, it usually doesn't need to be changed, if you find the CPU utilization is not full, you can appropriately increase it.In addition, starting from version0.4.1, a new parameterisolate_digitsis added, which defaults toFalse. When set toTrue, it ensures that all Arabic numbers are split into individual characters:trainer=Trainer(order=6,max_vocab_size=100000,min_count=32,isolate_digits=True)Starting from version0.6.0, a new parameterensure_unicodeis added, which can ensure that all multi-byte tokens can be restored to unicode. Since the current results show that enablingensure_unicodeoften results in a higher compression rate for the trained model, it is set toTrueby default. When set toFalse(equivalent to versions before 0.6.0), multi-byte tokens may needdecode(errors='ignore')to be restored to unicode:trainer=Trainer(order=6,max_vocab_size=100000,min_count=32,ensure_unicode=True)TokenizationAfter the training is completed, refer to the following usage:frombytepieceimportTokenizertokenizer=Tokenizer('bytepiece.model')text='Today'sweatherisgreat'tokens=tokenizer.tokenize(text)# Returns a list of bytesprint(b' '.join(tokens).decode(errors='ignore'))# Visualize the tokenization resultids=tokenizer.encode(text)# Returns the ids corresponding to tokensprint(tokenizer.decode(ids))# Decode the ids back to unicode textids=tokenizer.encode(text,iter=True)# Returns the generator of idstokens=tokenizer.tokenize(text,alpha=0.2)# Random Tokenizationprint(b' '.join(tokens).decode(errors='ignore'))# Visualize the tokenization resultComparisonComparison with small data volume:Training Time↓Maximum Memory Usage↓Compression Rate↑Tokenization Speed↑SP-BPE55.3 minutes5.2GB4.805.47SP-Unigram1.6 minutes2.5GB4.737.84BytePiece6.5 minutes4.3GB5.052.50Comparison with large data volume:Training Time↓Maximum Memory Usage↓Compression Rate (Homologous)↑Compression Rate (Heterologous)↑Tokenization Speed↑SP-BPE19.21 hours97GB4.524.461.27SP-Unigram2.02 hours384GB4.514.485.55BytePiece2.24 hours51GB5.394.511.92The unit of compression rate is "bytes/token", i.e., the average number of bytes per token; the unit of speed is "M bytes/second", i.e., the average number of bytes that can be segmented per second (in millions). For other details, please refer tohere.In the first table, the dataset has a shorter average length, BytePiece is slower than both SP-BPE and SP-Unigram; in the second table, the average length of the corpus is generally longer, resulting in BytePiece being faster than SP-BPE. This indicates that BPE's tokenization speed is significantly affected by length, and also indicates that BytePiece's tokenization speed, accelerated by Cython, can be compared with SentencePiece in terms of speed.DownloadTo download the open-source BytePiece model, please go tomodels.Citation@misc{bytepiece2023,
  title={BytePiece: A more pure and effective tokenizer},
  author={Jianlin Su},
  year={2023},
  howpublished={\url{https://github.com/bojone/bytepiece}},
}CommunicationQQ Group: 67729435, for WeChat group please add robot spaces_ac_cn