Package: tokenizers
TokenizersProvides an implementation of today's most used tokenizers, with a focus on performance and
versatility.Bindings over theRustimplementation.
If you are interested in the High-level design, you can go check it there.Otherwise, let's dive in!Main features:Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3
most common BPE versions).Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes
less than 20 seconds to tokenize a GB of text on a server's CPU.Easy to use, but also extremely versatile.Designed for research and production.Normalization comes with alignments tracking. It's always possible to get the part of the
original sentence that corresponds to a given token.Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.InstallationWith pip:pipinstalltokenizersFrom sources:To use this method, you need to have the Rust installed:# Install with:curlhttps://sh.rustup.rs-sSf|sh-s---yexportPATH="$HOME/.cargo/bin:$PATH"Once Rust is installed, you can compile doing the followinggitclonehttps://github.com/huggingface/tokenizerscdtokenizers/bindings/python# Create a virtual env (you can use yours as well)python-mvenv.envsource.env/bin/activate# Install `tokenizers` in the current virtual envpipinstall-e.Load a pretrained tokenizer from the HubfromtokenizersimportTokenizertokenizer=Tokenizer.from_pretrained("bert-base-cased")Using the provided TokenizersWe provide some pre-build tokenizers to cover the most common cases. You can easily load one of
these using somevocab.jsonandmerges.txtfiles:fromtokenizersimportCharBPETokenizer# Initialize a tokenizervocab="./path/to/vocab.json"merges="./path/to/merges.txt"tokenizer=CharBPETokenizer(vocab,merges)# And then encode:encoded=tokenizer.encode("I can feel the magic, can you?")print(encoded.ids)print(encoded.tokens)And you can train them just as simply:fromtokenizersimportCharBPETokenizer# Initialize a tokenizertokenizer=CharBPETokenizer()# Then train it!tokenizer.train(["./path/to/files/1.txt","./path/to/files/2.txt"])# Now, let's use it:encoded=tokenizer.encode("I can feel the magic, can you?")# And finally save it somewheretokenizer.save("./path/to/directory/my-bpe.tokenizer.json")Provided TokenizersCharBPETokenizer: The original BPEByteLevelBPETokenizer: The byte level version of the BPESentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePieceBertWordPieceTokenizer: The famous Bert tokenizer, using WordPieceAll of these can be used and trained as explained above!Build your ownWhenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer,
by putting all the different parts you need together.
You can check how we implemented theprovided tokenizersand adapt them easily to your own needs.Building a byte-level BPEHere is an example showing how to build your own byte-level BPE by putting all the different pieces
together, and then saving it to a single file:fromtokenizersimportTokenizer,models,pre_tokenizers,decoders,trainers,processors# Initialize a tokenizertokenizer=Tokenizer(models.BPE())# Customize pre-tokenization and decodingtokenizer.pre_tokenizer=pre_tokenizers.ByteLevel(add_prefix_space=True)tokenizer.decoder=decoders.ByteLevel()tokenizer.post_processor=processors.ByteLevel(trim_offsets=True)# And then traintrainer=trainers.BpeTrainer(vocab_size=20000,min_frequency=2,initial_alphabet=pre_tokenizers.ByteLevel.alphabet())tokenizer.train(["./path/to/dataset/1.txt","./path/to/dataset/2.txt","./path/to/dataset/3.txt"],trainer=trainer)# And Save ittokenizer.save("byte-level-bpe.tokenizer.json",pretty=True)Now, when you want to use this tokenizer, this is as simple as:fromtokenizersimportTokenizertokenizer=Tokenizer.from_file("byte-level-bpe.tokenizer.json")encoded=tokenizer.encode("I can feel the magic, can you?")