tokenizers (cargo): Provides an implementation of today's most used tokenizers, with a focus on performances and versatility. 
Keywords: tokenizer|nlp|wordpiece|huggingface|bpe|bert|gpt|language-model|natural-language-processing|natural-language-understanding|transformers