GitHub repository: npatsakula/rust-tokenizers
Primary language: Rust
Description: Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models