pyonmttok (pypi): Fast and customizable text tokenization library with BPE and SentencePiece support
Keywords: tokenization|opennmt|unicode|bpe|sentencepiece|subword|cpp|icu|machine-translation|natural-language-processing|python|tokenizer