GitHub repository: jungikim/Tokenizer
Primary language: C++
Description: Fast and customizable text tokenization library with BPE and SentencePiece support