GitHub repository: pytorch-labs/tokenizers
Primary language: C++
Description: C++ implementations for various tokenizers (sentencepiece, tiktoken etc).