GitHub repository: xai-org-shared/tiktoken-rs
Primary language: Rust
Description: Ready-made tokenizer library for working with GPT and tiktoken