GitHub repository: toeverything/tiktoken-rs
Primary language: Rust
Description: Ready-made tokenizer library for working with GPT and tiktoken