GitHub repository: kojix2/tiktoken-rs
Primary language: Rust
Description: Ready-made tokenizer library for working with GPT and tiktoken