GitHub repository: zurawiki/tiktoken-rs
Primary language: Rust
Created: 2023-02-02T17:20:47Z
Description: Ready-made tokenizer library for working with GPT and tiktoken