github.com/peterheb/gotoken (go): Package gotoken provides an OpenAI-compatible tokenization library similar to tiktoken. Its primary export is the Tokenizer interface, featuring Encode and Decode methods for converting strings to/from []int.  Tokenizer encodings, such as r50kbase or cl100kbase, are available in separate packages that implement the Tokenizer interface. This design mirrors the image/png and image/jpeg packages' integration with the standard image library. Encoding packages self-register with gotoken.  Encoding packages include built-in token dictionaries, which removes the need for external downloads or local file caches. However, these packages are relatively large (a few MB) and should only be imported when needed. At least one encoding package must be imported for gotoken to be able to tokenize text.  Example of importing gotoken and a tokenizer encoding:  The _ indicates that cl100kbase should be imported even without a direct reference in your code. Encoding packages have no public functions or types, but they do contain public constants defining special tokens.
Keywords: chatgpt|go|openai|tiktoken