openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Category: nlp
Language: Python
Topics: chatgpt, tokeniser, word-segmentation