Package: tiktoken
⏳ tiktokentiktoken is a fastBPEtokeniser for use with
OpenAI's models.importtiktokenenc=tiktoken.get_encoding("cl100k_base")assertenc.decode(enc.encode("hello world"))=="hello world"# To get the tokeniser corresponding to a specific model in the OpenAI API:enc=tiktoken.encoding_for_model("gpt-4")The open source version oftiktokencan be installed from PyPI:pip install tiktokenThe tokeniser API is documented intiktoken/core.py.Example code usingtiktokencan be found in theOpenAI Cookbook.Performancetiktokenis between 3-6x faster than a comparable open source tokeniser:Performance measured on 1GB of text using the GPT-2 tokeniser, usingGPT2TokenizerFastfromtokenizers==0.13.2,transformers==4.24.0andtiktoken==0.2.0.Getting helpPlease post questions in theissue tracker.If you work at OpenAI, make sure to check the internal documentation or feel free to contact
@shantanu.What is BPE anyway?Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
properties:It's reversible and lossless, so you can convert tokens back into the original textIt works on arbitrary text, even text that is not in the tokeniser's training dataIt compresses the text: the token sequence is shorter than the bytes corresponding to the
original text. On average, in practice, each token corresponds to about 4 bytes.It attempts to let the model see common subwords. For instance, "ing" is a common subword in
English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing"
(instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and
again in different contexts, it helps models generalise and better understand grammar.tiktokencontains an educational submodule that is friendlier if you want to learn more about
the details of BPE, including code that helps visualise the BPE procedure:fromtiktoken._educationalimport*# Train a BPE tokeniser on a small amount of textenc=train_simple_encoding()# Visualise how the GPT-4 encoder encodes textenc=SimpleBytePairEncoding.from_tiktoken("cl100k_base")enc.encode("hello world aaaaaaaaaaaa")Extending tiktokenYou may wish to extendtiktokento support new encodings. There are two ways to do this.Create yourEncodingobject exactly the way you want and simply pass it around.cl100k_base=tiktoken.get_encoding("cl100k_base")# In production, load the arguments directly instead of accessing private attributes# See openai_public.py for examples of arguments for specific encodingsenc=tiktoken.Encoding(# If you're changing the set of special tokens, make sure to use a different name# It should be clear from the name what behaviour to expect.name="cl100k_im",pat_str=cl100k_base._pat_str,mergeable_ranks=cl100k_base._mergeable_ranks,special_tokens={**cl100k_base._special_tokens,"
":100264,"
":100265,})Use thetiktoken_extplugin mechanism to register yourEncodingobjects withtiktoken.This is only useful if you needtiktoken.get_encodingto find your encoding, otherwise prefer
option 1.To do this, you'll need to create a namespace package undertiktoken_ext.Layout your project like this, making sure to omit thetiktoken_ext/__init__.pyfile:my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.pymy_encodings.pyshould be a module that contains a variable namedENCODING_CONSTRUCTORS.
This is a dictionary from an encoding name to a function that takes no arguments and returns
arguments that can be passed totiktoken.Encodingto construct that encoding. For an example, seetiktoken_ext/openai_public.py. For precise details, seetiktoken/registry.py.Yoursetup.pyshould look something like this:fromsetuptoolsimportsetup,find_namespace_packagessetup(name="my_tiktoken_extension",packages=find_namespace_packages(include=['tiktoken_ext*']),install_requires=["tiktoken"],...)Then simplypip install ./my_tiktoken_extensionand you should be able to use your
custom encodings! Make surenotto use an editable install.