open-index/open-npm huggingface dataset:
npm package: @just-moh-it/tiktoken-node
Description: We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thr
Latest version: 0.1.31

# @just-moh-it/tiktoken-node

We use our own fork for now because we are making changes that may not be useful to everyone. For example, we add support for special tokens, and we also add support for running tokenization asynchronously with the computation happening on a different thread.