Documentation improvement on tiktoken integration
Author: viktor-shcherb
Pull request: no
Labels: name: Core: Tokenization

I want to load tokenizers from tiktoken to use for model training. Right now, this page https://huggingface.co/docs/transformers/main/en/tiktoken is only info I could find on tiktoken integration.

Following the instructions on the page, I just need to add a `tokenizer.model` to model directory. In light of this, I have a few questions:
1. How do I generate this file from tiktoken's Encoding class?
2. What happens when I save the model? Does it get converted to HF format or is it saved as tiktoken tokenizer?

Would appreciate any help on this! 

Comments:
Through source code inspection, I see that these functions are used (from `tiktoken.load`):
```python
def dump_tiktoken_bpe(bpe_ranks: dict[bytes, int], tiktoken_bpe_file: str) -> None:
    try:
        import blobfile
    except ImportError as e:
        raise ImportError(
            "blobfile is not installed. Please install it by running `pip install blobfile`."
        ) from e
    with blobfile.BlobFile(tiktoken_bpe_file, "wb") as f:
        for token, rank in sorted(bpe_ranks.items(), key=lambda x: x[1]):
            f.write(base64.b64encode(token) + b" " + str(rank).encode() + b"\n")


def load_tiktoken_bpe(tiktoken_bpe_file: str, expected_hash: str | None = None) -> dict[bytes, int]:
    # NB: do not add caching to this function
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
    return {
        base64.b64decode(token): int(rank)
        for token, rank in (line.split() for line in contents.splitlines() if line)
    }

```

So I am guessing that `tokenizer.model` file should be generated with 
```python
dump_tiktoken_bpe(encoding._mergeable_ranks, 'tokenizer.model')
```

But I am still figuring out how to provide the special tokens and pattern string, since they are not stored in this bpe file.

So here is what I am going with:
```python
from pathlib import Path

import tiktoken
from tiktoken.load import dump_tiktoken_bpe
from transformers.convert_slow_tokenizer import TikTokenConverter

from transformers.tokenization_utils_fast import TIKTOKEN_VOCAB_FILE


def preload_tiktoken(tokenizer_name: str, output_dir: str | Path):
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)

    save_file = output_dir / 'tiktoken' / TIKTOKEN_VOCAB_FILE

    encoding = tiktoken.get_encoding(tokenizer_name)
    dump_tiktoken_bpe(encoding._mergeable_ranks, save_file)

    TikTokenConverter(
        vocab_file=save_file, 
        pattern=encoding._pat_str, 
        additional_special_tokens=encoding._special_tokens
    ).tokenizer().save(output_dir)
```

As far as I know, there is no way to change the pattern string without directly using the `TiktokenConverter`. If someone can verify if this is the intended approach, please let me know!

UPD: Well, now I have `tokenizer.json` file and have no idea what to do with it..

Thanks for the issue @ViktorooReps! cc @ArthurZucker 

Hey! Could you open a PR for this? 🤗 
The `tokenizer.json` file can now be used with `PreTrainedTokenizerFast.from_pretrained("path_to_folder")` which is the `transformers` format!

We can also add your function as a util to convert from an encoding / directly from tiktoken encoding name!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.