Error in tiktoken integration example
Author: kndtran
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-5.14.0-427.42.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.16
- Huggingface_hub version: 0.29.1
- Safetensors version: 0.5.3
- Accelerate version: 1.4.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No


- tiktoken==0.9.0

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

On the [tiktoken integration page](https://huggingface.co/docs/transformers/en/tiktoken), the `Create tiktoken tokenizer` section raises an error within the `convert_tiktoken_to_fast` function.

Full example:
```
from pathlib import Path

from transformers.integrations.tiktoken import convert_tiktoken_to_fast
from tiktoken import encoding_name_for_model

encoding = encoding_name_for_model("o1")
outdir = Path("tmp") / "tiktoken" / encoding
outdir.mkdir(parents=True, exist_ok=True)

convert_tiktoken_to_fast(encoding, outdir)
```

Raises this error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[24], line 10
      7 outdir = Path("tmp") / "tiktoken" / encoding
      8 outdir.mkdir(parents=True, exist_ok=True)
---> 10 convert_tiktoken_to_fast(encoding, outdir)

File ~/miniforge3/envs/tok/lib/python3.10/site-packages/transformers/integrations/tiktoken.py:42, in convert_tiktoken_to_fast(encoding, output_dir)
     37 except ImportError:
     38     raise ValueError("`tiktoken` is required to save a `tiktoken` file. Install it with `pip install tiktoken`.")
     40 tokenizer = TikTokenConverter(
     41     vocab_file=save_file_absolute, pattern=encoding._pat_str, additional_special_tokens=encoding._special_tokens
---> 42 ).converted()
     43 tokenizer.save(output_file_absolute)

File ~/miniforge3/envs/tok/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1630, in TikTokenConverter.converted(self)
   1623 tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
   1624     [
   1625         pre_tokenizers.Split(Regex(self.pattern), behavior="isolated", invert=False),
   1626         pre_tokenizers.ByteLevel(add_prefix_space=self.add_prefix_space, use_regex=False),
   1627     ]
   1628 )
   1629 tokenizer.decoder = decoders.ByteLevel()
-> 1630 tokenizer.add_special_tokens(self.additional_special_tokens)
   1632 tokenizer.post_processor = processors.ByteLevel(trim_offsets=False)
   1634 return tokenizer

TypeError: argument 'tokens': 'dict' object cannot be converted to 'PyList'
```

### Expected behavior

No error. A `tokenizer.json` is written to the directory and can be loaded into a `PreTrainedTokenizerFast` object as in the example.

Comments:
+1

I can confirm this bug in transformers 4.49.0 with tiktoken 0.8.0

gentle ping @arthurzucker @itazap 

I had the same issue, but checking the latest source code, this issue is fixed. Please upgrade to transformers 4.50.3. 

closing then! 🤗