Convert llama-2 to hf error
Author: xuanzhangyang
Pull request: no
Labels: name: Core: Tokenization

name: Fast Tokenizers

name: bug

### System Info

transformer version:   4.46.0  - install from master
pytorch version:  1.9
python version: 3.8

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

(pytorch1.9) xxx@xxx-G5500:~/code/pytorch/llama$ ls llama-2-7b

checklist.chk consolidated.00.pth params.json tokenizer_checklist.chk tokenizer.model
(pytorch1.9) xxx@xxx-G5500:~/code/pytorch/llama$ python ../hunningface/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir ./llama-2-7b --model_size 7B --output_dir ./llama-2-7b-hf

### Expected behavior

Error mesage:  
`You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Traceback (most recent call last):
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/convert_slow_tokenizer.py", line 1591, in convert_slow_tokenizer
    return TikTokenConverter(
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/convert_slow_tokenizer.py", line 1491, in converted
    tokenizer = self.tokenizer()
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/convert_slow_tokenizer.py", line 1483, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/convert_slow_tokenizer.py", line 1458, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
  File "/home/xuan/anaconda3/envs/pytorch1.9/lib/python3.8/site-packages/tiktoken/load.py", line 148, in load_tiktoken_bpe
    return {
  File "/home/xuan/anaconda3/envs/pytorch1.9/lib/python3.8/site-packages/tiktoken/load.py", line 150, in <dictcomp>
    for token, rank in (line.split() for line in contents.splitlines() if line)
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../hunningface/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 479, in <module>
    main()
  File "../hunningface/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 457, in main
    write_tokenizer(
  File "../hunningface/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 367, in write_tokenizer
    tokenizer = tokenizer_class(input_tokenizer_path)
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/models/llama/tokenization_llama_fast.py", line 157, in __init__
    super().__init__(
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/tokenization_utils_fast.py", line 138, in __init__
    fast_tokenizer = convert_slow_tokenizer(self, from_tiktoken=True)
  File "/home/xuan/code/pytorch/hunningface/transformers/src/transformers/convert_slow_tokenizer.py", line 1596, in convert_slow_tokenizer
    raise ValueError(
ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']
`

Comments:
cc @itazap if you have the bandwidth :)

hey @xuanzhangyang - I'm unable to reproduce the error, is your `llama-2-7b` a git clone of `https://huggingface.co/meta-llama/Llama-2-7b` ?

> hey @xuanzhangyang - I'm unable to reproduce the error, is your `llama-2-7b` a git clone of `https://huggingface.co/meta-llama/Llama-2-7b` ?

I use https://github.com/meta-llama/llama.git

This is not llama2 specific but probably related to the tokenizers 0.20.0 breaking change.

See minimal reproduction with the official deberta as well.

```
$ poetry show | grep -E 'transformer|tokenizer|torch'
tokenizers                    0.20.0          
torch                         2.4.1           Tensors and Dynamic neural ne...
transformers                  4.45.1          State-of-the-art Machine Lear...
```

```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained('microsoft/deberta-v3-base')
Traceback (most recent call last):
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1592, in convert_slow_tokenizer
    ).converted()
      ^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1489, in converted
    tokenizer = self.tokenizer()
                ^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1482, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1458, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/tiktoken/load.py", line 147, in load_tiktoken_bpe
    contents = read_file_cached(tiktoken_bpe_file, expected_hash)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/tiktoken/load.py", line 49, in read_file_cached
    cache_key = hashlib.sha1(blobpath.encode()).hexdigest()
                             ^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'encode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 926, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2216, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2450, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/models/deberta_v2/tokenization_deberta_v2_fast.py", line 103, in __init__
    super().__init__(
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 138, in __init__
    fast_tokenizer = convert_slow_tokenizer(self, from_tiktoken=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mjbommar/.cache/pypoetry/virtualenvs/kl3m-embedding-research-Or0d6wkg-py3.12/lib/python3.12/site-packages/transformers/convert_slow_tokenizer.py", line 1594, in convert_slow_tokenizer
    raise ValueError(
ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']
```

The `transformers` package is broken for now. It is not possible to run models from Hugging Face `transformers` at the moment.

I have the same issue! 

@mjbommar I can't reproduce

The warning is missleading, but you need `sentencepiece` installed if the tokenizer you are trying to load does not have a `tokenizer.json` (FYI @itazap )

how to solve the problem?

You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Traceback (most recent call last):
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1592, in convert_slow_tokenizer
    ).converted()
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1489, in converted
    tokenizer = self.tokenizer()
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1482, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1458, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/tiktoken/load.py", line 145, in load_tiktoken_bpe
    return {
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/tiktoken/load.py", line 147, in <dictcomp>
    for token, rank in (line.split() for line in contents.splitlines() if line)
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/li/reweight/convert.py", line 509, in <module>
    main()
  File "/home/li/reweight/convert.py", line 487, in main
    write_tokenizer(
  File "/home/li/reweight/convert.py", line 393, in write_tokenizer
    tokenizer = tokenizer_class(input_tokenizer_path)
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 157, in __init__
    super().__init__(
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 138, in __init__
    fast_tokenizer = convert_slow_tokenizer(self, from_tiktoken=True)
  File "/home/li/reweight/miniconda3/envs/eval/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py", line 1594, in convert_slow_tokenizer
    raise ValueError(
ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']

Getting the same error above. Running on Ubuntu 20.04.6 LTS

Here's my output from pip freeze 

```
aiohappyeyeballs==2.4.3
aiohttp==3.10.9
aiosignal==1.3.1
anyio==4.6.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
async-timeout==4.0.3
attrs==24.2.0
babel==2.16.0
beautifulsoup4==4.12.3
bleach==6.1.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.3.2
comm==0.2.2
datasets==3.0.1
debugpy==1.8.6
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.8
exceptiongroup==1.2.2
executing==2.1.0
fastjsonschema==2.20.0
filelock==3.16.1
fqdn==1.5.1
frozenlist==1.4.1
fsspec==2024.6.1
h11==0.14.0
httpcore==1.0.6
httpx==0.27.2
huggingface-hub==0.25.1
idna==3.10
ipykernel==6.29.5
ipython==8.28.0
ipywidgets==8.1.5
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.3
jupyter_core==5.7.2
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterlab==4.2.5
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
llama-jarvis==0.1.0
MarkupSafe==2.1.5
matplotlib-inline==0.1.7
mistune==3.0.2
mpmath==1.3.0
multidict==6.1.0
multiprocess==0.70.16
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.3
notebook==7.2.2
notebook_shim==0.2.4
numpy==2.1.2
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.77
nvidia-nvtx-cu12==12.1.105
overrides==7.7.0
packaging==24.1
pandas==2.2.3
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
platformdirs==4.3.6
prometheus_client==0.21.0
prompt_toolkit==3.0.48
protobuf==5.28.2
psutil==6.0.0
ptyprocess==0.7.0
pure_eval==0.2.3
pyarrow==17.0.0
pycparser==2.22
Pygments==2.18.0
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytz==2024.2
PyYAML==6.0.2
pyzmq==26.2.0
referencing==0.35.1
regex==2024.9.11
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.20.0
safetensors==0.4.5
Send2Trash==1.8.3
sentencepiece==0.2.0
six==1.16.0
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
sympy==1.13.3
terminado==0.18.1
tiktoken==0.8.0
tinycss2==1.3.0
tokenizers==0.20.0
tomli==2.0.2
torch==2.4.1
tornado==6.4.1
tqdm==4.66.5
traitlets==5.14.3
transformers==4.45.1
triton==3.0.0
types-python-dateutil==2.9.0.20241003
typing_extensions==4.12.2
tzdata==2024.2
uri-template==1.3.0
urllib3==2.2.3
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
widgetsnbextension==4.0.13
xxhash==3.5.0
yarl==1.13.1
```

Here's the error 

```
ValueError                                Traceback (most recent call last)
File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1592, in convert_slow_tokenizer(transformer_tokenizer, from_tiktoken)
   1588     logger.info("Converting from Tiktoken")
   1589     return TikTokenConverter(
   1590         vocab_file=transformer_tokenizer.vocab_file,
   1591         additional_special_tokens=transformer_tokenizer.additional_special_tokens,
-> 1592     ).converted()
   1593 except Exception:

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1489, in TikTokenConverter.converted(self)
   1488 def converted(self) -> Tokenizer:
-> 1489     tokenizer = self.tokenizer()
   1490     tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
   1491         [
   1492             pre_tokenizers.Split(Regex(self.pattern), behavior="isolated", invert=False),
   1493             pre_tokenizers.ByteLevel(add_prefix_space=self.add_prefix_space, use_regex=False),
   1494         ]
   1495     )

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1482, in TikTokenConverter.tokenizer(self)
   1481 def tokenizer(self):
-> 1482     vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
   1483     tokenizer = Tokenizer(BPE(vocab_scores, merges, fuse_unk=False))

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1458, in TikTokenConverter.extract_vocab_merges_from_model(self, tiktoken_url)
   1454     raise ValueError(
   1455         "`tiktoken` is required to read a `tiktoken` file. Install it with " "`pip install tiktoken`."
   1456     )
-> 1458 bpe_ranks = load_tiktoken_bpe(tiktoken_url)
   1459 byte_encoder = bytes_to_unicode()

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/tiktoken/load.py:145, in load_tiktoken_bpe(tiktoken_bpe_file, expected_hash)
    144 contents = read_file_cached(tiktoken_bpe_file, expected_hash)
--> 145 return {
    146     base64.b64decode(token): int(rank)
    147     for token, rank in (line.split() for line in contents.splitlines() if line)
    148 }

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/tiktoken/load.py:147, in <dictcomp>(.0)
    144 contents = read_file_cached(tiktoken_bpe_file, expected_hash)
    145 return {
    146     base64.b64decode(token): int(rank)
--> 147     for token, rank in (line.split() for line in contents.splitlines() if line)
    148 }

ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[25], line 6
      1 jarvis_config = JarvisConfig(
      2     BASE_LLM,
      3     SEAMLESS_MODEL
      4 )
      5 jarvis_model = JarvisModel(jarvis_config)
----> 6 jarvis_processor = JarvisProcessor(
      7     BASE_LLM,
      8     SEAMLESS_MODEL
      9 )

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/llama_jarvis/model.py:478, in JarvisProcessor.__init__(self, tokenizer, seamless_processor)
    475     tokenizer = AutoTokenizer.from_pretrained(tokenizer)
    477 if isinstance(seamless_processor, str):
--> 478     seamless_processor = SeamlessM4TProcessor.from_pretrained(
    479         seamless_processor
    480     )
    482 self.seamless_processor = seamless_processor
    483 self.tokenizer = prepare_tokenizer(tokenizer)

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/processing_utils.py:915, in ProcessorMixin.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, **kwargs)
    912 if token is not None:
    913     kwargs["token"] = token
--> 915 args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
    916 processor_dict, kwargs = cls.get_processor_dict(pretrained_model_name_or_path, **kwargs)
    918 return cls.from_args_and_dict(args, processor_dict, **kwargs)

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/processing_utils.py:961, in ProcessorMixin._get_arguments_from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    958     else:
    959         attribute_class = getattr(transformers_module, class_name)
--> 961     args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
    962 return args

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2216, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)
   2213     else:
   2214         logger.info(f"loading file {file_path} from cache at {resolved_vocab_files[file_id]}")
-> 2216 return cls._from_pretrained(
   2217     resolved_vocab_files,
   2218     pretrained_model_name_or_path,
   2219     init_configuration,
   2220     *init_inputs,
   2221     token=token,
   2222     cache_dir=cache_dir,
   2223     local_files_only=local_files_only,
   2224     _commit_hash=commit_hash,
   2225     _is_local=is_local,
   2226     trust_remote_code=trust_remote_code,
   2227     **kwargs,
   2228 )

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/models/seamless_m4t/tokenization_seamless_m4t_fast.py:357, in SeamlessM4TTokenizerFast._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, *init_inputs, **kwargs)
    343 @classmethod
    344 def _from_pretrained(
    345     cls,
   (...)
    355     **kwargs,
    356 ):
--> 357     tokenizer = super()._from_pretrained(
    358         resolved_vocab_files,
    359         pretrained_model_name_or_path,
    360         init_configuration,
    361         *init_inputs,
    362         token=token,
    363         cache_dir=cache_dir,
    364         local_files_only=local_files_only,
    365         _commit_hash=_commit_hash,
    366         _is_local=_is_local,
    367         **kwargs,
    368     )
    370     # ensure also set after from pretrained
    371     tokenizer.set_src_lang_special_tokens(tokenizer._src_lang)

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2450, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, trust_remote_code, *init_inputs, **kwargs)
   2448 # Instantiate the tokenizer.
   2449 try:
-> 2450     tokenizer = cls(*init_inputs, **init_kwargs)
   2451 except import_protobuf_decode_error():
   2452     logger.info(
   2453         "Unable to load tokenizer model from SPM, loading from TikToken will be attempted instead."
   2454         "(Google protobuf error: Tried to load SPM model with non-SPM vocab file).",
   2455     )

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/models/seamless_m4t/tokenization_seamless_m4t_fast.py:133, in SeamlessM4TTokenizerFast.__init__(self, vocab_file, tokenizer_file, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, src_lang, tgt_lang, additional_special_tokens, **kwargs)
    118 def __init__(
    119     self,
    120     vocab_file=None,
   (...)
    131     **kwargs,
    132 ):
--> 133     super().__init__(
    134         vocab_file=vocab_file,
    135         tokenizer_file=tokenizer_file,
    136         bos_token=bos_token,
    137         eos_token=eos_token,
    138         sep_token=sep_token,
    139         cls_token=cls_token,
    140         unk_token=unk_token,
    141         pad_token=pad_token,
    142         src_lang=src_lang,
    143         tgt_lang=tgt_lang,
    144         additional_special_tokens=additional_special_tokens,
    145         **kwargs,
    146     )
    148     self.vocab_file = vocab_file
    149     self._src_lang = f"__{src_lang}__" if "__" not in src_lang else src_lang

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py:138, in PreTrainedTokenizerFast.__init__(self, *args, **kwargs)
    136     self.vocab_file = kwargs.get("vocab_file", None)
    137     self.additional_special_tokens = kwargs.get("additional_special_tokens", [])
--> 138     fast_tokenizer = convert_slow_tokenizer(self, from_tiktoken=True)
    139     slow_tokenizer = None
    140 else:

File ~/anaconda3/envs/jarvis/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:1594, in convert_slow_tokenizer(transformer_tokenizer, from_tiktoken)
   1589     return TikTokenConverter(
   1590         vocab_file=transformer_tokenizer.vocab_file,
   1591         additional_special_tokens=transformer_tokenizer.additional_special_tokens,
   1592     ).converted()
   1593 except Exception:
-> 1594     raise ValueError(
   1595         f"Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path "
   1596         f"with a SentencePiece tokenizer.model file."
   1597         f"Currently available slow->fast convertors: {list(SLOW_TO_FAST_CONVERTERS.keys())}"
   1598     )

ValueError: Converting from Tiktoken failed, if a converter for SentencePiece is available, provide a model path with a SentencePiece tokenizer.model file.Currently available slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 'Qwen2Tokenizer', 'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer', 'RobertaTokenizer', 'RoFormerTokenizer', 'SeamlessM4TTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 'UdopTokenizer', 'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer', 'CodeLlamaTokenizer', 'GemmaTokenizer', 'Phi3Tokenizer']
```


Can anyone provide me with a reproducer? @digbangbang or @johnsutor ? 😓 I am down to fix anything but could nor reproduce previous snippers! 

BTW @johnsutor your error seems to indicate that loading the tiktoken file failed from the tiktoken library! 

Thanks, I have fixed it by `pip install sentencepiece` @ArthurZucker 

This seems like an error message that should be edited, feel free to open a PR!

@ArthurZucker I mentioned the issue to the tiktoken maintainers. Is there anything I should try in the interim? I tried installing sentencepiece, tried downgrading tiktoken, and more to no avail :( 

Update: uninstalled and then re-installed sentencepiece and now it works!

Restart the notebook and it will work 

I have tried all the mentioned solutions here and it doesn't work


@Wafaa014 can you provide a reproducer? 

here is the script:
```
import torch
from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("haoranxu/ALMA-13B")

device = "cuda" if torch.cuda.is_available() else "cpu"

prompt="Translate this from Chinese to English:\nChinese: 我爱机器翻译。\nEnglish:"
input_ids = tokenizer(prompt, return_tensors="pt", max_length=40, truncation=True).input_ids.to(device)
```


here is the pip freeze:
`accelerate==1.0.0
aiohappyeyeballs==2.4.3
aiohttp==3.10.9
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.6.0
async-timeout==4.0.3
attrs==24.2.0
blobfile==3.0.0
certifi==2024.8.30
charset-normalizer==3.4.0
click==8.1.7
cloudpickle==3.0.0
datasets==3.0.1
dill==0.3.8
diskcache==5.6.3
distro==1.9.0
einops==0.8.0
exceptiongroup==1.2.2
fastapi==0.115.0
filelock==3.16.1
frozenlist==1.4.1
fsspec==2024.6.1
gguf==0.10.0
h11==0.14.0
httpcore==1.0.6
httptools==0.6.1
httpx==0.27.2
huggingface-hub==0.25.2
idna==3.10
importlib_metadata==8.5.0
interegular==0.3.3
Jinja2==3.1.4
jiter==0.6.1
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
lark==1.2.2
llvmlite==0.43.0
lm-format-enforcer==0.10.6
lxml==5.3.0
MarkupSafe==3.0.1
mistral_common==1.4.4
mpmath==1.3.0
msgpack==1.1.0
msgspec==0.18.6
multidict==6.1.0
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.3
numba==0.60.0
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.560.30
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.77
nvidia-nvtx-cu12==12.1.105
openai==1.51.2
outlines==0.0.46
packaging==24.1
pandas==2.2.3
partial-json-parser==0.2.1.1.post4
peft==0.13.1
pillow==10.4.0
prometheus-fastapi-instrumentator==7.0.0
prometheus_client==0.21.0
propcache==0.2.0
protobuf==5.28.2
psutil==6.0.0
py-cpuinfo==9.0.0
pyairports==2.1.1
pyarrow==17.0.0
pycountry==24.6.1
pycryptodomex==3.21.0
pydantic==2.9.2
pydantic_core==2.23.4
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.2
PyYAML==6.0.2
pyzmq==26.2.0
ray==2.37.0
referencing==0.35.1
regex==2024.9.11
requests==2.32.3
rpds-py==0.20.0
safetensors==0.4.5
sentencepiece==0.2.0
six==1.16.0
sniffio==1.3.1
starlette==0.38.6
sympy==1.13.3
tiktoken==0.7.0
tokenizers==0.20.1
torch==2.4.0
torchvision==0.19.0
tqdm==4.66.5
transformers==4.45.2
triton==3.0.0
typing_extensions==4.12.2
tzdata==2024.2
urllib3==2.2.3
uvicorn==0.31.1
uvloop==0.20.0
vllm==0.6.2
watchfiles==0.24.0
websockets==13.1
xformers==0.0.27.post2
xxhash==3.5.0
yarl==1.14.0
zipp==3.20.2
`

I have been doing some debugging. When I load the `haoranxu/ALMA-13B` tokenizer, `LlamaTokenizer.from_pretrained` returns a `<class 'bool'>` which is not callable. when I change the model to another llama model `meta-llama/Llama-2-7b`, It works. So this issue has to do with ALMA 13B.

Okay, will have a look but I don't understand why you are not using LlamaTokenizerFast, as it should be faster 

<img width="729" alt="image" src="https://github.com/user-attachments/assets/7c984c39-f95a-4846-81e6-f5a7890c7f4b">
Could not reproduce sadly btw

the fast tokenizer throws the tiktoken error. 
This is weird. Are you creating a new env with the pip freeze to reproduce?

@ArthurZucker  

I can't reproduce but I did resolve the issue by removing my locally cached hub artifacts (`~/.cache/huggingface/hub/...` for affected tokenizers/models).

I'm guessing it has to do with first retrieving a model with the new `tokenizers` library or the rust library directly, then using an older version to load the same `repo_id` from cache.

Hope that helps

Oh thank you so much! Deleting the cache does solve it! I appreciate your help both @mjbommar @ArthurZucker 

Cool closing this as solved then! 🤗 

> Update: uninstalled and then re-installed sentencepiece and now it works!

I found that I didn't have sentencepiece installed, so I ran `pip install sentencepiece` to fix the problem. And it works.