llama 2 conversion script unknown error
Author: liboliba
Pull request: no

### System Info

Hi,
I have downloaded llama 2 weights and installed the transformer package. I plan to use it under transformer package and applied the conversion script.

The conversion script does not work:
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/tomyfilepath

File "...path/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 126
    print(f"Fetching all parameters from the checkpoint at {input_base_path}.")
                                                                             ^
SyntaxError: invalid syntax

On Linux when I do for example:  
ls /path/to/downloaded/llama/llama-2-7b-chat 
I get:
checklist.chk  consolidated.00.pth  params.json

I assume I have the correct files. Any advise would be grateful. 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/tomyfilepath

### Expected behavior

It is expected tokenizer and model be converted so that they are usable for transformer package. 

Comments:
Hi @liboliba, thanks for raising an issue! 

So that we can best help you, please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) including: 
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* The full error traceback
* A minimal code reproducer. Here we don't have access to the weights. Are there weights you could share which reproduce this error?  

Thank you for the advise!
transformers-cli env returns:
- `transformers` version: 4.36.2
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.5
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes & No, I am working under HPC environment, my current compute node has no GPU, but it shared hard disk with a GPU node. A GPU node has no internet connection and can only load model via the shared hard disk with the compute node I work on now. 
- Using distributed or parallel set-up in script?: No

For the other two bullets, sorry I am less sure how to respond because what I did was to download the official meta llama 2 into a folder, and then I git clone the transformer source code and try to run the conversion code. The error I get now is:

python /scratch/ll1d19/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir /scratch/ll1d19/llama/llama/llama-2-7b-chat/ --model_size 7B --output_dir /scratch/ll1d19/hf_llama2/Llama-2-7b-chat-hf/
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Traceback (most recent call last):
  File "/scratch/ll1d19/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 319, in <module>
    main()
  File "/scratch/ll1d19/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 307, in main
    write_model(
  File "/scratch/ll1d19/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 109, in write_model
    tokenizer = tokenizer_class(tokenizer_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 124, in __init__
    super().__init__(
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__
    slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 178, in __init__
    self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/transformers/models/llama/tokenization_llama.py", line 203, in get_spm_processor
    tokenizer.Load(self.vocab_file)
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ll1d19/.conda/envs/myiai/lib/python3.11/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Not found: "/scratch/ll1d19/llama/llama/llama-2-7b-chat/tokenizer.model": No such file or directory Error #2

The weights are too large to share as it is about 13GB, the json file is around 100bytes.
Any advise would be grateful!

Hi @liboliba, thanks for the update! 

Based on the error, I'd suggest making sure you have the latest versions of `tokenizers` and `sentencepiece` installed in your environment. 

There's no need to convert the official checkpoints though - there's many already available on the hub e.g. [here](https://huggingface.co/huggyllama/llama-7b) which you can access provided you've filled out the access form; or [meta-llama/Llama-2-70b-hf](https://huggingface.co/meta-llama/Llama-2-70b-hf) for llama 2.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.