unable to convert llama 3.3 weights to hf.py
Author: AshishMulupuri
Pull request: no
Labels: name: bug

### System Info

python == 3.10.12
transformers == 4.45.1


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I ran the `convert_llama_weightsto_hf.py` script on the llama-3.3-70B-Instruct checkpoint weights 
`!python convert_llama_weights_to_hf.py --input_dir "llama_3_3_model/Llama-3.3-70B-Instruct/original" --model_size 70B --output_dir "llama_3_3_model/Llama-3.3-70B-Instruct/hf-converted"`, 
and got the 
`ValueError: Failed to instantiate tokenizer.Pleae, make sure you have sentencepiece and protobuf.`
I tried upgrading both sentencepiece and protobuf but same error is occurring again


### Expected behavior

downloaded from HuggingFace Files.

Comments:
# I think the issue here is with your dependencies 

## 1. Upgrade Dependencies
Ensure that the necessary dependencies are installed and up-to-date. Run the following command:

```bash
pip install --upgrade sentencepiece protobuf
```

```bash

pip show sentencepiece protobuf

```
I think it should run like normal






I tried it, upgraded versions `protobuf-5.29.2 and sentencepiece-0.2.0`
but still got the same error.

Were you using a notebook? Did you remember to restart the notebook session after installing the dependencies? I was able to convert the llama weights by using a dev install of transformers.

Maybe environmental problems? 

```
 .\<your_env>\Scripts\activate 
 
 ````
 
 
 another option could be downgrade your library but I'm not sure if it's going to work  

> Were you using a notebook? Did you remember to restart the notebook session after installing the dependencies? I was able to convert the llama weights by using a dev install of transformers.

Yes, I'm using the notebook. I restarted it after upgrading the `sentencepiece and protobuf` before running the `convert_llama_weights_to_hf.py`, but I still ended up with the same error.

transformers version 4.47.1
python version 3.10.12
updated sentencepiece protobuf

I have the same problem converting Llama3.3-70B-Instruct from the command line.  It seems to keep running out of memory on the large model conversion (I have 32GB).

python3 ~/ai/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir ~/.llama/checkpoints/Llama3.3-70B-Instruct --model_size 70B --llama_version 3 --output_dir ~/.llama/checkpoints/Llama3.3-70B-Instruct-hf

Converting the tokenizer.
Saving a LlamaTokenizerFast to /home/chad/.llama/checkpoints/Llama3.3-70B-Instruct-hf.
Converting the model.
Fetching all parameters from the checkpoint at /home/chad/.llama/checkpoints/Llama3.3-70B-Instruct.
Loading in order: ['consolidated.00.pth', 'consolidated.01.pth', 'consolidated.02.pth', 'consolidated.03.pth', 'consolidated.04.pth', 'consolidated.05.pth', 'consolidated.06.pth', 'consolidated.07.pth']
/mnt/c/Users/chad/ai/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py:236: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  loaded = [torch.load(os.path.join(input_base_path, file), map_location="cpu") for file in checkpoint_list]
**Killed**

@verbowski Your issue is most likely due to running out of memory. The model has to be fully loaded in memory and 70B is rather large. Swap will suffice, but you're going to need approximately 200 GB of pageable memory (RAM + swap).

@AshishMulupuri It's difficult to debug this remotely. Two tips that may or may not help; try using a .py script instead of a notebook, and use the dev version of transformers; clone the git repo and install with `pip install -e '.[dev]'` in a new environment.

@hvaara I'll try to use dev version of transformers. Meanwhile here's the full error I'm getting even running it through terminal.

amulupu@f83c3ae8b4f4:~/llama_3_3_model$ python convert_llama_weights_to_hf.py --input_dir "llama_3_3_model/Llama-3.3-70B-Instruct/original/" --model_size 70B --output_dir " llama_3_3_model/Llama-3.3-70B-Instruct/hf-converted"
Converting the tokenizer.
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
Traceback (most recent call last):
  File "/home/amulupu/llama_3_3_model/convert_llama_weights_to_hf.py", line 502, in write_tokenizer
    tokenizer = tokenizer_class(input_tokenizer_path)
  File "/home/amulupu/.local/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 157, in __init__
    super().__init__(
  File "/home/amulupu/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 132, in __init__
    slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs)
  File "/home/amulupu/.local/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 171, in __init__
    self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
  File "/home/amulupu/.local/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 198, in get_spm_processor
    tokenizer.Load(self.vocab_file)
  File "/home/amulupu/.local/lib/python3.10/site-packages/sentencepiece/__init__.py", line 961, in Load
    return self.LoadFromFile(model_file)
  File "/home/amulupu/.local/lib/python3.10/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "llama_3_3_model/Llama-3.3-70B-Instruct/original/tokenizer.model": No such file or directory Error #2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/amulupu/llama_3_3_model/convert_llama_weights_to_hf.py", line 601, in <module>
    main()
  File "/home/amulupu/llama_3_3_model/convert_llama_weights_to_hf.py", line 576, in main
    write_tokenizer(
  File "/home/amulupu/llama_3_3_model/convert_llama_weights_to_hf.py", line 504, in write_tokenizer
    raise ValueError(
ValueError: Failed to instantiate tokenizer. Please, make sure you have sentencepiece and protobuf installed.

@AshishMulupuri what's in your `llama_3_3_model/Llama-3.3-70B-Instruct/original/` folder? Looks like the tokenizer can't be found (ref `OSError: Not found: "llama_3_3_model/Llama-3.3-70B-Instruct/original/tokenizer.model": No such file or directory Error #2`).

Can you try downloading the files to that same folder again using the instructions from Meta? Any files you already have downloaded should be recognized as alrasdy downloaded and skipped. Then try running the conversion script again.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

# Resolving Missing Tokenizer File (`tokenizer.model`) Error

The error indicates that the `tokenizer.model` file is missing from the directory `llama_3_3_model/Llama-3.3-70B-Instruct/original/`. This file is essential for tokenizing input text before passing it to the model. Below are the steps to resolve the issue:

---

## 1. **Verify Directory Contents**
- Check if the `tokenizer.model` file exists in the `original/` directory.
- If the file is missing, confirm whether the downloaded model files are complete.

---

## 2. **Recheck Model Download**
- **Manual Download**: If the model files were downloaded manually, re-download them from the official source to ensure completeness.
- **Script-Based Download**: Verify that the download script includes the tokenizer file.

---

## 3. **Correct Path**
- Ensure your script or code points to the correct directory.
- Double-check whether you're using relative or absolute paths for the directory.

---

## 4. **Use Hugging Face's `transformers` Library (if applicable)**
If you are using Hugging Face's `transformers`, you can load the tokenizer and model with the following code:

```python
from transformers import LlamaTokenizer, LlamaForCausalLM

# Update with the correct path to your model directory
model_path = "path_to_model_directory"

# Load tokenizer and model
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(model_path)


@AshishMulupuri I know I'm late, but I hope this comment helps others facing the same issue. You can simply pass the argument `--llama_version 3`, which changes the [order flow](https://github.com/huggingface/transformers/blob/6b550462139655d488d4c663086a63e98713c6b9/src/transformers/models/llama/convert_llama_weights_to_hf.py#L493-L506).

Though the script claims it only supoort  llama 3.1 and 3.2, the flag works for 3.3 too.