Add llama3
Author: ArthurZucker
Pull request: yes

# TODOs
- [x] docs/source/en/model_doc/llama3.md

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30334). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Managed to get this commit working, but it requires setting flag --llama_version 3 when calling convert_llama_wieghts_to_hf.py

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir /mnt/nvme1n1/Models/llama3/Meta-Llama-3-8B-Instruct \
--model_size 8B \
--output_dir /mnt/nvme1n1/Models/llama3/Meta-Llama-3-8B-Instruct-hf \
--llama_version 3

```

Additionally, it tries to delete the tmp folder before it's empty, so throws an error at the end, even though conversion is successful.

>  it requires setting flag --llama_version 3 when calling convert_llama_wieghts_to_hf.py

that is expected since you are converting version3, but should still work for earlier versions. 

Hello, I notice the mask procedure ensuring self-attention does not cross document boundaries in meta's llama3 blog, as the following:
![image](https://github.com/huggingface/transformers/assets/15059072/2f9d129d-71f5-4e61-8dda-e8f1564d5dc3)

Should llama3 have an extra attention mask for this?

In `transformers` we usually let the user create the custom mask. The trainer might even support it. but no it's not a modeling code change we want 😉 you can already pass any 4d or 2d mask to Llama

Failing tests are unrelated

I meet this error when using the convert_llama_weights_to_hf.py. ImportError: cannot import name 'TikTokenConverter' from 'transformers.convert_slow_tokenizer', how to solve it ? The version of my transformers is 4.40.1

Hello @lhanchao777, conversion scripts should be used with a source install of `transformers`. You can install from source with the following:

```
pip install git+https://github.com/huggingface/transformers
```
You could also clone the repo and add it as an editable installation:
```
git clone https://github.com/huggingface/transformers
pip install -e ./transformers
```

> I meet this error when using the convert_llama_weights_to_hf.py. ImportError: cannot import name 'TikTokenConverter' from 'transformers.convert_slow_tokenizer', how to solve it ? The version of my transformers is 4.40.1

Hi, @lhanchao777  i download model from modelscop and meet the same problem when trans model format; 
To solve this, you need to instal transformer from source; 
 cause transformer source code uodated to 4.41.0,  but transformer of huggingface updated to 4.40.1，so we need to install transformer from source; 
And install from conda not pip

> Transformers can be installed using conda as follows:
        conda install conda-forge::transformers
        NOTE: Installing transformers from the huggingface channel is deprecated.
        # todo: https://github.com/huggingface/transformers/tree/main

but even thoughi solve this problem, i still get other bugs which i have no clue; so finally i still get llama3 direclly from huggingface

``import transformers
import torch
from huggingface_hub import login
login(token="your access token")

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
pipeline = transformers.pipeline("text-generation",
  model=model_id,
  model_kwargs={"torch_dtype": torch.float16},
  device="cuda",
)
pipeline("Hey how are you doing today?")``

<img width="820" alt="微信图片_20240429172839" src="https://github.com/huggingface/transformers/assets/19632448/ee252033-facf-4544-abce-91844282a729">
<img width="821" alt="微信图片_20240429172845" src="https://github.com/huggingface/transformers/assets/19632448/bdf44065-9b54-415a-adbd-cdcb8894aa50">

hope it helps!

Hi there!  Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion:

python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2
 
--llama_version 3 works fine, but that result is not what I'm looking for in my application.

 I keep getting this error:
 
 `RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model`
 
 Thanks!

you want llama2, but your error path is llama    "./Meta-Llama-3-8B/original/tokenizer.model", check your path please
________________________________
发件人: danieltmeta ***@***.***>
发送时间: 2024年4月30日 4:23
收件人: huggingface/transformers ***@***.***>
抄送: ZJL0111 ***@***.***>; Comment ***@***.***>
主题: Re: [huggingface/transformers] Add llama3 (PR #30334)


Hi there! Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion:

python src/transformers/models/llama/convert_llama_weights_to_hf.py
--input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2

--llama_version 3 works fine, but that result is not what I'm looking for in my application.

I keep getting this error:

RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model

Thanks!

―
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/pull/30334#issuecomment-2083595969>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEVZCQDAT35B76KQRCX7FTTY72T2LAVCNFSM6AAAAABGOSEOTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBTGU4TKOJWHE>.
You are receiving this because you commented.Message ID: ***@***.***>


Hi there! I followed the above instructions to convert Meta-Llama-3-8B to hf format but still got errors as follows:

```
Traceback (most recent call last):
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 407, in <module>
    main()
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 394, in main
    vocab_size = len(write_tokenizer(args.output_dir, spm_path, llama_version=args.llama_version))
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 358, in write_tokenizer
    tokenizer = Llama3Converter(input_tokenizer_path).tokenizer
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 319, in __init__
    tokenizer = self.converted()
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1534, in converted
    tokenizer = self.tokenizer()
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1527, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1503, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
  File "/home/xxx/miniconda3/envs/pytorch/lib/python3.8/site-packages/tiktoken/load.py", line 115, in load_tiktoken_bpe
    return {
  File "/home/xxx/miniconda3/envs/pytorch/lib/python3.8/site-packages/tiktoken/load.py", line 117, in <dictcomp>
    for token, rank in (line.split() for line in contents.splitlines() if line)
ValueError: too many values to unpack (expected 2)
```

I installed transformers from the source file under the folder `/data/models/origin-format/transformers/`; the version is 4.41.0.dev0. By the way, I follow the instruction to install it.

> Hello @lhanchao777, conversion scripts should be used with a source install of `transformers`. You can install from source with the following:
> 
> ```
> pip install git+https://github.com/huggingface/transformers
> ```
> 
> You could also clone the repo and add it as an editable installation:
> 
> ```
> git clone https://github.com/huggingface/transformers
> pip install -e ./transformers
> ```



You are most probably not using the correct original tokenizer.model. We proof tested the script many times 😉 

@ArthurZucker You are correct, I was not using this correctly.  Thanks!

> Hi there! Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion:
> 
> python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2
> 
> --llama_version 3 works fine, but that result is not what I'm looking for in my application.
> 
> I keep getting this error:
> 
> `RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model`
> 
> Thanks!

I also have the error "Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model", I found the "Meta-Llama-3-8B-chat" don't have the  "tokenizer.model", how to solve it ?


You should use `PreTrainedTokenizerFast` or just `AutoTokenizer`! 

Traceback (most recent call last):
  File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 407, in <module>
    main()
  File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 396, in main
    write_model(
  File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 178, in write_model
    f"model.layers.{layer_i}.self_attn.k_proj.weight": permute(
                                                       ^^^^^^^^
  File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 154, in permute
    return w.view(n_heads, dim1 // n_heads // 2, 2, dim2).transpose(1, 2).reshape(dim1, dim2)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[32, 2, 2, 4096]' is invalid for input of size 16777216
How to solve it?

Could you share which version of `transformers` you are using? And which version of `llama` you are trying to convert