isses with converting llama-70B-chat model with convert_llama_weights_to_hf.py
Author: Braxtogoo
Pull request: no

### System Info

Howdy.
I'm trying to convert the new llama 70B chat model with convert_llama_weights_to_hf.py
but I'm getting the following error during loading the checkpoint shards:

`ValueError: Trying to set a tensor of shape torch.Size([1024, 8192]) in "weight" (which has shape torch.Size([8192, 8192])), this look incorrect.`

I'm running the script on Ubuntu 22.04.1 LTS and python 3.11.4 with Transformers version 4.33.2.

Thank you in advance.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This was the command I ran.
`python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir . --model_size 70B --output_dir llama-2-70b-chat-hf --safe_serialization true`

### Expected behavior

At the end of the process, I should have the model in a bin format.
With config.json and tokenizer file.

Comments:
Hey, could you confirm that the checkpoints you download are from https://huggingface.co/meta-llama ? 

They came from Meta themselves, the files I have seem to match the checksums on the huggingface repo though I am missing some files like the tokenizer and such. Would I need to grab the whole thing off of huggingface again for it to work? If so I can and I'd be happy to report back with the results.

Probably the `params.json` file. The weights were converted quite a few number of times without problem so that will be the first culprit 

Looks like mine matches the one on huggingface, still downloaded it and retried. Same error.

Downgrading to `4.33.1` beats this boss and lets you advance to the next level (at least in my case).

Arf that's not normal! @Braxtogoo could you share the full traceback please? 😉 
There were not many changes to the script, one most notable is from #25740 ! 

Sorry for the delay.

@tycrimm Downgrading appears to have the same unbeatable boss bug. :)

@ArthurZucker Here's the trace back from my last attempt with the downgraded version. It appears to be the same.

Traceback (most recent call last):
  File "/pathToLlama2/./transformers-4.33.1/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 318, in <module>
    main()
  File "/pathToLlama2/./transformers-4.33.1/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 306, in main
    write_model(
  File "/pathToLlama2/./transformers-4.33.1/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 270, in write_model
    model = LlamaForCausalLM.from_pretrained(tmp_model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homePath/anaconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2777, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homePath/anaconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3118, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homePath/anaconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 702, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/homePath/anaconda3/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 281, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([1024, 8192]) in "weight" (which has shape torch.Size([8192, 8192])), this look incorrect.

Hey, I'm reproducing right now, will let you know 😉 

Here are the steps I used: (transformers==4.34.1)
```bash 
pip install transformers
huggingface-cli login
huggingface-cli download  meta-llama/Llama-2-70b --local-dir="70B" --cache-dir="./cache"
huggingface-cli download  meta-llama/Llama-2-70b-hf tokenizer.model --local-dir="70B" --cache-dir="./cache"
python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir absolute/path/to/moved/repo --model_size 70B --output_dir llama-2-70b-chat-hf --safe_serialization true
```
this worked as expected. I'll close this as I can't reproduce and we have quite a lot of users who successfully converted the checkpoints. 
Might be a transformers version issue 😉 

Maybe you can try 4-bit quantization.