Failing to load 'Llama-2-70b-chat-hf' in transformers.LlamaForCausalLM
Author: mmc31
Pull request: no

### System Info

Ubuntu 22.04
Python 3.9.18
torch.__version__ = '2.0.1+cu117'
transformers.__version__ = '4.31.0'

dowloaded files from here:
https://huggingface.co/meta-llama/Llama-2-70b-chat-hf/tree/main

run this:
```

import torch
from transformers import LlamaForCausalLM
base_model = "Llama-2-70b-chat-hf"
model = LlamaForCausalLM.from_pretrained(
    base_model,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    local_files_only=True,
)

```
and the error:
*** ValueError: weight is on the meta device, we need a `value` to put in on 1.

/lib/python3.9/site-packages/accelerate/hooks.py(253)
`set_module_tensor_to_device(module, name, self.execution_device)`
module is a LlamaDecoderLayer
name = 'mlp.up_proj.weight'
self.execution_device = 1 

```
module.mlp.up_proj.weight
Parameter containing:
Parameter(Int8Params(..., device='meta', size=(28672, 8192), dtype=torch.float16))
```

so it appears this parameter is empty and has no data. Why is this not happening to anyone else??

I have enough GPU memory
nvidia-smi
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    Off  | 00000000:18:00.0 Off |                  Off |
| 30%   41C    P0    83W / 300W |      0MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000    Off  | 00000000:C3:00.0 Off |                  Off |
| 30%   42C    P0    86W / 300W |      0MiB / 48685MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```
@ArthurZucker @younesbelkada
Any ideas?

thanks!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import LlamaForCausalLM
base_model = "Llama-2-70b-chat-hf"
model = LlamaForCausalLM.from_pretrained(
    base_model,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    local_files_only=True,
)
```

### Expected behavior

The model should load rather than error.

Comments:
I should also mention that this works fine for 7b and 13b

cc @SunMarc as well!

hi @mmc31 
Out of curiosity does the issue still persists with the latest transformers and accelerate?

```bash
pip install -U accelerate transformers
```

Still the same error. Now i have:
transformers.__version__  = '4.34.0'
accelerate.__version__ = '0.23.0'

@younesbelkada any other ideas why this layer's parameter is empty?

I'm going to close this as it seems to be related to your setup and I cannot reproduce it on `main`. Your local files are probably corrupted