When I use the command "python convert_llama_weights_to_hf.py --input_dir /xxx/llama/ --model_size 70B --output_dir /xxxx/Llama-2-70b-chat-hf" killed
Author: cm-liushaodong
Pull request: no

### System Info

Hi,@ArthurZucker,@younesbelkada
When I use the command "python convert_llama_weights_to_hf.py --input_dir /xxx/llama/ --model_size 70B --output_dir /xxxx/Llama-2-70b-chat-hf", the following error occurred:
<img width="1342" alt="image" src="https://github.com/huggingface/transformers/assets/44772254/58946c17-e65d-4d56-b675-361ff0832576">
Additional Information: Memory Size：116GB
I noticed: Important note: you need to be able to host the whole model in RAM to execute this script (even if the biggest versions 
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).
I would like to ask, is there a parameter control, I can use disk instead of memory to complete the task? I can live with him running slower.

### Who can help?

@ArthurZucker
@younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

  python convert_llama_weights_to_hf.py  --input_dir /home/xxxx/llama/ --model_size 70B --output_dir /home/xxxx/llama/Llama-2-70b-chat-hf


### Expected behavior

it can execute successfully

Comments:
Hi @cm-liushaodong 
The llama-70B model weights file is at least 140GB in half precision, sadly I think that you need an instance of at least that CPU memory size to download the weights and load them in CPU memory. Maybe @ArthurZucker can confirm as he has used that script

Yes, as[ the documentation mentions, ](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py#L52):
```python 
Important note: you need to be able to host the whole model in RAM to execute this script (even if the biggest versions
come in several checkpoints they each contain a part of each weight of the model, so we need to load them all in RAM).
```
`killled` means you probably had a OOM issue. If you are on linux you can check with `dmesg -T| grep -E -i -B100 'killed process'` 

Thanks all, I used a machine with a larger memory to transcode successfully.

After you transcode the code on a machine with a larger memory, were you able to take it back and fine-tune (train) the model on you older machine (which had lesser memory)? If so, please share your transcoded model, since I have the exact same problem. 

I don’t have a machine with a larger memory to transcode the code (i.e. to change the 70B model weights to Huggingface format using the convert_llama_weights_to_hf.py Python code). However, if can already get the 70B model weights in Huggingface format (i.e. your  transcoded model you just made with a machine with a larger memory), I can then train (fine-tune) it on my 4090 GPU machine. 

I don't care how long it takes (can live with it being slow to train).

Or does anyone know where I can download the 70B model weights ALREADY converted in Huggingface format? It is probably around 250 GB

@BramVanroy You can search for "Llama 70B" on hf.co and find a list of checkpoints: https://huggingface.co/models?search=llama%2070b

@amyeroberts Not sure why you tagged me - perhaps by accident?

Yes, sorry, my apologies! I meant to tag @BrookMakF 

 I'm new to this so please break it down for me. Here is what I tried:
 
* I went to the list of checkpoints on hf.co and chose this one: https://huggingface.co/meta-llama/Llama-2-70b-hf/tree/main

* I then went to the "Files and versions" tab and downloaded these two files ('pytorch_model.bin.index.json' and 'config.json') into a new folder called "new_70B_hf_format" under my Llama 2 folder (path: ‘llama-2_for_70B/llama/new_70B_hf_format/’)

*Then I tried to load this 70B model in Python using transformers as below:
 `from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("../new_70B_hf_format") ` 

However, the kernel dies. Is there a way to PRE-DOWNLOAD the 70B model in Huggingface format, instead of loading the model in Python? I’m not sure what I’m doing wrong. I have attached screenshots to better explain, thank you for the help. 


![Screenshot-2023-09-05-21-41-16](https://github.com/huggingface/transformers/assets/5878134/421c43c9-854c-420d-a5ee-c402b3331897)
![Screenshot-2023-09-05-21-40-36](https://github.com/huggingface/transformers/assets/5878134/1d31b331-d3a9-44d7-80b0-2af4ddfdb4af)
![Screenshot-2023-09-05-21-43-54](https://github.com/huggingface/transformers/assets/5878134/9205e7da-ed29-4d17-b968-15775c352a26)


@amyeroberts 🤗

@BrookMakF Please have patience. Pinging maintainers when they don't immediately reply to your issue isn't sustainable or scalable behaviour - if everyone did it, it wouldn't be impossible for us to manage our notifications. We're all very busy on different pieces of work across different repos and, importantly, on different timezones too. 

The files you downloaded aren't the model weights - `pytorch_model.bin.index.json` is a json file which contains the mapping of model weights to their respective shard, and config.json contains the model config file. 

When you call `from_pretrained`, the model weights are first downloaded to a cache folder on your local machine and the loaded into the model. You can specify the model checkpoint from the hub directly: 

```python
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-70b-hf")
```

Next time it is called, it will load the downloaded weights from your local machine. 

If you want to copy all of the weights and model files outside of a python session, you can clone the checkpoint repo. Make sure you have git lfs installed. 

```
git clone https://huggingface.co/meta-llama/Llama-2-70b-hf
```

And the pass the path to the repo when calling `from_pretrained` 


Thank you for taking the time to provide guidance on this matter. Your insights on managing notifications and considering the global nature of contributors' engagements are duly noted.

I downloaded all of the weights and model files outside of a python session, since I was having RAM memory issue. I only have one 4090 GPU server with 24GB VRAM (which is an online rented from Hostkey). I cloned the 513 GB Llama-2-70b-hf checkpoint repo since I had lots of disk space on my server. However, I still have the same issue, it seems to run out of VRAM memory, it says “The kernel for llama-recipes/quickstart.ipynb appears to have died. It will restart automatically”

The ‘llama-recipes/quickstart.ipynb’ wrote “This notebook shows how to train a Llama 2 model on a single GPU (e.g. A10 with 24GB) using int8 quantization and LoRA”. 
However, I couldn’t train / fine-tune the 70B model on my 4090 GPU server with 24GB. 

Though, I was able to fine-tune the 7B & 13B models, I couldn’t fine-tune the 70B on this machine. 

Did anyone mange to fine-tune 70B on such machine? Are there things I can change?

Here are things I tried and failed:

1. I manually specified which layers get offloaded to cpu using the device_map argument

```python
device_map = {     
"model.layers.0.self_attn.o_proj.weight": "cpu",
"model.layers.0.self_attn.q_proj.weight": "cpu",
"model.layers.0.self_attn.rotary_emb.inv_freq": "cpu",
"model.layers.0.self_attn.v_proj.weight": "cpu",
"model.layers.1.input_layernorm.weight": "cpu",
"model.layers.1.mlp.down_proj.weight": "cpu",
"model.layers.1.mlp.gate_proj.weight": "cpu",
"model.layers.1.mlp.up_proj.weight": "cpu",
"model.layers.1.post_attention_layernorm.weight": "cpu",
#more layers …
}
model = LlamaForCausalLM.from_pretrained(model_id, device_map=device_map, torch_dtype=torch.float16)
```
But I keep getting the error:

```
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
```


2. I tried to copy the model back to cuda

```python
model = model.to('cuda')
```
However, it runs out of VRAM memory, it says:
```
OutOfMemoryError: CUDA out of memory. Tried to allocate 448.00 MiB (GPU 0; 23.65 GiB total capacity; 23.11 GiB already allocated; 61.69 MiB free; 23.11 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```
![Screenshot-2023-09-07-20-59-54_3](https://github.com/huggingface/transformers/assets/5878134/dc662232-ab1a-4e88-bd69-8ab2a02bbd6e)
![Screenshot-2023-09-07-20-59-54_2](https://github.com/huggingface/transformers/assets/5878134/10bdafae-0048-4898-a5b0-a9bab9113d0b)
![Screenshot-2023-09-07-20-59-54](https://github.com/huggingface/transformers/assets/5878134/a0343987-7697-420a-b58b-0f83c3e05954)


Hi @BrookMakF, 

Questions such as these: how to fit your model onto a device are best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports. I recommend reading through the docs to see the recommended ways for loading models e.g. https://huggingface.co/docs/accelerate/usage_guides/big_modeling.