如果在单个GPU上out of memory 如何用两个GPU加载推理同一个模型？
Author: zhaoyuchen1128
Pull request: no

### System Info

latest

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

我用的是transformers的from_pretrained方法，单个GPU加载模型out of memory,如何用另一块一起加载？

### Expected behavior

成功加载较大的模型

Comments:
Hi @zhaoyuchen1128 you can do that as follows for single GPU out of memory issue (對於單 GPU 記憶體不足問題，您可以執行以下操作):
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Trelis/Llama-2-7b-chat-hf-function-calling-v2", low_cpu_mem_usage=True,offload_state_dict=True
)
```
`low_cpu_mem_usage` : it will load the model using ~1x model size CPU memory (它將使用約 1 倍模型大小的 CPU 記憶體載入模型)
`offload_state_dict` : it will temporarily offload the CPU state dict to hard drive and will prevent getting out of RAM (它將暫時將 CPU 狀態指令卸載到硬碟並防止記憶體溢出)

For multiple GPUs model loading use (用於多GPU模型載入使用) : `device_map = 'auto'` as follows:
```
model = AutoModelForCausalLM.from_pretrained(
    "Trelis/Llama-2-7b-chat-hf-function-calling-v2", low_cpu_mem_usage=True,offload_state_dict=True, device_map = 'auto'
)
```
`device_map` : It loads a model onto multiple GPUs (它將模型載入到多個 GPU 上)

Cheers!


> Hi @zhaoyuchen1128 you can do that as follows for single GPU out of memory issue (對於單 GPU 記憶體不足問題，您可以執行以下操作):
> 
> ```
> from transformers import AutoModelForCausalLM
> model = AutoModelForCausalLM.from_pretrained(
>     "Trelis/Llama-2-7b-chat-hf-function-calling-v2", low_cpu_mem_usage=True,offload_state_dict=True
> )
> ```
> 
> `low_cpu_mem_usage` : it will load the model using ~1x model size CPU memory (它將使用約 1 倍模型大小的 CPU 記憶體載入模型) `offload_state_dict` : it will temporarily offload the CPU state dict to hard drive and will prevent getting out of RAM (它將暫時將 CPU 狀態指令卸載到硬碟並防止記憶體溢出)
> 
> For multiple GPUs model loading use (用於多GPU模型載入使用) : `device_map = 'auto'` as follows:
> 
> ```
> model = AutoModelForCausalLM.from_pretrained(
>     "Trelis/Llama-2-7b-chat-hf-function-calling-v2", low_cpu_mem_usage=True,offload_state_dict=True, device_map = 'auto'
> )
> ```
> 
> `device_map` : It loads a model onto multiple GPUs (它將模型載入到多個 GPU 上)
> 
> Cheers!

Thank you.But when I load a model ,it seems too slowly with two 4090 RTX,is that normal? The model is llama3:70b of 140GB

@zhaoyuchen1128 if you feel the model loading as pretty slow then please go through this [guide](https://huggingface.co/docs/accelerate/v0.11.0/en/big_modeling) of improving the speed with `accelerate` library. It gives all the details regarding it and will pretty much answer all the questions and if any remains the issues are always open.

Cheers!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.