[Bug] `low_cpu_mem_usage=True` is not working for LLAMA2-70B
Author: dc3671
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.14.21-150400.24.55-default-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0a0+git4a3d0d4 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@pacman100 @sgugger @muellerzr 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I use following lines along with deepspeed tp_size=4 on single node (Although I don't think it's related) to load.
```
model_name = <my_absolute_path_to_llama2>
config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, config=config, low_cpu_mem_usage=True, torch_dtype=torch.float16, trust_remote_code=True)
```

But the cpu memory usage is still increasing rapidly and seems never release, until it reached max memory (512GB). For Bloom-176B I can see the low memory usage and it will maintain at a low level less than 10GB.

![image](https://github.com/huggingface/transformers/assets/5948851/17737960-7b85-464a-95ba-83e9c6f23bb8)

log line before OOM:
```
Loading checkpoint shards:  87%|████████▋ | 13/15 [03:46<00:38, 19.27s/it]
```

I already investigated some code of your `from_pretrained` method in `src/transformers/modeling_utils.py`. Is it related to meta tensor load support? Because I know it's not supported in deepspeed for LLAMA2.

### Expected behavior

Like bloom-176B loading, memory usage will maintain at a low level less than 10GB.

Comments:
You must be comparing two different codes as you are not asking to put the model on the GPU, so it's completely normal to see an increase on the CPU RAM. If using DeepSpeed, please also provide your DeepSpeed config as requested in the template.

@sgugger well, yes, I found that when loading bloom-176B, I use `with deepspeed.OnDevice(dtype=load_dtype, device="meta", enabled=True)` scope, and for LLAMA2, `enabled=False`.

So `low_cpu_memory_usage=True` won't decrease memory to less than 1x model size and just avoid more usage than that? Maybe the acutual solution is to add meta `deepspeed.OnDevice` support for LLAMA2? 

Or can I use `device_map` arguments to directly load it to GPU?

Yes, if you load it directly on the GPU you will avoid using CPU RAM.

> Or can I use `device_map` arguments to directly load it to GPU?

Well, I think at the time of `from_pretrained`, the model won't be splitted. And 70*2=140GB still cannot fit into one GPU card. So I must either load it into CPU memory (use 1T memory machine) and then let deepspeed split the model and move into GPU, or I must enable `deepspeed.OnDevice(device="meta")` support. Am I right @sgugger ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> > Or can I use `device_map` arguments to directly load it to GPU?
> 
> Well, I think at the time of `from_pretrained`, the model won't be splitted. And 70*2=140GB still cannot fit into one GPU card. So I must either load it into CPU memory (use 1T memory machine) and then let deepspeed split the model and move into GPU, or I must enable `deepspeed.OnDevice(device="meta")` support. Am I right @sgugger ?

@dc3671 
Hello, did you load the 70B model successfully?
When I load 70B model on 8 GPU, the required CPU memory > 1T, so cannot be load successfully.