low_cpu_mem_usage would throw error when using newly defined model
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.38.0
- Platform: Linux-5.15.0-1050-azure-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.21.1
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@SunMarc @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Thanks so much for the convenient `low_cpu_mem_usage` option for loading super large models. However, I found a problem when using this flag with a newly defined model, here is a short reproducible snippet:
```python
from transformers import LlamaForCausalLM
import torch.nn as nn

class NewModel(LlamaForCausalLM):
    def __init__(self,config):
        super().__init__(config)
        self.new_module = nn.Linear(config.hidden_size,config.hidden_size)

model = NewModel.from_pretrained("meta-llama/llama-2-7b-hf",low_cpu_mem_usage=True)
model = model.to("cuda")
```
This snippet would throw the following error:
```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/azureuser/miniconda3/envs/xrag/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2556, in to
    return super().to(*args, **kwargs)
  File "/home/azureuser/miniconda3/envs/xrag/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1160, in to
    return self._apply(convert)
  File "/home/azureuser/miniconda3/envs/xrag/lib/python3.9/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  File "/home/azureuser/miniconda3/envs/xrag/lib/python3.9/site-packages/torch/nn/modules/module.py", line 833, in _apply
    param_applied = fn(param)
  File "/home/azureuser/miniconda3/envs/xrag/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1158, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
```
After removing `low_cpu_mem_usage`, it works fine. 

So I am asking if this is a potential bug or the current transformers library doesn't support newly define model with `low_cpu_mem_usage=True`? If the latter, could  you please recommend some workarounds? Thanks! 

### Expected behavior

Successfully loading the model.

Comments:
Hi @Hannibal046 
Thaks for the issue ! This because in the `"meta-llama/llama-2-7b-hf"` checkpoint the state dict does not contain `new_module` weights. What you could do is to push a checkpoint that contain the llama 2 weights and the `new_modules` weights and do something like the following: 
```python
from transformers import LlamaForCausalLM, AutoConfig
import torch
import torch.nn as nn

class NewModel(LlamaForCausalLM):
    def __init__(self,config):
        super().__init__(config)
        self.new_module = nn.Linear(config.hidden_size,config.hidden_size)

config = AutoConfig.from_pretrained("meta-llama/llama-2-7b-hf")

with torch.device("meta"):
    model = NewModel(config)

state_dict = xxxx # here comes your loaded state dict
model.load_state_dict(strict=True, assign=True)
model = model.to("cuda")
```

This snippet would have the same effect as `low_cpu_mem_usage` as the layers won't get initialized 

Nice solution, greatly appreciate!