Initialization of pytorch_util's Conv1D takes long time regardless of init_empty_weights when loading pretrained gpt2
Author: twaka
Pull request: no

### System Info

- `transformers` version: 4.26.1
- `accelerate` version: 0.16.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

gpt2-xl.py
```py
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
        "gpt2-xl",
        torch_dtype=torch.half,
        low_cpu_mem_usage=True)
```

```
$ python -m cProfile -s tottime gpt2-xl.py | head
         1264809 function calls (1209396 primitive calls) in 24.274 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      194   19.016    0.098   19.016    0.098 {method 'normal_' of 'torch._C._TensorBase' objects}
     1256    2.083    0.002    2.083    0.002 {method '_set_from_file' of 'torch._C.StorageBase' objects}
        2    0.684    0.342    0.684    0.342 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        2    0.355    0.178    0.355    0.178 {method 'read' of '_ssl._SSLSocket' objects}
```

https://github.com/huggingface/transformers/blob/b29e2dcaff114762e65eaea739ba1076fc5d1c84/src/transformers/pytorch_utils.py#L105-L110

`w` is constructed in `device: cpu` and actually computes `normal_`.
It's problematic when loading pretrained gpt2 models with larger number of parameters.

### Expected behavior

https://github.com/huggingface/transformers/blob/b29e2dcaff114762e65eaea739ba1076fc5d1c84/src/transformers/modeling_utils.py#L2491-L2492
- by change this line to `init_contexts.append(init_empty_weights(include_buffers=True))`
- `w` will be constructed in `device: meta` according to https://github.com/huggingface/accelerate/pull/699
- as a result, actual computation of `normal_` will be skipped and faster model loading time

```
$ python -m cProfile -s time gpt2-xl.py | head
         1265651 function calls (1210238 primitive calls) in 4.692 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1256    1.766    0.001    1.766    0.001 {method '_set_from_file' of 'torch._C.StorageBase' objects}
        2    0.684    0.342    0.684    0.342 {method 'do_handshake' of '_ssl._SSLSocket' objects}
        2    0.357    0.178    0.357    0.178 {method 'read' of '_ssl._SSLSocket' objects}
        2    0.342    0.171    0.342    0.171 {method 'connect' of '_socket.socket' objects}
```

Though, I don't know whether it's safe to set `include_buffers=True` for all models.

Comments:
It would be easier to move the initialization after initializing the parameter (so doing `self.weight = nn.Parameter(torch.empty(nx, nf))` and then apply the init normal. Would you like to make a PR with this change?

Even better, the initialization should be completely left to the `_init_weights` method of the PreTrainedModel using Conv1D and not present in this class at all, but it is a bit more work.

Thank you for your suggestion for reordering of initialization.
It makes sense to me. I'll make a PR soon.