Failed to load santacoder model with transformer 4.51.3, it's a similar issue like #37737
Author: nv-guomingz
Pull request: no
Labels: name: bug


### System Info

Hi, I just installed transformers 4.51.3 on linux system with nvidia gpu.

When I tried to run below code snippet 
```python
import transformers

model = transformers.AutoModelForCausalLM.from_pretrained('bigcode/santacoder', device_map='auto', trust_remote_code=True)
```

Meanwhile, I set below envs
```bash
export WORLD_SIZE=1
export RANK=0
export LOCAL_RANK=0
export MASTER_PORT=63563
export MASTER_ADDR=`your localhost` 
```

I got below error message:
```bash
[rank0]:   File "/home/guomingz/.local/lib/python3.12/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/guomingz/.local/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4401, in from_pretrained
[rank0]:     ) = cls._load_pretrained_model(
[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/guomingz/.local/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4830, in _load_pretrained_model
[rank0]:     disk_offload_index, cpu_offload_index = _load_state_dict_into_meta_model(
[rank0]:                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/guomingz/.local/lib/python3.12/site-packages/transformers/modeling_utils.py", line 777, in _load_state_dict_into_meta_model
[rank0]:     shard_and_distribute_module(
[rank0]:   File "/home/guomingz/.local/lib/python3.12/site-packages/transformers/integrations/tensor_parallel.py", line 669, in shard_and_distribute_module
[rank0]:     param = torch.nn.Parameter(param)
[rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/parameter.py", line 46, in __new__
[rank0]:     return torch.Tensor._make_subclass(cls, data, requires_grad)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: Only Tensors of floating point and complex dtype can require gradients
[rank0]:[W424 16:00:01.852231233 ProcessGroupNCCL.cpp:1497] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
```

If I unset **WORLD_SIZE**, the above code snippet could run succesfully.

It looks like one similar issue like #37737




### Who can help?
@Cyrilvallez 

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install transformers 4.51.3 via `pip install transformers`
2. Run below code snippet
`import transformers

model = transformers.AutoModelForCausalLM.from_pretrained('bigcode/starcoder', device_map='auto')`

### Expected behavior

Load model sucessfully.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install transformers 4.51.3 via pip install transformers
2. Run below code snippet

```python
import transformers
model = transformers.AutoModelForCausalLM.from_pretrained('openai-community/gpt2-medium', device_map='auto', trust_remote_code=True)
```

### Expected behavior

Load model sucessfully.

Comments:
Hi @Cyrilvallez ,could u plz review this fixing PR? https://github.com/huggingface/transformers/pull/37767