device_map=auto fails to load the dequantized model on gpu+cpu offload
Author: IlyasMoutawwakil
Pull request: no
Labels: name: bug

### System Info

On a A100 using transformers from main and latest torch

### Who can help?

@Cyrilvallez 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

on an A100 or any single gpu with insufficient memory to load the dequantized gpt_oss:

```python
from transformers import AutoModelForCausalLM, Mxfp4Config


model_id = "openai/gpt-oss-120b"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype="auto",
    use_kernels=False,
    device_map="auto",
    quantization_config=Mxfp4Config(dequantize=True),
).eval()
```

for example on the dgx `CUDA_VISIBLE_DEVICES=1 python script.py`

```
(.transformers) (base) ilyas@hf-dgx-01:~/transformers$ CUDA_VISIBLE_DEVICES=1 python baddbmm_vs_bmm.py 
Loading weights: 100%|███████████████████████████████████████████████████████████| 615/615 [00:06<00:00, 92.59it/s, Materializing param=model.norm.weight]
Traceback (most recent call last):
  File "/home/ilyas/transformers/baddbmm_vs_bmm.py", line 5, in <module>
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ilyas/transformers/src/transformers/models/auto/auto_factory.py", line 372, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ilyas/transformers/src/transformers/modeling_utils.py", line 4042, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, offload_index, error_msgs = cls._load_pretrained_model(
                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ilyas/transformers/src/transformers/modeling_utils.py", line 4212, in _load_pretrained_model
    model._move_missing_keys_from_meta_to_device(missing_and_mismatched, device_map, device_mesh, hf_quantizer)
  File "/home/ilyas/transformers/src/transformers/modeling_utils.py", line 4456, in _move_missing_keys_from_meta_to_device
    value = torch.empty_like(param, device=param_device)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ilyas/transformers/.transformers/lib/python3.12/site-packages/torch/_prims_common/wrappers.py", line 309, in _fn
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/ilyas/transformers/.transformers/lib/python3.12/site-packages/torch/_refs/__init__.py", line 5113, in empty_like
    return torch.empty_permuted(
           ^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.96 GiB. GPU 0 has a total capacity of 79.25 GiB of which 84.62 MiB is free. Including non-PyTorch memory, this process has 79.16 GiB memory in use. Of the allocated memory 77.45 GiB is allocated by PyTorch, and 1.23 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
```

### Expected behavior

To load successfully.

Comments: