Fix state_dict map location when quantized
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Fix following https://github.com/huggingface/transformers/pull/35926. If we now allow to load on meta with quantization as well, we need to be coherent on both sides


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hey @Cyrilvallez, this PR caused on error with HQQ (0.2.5) for me:

```python
import torch
from transformers import AutoModelForCausalLM, HqqConfig

device = "cuda"
causal_lm_model_id = "trl-internal-testing/tiny-random-LlamaForCausalLM"

quant_config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    causal_lm_model_id,
    device_map=device,
    quantization_config=quant_config,
)
```

With the previous transformers commit (`d6064754ea12661d2e60417fc9598dbc117c7712`), this passes, but after this one, I get the following error:

```
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/work/forks/transformers/src/transformers/models/auto/auto_factory.py", line 573, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/work/forks/transformers/src/transformers/modeling_utils.py", line 273, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/work/forks/transformers/src/transformers/modeling_utils.py", line 4503, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/work/forks/transformers/src/transformers/modeling_utils.py", line 4931, in _load_pretrained_model
    disk_offload_index, cpu_offload_index = _load_state_dict_into_meta_model(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/anaconda3/envs/peft/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/work/forks/transformers/src/transformers/modeling_utils.py", line 873, in _load_state_dict_into_meta_model
    hf_quantizer.create_quantized_param(
  File "/home/name/work/forks/transformers/src/transformers/quantizers/quantizer_hqq.py", line 285, in create_quantized_param
    module = module.to(dtype=self.torch_dtype, device=target_device)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/name/anaconda3/envs/peft/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1343, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/name/anaconda3/envs/peft/lib/python3.12/site-packages/torch/nn/modules/module.py", line 930, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/home/name/anaconda3/envs/peft/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1336, in convert
    raise NotImplementedError(
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

```

Could you please check?

Same issue as @BenjaminBossan 

Looking into it!

Fix in https://github.com/huggingface/transformers/pull/37133

Fixing it in https://github.com/huggingface/transformers/pull/37136 in the end -- standardization for quantizers (to avoid using the full state_dict) will follow later thanks to @MekkCyber who offered to look into it! 🤗