Detr Models cannot be loaded with `device_map="auto"`
Author: chiragjn
Pull request: no
Labels: name: Good Difficult Issue

### System Info

- `transformers` version: 4.28.1
- Platform: macOS-13.1-x86_64-i386-64bit
- Python version: 3.9.2
- Huggingface_hub version: 0.12.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import pipeline

p = pipeline(
    "object-detection", 
    model="facebook/detr-resnet-50", 
    image_processor="facebook/detr-resnet-50", 
    device_map="auto"
)
```

### Expected behavior

This does not work because the `transformers.models.detr.modeling_detr.DetrConvEncoder` model init involves copy weights from `nn.BatchNorm2d` to `DetrFrozenBatchNorm2d` which is not allowed when on a meta device.

```
 File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/pipelines/__init__.py", line 779, in pipeline
    framework, model = infer_framework_load_model(
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/pipelines/base.py", line 262, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 471, in from_pretrained
    return model_class.from_pretrained(
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/detr/modeling_detr.py", line 1373, in __init__
    self.model = DetrModel(config)
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/detr/modeling_detr.py", line 1205, in __init__
    backbone = DetrConvEncoder(config)
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/detr/modeling_detr.py", line 354, in __init__
    replace_batch_norm(backbone)
  File "/Users/chiragjn/venv39/lib/python3.9/site-packages/transformers/models/detr/modeling_detr.py", line 314, in replace_batch_norm
    frozen.weight.data.copy_(bn.weight)
NotImplementedError: Cannot copy out of meta tensor; no data!
```

The model loads fine with a specific device with `device` argument. 

Comments:
cc @alaradirik and @amyeroberts 

Hi @chiragjn, I was able to replicate the error on my local (also macOS-13.1-x86_64-i386-64bit) and I'm looking into the issue.

A quick update - I tracked down the issue to the accelerate library, setting `device_map=True` sets `low_cpu_mem_usage` to True. This causes the model parameters to be initialized as meta tensors, which can not be copied to CPU or GPU without tensor conversion.

This issue also affects DETA, Conditional DETR, Deformable DETR and Table Transformers as they have identical frozen modules that are initialized by copying the parameters of their respective backbone models. We will be opening a fix PR shortly!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey, is there any progress with this issue?

Hi @AlonZolfi, @alaradirik has now left Hugging Face, so I'm picking this up. 

As @alaradirik mentions, this arises as a consequence the replacement of the batch norm in the backbone of these models. I'll be digging into it properly next week when I have a bit more time. 

Re-opening the issue as it's not yet solved and will keep you posted! 

It closed again, there was some progress with the issue?

@amyeroberts the problem is indeed annoying, I have similar problem fine-tuning some models like llama.  anyone working to solve it?

Hey @amyeroberts, was this issue solved already?

@AlonZolfi @ranchlai No, I unfortunately haven't had bandwidth to address this yet. I'm marking it as a difficult issue that anyone in the community can try and tackle if they wish. 

I got the same problem when using accelerate, doing `model.cuda()` worked as expected. 
The related PR is #26150
where:
```python
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained(model_name, device_map="auto")
```

so pinging @muellerzr as this is probably related to our hf hooks. Now I might be creating the buffer and tensors in a wrong way but can´t get it to load so help is appreciated! (See the `UMTRelativePositionalBias` class)  

(using accelerate 0.22.0)

cc @SunMarc 

Hi @ArthurZucker , I left a few comments on the PR to explain the issue. Hope that you have enough context to fix the problem ;)