NotImplementedError: Cannot copy out of meta tensor; no data when embedding to meta
Author: DonggeunYu
Pull request: no

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.2
- Accelerate version: 0.31.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. To send [query_position_embeddings](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deformable_detr/modeling_deformable_detr.py#L1544) to meta, modify it as follows: `max_memory = {0: max_size // 2, "cpu": max_size * 2}`
https://github.com/huggingface/transformers/blob/main/tests/test_modeling_common.py#L30852
2. Run `python3 -m pytest tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_disk_offload_safetensors`


### Expected behavior

~~~
FAILED tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_disk_offload_safetensors - NotImplementedError: Cannot copy out of meta tensor; no data!
~~~

Comments:
Hi @DonggeunYu, thanks for reporting! 

We'll look into it. Out of interest, how did you discover this? Was is modifying the tests, or are the tests just an easy way to demonstrate this behaviour? 

Easy way to demonstrate this behavior.
While using a private model, I discovered that there was a problem with nn.Embedding.

I may be wrong, as I still need to understand the transformers and accelerate code fully.
When offload is used, it becomes a meta device during the init process. The weight of nn.Embedding created in `__init__` becomes the meta device. If i use `nn.Embedding` callable, the `pre_forward` hook of accelerate will match the device of args, kwrags, and embedding.

However, because the embedding weight is inserted into the forward of another module, it enters the pre_forward hook as a meta device.
To prove this, the log that the `pre_forward` of accelerate hook.py outputs the module device and the device of the args.
Until the `nn.embedding` weight in the problem, the module device is meta device, and the args device is cuda. If the `nn.embedding` weight in the problem enters another module's args, the module device is meta and the args device is meta (embedding weight). An error occurs when performing meta to cuda using `send_to_device (args, self.execution_device)`.

~~~
module.__class__.__name__, device of module, device of args
Linear [device(type='meta')] [device(type='cuda', index=0)]
LayerNorm [device(type='meta')] [device(type='cuda', index=0)]
Linear [device(type='meta')] [device(type='cuda', index=0)]
Linear [device(type='meta')] [device(type='cuda', index=0)]
LayerNorm [device(type='meta')] [device(type='cuda', index=0)]
Linear [device(type='meta')] [device(type='meta')]
~~~

[def pre_forward of accelerate](https://github.com/huggingface/accelerate/blob/main/src/accelerate/hooks.py#L322)
[nn.Embedding of transformers](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deformable_detr/modeling_deformable_detr.py#L1544)

@DonggeunYu Thanks for the update. Indeed, the structure of using the embedding weights rather than the layer in the forward pass is quite odd. cc @muellerzr who knows more about the `pre_forward` hook of accelerate

@amyeroberts @muellerz 
How is the progress?

cc @SunMarc 

@SunMarc
I would appreciate it if you could check the issue.

I would appreciate it if you could check the issue. 🥲

Sorry for the delay @DonggeunYu ! As you saw from your investigation, the issue is that we are using the weight of the embedding layer that is offloaded to cpu, meaning that the weights are on meta device. To fix the issue, we would need to use the layer instead of just getting the weights. Another solution would be to make sure that this specific layer stays on the cuda. 

@SunMarc 
Is there any solution?
Is it possible to keep certain layers in cuda when using accelerate offload?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.