fix "Cannot copy out of meta tensor; no data!" issue for BartForConditionalGeneration model
Author: yao-matrix
Pull request: yes

fix issue: https://github.com/huggingface/transformers/issues/36247

cause of this issue:

[bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn/tree/main)  checkpoint only has `model.decoder.embed_tokens.weight` while `load_state_dict`, but in BartModel implementation, both `model.decoder.embed_tokens.weight` and `model.encoder.embed_tokens.weight` tie to the `shared.weights` which doesn't have corresponding real weights in checkpoints, so when `tie_weights` are called, things are back to a void `shared.weights` which in `meta` device, then lead to the model.to issue

@SunMarc @Rocketknight1 , I don't know whether this PR meet your standard since it changed modeling code, pls let me know your thoughts, thx.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

hi @yao-matrix, this causes several other BART tests to fail! You can see the failures in the CI at the bottom of this post.

Some of the failures are just code style - these are simple and can be fixed with `make style` or `make fixup` after `pip install transformers[quality]`. However, the failures in `tests_torch` indicate that this PR breaks BART weight loading, probably because the old `shared` tensor doesn't match up with the architecture after your PR. We won't be able to merge the PR unless those errors are resolved, and I'm not sure that will be possible while `self.shared` is removed.

@Rocketknight1 @SunMarc , Marc's method is better, I used it to fix the issue, pls help review and comment, thx.

@ArthurZucker , pls take a look, thx.

Yes, this is relatively small but it might affect the transformers philosophy a bit, so cc @cyrilvallez @ArthurZucker!

the ci failure is not introduced by my PR

@Cyrilvallez, could you pls help review and comment? Thx.

Hey! Thanks for opening the PR. The proposed change is only one part of the solution for the given checkpoint 🤗 You will need to add the below method to `BartForConditionalGeneration` as well, as the `lm_head` is supposed to be tied to `model.shared`, but `model.shared` is saved as `model.decoder.embed_tokens` for this checkpoint.

```python
def _tie_weights(self):
    if self.config.tie_word_embeddings:
        # Some model checkpoints like "facebook/bart-large-cnn"'s embedding weight is in decoder.embed_tokens, need check here, see issue #36247
        if self.model.shared.weight.device == torch.device(
            "meta"
        ) and self.model.decoder.embed_tokens.weight.device != torch.device("meta"):
            self._tie_or_clone_weights(self.lm_head, self.model.decoder.embed_tokens)
        else:
            self._tie_or_clone_weights(self.lm_head, self.model.shared)
```

However, we really dislike changing modeling code because of a single faulty checkpoint on the hub. Are you aware of any other checkpoints with the same issue? 🤔 It is fine for now, but if it is only this one, I will open a PR directly on the Hub, then revert those changes 🤗


> Hey! Thanks for opening the PR. The proposed change is only one part of the solution for the given checkpoint 🤗 You will need to add the below method to `BartForConditionalGeneration` as well, as the `lm_head` is supposed to be tied to `model.shared`, but `model.shared` is saved as `model.decoder.embed_tokens` for this checkpoint.
> 
> ```python
> def _tie_weights(self):
>     if self.config.tie_word_embeddings:
>         # Some model checkpoints like "facebook/bart-large-cnn"'s embedding weight is in decoder.embed_tokens, need check here, see issue #36247
>         if self.model.shared.weight.device == torch.device(
>             "meta"
>         ) and self.model.decoder.embed_tokens.weight.device != torch.device("meta"):
>             self._tie_or_clone_weights(self.lm_head, self.model.decoder.embed_tokens)
>         else:
>             self._tie_or_clone_weights(self.lm_head, self.model.shared)
> ```
> 
> However, we really dislike changing modeling code because of a single faulty checkpoint on the hub. Are you aware of any other checkpoints with the same issue? 🤔 It is fine for now, but if it is only this one, I will open a PR directly on the Hub, then revert those changes 🤗

@Cyrilvallez I updated the code per your comments. And I don't like these changes either. If you can change the model checkpoint to make it work, it's great! 

All right, thanks! 🤗 Merging for now as it's blocking and not a large change! However, I will still open a Hub PR immediately to fix the faulty weights. If they answer and merge it, I will revert those changes!

In the meantime, if you know of any other similar checkpoints with the same issue, let me know! If it's too many, we might need to keep the modeling as it is!

Hub PR is here https://huggingface.co/facebook/bart-large-cnn/discussions/89