Persist embedding type of BART and mBART models after resize
Author: AbdiHaryadi
Pull request: yes

# What does this PR do?
This pull request changes `BartModel.shared` and `MBartModel.shared` property initialization from `torch.nn.Embeddings` to custom scaled embedding type. This change persists embedding type after resize (with `resize_token_embeddings` method), especially for `BartForConditionalGeneration` and `MBartForConditionalGeneration`.

Fixes #31835 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?
@zucchini-nlp 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32242). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Great, thanks for fixing and adding a test!
> 
> Wondering if you're willing to work on other models with the same problem (Bart, SeamlessM4TForTextToText, SeamlessM4TForTextToSpeech, SeamlessM4TModel and same models for Seamless-v2). It's okey if you don't have bandwidth, I'll put it in my TODO list :)

Your welcome~ ;)

For other models, well because it's still related, okay, let me check and do it for one or two weeks. :)
I'll make a comment if there's any issue.

Wait, a little bit of warning for this fix.

Because this pull request can change how to calculate the embedding, __all old, resized MBART models with scale embedding can give different result in inference__. The default `scale_embedding` value is `True`, but the implementation before this change and after introducing `MBartScaledWordEmbeddings` is always not using scale embedding, no matter what the value of `scale_embedding` is, if the model has been resized (or using resize method) before training.

@AbdiHaryadi I tracked that this was introduced in previous PR that added `ScaledEmbedding` to models, and all models before that were scaled by default. So imo this PR shouldn't be breaking, but returning the expected behavior we had earlier.

> > Great, thanks for fixing and adding a test!
> > Wondering if you're willing to work on other models with the same problem (Bart, SeamlessM4TForTextToText, SeamlessM4TForTextToSpeech, SeamlessM4TModel and same models for Seamless-v2). It's okey if you don't have bandwidth, I'll put it in my TODO list :)
> 
> Your welcome~ ;)
> 
> For other models, well because it's still related, okay, let me check and do it for one or two weeks. :) I'll make a comment if there's any issue.

@zucchini-nlp,
I've checked BART and SeamlessM4T (including v2) models. BART seems have an issue like mBART, so I can fix it. However, there is no issue in SeamlessM4T models because it can pass the same embedding type persistence test (like I did it in mBART). It has been fixed, but with different method.

As a comparison, here is the implementation of [`BartModel`](https://github.com/huggingface/transformers/blob/v4.43.3/src/transformers/models/bart/modeling_bart.py#L1427) (and the same as mBART, before fix):
```python
class BartModel(BartPreTrainedModel):
    ...

    def __init__(self, config: BartConfig):
        super().__init__(config)

        padding_idx, vocab_size = config.pad_token_id, config.vocab_size
        self.shared = nn.Embedding(vocab_size, config.d_model, padding_idx)

        self.encoder = BartEncoder(config, self.shared)
        self.decoder = BartDecoder(config, self.shared)

        ...

    ...

    def get_input_embeddings(self):
        return self.shared

    def set_input_embeddings(self, value):
        self.shared = value
        self.encoder.embed_tokens = self.shared
        self.decoder.embed_tokens = self.shared

    ...
```

Here is the implementation of [`SeamlessM4TModel`](https://github.com/huggingface/transformers/blob/v4.43.3/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py#L3952) (and the same as v2 variation):
```python
class SeamlessM4TModel(SeamlessM4TPreTrainedModel):
    ...

    def __init__(self, config, current_modality="text"):
        super().__init__(config)

        self.shared = nn.Embedding(config.vocab_size, config.hidden_size, config.pad_token_id)

        self.text_encoder = SeamlessM4TEncoder(config, self.shared)
        self.speech_encoder = SeamlessM4TSpeechEncoder(config)
        self.text_decoder = SeamlessM4TDecoder(config, self.shared)
        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

        ...

    ...

    def get_input_embeddings(self):
        return self.text_decoder.embed_tokens

    def set_input_embeddings(self, value):
        self.text_encoder.embed_tokens = value
        self.text_decoder.embed_tokens = value
        self.shared = value

    ...
```

The main difference is in the `get_input_embeddings` method. In BART model, it retrieves from `self.shared` attribute which has standard embedding type, so resizing it changes the embedding type with that implementation. In SeamlessM4T model, it retrieves directly from `self.text_decoder.embed_tokens` which has custom embedding type (see [here](https://github.com/huggingface/transformers/blob/v4.43.3/src/transformers/models/seamless_m4t/modeling_seamless_m4t.py#L1833-L1835)). So, I think SeamlessM4T models don't need to be fixed in this case.

Because of that founding, it leaves two alternatives for fixing BART and mBART:
1. Change with the same implementation as before (change `self.shared` initialization type). The downside here is the implementation is slightly inconsistent with SeamlessM4T.
2. Change with the same implementation as SeamlessM4T did.

Any recommendation which alternative should I choose, and is it okay to keep SeamlessM4T unchanged?

Right, thanks for digging into this! Didn't notice that. 

Anyone who tries to `get_input_embedding` should get the scaled embedding layer since people usually assume that passing ids through `get_input_embedding` and feeding into model is same as passing ids directly. So either way should be okay in theory, but I prefer the option-1 which is more explicit and doesn't raise question at first glance about embedding types. 

We can modify Bart and MBart models only, and leave Seamless as is because it behaves as expected as you pointed out

@zucchini-nlp,
I think this pull request is complete. Feel free to tell me if there is something that I missed in this pull request.

@jswebguru Seamless has no problem when resizing the embedding as the `get_input_embeddings` return Scaled type and sets it for all embeddings in `set_input_embeddings`