Natively support SONAR text models as M2M100 encoder and decoder models
Author: avidale
Pull request: yes

# What does this PR do?

This PR adds native support for SONAR text encoders are decoders (https://github.com/facebookresearch/SONAR).

SONAR for text is architecturally an NLLB model, but with the encoder representations mean-pooled into a single fixed-sized vector before passing them to the decoder. Thus, SONAR encoder works as a sentence embedder, and thanks to pretraining on translation data, it is massively multilingual and language-agnostic. And, unlike other sentence encoder, this one has a decoder that can reconstruct the original texts back from their embeddings.

To supports such models natively, the easiest way would be to create ~~NLLB~~ M2M100 model classes with encoder only or decoder only, similarly to the existing classes `T5EncoderModel` or `MT5EncoderModel`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

# Details 

- I add two new public model classes, `M2M100EncoderModel` and `M2M100DecoderModel`. 
- `M2M100EncoderModel` is a typical encoder-only model (BERT-style), with an additional option of applying mean pooling to its outputs (this is how SONAR text embeddings are computed)
- `M2M100DecoderModel` is a module consisting of M2M100Decoder and an output projection layer. As the input, it always expects the `encoder_outputs` argument to be present, and ignores its `input_ids`. 
- Unlike `M2M100ForConditionalGeneration`,  `M2M100DecoderModel` always has its decoder input embedding and decoder output projection layers tied, because this is how SONAR decoder originally was implemented.
- I add a script for creating these models from the original `fairseq2` checkpoints  (it doesn't require `fairseq2` as a dependency; instead, it just reads and reformats the torch model state dicts).
- I add specialized unit tests for the encoder-only model (implemented following `T5EncoderOnlyModelTest`, see https://github.com/huggingface/transformers/pull/8717), and for the decoder-only model (based loosely on similar ideas, but with more tweaks). 
- I add an integration test based on the checkpoints that I published to the HF hub. They reproduce the example sentence encoding and decoding from the readme in the SONAR repo: https://github.com/facebookresearch/SONAR/tree/main.

# Testing

All the unit tests I added are run by 
```
python -m pytest tests/models/m2m_100/test_modeling_m2m_100.py
```

The integration tests that I added are marked as slow, so they could be run with

```RUN_SLOW=1 python -m pytest tests/models/m2m_100/test_modeling_m2m_100.py::SonarIntegrationTests```

Comments:
Hi @avidale, thanks for opening this PR! Let us know when it's ready for review 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey, I want to continue this work!
How to I reopen it?

@avidale I can reopen for you

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29646). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Commenting to avoid it getting closed


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still planning to return to it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @amyeroberts!
The PR is mostly ready for the review.
I am still uncertain about several things, but I would appreciate if you commented on the current design!

One of the main questions: is it alright to always demand `encoder_outputs` in the `M2M100DecoderModel`, or a different presentation of its inputs is preferrable?

Also tagging @ArthurZucker as a potential reviewer

Tagging @ArthurZucker, @younesbelkada, and @amyeroberts once more, as potential reviewers.
Could you please give some feedback?

Yes! Sorry it has been on my stack for a while, and summer vacations hit! 

@ArthurZucker any other comments?
What do you think about my responses?

Hi @ArthurZucker!
I have responded to your last comments. 
Some tests are failing, but they all seem to be unrelated to my code. 

Will review ASAP

@ArthurZucker do you think this PR has a chance?

What exactly am I supposed to do with `modular` approach here?
My guess is that you suggest creating a "modular" `sonar` directory that imports some code from `m2m100`, instead of adding more code to `m2m100`. Is it correct?

Yes @avidale ! This should make it easier to isolate the changes and iterate!

@avidale do you plan to update this, would really appreciate using this

Hi @danikhan632!
This PR grew very stale; I think it might need a complete rewriting from scratch. 
I might return to it nearer to the end of the year, but if you are willing to revive it sooner, I'd be happy to hand it over to you!