Add Seamless M4T model
Author: ylacombe
Pull request: yes

# What does this PR do?

Meta recently introduced [Seamless M4T](https://ai.meta.com/blog/seamless-m4t/),  a collection of models designed to provide high quality translation, allowing people from different linguistic communities to communicate effortlessly through speech and text. 

SeamlessM4T supports multiple audio and/or translation tasks, namely S2TT, S2ST, T2TT, T2ST, where the last T stands for translation. 
In other words, this model _seamlessly_ supports audio|text to translated audio|text.

SeamlessM4T weights are already available on the hub ([large](https://huggingface.co/facebook/seamless-m4t-large) and [medium](https://huggingface.co/facebook/seamless-m4t-medium)) and the code is available on the [seamless_communication git repo](https://github.com/facebookresearch/seamless_communication).

In terms of architecture, and after having discussed with @sanchit-gandhi, I've came up with 4 differents models for the 4 tasks and one model that can do each task.

I've been working on the integration for a couple of days already. At the moment, the converting script is more or less ready and the different models can generate.

Here is a TODO of what's left to be done:

- [x] Agree on the current architecture and some modeling details (for example what outputs)
- [x] integrate feature extraction (fbank) and tokenizer (similar to NLLB)
- [x] Integrate their vocoder
- [x] Write and format docstrings
- [x] Do integration tests - there's probably some modeling discrepancy at the moment (except for the speech encoder with a one-to-one correspondance)
- [x] Finish regular tests
- [x] There's probably some work to be done for the optimal generation config 

cc @sanchit-gandhi and @ArthurZucker !

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @sanchit-gandhi, thanks for your thorough review! I've addressed or answered almost every comment, except your request on the nested configuration. 

For the moment being, I'd rather use a formatting with clear delimitation, a bit like the one you can find on the [PretrainedConfig doc](https://huggingface.co/docs/transformers/v4.33.2/en/main_classes/configuration#transformers.PretrainedConfig). Here is how it looks like with my config: [docs](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25693/en/model_doc/seamless_m4t#transformers.SeamlessM4TConfig). I believe that it renders pretty well, and avoid the complexity of adding a nested config to an already pretty heavy PR. WDYT ?

I would really like @ArthurZucker point of view on that! This is also the opportunity to ask @ArthurZucker for a review!

EDIT: Rebased and modified on the new tokenizer PR (#23909)

Hi @ydshieh! 

It seems that the [documentation tests](https://github.com/huggingface/transformers/pull/26212) fail because it tries running `src/transformers/models/seamless_m4t/convert_fairseq2_to_hf.py`.
I saw that you addressed a similar issue in #26212, and I was wondering if that was a normal behavior ! I believe that precedent PRs didn't try running the converting script.

I'll update `utils/not_doctested.txt` if this is the expected fix!


Hi

Usually, if the script has `__main__`, it should be fine. But here is an import error, and put it under `utils/not_doctested.txt` is the way to go.

Hi @ydshieh, thanks for your help. The script had  `__main__` but was still throwing some errors, so I added it to `utils/not_doctested.txt`

Following #26182 merging, I've updated the FE, thus removing dependency to torchaudio. 

BTW, this is also a gentle reminder to review this PR when you have time @ArthurZucker :hugs: 

Hey @sanchit-gandhi , thanks for your review, I've corrected everything!

Thanks for the quick review, I'll address most your comments this morning.

I still have this pending questions if you have time:
- https://github.com/huggingface/transformers/pull/25693#discussion_r1350681175
- That one's for @ydshieh : https://github.com/huggingface/transformers/pull/25693#discussion_r1350682649



Hey @ArthurZucker, your suggestions made it even cleaner! I've also added some copied from statements! Thanks for that!

Do you think you can review a bit more thoroughly the modeling code soon ?

on it tomorrow! 

Thanks for the last review @ArthurZucker! 
Two last things to address before merging:
1.~~`SeamlessM4TConfig` formatting: https://github.com/huggingface/transformers/pull/25693#discussion_r1362184879 - cc @ydshieh , could you take a look at it?~~
2. ~~Assisted generation test: I can't seem to pass `SeamlessM4TModelWithTextInputTest::test_assisted_decoding_sample` and this might be out of my scope. Could you take a look at it @gante ? It's basically testing with SeamlessM4TForTextToText which is a basic seq2seq model!~~ [UPDATE: ignore test after offline discusions]