Add Speech-to-Speech Translation (S2ST)
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Adds the S2ST models from the paper [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (Popuri et al. 2022)](https://arxiv.org/abs/2204.02967).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
Pipeline:

![pipeline](https://user-images.githubusercontent.com/93869735/201135128-13ccac3b-ab96-43f3-b906-0ed5cfc56a7b.png)

This is effectively an encoder-decoder-vocoder configuration:

- The feature extractor normalises the audio inputs.
- The encoder maps the (normalised) audio inputs to a sequence of encoder hidden-states.
- The decoder auto-regressively generates a sequence of tokens (interpreted as speech ‘hidden units’).
- The vocoder maps these discrete tokens to a sequence of continuous audio outputs.

The encoder-decoder portion is a standard seq2seq mapping, entirely equivalent to the [speech encoder-decoder model](https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder#transformers.SpeechEncoderDecoderModel) we have in Transformers. The vocoder aspect is new.



Question 1 - Explicit modelling code?

The pre-trained checkpoints only use a Wav2Vec2 encoder and mBART decoder (see [enhanced_direct_s2st_discrete_units.md](https://github.com/facebookresearch/fairseq/blob/main/examples/speech_to_speech/docs/enhanced_direct_s2st_discrete_units.md#finetuned-model-checkpoints)). There are no other encoder or decoder checkpoints/architectures used. Thus, we have two options:

1. Explicitly add the modelling code for Wav2Vec2 and the mBART decoder to `modeling_speech_to_speech.py` -> there are no abstractions, all the relevant modelling code for the encoder and decoder is in one file
2. Follow what’s done in speech encoder-decoder model and add modelling code for a generic encoder and generic decoder -> there is one layer of abstraction, neither the modelling code for the encoder or decoder are in `modeling_speech_to_speech.py`, they are instead called through AutoModel:
https://github.com/huggingface/transformers/blob/6dda14dc47d82f0e32df05fea8ba6444ba52b90a/src/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py#L213-L217
-> this is quite nasty as it abstracts the encoder and decoder, when really they can be in the same file if we know that the encoder is always Wav2Vec2 and the decoder always mBART.

Option 1 is more clear to the user -> all the relevant code sits in the relevant modelling file. 

Option 2 is what we have for speech encoder decoder. It facilitates for different combinations of models should people wish to train different encoder-decoder combos themselves, but this is highly unlikely to ever happen! Option 2 gives one layer of abstraction that makes the code much harder to follow -> this is similar to what they do in fariseq, and it’s a struggle to jump around and find the right modelling files. 

We currently have option 2 in the PR, but my preference would be for 1 unless there are any objections.

Question 2 - Where should the vocoder go?

The model is trained to predict target tokens (speech ‘hidden units’). These target tokens are converted to continuous speech by action of a vocoder. This vocoder is not trained. It is loaded standalone to the seq2seq model after training.

![Untitled Diagram](https://user-images.githubusercontent.com/93869735/201135650-b8ba3002-2e3b-4527-ab72-e671366fff30.png)

Should the vocoder be included in the modelling file as part of the pre-trained model? Or should it operate in a similar way to a tokenizer (an object that isn’t trained, purely used to map generated tokens to the final output)?

My preference would be to include it in the modelling file as an `nn.Module`. If we go for option 1 from the previous question (explicitly adding the Wav2Vec2 and mBART code to modling_speech_to_speech), we would then have the following structure:

- Wav2Vec2 encoder

- mBART decoder

- CodeHiFiGAN vocoder

- SpeechToSpeechTranslationModel (Wav2Vec2 encoder - mBART decoder)

- SpeechToSpeechTranslationWithCodeHiFiGANVocoder (encoder-decoder-vocoder)

 -> this design treats the vocoder like a head to the base model


Question 3 - What about the configs?

Speech encoder decoder partitions its configuration into an encoder config and decoder config (see [speech-encoder-decoder-config](https://huggingface.co/docs/transformers/model_doc/speech-encoder-decoder#transformers.SpeechEncoderDecoderConfig)):

* Encoder config
* Decoder config

Do we do the same with the encoder-decoder-vocoder model:

* Encoder config
* Decoder config
* Vocoder config -> note: not used for the SpeechToUnitTranslationModel, only the SpeechToSpeechWithCodeHiFiGANVocoder. We’ll have to handle it differently in each case.

Or combine them into a single config file for all the modelling components (as is done with [T5Config](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Config) for example, the encoder and decoder parts of the config are prefixed by `encoder_` and `decoder_` respectively).

My preference would be for combining them into a single config and prefixing by `encoder_`, `decoder_` and `vocoder_` -> I think this is cleaner than having three different sub-configs on the go.

Question 4 - How to make compatible with generation?

This model currently isn’t a good fit for Transformers with regards to generation: we want to auto-regressively generate using the decoder and then pass the generation outputs through **another** stage of the model (vocoder) -> this currently isn't possible with `.generate` alone. We either have to add this functionality to generate, or override the generate method for SpeechToSpeechWithCodeHiFiGANVocoder.

We're currently doing something very hacky to make this work:
https://github.com/huggingface/transformers/blob/43b744283b57d156845af0eda77b806b88957395/src/transformers/models/speech_to_speech/modeling_speech_to_speech.py#L1012-L1022

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.