Add FastSpeech2Conformer
Author: connor-henderson
Pull request: yes

# What does this PR do?

Adds the TTS (text-to-speech) conformer version of the FastSpeech2 model. Closest related issue is [#15166](https://github.com/huggingface/transformers/issues/15166) though this implements ESPnet's conformer version instead of Fairseq's version as suggested in https://github.com/huggingface/transformers/pull/15773#issuecomment-1529558164.

[FastSpeech2 paper (Microsoft)](https://arxiv.org/pdf/2006.04558.pdf)
[Conformer version paper (ESPnet)](https://arxiv.org/pdf/2010.13956.pdf)

Conformer version code implementation: https://github.com/espnet/espnet/tree/master/espnet2/tts/fastspeech2
Additional conformer version code code implementation: https://github.com/DigitalPhonetics/IMS-Toucan/blob/ToucanTTS/TrainingInterfaces/Text_to_Spectrogram/FastSpeech2

The paper abstracts say most of this, but the main points of what makes this model an interesting addition are:
- It's non auto-regressive, leading to faster inference since it doesn't have to make predictions sequentially (hence the name `FastSpeech`)
- Uses a variance predictor in between the encoder and decoder to explicitly predict duration, pitch, and energy, leading to more accurate results
- Conformer architectures have been shown to improve performance in text to speech tasks, with the convolutions learning close range speech patterns and transformer attention helping with understanding longer range contexts
- There is currently only one other text-to-speech model in `transformers` (`SpeechT5ForTextToSpeech`)


## To do
- [x] Prepared 🤗 Transformers dev environment
- [x] Set up debugging environment of the original repository
- [x] Created script that successfully runs the forward() pass using the original repository and checkpoint
- [x] Successfully added the model skeleton to 🤗 Transformers (+ vocoder)
- [x] Successfully converted original checkpoint to 🤗 Transformers checkpoint (+ vocoder)
- [x] Successfully ran forward() pass in 🤗 Transformers that gives identical output to original checkpoint (+ vocoder)
- [x] Finished model tests in 🤗 Transformers
- [x] Successfully added tokenizer in 🤗 Transformers
- [x] Run end-to-end integration tests
- [x] Finished docs
- [x] Uploaded model weights to the Hub (will ask they're moved to just `fastspeech2_conformer` when ready)

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->
@hollance @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23439). All of your documentation changes will be reflected on that endpoint.

Thanks for the review @hollance! Addressed the comments above, the only part that might need follow-up discussion is making the `labels` compatible with the `Trainer`
>Re labels, FastSpeech2 is somewhat unique in that it takes in many labels (spectrograms, pitch, energy, and duration) for training. I'm not sure exactly what this means for compatibility with Trainer since I haven't had time to do a deeper dive, but for now I've changed the "targets" to include _labels in their name, left the training test as skipped, and planning to look into it more when I do the demo notebook.

Appreciate the comments @hollance, @ArthurZucker @sanchit-gandhi this should be ready for review for you now

Thank you for the review @sanchit-gandhi, comments should be addressed now. 

Centralizing a note on passing the config instead of args here since there were a few comments on that - the other modules mentioned are all instantiated twice with different arg values so they can’t solely be passed the config.  Lmk if you think there’s something I missed or if you’d prefer something else like duplicating the modules in order to pass just the config.

Reviewing now! 

Thanks @ArthurZucker for the review, your comments should be addressed now

Looking very clean, thanks for iterating @connor-henderson! Would be great to get a look from @ylacombe to check this model will be compatible with the new TTS pipeline (e.g. with the auto mapping names), alongside @ArthurZucker for the final review

I'll review this today as well 😉 

@ArthurZucker comments are addressed, left two follow up questions: first one on if changes requested on `# Copied from …` code in speecht5 hifigan should in fact be made, second one regarding self.training vs is_inference and checking for labels

Answered in-line! Thanks for iterating here @connor-henderson!

Thanks for clarifying @sanchit-gandhi, addressed that. I believe the only lingering question is the one from this comment https://github.com/huggingface/transformers/pull/23439#discussion_r1302119798 around whether I should remove the tokenizer from the docstring examples to pass the doc tests since it requires g2p_en backend

Hey! You can also add `g2p` to the list of package needed to run the PR documentation tests in the CI job! Do you need help with that? 

See @ydshieh's comment for reference: https://github.com/huggingface/transformers/pull/23439#discussion_r1333918558

> Hey! You can also add `g2p` to the list of package needed to run the PR documentation tests in the CI job! Do you need help with that?

> See @ydshieh's comment for reference: https://github.com/huggingface/transformers/pull/23439#discussion_r1333918558

@sanchit-gandhi @ArthurZucker all good thanks was on vacation, just added

bump!

I'll help to get this merge @connor-henderson as seen offline! I gotta finish catching up and should come back to this tomorrow ! Sorry again 🤗 

Here is [how I dealt with the `CLIPVisionModel` ](https://github.com/huggingface/transformers/pull/27662#discussion_r1415447950). In the PR I added the config and model to the config mappings and also the correct places to import, but not in the readmes or in the index file. 

It's been a long ride, but merging now!

Thanks again for the great work and your patience !