Adding FastSpeech2
Author: huu4ontocord
Pull request: yes

# What does this PR do?

This is a draft PR for Fastspeech2 which includes melgan and a custom g2p pythorch module.  See https://huggingface.co/ontocord/fastspeech2-en

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@patil-suraj 

Models:
- Fastspeech2


Comments:
Hey @ontocord,

Thanks a lot for opening this pull request :-) 

We are very much open to adding FastSpeech2 to `transformers`! One thing that is quite important to us is that we stick as much as possible to the original implementation of the model. 

IMO, the easiest way to approach this would be to translate [this](TF code) to PyTorch: https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/models/fastspeech2.py and add it to our lib.

Let me know if it would also be ok/interesting for you to stay closer to the official code -> I'm very happy to help you get this PR merged then :-)

Happy for you to close this PR in favor of an official implementation. Just as an aside, the G2P (GRU->GRU) code is actually based on the original impementation from the Fastspeech2 paper. But it uses https://github.com/Kyubyong/g2p  which is slower than pytorh and based on Numpy. I re-wrote the G2P in pytorch based on the G2P author's notes, and retrained it so it's faster.

From the paper: 

"To alleviate the mispronunciation problem, we convert the
text sequence into the phoneme sequence (Arik et al., 2017; Wang et al., 2017; Shen et al., 2018;
Sun et al., 2019) with an open-source grapheme-to-phoneme tool5
...
5https://github.com/Kyubyong/g2p"

I think this module is really one of the things that keeps the Fastspeech2 model (and tacotron 2 and similar models) from generalizing to more languages. In theory you could just train on character level, but it's harder. DM if you want to discuss work arounds...

Hey @ontocord @patrickvonplaten, I was wondering if there has been a followup to this PR. I'd love to see transformer TTS models like FastSpeech2 in this library and would be more than happy to help contribute if possible!

I also think we should eventually add models like FastSpeech2 to the library. Gently ping to @anton-l here who was interested in this addition as well.

@patrickvonplaten @anton-l  Do we only add models with official weights from the paper authors? AFAIK FastSpeech2 has plenty of unofficial implementations with weights, but there is no official repository ([PwC](https://paperswithcode.com/paper/fastspeech-2-fast-and-high-quality-end-to-end)). I think we should reach out to the author (Yi Ren is on GitHub), and if that doesn't work out, consider which implementation/weights we want to port. What do you think?

Also if you'd prefer, I'll open a new issue dedicated to this discussion instead of hijacking this PR.

I think we should definitely reach out to the original authors! Feel free to contact them :-)

Just emailed the first author and cc'd both you and Anton! I'll keep you posted. 

I would be interested in working on something more generic than fastspeech2 which needs a g2p module. It’s not truly end to end. 

> On Jan 3, 2022, at 7:55 AM, Jake Tae ***@***.***> wrote:
> 
> ﻿
> Just emailed the first author and cc'd both you and Anton! I'll keep you posted.
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> Triage notifications on the go with GitHub Mobile for iOS or Android. 
> You are receiving this because you were mentioned.