[Community] Add fairseq FastSpeech2
Author: jaketae
Pull request: yes
Labels: name: Good Second Issue

name: WIP

name: Good Difficult Issue

# What does this PR do?

This PR adds FastSpeech2, a transformer-based text-to-speech model. Fixes #15166.

## Motivation

While HF transformers has great support for ASR and other audio processing tasks, there is not much support for text-to-speech. There are many transformer-based TTS models that would be great to have in the library. FastSpeech2 is perhaps the most well-known and popular of transformer-based TTS models. With TTS support, we could potentially have really cool demos that go from ASR -> NLU -> NLG -> TTS to build full-fledged voice assistants! 

## Who can review?

Anton, Patrick (will tag when PR is ready)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15773). All of your documentation changes will be reflected on that endpoint.

Here is a (tried-to-keep-short-but-long) update on this PR, summarizing some of the things discussed with @anton-l for public visibility.

1. Checkpoint loading: The model weight loads properly and produces roughly the same output as the fairseq model. On preliminary tests, I've found that `torch.allclose` passes with `atol=1e-6`. On this branch, the model can be loaded via
    ```python
    from transformers import AutoModel
    model = AutoModel.from_pretrained("jaketae/fastspeech2-lj")
    ```
2. Grapheme-to-phoneme: Fairseq uses [`g2p_en`](https://github.com/Kyubyong/g2p). I've added this as a dependency via `requires_backend` in the tokenizer. One thing I'm not too sure about is whether to enforce this from the get-go at `__init__`, or only require it in the `phonemize` function. 
3. Tokenizer: The tokenizer can be loaded as shown. For some reason, `AutoTokenizer` doesn't work at the moment, but this can likely be fixed by reviewing import statements. One consideration is that the fairseq code seems to add an EOS token at the end of every sequence, whereas HF's tokenizers do not by default (correct me if I'm wrong). In my code, I manually add the EOS token at the end for consistency.
    ```python
    from transformers.models.fastspeech2 import FastSpeech2Tokenizer
    tokenizer = FastSpeech2Tokenizer.from_pretrained("jaketae/fastspeech2-lj")
    ```
4. Vocoder: (Note: This is not directly related to the source code in this PR.) Using Griffin-Lim for vocoding yields very poor audio quality and is likely a bad idea for demos. An alternative is to use neural vocoders like HiFi-GAN. I've added HiFi-GAN on the model hub, which can be imported via `vocoder = AutoModel.from_pretrained("jaketae/hifigan-lj-v1", trust_remote_code=True)`. At the moment, for some reason the model weights do not load properly, debugging required.

I have some questions for discussion.

1. Model output: What should the output of the model be? At the moment, it returns a tuple of mel-spectrograms, mel-spectrogram after postnet, output lengths (number of spectrogram frames), duration predictor output, predicted pitch, and predicted energy. I assume we need to subclass `BaseModelOutput` for something like `TTSModelOutput`, and also optionally return loss when duration, pitch, and energy ground-truth labels are supplied in `forward`.
2. Mean and standard deviation: Fairseq's FastSpeech2 model technically does not predict a mel-spectrogram, but a normalized mean spectrogram according to mu and sigma values extracted from the training set. In fairseq's TTS pipeline, FS2 output is denormalized by some mu and sigma before being vocoded. I've added the mu and sigma values as `register_buffer` to the model, and created `set_mean` and `set_std` functions so that users can set these values according to their own training set. By default, mean and variance are set to 0 and 1 so that no rescaling is performed. Is this good design?
3. TTS pipeline: At the moment, there is only `FastSpeech2Model`, which outputs mel-spectrograms as indicated above. While people might want to use acoustic feature generators for mel-spectrogram synthesis, more often they will want to obtain raw audio in waveform. As mentioned, this requires a vocoder, and we need to discuss ways to integrate this. Some API ideas I have include:
    - Introducing a `FastSpeech2ForAudioSynthesis` class with an integrated vocoder
    - Adding a `pipeline` that supports full-fledged TTS via something like `pipeline("text-to-speech", model="fastspeech2", vocoder="hifigan")`. But this is again contingent on how we want to integrate the vocoder.

To-do's on my end (will be updated as I find more).
- [x] Writing tests
- [ ] Documentation
- [x] Cleaning up imports
- [x] Debugging why `AutoTokenizer` fails
- [ ] Debugging why HiFi-GAN weights do not load

cc @anton-l @patrickvonplaten 

Hey @anton-l, here are some long overdue updates:

* Incorporated LJSpeech and Common Voice checkpoints to the model hub
* Wrote unit and fairseq integration tests

At the moment, `test_initialization` is failing, and I'm still investigating how to fix it. I also skipped a number of tests (e.g., ones that test for `output_attentions` or `hidden_states`), because FS2 does not return these keys.

I've added one simple `test_inference_integration` test to check that the expected mel-spectrogram output by the fairseq model is `torch.allclose` to the HF port. However, mel-spectrograms are too big in dimension, e.g., `(batch_size, seq_len, mel_dim)`, where `mel_dim` is typically 80. Therefore, I've sliced the first 100 elements for inspection instead. If you think there is a more robust check (i.e., more elements, different text, etc), please let me know.

An updated to-do's:

- [ ] docstrings/documentation
- [ ] debug `test_initialization`
- [ ] debug any failing tests

Thank you!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15773). All of your documentation changes will be reflected on that endpoint.

@LysandreJik @sgugger would be awesome if you take a look at this PR too, since it introduces a novel modality :) 

@patrickvonplaten after a brief discussion with @jaketae, I think we should keep the spectrogram2waveform generators as separate models on the Hub for several reasons:
* the models for waveform generation are more or less interchangeable between different models (maybe with some finetuning) and are trained separately from the TTS models
* since the generators can be quite diverse, it would be cumbersome to support training for them, e.g. we probably don't have GANs or diffusion models in `transformers` for that very reason
* if I understand correctly, the generators usually don't support batched inference on inputs of varying lengths, so they're more suitable for `pipelines` rather than an integrated module

Having said that, I suggest that we finish this PR as a first step without adding any waveform generators. After that we can work on a TTS `pipeline` that runs inference on FastSpeech2 and decodes the output using either a Griffin-Lim synthesizer or any of the GAN models from the Hub (e.g. HiFiGAN) that we'll port.


Hi, may I ask a couple of questions:
1. Is there some estimated time of merging for this PR?
2. Are there any experiments/examples of training using this implementation Fastspeech2?
And somewhat more general question:
3. Is there work in progress related to `text-to-speech` pipeline?

I'm working training/fine-tuning fastspeech2 models and would like to deploy using HF inference API.
Would be happy to help out with testing and contribute to pipeline and/or porting HiFiGAN or provide any assistance I can to speed things up with TTS

@anton-l, think it's not high priority but once you have a free day let's maybe add the model :-)

Wish I could have iterated faster to finish it earlier. Let me know if you need anything from me @anton-l @patrickvonplaten!

> Wish I could have iterated faster to finish it earlier. Let me know if you need anything from me @anton-l @patrickvonplaten!

@jaketae if you want, would you like to finish this PR maybe ? E.g. fix the failing tests and add **all** the necessary code (including the vocoders) to the modeling file?

@anton-l I still think we should go ahead with this PR - it's been stale for a while now. Wdyt?

@patrickvonplaten definitely! I'm trying to find some time alongside other tasks to work on FS2 :sweat_smile: 

Hey @anton-l @patrickvonplaten, hope you're doing well! I was wondering where FS2 fits within the transformers road map, and whether it would make sense for me to update this branch and wrap up the work. I feel responsible for leaving this stale, and I still think it would be cool to have this feature pushed out. Thank you!

Hey @jaketae, 

We definitely still want to add the model - would you like to go ahead and update the branch?

I don't think anybody will have time to work on it soon. If anybody from the community wants to give it a shot, please feel free to do so! 

> I don't think anybody will have time to work on it soon. If anybody from the community wants to give it a shot, please feel free to do so!

Can I try working on this?

@JuheonChu I'm also interested in working on this, any interest in collaborating? Planning to get up to speed then open a fresh pr.

Hi @JuheonChu @connor-henderson, internally at HF we've been discussing what to do with this model. This PR has a lot of good work done already and FastSpeech 2 is a popular model, so it makes a lot of sense to add this to Transformers. If you want to help with this, that's great. 😃 

There are a few different implementations of FastSpeech 2 out there and we'd like to add the best one to Transformers. As far as we can tell, there is a Conformer-based FastSpeech 2 model in ESPNet that outperforms regular FastSpeech 2, so that might be an interesting candidate. You can find the [Conformer FastSpeech 2 checkpoints here](https://huggingface.co/models?sort=downloads&search=espnet+fastspeech2+conformer).

In addition, PortaSpeech is very similar to FastSpeech 2 but it replaces the post-net and possibly a few other parts of the model. So that might be an interesting variation of FS2 to add as well.

We're still trying to figure out which of these models we'd prefer to have, but since you've shown an interest in working on this, I wanted to point out the direction we were thinking of taking these models internally. And I'd like to hear your opinions on this as well. 😄 

P.S. Make sure to check out the implementation of SpeechT5. It has many features in common with FastSpeech2 (such as the vocoder and the generation loop) and so we'd want to implement FS2 mostly in the same manner.