Add Text-To-Speech pipeline
Author: ylacombe
Pull request: yes

# What does this PR do?

Until recently, there was only one TTS model in Transformers. Recent ([Bark](https://huggingface.co/docs/transformers/model_doc/bark)) and future ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439)) additions have and will further enrich the number of TTS models in Transformers. 
This may be the best time to add a text-to-speech pipeline to Transformers.

This PR tentatively proposes:

- The addition of a text-to-speech pipeline whose design could be modified in line with future TTS additions.
- Add a class AutoModelForTextToSpeech
- Add a `processor` task to the pipeline code to facilitate use of the `processor`.

My conception of the architecture for now:

- Backward compatibility with [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439), retaining the ability to use its hacked `generate_speech` method.
- Future compatibility with future TTS models, counting on the fact that these models will use a `generate` method to generate audio.
- Possible compatibility with other TTA (text-to-audio) models such as [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen).

What I'm counting on:
- future models should have a `generate` method, even if they are not AR models per se (for the moment, [FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) is not AR and has no such method) or counts on an additional head model ([FastSpeechConformer2](https://github.com/huggingface/transformers/pull/23439) needs a vocoder on top to pass from a spectrogram to an audio - see [discussion here](https://github.com/huggingface/transformers/pull/23439#discussion_r1258660411)).
- future models will use a `Processor` even if they only use a tokenizer, to allow easy use of other conditional inputs such as audio or speaker embeddings.  And the processor must be added to `PROCESSOR_MAPPING` (not the case of [MusicGen](https://huggingface.co/docs/transformers/model_doc/musicgen) atm).

I'm open to further discuss the architecture and to make some changes!

EDIT: for reference, I've made another design choice following internal discussions. It is discussed [here](https://github.com/huggingface/transformers/pull/24952/#pullrequestreview-1556507240).

Fixes #22487

*Note:* I was inspired by @LysandreJik draft of a [TTS pipeline](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? [LINK](https://github.com/huggingface/transformers/issues/22487#issuecomment-1496312713)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Hey @sanchit-gandhi and @Narsil, I think you're the right people to talk to before the core review!


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @Narsil , thanks for your fast review!
Basically, I will refactor my code to meet your expectations !
There are still 2 things I'd like to discuss before and that I talked about in the comments:

1. **`speechT5` specific case:** `speechT5` was introduced 5 months ago, and has two issues - it uses a `.generate_speech` method instead of a `.generate`, and it needs an additional vocoder on top in order to actually produce audio signals instead of a spectrogram. What's the best way to stay consistent with it and with the pipeline logic? Should I still introduce model specific code or should I work on modifying `speechT5` instead?  Modying `speechT5` might be problematic since it was the first TTS model introduced so users might be used to its API and because it would leave `BarkModel` has the only TTS model supported in the pipeline for a short time
2. **`speaker_embeddings`** and other `Processor`-related utilities: how to stay consistent with the library and continue to use some of the benefits of the Processor or continue to use speaker embeddings in an easy way? I fear that it might add unnecessary difficulties for the users to forward `speaker_embeddings` arguments, WDYT?

Anyways, many thanks again for the review!

 

> Hi @Narsil , thanks for your fast review! Basically, I will refactor my code to meet your expectations ! There are still 2 things I'd like to discuss before and that I talked about in the comments:
> 
>     1. **`speechT5` specific case:** `speechT5` was introduced 5 months ago, and has two issues - it uses a `.generate_speech` method instead of a `.generate`, and it needs an additional vocoder on top in order to actually produce audio signals instead of a spectrogram. What's the best way to stay consistent with it and with the pipeline logic? Should I still introduce model specific code or should I work on modifying `speechT5` instead?  Modying `speechT5` might be problematic since it was the first TTS model introduced so users might be used to its API and because it would leave `BarkModel` has the only TTS model supported in the pipeline for a short time

I will let maintainers focusing on audio answer to that @sanchit-gandhi I think.
But what I do know is that not relying on invariants within `transformers` makes pipelines play the never ending game of `catch-up` for every model thrown into the mix. pipelines see `AutoModelFor` which should have consistent API which we can rely on.

I remember talks about splitting `generate` and `generate_speech` to allow differentation between the 2.

For the `vocoder`, I don't know how, but it should be invinsible to users.
In ASR we've had ngram being added to the configuration for instance, which makes it loadable automatically.

> 
>     2. **`speaker_embeddings`** and other `Processor`-related utilities: how to stay consistent with the library and continue to use some of the benefits of the Processor or continue to use speaker embeddings in an easy way? I fear that it might add unnecessary difficulties for the users to forward `speaker_embeddings` arguments, WDYT?

Again, there might already be solutions. 
But loading from a random dataset some random data within `preprocess` is not really sustainable.

My suggestion to put this in usercode alleviates that contraint.

But in general having speaker_embedding for TTS should always be purely optional imo.

> 
> 
> Anyways, many thanks again for the review!



Thanks @Narsil , I will wait for @sanchit-gandhi opinion on it then!
What about [this comment](https://github.com/huggingface/transformers/pull/24952#discussion_r1270368324) ?

I haven't generated the tiny models for `bark`. I will do it today 🙏 (I can't guarantee it will be able to be generated smoothly - usually they should be already on the Hub, and if not, it means the creation process has some issue for this model)

Hi @ylacombe . There are a few issues that blocks the creation of tiny model (of `bark` for pipeline testing).

The first one is `tests/models/bark/test_modeling_bark.py` has no `BarkModelTest` and `BarkModelTester`. Only the component models (fine, coarse, semantic).

Are those component models also used as standalone models? Or they are really just components for `BarkModel` and we expect the users to use `BarkModel` rather than those components?

More importantly, for the pipeline implemented in this PR, which model types do it needs.

Thanks in advance.

Hi @ydshieh, thanks for your help on the matter!

There's only `BarkModelIntegrationTests` for now. The other sub-models are used as components for `BarkModel`. Users are expected to use `BarkModel`, which will ultimately be used in the pipeline.

Let me know if I can help you with anything!

>  Users are expected to use BarkModel

In this case, it means we need to create a tiny model for `BarkModel` or models with head (if any) on top of it. This implies we need a `BarkModelTest` and `BarkModelTester`. For the creation, we don't really need to implement test methods in `BarkModelTest`, but there should be later (I would not say it's urgent though).

I will open a PR to quickly add something necessary so we can create tiny model for `bark`. I will ping you for a review so you know what's necessary (in the future for new models you will add 🤗 ).


LGTM ! Thanks for the rehaul 

Hey @ylacombe 

If you don't mind, let's have the tiny model ready and try with it first before merge.
I am just able to create it.

Just uploaded

https://huggingface.co/hf-internal-testing/tiny-random-BarkModel/

But I haven't tried it with the tests implemented in this PR.