Add bark
Author: ylacombe
Pull request: yes

This PR aims at integrating Bark, a TTS model, to `transformers`.
`Bark` was designed and trained by [Suno-AI team](https://github.com/suno-ai/bark) and is made of 4 main components:

- A `semantic model ` (also named `text model`), i.e a causal autoregressive transformer (GPT2-like), which takes into input a tokenized text
- A `coarse acoustics model` (also named `coarse model`), also a causal autoregressive transformer, taking into input the results of the last model. It aims at regressing the first two audio codebooks necessary to `encodec`.
- A `fine acoustics model` (`fine model`), this time a non-causal autoencoder transformer, which iteratively predicts the 6 last codebooks based on the sum of the previous codebooks embeddings.
- having predicted 8 codebooks channels of `encodec`, Bark uses `encodec` to generate the output audio array.

Note that each of the first 3 modules can take optional conditional speaker embeddings aiming at conditioning the output audio according to `specific preset voices.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sanchit-gandhi 

PR supersedes #23375

Hi @sanchit-gandhi , I think it's finally time for the final review! You might want to check the refactoring of `generate_text_semantic`, `generate_coarse`, `generate_fine`, but otherwise, sounds good!

Hi @amyeroberts, the PR is ready for review ! I'd be delighted to get your feedback on this when you have a chance. Let me know if I can help with anything!

@ylacombe Great! Could you resolve the conflicts? Once that's done I'll review 👍 

Hi @amyeroberts, as demanded, I resolved the merge conflicts! I've also updated the `speaker_embeddings` processing in `BarkProcessor`. Could you take a look when you have time ? 
Thanks! 

Before I start a full review of this model, could you explain why the model was written in this structure - with no `forward` method and no task specific model e.g. `BarkForTextToSpeech`? 

> Before I start a full review of this model, could you explain why the model was written in this structure - with no `forward` method and no task specific model e.g. `BarkForTextToSpeech`?

Of course! 

You can't really `forward` through `BarkModel` because it uses the `generate` methods of its sub-models, each one with its one `GenerationConfig`.

To be a little bit more specific,  `BarkModel` is a nested model composed of 4 sub-models.

The first 3 submodels follow a classic `transformer` architecture - hence the existence of a `forward` method for these submodels. However, when used by `BarkModel` in `generate_speech`, they are used in a non-traditional way (sliding windows, addition of input vectors alongside dimensions) and directly with their `generate` methods, model by model.

To be more in line with  `transformers` paradigm, we decided to keep the classical architecture for the 3 sub-models (the fourth, [`Encodec`](https://huggingface.co/docs/transformers/main/model_doc/encodec), being already implemented) and to provide a `generate_speech` method for the final model, with nested configs and nested generation configs.

Using `forward` would have meant a much messier and probably slightly more confusing path, since it's not actually a matter of `forwarding` a list of tokens to generate new audio tokens one by one, but of generating the audio all at once!

@sanchit-gandhi, I might have missed some arguments here, feel free to contribute!

@amyeroberts, let me know if this answers your question!

Regarding the last part of your question, the `BarkModel` architecture can't really be used for anything other than this specific task, so it's a kind of single-purpose model.

We considered adding task-specific sub-models, since the two first sub-models are GPT2-like auto-regressive models, but we decided not to move forward, for multiple reasons imo:

1. The first two sub-models, `BarkSemanticModel` and `BarkCoarseModel` could have had task-specific sub-classes, but I think this complicates both the code and users' understanding of the model architecture. What's more, although their architecture is general, it's only used here with an `lm_head`, and I can't think of any other use for them.
2. The third sub-model, `BarkFineModel`, needs [multiple embeddings  layers and lm_heads](https://github.com/ylacombe/transformers/blob/33081dc7e3650ff07f2c766c3aa69bc7b6c82351/src/transformers/models/bark/modeling_bark.py#L968C2-L985), one per codebook channels of `Encodec`. So it's a non-regular type of task.



In reality, the Bark TTS model is just three auto-regressive models concatenated together.

To generate with the Bark TTS model, you first have to generate **all** the ids for the first model. You then forward **all** of these ids to the coarse model to generate a new set of ids. You subsequently forward **all** of these generated ids to the third model to get your audio code outputs. So we can't just define one `forward` call and then auto-regressively generate with it (this would only get you one set of ids, not the three stages that you need).

-> for this reason it doesn't really make sense to have a `forward` call for the `BarkModel`, since the model is just a placeholder to hold the three submodes together, and pipe the generated outputs from one model into the next

Regarding why this the model isn't called `BarkForTextToSpeech`, it's the same argument as VITS: https://github.com/huggingface/transformers/pull/24085#discussion_r1252222434

Happy to rename to `BarkForTextToSpeech` if you feel that this gives a more unified API between models, but Bark can **only** do text-to-speech, so this part of the name is somewhat redundant

@sanchit-gandhi @ylacombe Thanks for the detailed explanations! 

Hi @amyeroberts, you're welcome!
Have you had time to look into the PR? I'd be happy to answer any questions you might have about my code, as it's a rather atypical model!

Hi @amyeroberts ,
Thanks for the comprehensive review!
I've answered most of your comments, but there are still a few I've asked questions/clarifications about! 


Hi @amyeroberts and @sgugger!

Many thanks for the additional review (and thanks @sanchit-gandhi for your insights)!
I've addressed most of your comments, especially those requiring more consistency with transformers regarding naming the `generate_xxx`. I still have a few comments to resolve, I'll wait for your returns on that!



Hi @amyeroberts, 
there was a time-out when executing the python snippet of the `generate` docstrings.
I took advantage of this to [add the ability to specify sub-model specific parameters](https://github.com/huggingface/transformers/pull/24086/commits/97cdc38e66b600d3bc1d82c56099acf3cdc6a0f8) in `BarkModel.generate`. 

To give a concrete example, you can specify now how many `max_new_tokens` you want for the `semantic` part of the model:
```audio_array = model.generate(**inputs, semantic_max_new_tokens=100)```

Now that it is done, there are still a few comments to resolve, so I look forward to hearing from you!

Hey @amyeroberts,
I've addressed your last remarks! Does that work with you?
Many thanks!

@ylacombe LGTM! I think we're good to merge 👍