Fix processor usage + add chat_template support to TTS pipeline, and shift common chat template logic to base class.
Author: ebezzam
Pull request: yes
Labels: name: Audio

name: for_v5?

# What does this PR do?

Processor usage within TTS pipelines is faulty.


Moreover, it does not support chat template inputs with/without audio, which is of interest for the following models for generating conversations:
- [CSM](https://huggingface.co/sesame/csm-1b) (already in Transformers) 
- [Higgs](https://huggingface.co/bosonai/higgs-audio-v2-generation-3B-base) (upcoming in Transformers, cc @eustlb)
- [VibeVoice](https://huggingface.co/microsoft/VibeVoice-1.5B) (upcoming in Transformers)

Example usage [for CSM](https://huggingface.co/sesame/csm-1b#csm-sounds-best-when-provided-with-context) and inputs like such for VibeVoice:
```python
conversation = [
    {"role": "0", "content": [
        {"type": "text", "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me."},
        {"type": "audio", "path": https://hf.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Alice_woman.wav}
    ]},
    {"role": "1", "content": [
        {"type": "text", "text": "Thanks so much for having me, Linda. You're absolutely right—this question always brings out some seriously strong feelings."},
        {"type": "audio", "path": https://hf.co/datasets/bezzam/vibevoice_samples/resolve/main/voices/en-Frank_man.wav}
    ]},
    {"role": "0", "content": [
        {"type": "text", "text": "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the Finals, six championships. That kind of perfection is just incredible."},
    ]},
    {"role": "1", "content": [
        {"type": "text", "text": "Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it"},
    ]},
]
```

This PR is related to #39796, but this one is different/simpler (and I'd say more urgent) in its objective 👉 _fixing processor usage and enabling chat_template inputs like above_.

So I think it's worth a separate PR as #39796 requires more testing/review.

# Context for current error

[This line](https://github.com/huggingface/transformers/blob/f2738ee3756483538c20e10e4f104324675fb406/src/transformers/pipelines/text_to_audio.py#L153) fails when trying to use the processor because it [isn’t loaded](https://github.com/huggingface/transformers/blob/f2738ee3756483538c20e10e4f104324675fb406/src/transformers/pipelines/text_to_audio.py#L84) (so it is `None` instead)

Minimal failing example:
```python
import torch
from transformers import pipeline

model_id = "sesame/csm-1b"
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = pipeline("text-to-speech", model=model_id, device=device, no_processor=False)

# prepare the inputs
text = "[0]Hello from Sesame." # `[0]` for speaker id 0

# apply pipeline
output = pipe(text)

"""
Traceback (most recent call last):
  File "/home/eric_bezzam/transformers/src/transformers/pipelines/test_pipeline_chat_template.py", line 34, in <module>
    output = pipe(text)
             ^^^^^^^^^^
  File "/home/eric_bezzam/transformers/src/transformers/pipelines/text_to_audio.py", line 218, in __call__
    return super().__call__(text_inputs, **forward_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric_bezzam/transformers/src/transformers/pipelines/base.py", line 1261, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric_bezzam/transformers/src/transformers/pipelines/base.py", line 1267, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric_bezzam/transformers/src/transformers/pipelines/text_to_audio.py", line 153, in preprocess
    output = preprocessor(text, **kwargs, return_tensors="pt")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
"""
```

# After changes

No need to explicitly ask for processor usage (auto-detected internally).

TTS example
```python
import torch
from transformers import pipeline
import soundfile as sf

model_id = "sesame/csm-1b"
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = pipeline("text-to-speech", model=model_id, device=device)

# prepare the inputs
text = "[0]Hello from Sesame." # `[0]` for speaker id 0

# apply pipeline
output = pipe(text, generate_kwargs={"output_audio": True})

# save the audio to a file
audio = output["audio"][0].squeeze()
fn = "csm_pipeline_output.wav"
sf.write(fn, audio, output["sampling_rate"])
print(f"Audio saved to {fn}")
```

Conversation example with chat template, which was not possible before! Again auto-detected [like in text-generation pipeline](https://github.com/huggingface/transformers/blob/f2738ee3756483538c20e10e4f104324675fb406/src/transformers/pipelines/text_generation.py#L359). Below example mimics [this usage](https://huggingface.co/sesame/csm-1b#csm-sounds-best-when-provided-with-context) from CSM:
```python
import torch
from transformers import pipeline
import soundfile as sf
from datasets import load_dataset, Audio

model_id = "sesame/csm-1b"
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = pipeline("text-to-speech", model=model_id, device=device)

# prepare the inputs like here: https://huggingface.co/sesame/csm-1b#csm-sounds-best-when-provided-with-context
ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train")
ds = ds.cast_column("audio", Audio(sampling_rate=24000))
conversation = []
for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]):
    conversation.append(
        {
            "role": f"{speaker_id}",
            "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}],
        }
    )
conversation.append({"role": f"{ds[4]['speaker_id']}", "content": [{"type": "text", "text": ds[4]["text"]}]})

# apply pipeline
output = pipe(conversation, generate_kwargs={"output_audio": True})

# save the audio to a file
audio = output["audio"][0].squeeze()
fn = "csm_pipeline_output_chat.wav"
sf.write(fn, audio, output["sampling_rate"])
print(f"Audio saved to {fn}")

```



---


@Rocketknight1 since it is pipeline related you can take a look if you want! but will definitely ask @vasqu an @eustlb for audio-specific feedback 🙂 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42326). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Failed tests are unrelated, concatenating below:
```
# https://app.circleci.com/pipelines/github/huggingface/transformers/154274/workflows/48600cd6-5d2a-455a-a2e0-96f3b9bae8ae/jobs/2028466
FAILED tests/models/efficientloftr/test_image_processing_efficientloftr.py::EfficientLoFTRImageProcessingTest::test_post_processing_keypoint_matching_with_padded_match_indices - AssertionError: 2 != 1
===== 1 failed, 549 passed, 362 skipped, 24 warnings in 115.22s (0:01:55) ======

# https://app.circleci.com/pipelines/github/huggingface/transformers/154274/workflows/48600cd6-5d2a-455a-a2e0-96f3b9bae8ae/jobs/2028474
FAILED tests/models/resnet/test_modeling_resnet.py::ResNetModelTest::test_can_load_ignoring_mismatched_shapes - AssertionError: 0.14109472930431366 not less than or equal to 0.1 : Issue with classifier.1.bias
==== 1 failed, 3959 passed, 6611 skipped, 93 warnings in 187.89s (0:03:07) =====

# https://app.circleci.com/pipelines/github/huggingface/transformers/154274/workflows/48600cd6-5d2a-455a-a2e0-96f3b9bae8ae/jobs/2028467
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoModelTest::test_generate_with_static_cache - AssertionError: False is not true
===== 1 failed, 625 passed, 218 skipped, 16 warnings in 118.43s (0:01:58) ======
```

Failing tests are either flaky (loftr one is handled elsewhere)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: bark