MusicGen with TextToAudioPipeline issues
Author: osanseviero
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14

### Who can help?

@sanchit-gandhi @ylacombe @Vaibhavs10 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

#### Issue 1 - logging error

Load musicgen with pipeline will have a logging error

```python
from transformers import pipeline

pipe = pipeline("text-to-audio", model="facebook/musicgen-small")
data = pipe("latin salsa with violins")
```

error

```
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/colab_kernel_launcher.py", line 37, in <module>
    ColabKernelApp.launch_instance()
  File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 992, in launch_instance
    app.start()
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py", line 619, in start
    self.io_loop.start()
  File "/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py", line 195, in start
    self.asyncio_loop.run_forever()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 685, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 738, in _run_callback
    ret = callback()
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 825, in inner
    self.ctx_run(self.run)
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
    yielded = self.gen.send(value)
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 361, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
    yielded = ctx_run(next, result)
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 261, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
    yielded = ctx_run(next, result)
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 539, in execute_request
    self.do_execute(
  File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
    yielded = ctx_run(next, result)
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py", line 302, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 539, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 2975, in run_cell
    result = self._run_cell(
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3030, in _run_cell
    return runner(coro)
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
    coro.send(None)
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3257, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3473, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-d27dc111b0bf>", line 4, in <cell line: 4>
    data = pipe("latin salsa with violins")
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_to_audio.py", line 138, in __call__
    return super().__call__(text_inputs, **forward_params)
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py", line 1140, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py", line 1147, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_to_audio.py", line 112, in _forward
    output = self.model.generate(**model_inputs, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/musicgen/modeling_musicgen.py", line 2335, in generate
    logger.warning(
Message: 'Using the model-agnostic default `max_length` (=1500) to control the generation length.  recommend setting `max_new_tokens` to control the maximum length of the generation.'
Arguments: (<class 'UserWarning'>,)
```

#### Issue 2 - Not clear how to specify max_new_tokens

```
data = pipe("latin salsa with violins", max_new_tokens=256)
```

will give an error

```
TypeError: TextToAudioPipeline._sanitize_parameters() got an unexpected keyword argument 'max_new_tokens'
```

I would have expected the kwargs to be passed

### Expected behavior

Pipeline works for MusicGen

Comments:
cc @Narsil as well :)

Re: the issue with `max_new_tokens` - We should take care of this in the `santize_parameters` function, right?

`max_new_tokens` is part of the `generate_kwargs` - It should be considered as here: https://github.com/huggingface/transformers/blob/546e7679e7f692ebeefcfc5063cec271a55bae20/src/transformers/pipelines/automatic_speech_recognition.py#L384

Currently, `max_new_tokens` works with `forward_params`: https://github.com/Vaibhavs10/scratchpad/blob/main/text_to_audio_pipeline_repro.ipynb

cc: @sanchit-gandhi @ylacombe 

The Automatic Speech Recognition pipeline takes `max_new_tokens` as a standalone argument: https://github.com/huggingface/transformers/blob/033ec57c038bbc1a85a936196c9a63072088d221/src/transformers/pipelines/automatic_speech_recognition.py#L344-L345

This is in contrary to the Text To Speech pipeline, which expects it to be passed as a `forward_param`: https://github.com/huggingface/transformers/blob/033ec57c038bbc1a85a936196c9a63072088d221/src/transformers/pipelines/text_to_audio.py#L129-L130

This again is different to the Text Generation pipeline, which expects `generate_kwargs`:
https://github.com/huggingface/transformers/blob/033ec57c038bbc1a85a936196c9a63072088d221/src/transformers/pipelines/text_generation.py#L193-L195

IMO we can unify them all to have the same argument for the forward params - WDYT @Narsil? At least for the TTS pipeline, we can accept `generate_kwargs`, since these are used in all the other generation based pipelines (cc @ylacombe)

IMHO, we should unify `max_new_tokens` to top-level calls for all the pipelines. Pipelines are supposed to reduce friction for our end-user. Expecting them to know the different ways to do something as basic as controlling the length of generation causes a great deal of friction.

If there is disagreement with the above, then I think we should consider unifying all under `generate_kwargs` - FYI I've been recommending `generate_kwargs` for ASR pipeline as it also allows user to play around with other generation strategies directly through there.

I'm not sure if using `forward_params` in TTS pipeline is such a good idea. Can these be folded into `generate_kwargs`? Would that make much sense?


I'm not sure about the correct naming (`forward_params` or `generate_params`), but keep in mind that the TTS/TTA pipeline is also compatible with models that are not generative, so using `generate_params` or having a `max_new_tokens` param can be misleading in some cases.

However, I would be in favor of adding these naming and common parameters, provided that the documents are clear enough to specify that they will only be used with generative models.


> IMO we can unify them all to have the same argument for the forward params - WDYT @Narsil? At least for the TTS pipeline, we can accept generate_kwargs, since these are used in all the other generation based pipelines (cc @ylacombe)

`max_new_tokens`  is what I call a lifted arg. It's a top-level one because it's very useful one in text-generation (basically to prevent long generations).

Is is really that useful here  ? AFAIK it's not defined in ASR for instance, where you want all the generated text and don't really care how many tokens that is. For TTA, maybe we want to generate audio of a specific length ? Feels here you just tried something because you saw the warning. Maybe the warning is the issue ? (And is fixed by modifying the generate_config.json iiuc).

In general for pipeline, which is aimed at non ML users:

 -> No param is best, we provide sane defaults wherever it fits (we expect 80% of calls are this)
 -> Simple param control, which are model agnostic (they work, no matter what's underneath, like top_k for fill-mask)   (10% of the calls)
 -> Advanced param control (everything else including `generate_kwargs`) . This should be quite rare imho, and if anything is missing, dropping down to Model + tokenizer + feature_extractor +... is the actual way to go, not overparametrizing the pipeline.
 As long as their support is easy for the pipeline, they are a nice escape hatch for advanced usage, but they are not by any means core intended usage imho).
 
 Note that I didn't say if `max_new_tokens` is or isn't relevant here, I'm raising the question that maybe it isn't, you can decide if it actually is.
 
+1 to making it top-level whereever it might fit -> WITH proper sanitation (which is a bit tedious, it must be defined at top-level, not in generate_kwargs at the same time, and it must actually be do something special if the pipeline is not going to use it, for instance if the model is not generative)

`{generate|forward}_kwargs` are "drop-all" kind of kwargs, enabling powerusers to do all the customization they want. They are an escape hatch, no need to overthink them I think.