EncodecFeatureExtractor.__call__() got an unexpected keyword argument 'pad_to_multiple_of'
Author: auli-aziz
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.0
- Platform: Windows-10-10.0.26200-SP0
- Python version: 3.10.18
- Huggingface_hub version: 0.35.3
- Safetensors version: 0.5.3
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.4.0+cpu (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

Hello  @eustlb @ebezzam @vasqu

Just want to report a suspected bug. I keep getting an error when i try to inference the CSM-1B model using transformers with context. The error that i got is `Error with generated segments: EncodecFeatureExtractor.__call__() got an unexpected keyword argument 'pad_to_multiple_of'`. Perhaps the issue is particularly with how the audio is stored (doesn't give error when i apply_chat_template using one text input). I tried many things to store the audio such as storing it in Torch.tensor format and referencing it from a save wav file, but it still failed.

Here is how i saved the generated_segments:
```
audio_file = f"temp_vad_recording_{int(time.time())}.wav"
audio_tensor = torch.tensor(audio_data, dtype=torch.float32).unsqueeze(0)
torchaudio.save(audio_file, audio_tensor, sample_rate)

user_input = self.transcribe_audio(audio_file)

self.generated_segments.append(
    {
        "role": "1",
        "content": [{"type": "text", "text": user_input}, {"type": "audio", "path": audio_file}],
    }
)
```

Here is how i applied chat template:
```
inputs = self.tts_processor.apply_chat_template(
                    self.generated_segments,
                    tokenize=True,
                    return_dict=True,
                ).to(self.tts_generator.device)
```


### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. run the code using a vad and whisper for transcribing into text:
```
audio_file = f"temp_vad_recording_{int(time.time())}.wav"
audio_tensor = torch.tensor(audio_data, dtype=torch.float32).unsqueeze(0)
torchaudio.save(audio_file, audio_tensor, sample_rate)

user_input = self.transcribe_audio(audio_file)

self.generated_segments.append(
    {
        "role": "1",
        "content": [{"type": "text", "text": user_input}, {"type": "audio", "path": audio_file}],
    }
)
```

### Expected behavior

apply_chat_template should work when referencing local audio files in the path, like mentioned in the the error `Incorrect format used for `audio`. Should be an url linking to an audio, a local path, or numpy array.`

Comments:
Hey `@auli-aziz`, this is a bug within the library, not an issue with how you are storing your audio. The `processor.apply_chat_template` method is incorrectly forwarding keyword arguments intended for the text tokenizer, like `pad_to_multiple_of`, down to the `EncodecFeatureExtractor`, which doesn't accept it and causes the `TypeError`. As an immediate workaround, you can bypass this by processing the text and audio modalities separately using the `.tokenizer` and `.feature_extractor` attributes of your processor, and then manually combine the inputs for the model. This is a clear bug in the library's argument dispatching logic that we'll need to patch. Thanks again for flagging this!

Hi, is this fixed? Can I work on this ?

> As an immediate workaround, you can bypass this by processing the text and audio modalities separately using the .tokenizer and .feature_extractor attributes of your processor, and then manually combine the inputs for the model. This is a clear bug in the library's argument dispatching logic that we'll need to patch. Thanks again for flagging this!

can you please give an example of how to implement this workaround?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Apologies for the delay @auli-aziz!

Since `apply_chat_template` handles complex logic (like formatting text and interleaving audio tokens) that is difficult to replicate manually, the cleanest workaround is to temporarily "patch" the feature extractor to ignore the problematic argument.

You can add this small snippet to your script right before you call `apply_chat_template`:

```python
# 1. Save the original __call__ method of the feature extractor
_original_call = self.tts_processor.feature_extractor.__call__

# 2. Define a wrapper that filters out the problematic argument
def safe_feature_extractor_call(*args, **kwargs):
    # Remove 'pad_to_multiple_of' if present, as EncodecFeatureExtractor doesn't accept it
    kwargs.pop("pad_to_multiple_of", None)
    return _original_call(*args, **kwargs)

# 3. Apply the patch to the processor instance
self.tts_processor.feature_extractor.__call__ = safe_feature_extractor_call
# --- Workaround End ---

# Now your existing code will run without the TypeError:
inputs = self.tts_processor.apply_chat_template(
    self.generated_segments,
    tokenize=True,
    return_dict=True,
).to(self.tts_generator.device)