Device selection for Qwen2Audio
Author: ifsheldon
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution. And I could not fix this problem without modifying the relevant code in `transformers`.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes runtime errors like "inputs are on different devices" when Qwen2 Audio runs on devices like "mps". This problem occurs when I tried to run the model on my Mac using `mps` device.

Tests in `transformers/tests/models/qwen2_audio` have passed and I have also test it with the official demo from Qwen 2 Audio with a few modification to run it on MPS device (see below).

## Problem and Code References

https://github.com/huggingface/transformers/blob/342e3f9f2067d45c27c30fbd4d748d233bca3adc/src/transformers/models/qwen2_audio/processing_qwen2_audio.py#L94

Here it calls
https://github.com/huggingface/transformers/blob/342e3f9f2067d45c27c30fbd4d748d233bca3adc/src/transformers/models/whisper/feature_extraction_whisper.py#L180

which has an optional argument `device` that defaults to "cpu". So, the output of the whisper feature extractor will by default 

But we can't just pass `device="mps"` when calling `Qwen2AudioProcessor.__call__`, which will cause another runtime error that says `self.tokenizer()` does not have a `device` argument.

## Who can review?

Probably @faychu @ylacombe  can take a look because of #32137?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

## Modified Demo Code

Modified from https://github.com/QwenLM/Qwen2-Audio?tab=readme-ov-file#audio-analysis-inference

```python
from io import BytesIO
from urllib.request import urlopen
import librosa
from transformers import Qwen2AudioForConditionalGeneration, AutoProcessor

DEFAULT_DEVICE = "mps"  # or "cuda", "cpu", NEW

processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct")
model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct", device_map="auto")

conversation = [
    {'role': 'system', 'content': 'You are a helpful assistant.'}, 
    {"role": "user", "content": [
        {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3"},
        {"type": "text", "text": "What's that sound?"},
    ]},
    {"role": "assistant", "content": "It is the sound of glass shattering."},
    {"role": "user", "content": [
        {"type": "text", "text": "What can you do when you hear that?"},
    ]},
    {"role": "assistant", "content": "Stay alert and cautious, and check if anyone is hurt or if there is any damage to property."},
    {"role": "user", "content": [
        {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac"},
        {"type": "text", "text": "What does the person say?"},
    ]},
]
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios = []
for message in conversation:
    if isinstance(message["content"], list):
        for ele in message["content"]:
            if ele["type"] == "audio":
                audios.append(
                    librosa.load(
                        BytesIO(urlopen(ele['audio_url']).read()), 
                        sr=processor.feature_extractor.sampling_rate)[0]
                )

inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True, device=DEFAULT_DEVICE)  # NEW
# inputs.input_ids = inputs.input_ids.to("cuda") # COMMENTED OUT, NOT NEEDED ANYMORE

generate_ids = model.generate(**inputs, max_length=256)
generate_ids = generate_ids[:, inputs.input_ids.size(1):]

response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
```


Comments:
Hey @ifsheldon, thanks for opening this PR!
have you considered using the following way of switching devices ?

`inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True).to(DEFAULT_DEVICE)`

I'm not sure adding `device` to the processor fits with our current design

@ylacombe 
> inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True).to(DEFAULT_DEVICE)

That works as well, but then the `WhisperFeatureExtractor` will run on cpu, so the performance is not optimal.

> I'm not sure adding device to the processor fits with our current design

I don't know what you mean by "current design". By looking at `class Qwen2AudioProcessor(ProcessorMixin)`, it seems there's no restriction on `__call__`, so I guess it's free to add any arguments that make sense for doing processing?