Add Qwen2-Audio
Author: faychu
Pull request: yes
Labels: name: Audio

name: run-slow

This PR adds the support of codes for the Qwen2-Audio models. For information about Qwen2-Audio, please visit https://github.com/QwenLM/Qwen2-Audio. @ArthurZucker 

Comments:
cc @sanchit-gandhi @kamilakesbi 

Super cool @faychu! Let us know if you have any questions regarding the model integration - more than happy to help here. Otherwise, ping us when this one is ready for review and we can take a look!

Also cc @ylacombe @Vaibhavs10 for viz

Hi, @sanchit-gandhi 

`Qwen/Qwen2-Audio-7B` is utilized in the tests, and I have created a private repository to host the corresponding model at https://huggingface.co/Qwen/Qwen2-Audio-7B. 

However, I encountered an issue while running the tests:
![20240723203639](https://github.com/user-attachments/assets/ee54c9f1-57fc-44c2-9f43-501d3f5075f8)

How can I resolve this problem?

Thank you in advance.



Hey @faychu, thanks for adding the model directly into transformers!

If your testing the model locally, you need to follow the instruction (`huggingface-cli login`) in order to be able to load the weights from the hub. Don't hesitate to read more about it [here](https://huggingface.co/docs/huggingface_hub/en/guides/cli#huggingface-cli-login) or ping us again in this PR.

If you're talking about the online CI tests, I'm not aware of being able to resolve this issue without making the model public (maybe @amyeroberts or @ydshieh have a solution?). 
But this shouldn't be a blocker for you if you can test everything locally. We can review the PR without having a fully green CI online, and in the meantime you can run slow tests locally, as long as you have a GPU.

Once the PR is approved, and every slow tests pass locally on your side, we can make the model public at the last moment just before integrating the PR, what do you think?

Hey @ylacombe 

Great! Thank you. I have successfully passed all the tests locally following the instructions on https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request, and I believe the code is ready.



Hey @ylacombe , I've made the changes you suggested to the code. Could you please take another look to see if it resolves the issues mentioned?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32137). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @amyeroberts or @ArthurZucker, thanks to @faychu's great work, I think the PR is ready for your review!

BTW, there's still the pending question about input ids being part of the generated ids (cf [here](https://github.com/huggingface/transformers/pull/32137#discussion_r1695247851)) that needs your insights !

@ylacombe That's great news! 

Regarding final review from a core maintainer: 
* If you think it's mergeable then it helps us to know this with a final review and approval from someone else 
* All tests should be passing. I triggered a re-run of the CI tests which looked like they were transient. At the moment, there's still a model-specific test which is failing `tests/models/qwen2_audio/test_modeling_qwen2_audio.py::Qwen2AudioForConditionalGenerationModelTest::test_batching_equivalence` 

For the input_ids in the outputs, this is unfortunately inconsistent across models in the library cc @gante here - should we be aiming to just return the generated ids? 

Hey @amyeroberts, thanks for your insights!

In that case, let's wait a bit for your review, I'll provide a final review when the current failing test is green again, and when the generated ids issue is resolved.

> For the input_ids in the outputs, this is unfortunately inconsistent across models in the library cc @gante here - should we be aiming to just return the generated ids?

This is what I thought as well, but what about @faychu's [rightful observation](https://github.com/huggingface/transformers/pull/32137#discussion_r1695391004) that Llava also includes the input ids ?
 

> This is what I thought as well, but what about @faychu's https://github.com/huggingface/transformers/pull/32137#discussion_r1695391004 that Llava also includes the input ids ?

Sorry, I wasn't clear. Yes, @faychu is right, some models do return the input_ids in the generated sequence, some don't. This has been a problem for building the mulitmodal pipelines involving VLMs. To the best of my knowledge, we don't normally return the input input_ids, but I'm not 100% sure tbh so was pinging @gante as he's more likely to know :) 

@amyeroberts I've addressed all your comments, and pushed a commit with the message `[run_slow] qwen2_audio`, and made all the tests green. But I don't know how to add the `run-slow` label. Have we met the requirements for merging?

@faychu I just added the label (you can find the `Labels` section on the right-hand side panel at the top of the PR). Could you push another commit with the `[run_slow] qwen2_audio` message to trigger the run?

@faychu Great - thanks for pushing that commit! At the moment - some of the multi-gpu tests are failing for the model, so these will need to be resolved before we can merge.

Hey @amyeroberts, I've resolved the multi-gpu tests locally, and pushed a commit with `[run_slow] qwen2_audio`, but it seems that the multi-gpu tests have not been triggered. What should I do now?

@ylacombe @amyeroberts It looks like the single-gpu test has been stuck for 15 hours and we need your help. 

@faychu Hmmmm - I'm not really sure what caused this. Sometimes there aren't enough machines available for the runs and they get stuck even if you try to trigger a re-run from the workflow page. It's more important that the tests pass for the multi-gpu runs, so if they're OK then I'm pretty sure the single GPU ones will be. 

@ylacombe In this case, are you happy to merge and be the one to follow up on tests if we find there's failures for the single GPU runs? 

Hey @amyeroberts, totally, I'd be happy to follow up on tests for this model.

@faychu, I see that there still some failling tests, are you sure you propagated the model repository id to every cases?

Merging now!