Qwen2-Audio Technical Report
Authors: name: Yunfei Chu; name: Jin Xu; name: Qian Yang; name: Haojie Wei; name: Xipin Wei; name: Zhifang Guo; name: Yichong Leng; name: Yuanjun Lv; name: Jinzheng He; name: Junyang Lin; name: Chang Zhou; name: Jingren Zhou
Published: 2024-07-15 14:38:09

Abstract: We introduce the latest progress of Qwen-Audio, a large-scale audio-language
model called Qwen2-Audio, which is capable of accepting various audio signal
inputs and performing audio analysis or direct textual responses with regard to
speech instructions. In contrast to complex hierarchical tags, we have
simplified the pre-training process by utilizing natural language prompts for
different data and tasks, and have further expanded the data volume. We have
boosted the instruction-following capability of Qwen2-Audio and implemented two
distinct audio interaction modes for voice chat and audio analysis. In the
voice chat mode, users can freely engage in voice interactions with Qwen2-Audio
without text input. In the audio analysis mode, users could provide audio and
text instructions for analysis during the interaction. Note that we do not use
any system prompts to switch between voice chat and audio analysis modes.
Qwen2-Audio is capable of intelligently comprehending the content within audio
and following voice commands to respond appropriately. For instance, in an
audio segment that simultaneously contains sounds, multi-speaker conversations,
and a voice command, Qwen2-Audio can directly understand the command and
provide an interpretation and response to the audio. Additionally, DPO has
optimized the model's performance in terms of factuality and adherence to
desired behavior. According to the evaluation results from AIR-Bench,
Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests
focused on audio-centric instruction-following capabilities. Qwen2-Audio is
open-sourced with the aim of fostering the advancement of the multi-modal
language community.