Qwen-Audio: Advancing Universal Audio Understanding via Unified
  Large-Scale Audio-Language Models
Authors: name: Yunfei Chu; name: Jin Xu; name: Xiaohuan Zhou; name: Qian Yang; name: Shiliang Zhang; name: Zhijie Yan; name: Chang Zhou; name: Jingren Zhou
Published: 2023-11-14 05:34:50

Abstract: Recently, instruction-following audio-language models have received broad
attention for audio interaction with humans. However, the absence of
pre-trained audio models capable of handling diverse audio types and tasks has
hindered progress in this field. Consequently, most existing works have only
been able to support a limited range of interaction capabilities. In this
paper, we develop the Qwen-Audio model and address this limitation by scaling
up audio-language pre-training to cover over 30 tasks and various audio types,
such as human speech, natural sounds, music, and songs, to facilitate universal
audio understanding abilities. However, directly co-training all tasks and
datasets can lead to interference issues, as the textual labels associated with
different datasets exhibit considerable variations due to differences in task
focus, language, granularity of annotation, and text structure. To overcome the
one-to-many interference, we carefully design a multi-task training framework
by conditioning on a sequence of hierarchical tags to the decoder for
encouraging knowledge sharing and avoiding interference through shared and
specified tags respectively. Remarkably, Qwen-Audio achieves impressive
performance across diverse benchmark tasks without requiring any task-specific
fine-tuning, surpassing its counterparts. Building upon the capabilities of
Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from
various audios and text inputs, enabling multi-turn dialogues and supporting
various audio-central scenarios.