WavJourney: Compositional Audio Creation with Large Language Models
Authors: name: Xubo Liu; name: Zhongkai Zhu; name: Haohe Liu; name: Yi Yuan; name: Meng Cui; name: Qiushi Huang; name: Jinhua Liang; name: Yin Cao; name: Qiuqiang Kong; name: Mark D. Plumbley; name: Wenwu Wang
Published: 2023-07-26 17:54:04

Abstract: Large Language Models (LLMs) have shown great promise in integrating diverse
expert models to tackle intricate language and vision tasks. Despite their
significance in advancing the field of Artificial Intelligence Generated
Content (AIGC), their potential in intelligent audio content creation remains
unexplored. In this work, we tackle the problem of creating audio content with
storylines encompassing speech, music, and sound effects, guided by text
instructions. We present WavJourney, a system that leverages LLMs to connect
various audio models for audio content generation. Given a text description of
an auditory scene, WavJourney first prompts LLMs to generate a structured
script dedicated to audio storytelling. The audio script incorporates diverse
audio elements, organized based on their spatio-temporal relationships. As a
conceptual representation of audio, the audio script provides an interactive
and interpretable rationale for human engagement. Afterward, the audio script
is fed into a script compiler, converting it into a computer program. Each line
of the program calls a task-specific audio generation model or computational
operation function (e.g., concatenate, mix). The computer program is then
executed to obtain an explainable solution for audio generation. We demonstrate
the practicality of WavJourney across diverse real-world scenarios, including
science fiction, education, and radio play. The explainable and interactive
design of WavJourney fosters human-machine co-creation in multi-round
dialogues, enhancing creative control and adaptability in audio production.
WavJourney audiolizes the human imagination, opening up new avenues for
creativity in multimedia content creation.