SongCreator: Lyrics-based Universal Song Generation
Authors: name: Shun Lei; name: Yixuan Zhou; name: Boshi Tang; name: Max W. Y. Lam; name: Feng Liu; name: Hangyu Liu; name: Jingcheng Wu; name: Shiyin Kang; name: Zhiyong Wu; name: Helen Meng
Published: 2024-09-09 19:37:07

Abstract: Music is an integral part of human culture, embodying human intelligence and
creativity, of which songs compose an essential part. While various aspects of
song generation have been explored by previous works, such as singing voice,
vocal composition and instrumental arrangement, etc., generating songs with
both vocals and accompaniment given lyrics remains a significant challenge,
hindering the application of music generation models in the real world. In this
light, we propose SongCreator, a song-generation system designed to tackle this
challenge. The model features two novel designs: a meticulously designed
dual-sequence language model (DSLM) to capture the information of vocals and
accompaniment for song generation, and an additional attention mask strategy
for DSLM, which allows our model to understand, generate and edit songs, making
it suitable for various song-related generation tasks. Extensive experiments
demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
competitive performances on all eight tasks. Notably, it surpasses previous
works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
it is able to independently control the acoustic conditions of the vocals and
accompaniment in the generated song through different prompts, exhibiting its
potential applicability. Our samples are available at
https://songcreator.github.io/.