MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
  Generation
Authors: name: Yun-Han Lan; name: Wen-Yi Hsiao; name: Hao-Chung Cheng; name: Yi-Hsuan Yang
Published: 2024-07-21 05:27:53

Abstract: Existing text-to-music models can produce high-quality audio with great
diversity. However, textual prompts alone cannot precisely control temporal
musical features such as chords and rhythm of the generated music. To address
this challenge, we introduce MusiConGen, a temporally-conditioned
Transformer-based text-to-music model that builds upon the pretrained MusicGen
framework. Our innovation lies in an efficient finetuning mechanism, tailored
for consumer-grade GPUs, that integrates automatically-extracted rhythm and
chords as the condition signal. During inference, the condition can either be
musical features extracted from a reference audio signal, or be user-defined
symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
on two datasets -- one derived from extracted features and the other from
user-created inputs -- demonstrates that MusiConGen can generate realistic
backing track music that aligns well with the specified conditions. We
open-source the code and model checkpoints, and provide audio examples online,
https://musicongen.github.io/musicongen_demo/.