Audio Conditioning for Music Generation via Discrete Bottleneck Features
Authors: name: Simon Rouard; name: Yossi Adi; name: Jade Copet; name: Axel Roebel; name: Alexandre Défossez
Published: 2024-07-17 13:47:17

Abstract: While most music generation models use textual or parametric conditioning
(e.g. tempo, harmony, musical genre), we propose to condition a language model
based music generation system with audio input. Our exploration involves two
distinct strategies. The first strategy, termed textual inversion, leverages a
pre-trained text-to-music model to map audio input to corresponding
"pseudowords" in the textual embedding space. For the second model we train a
music language model from scratch jointly with a text conditioner and a
quantized audio feature extractor. At inference time, we can mix textual and
audio conditioning and balance them thanks to a novel double classifier free
guidance method. We conduct automatic and human studies that validates our
approach. We will release the code and we provide music samples on
https://musicgenstyle.github.io in order to show the quality of our model.