FLUX that Plays Music
Authors: name: Zhengcong Fei; name: Mingyuan Fan; name: Changqian Yu; name: Junshi Huang
Published: 2024-09-01 02:43:33

Abstract: This paper explores a simple extension of diffusion-based rectified flow
Transformers for text-to-music generation, termed as FluxMusic. Generally,
along with design in advanced
Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
into a latent VAE space of mel-spectrum. It involves first applying a sequence
of independent attention to the double text-music stream, followed by a stacked
single music stream for denoised patch prediction. We employ multiple
pre-trained text encoders to sufficiently capture caption semantic information
as well as inference flexibility. In between, coarse textual information, in
conjunction with time step embeddings, is utilized in a modulation mechanism,
while fine-grained textual details are concatenated with the music patch
sequence as inputs. Through an in-depth study, we demonstrate that rectified
flow training with an optimized architecture significantly outperforms
established diffusion methods for the text-to-music task, as evidenced by
various automatic metrics and human preference evaluations. Our experimental
data, code, and model weights are made publicly available at:
https://github.com/feizc/FluxMusic.