DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End
  Full-Length Song Generation with Latent Diffusion
Authors: name: Ziqian Ning; name: Huakang Chen; name: Yuepeng Jiang; name: Chunbo Hao; name: Guobin Ma; name: Shuai Wang; name: Jixun Yao; name: Lei Xie
Published: 2025-03-03 05:15:34

Abstract: Recent advancements in music generation have garnered significant attention,
yet existing approaches face critical limitations. Some current generative
models can only synthesize either the vocal track or the accompaniment track.
While some models can generate combined vocal and accompaniment, they typically
rely on meticulously designed multi-stage cascading architectures and intricate
data pipelines, hindering scalability. Additionally, most systems are
restricted to generating short musical segments rather than full-length songs.
Furthermore, widely used language model-based methods suffer from slow
inference speeds. To address these challenges, we propose DiffRhythm, the first
latent diffusion-based song generation model capable of synthesizing complete
songs with both vocal and accompaniment for durations of up to 4m45s in only
ten seconds, maintaining high musicality and intelligibility. Despite its
remarkable capabilities, DiffRhythm is designed to be simple and elegant: it
eliminates the need for complex data preparation, employs a straightforward
model structure, and requires only lyrics and a style prompt during inference.
Additionally, its non-autoregressive structure ensures fast inference speeds.
This simplicity guarantees the scalability of DiffRhythm. Moreover, we release
the complete training code along with the pre-trained model on large-scale data
to promote reproducibility and further research.