Presto! Distilling Steps and Layers for Accelerating Music Generation
Authors: name: Zachary Novack; name: Ge Zhu; name: Jonah Casebeer; name: Julian McAuley; name: Taylor Berg-Kirkpatrick; name: Nicholas J. Bryan
Published: 2024-10-07 16:24:18

Abstract: Despite advances in diffusion-based text-to-music (TTM) methods, efficient,
high-quality generation remains a challenge. We introduce Presto!, an approach
to inference acceleration for score-based diffusion transformers via reducing
both sampling steps and cost per step. To reduce steps, we develop a new
score-based distribution matching distillation (DMD) method for the EDM-family
of diffusion models, the first GAN-based distillation method for TTM. To reduce
the cost per step, we develop a simple, but powerful improvement to a recent
layer distillation method that improves learning via better preserving hidden
state variance. Finally, we combine our step and layer distillation methods
together for a dual-faceted approach. We evaluate our step and layer
distillation methods independently and show each yield best-in-class
performance. Our combined distillation method can generate high-quality outputs
with improved diversity, accelerating our base model by 10-18x (230/435ms
latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA) --
the fastest high-quality TTM to our knowledge. Sound examples can be found at
https://presto-music.github.io/web/.