Presto! Distilling Steps and Layers for Accelerating Music Generation

## Abstract

Abstract Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge.
We introduce Presto! , an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step.
To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffus ion models, the first GAN-based distillation method for TTM.
To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method
that improves learning
via better preserving hidden state variance.
Finally, we combine our step and layer distillation methods together for a dual-faceted approach.
We evaluate our step and layer distillation methods independently and show each yield best-in-class performance.
Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than comparable SOTA)
— the fastest high-quality TTM to our knowledge.

## 1 Introduction

We have seen a renaissance of audio-domain generative media (Chen et al., ; Agostinelli et al., ; Liu et al., ; Copet et al., ), with increasing capabilities for both Text-to-Audio (TTA) and Text-to-Music (TTM) generation.
This work has been driven in-part by audio-domain *diffusion models* (Song et al., ; Ho et al., ; Song et al., ), enabling considerably better audio modeling than generative adversarial network (GAN) or variational autoencoder (VAE) methods (Dhariwal & Nichol, ).
Diffusion models, however, suffer from long inference times due to their iterative denoising process, requiring a substantial number of function evaluations (NFE)
during inference (i.e. sampling) and resulting in $\approx$5-20 seconds at best for non-batched latency.

Accelerating diffusion inference typically focuses on *step distillation*,
i.e. the process of reducing the *number* of sampling steps by distilling the diffusion model into a few-step generator. Methods include consistency-based  (Salimans & Ho, ; Song et al., ; Kim et al., ) and adversarial (Sauer et al., ; Yin et al., ; ; Kang et al., ) approaches.
Others have also investigated *layer-distillation* (Ma et al., ; Wimbauer et al., ; Moon et al., ), which draws from transformer early exiting (Hou et al., ; Schuster et al., ) by dropping interior layers to reduce the *cost* per sampling step for image generation.
For TTA/TTM models, however, distillation techniques have only been applied to shorter or lower-quality audio (Bai et al., ; Novack et al., ),
necessitate
$\approx$10 steps (vs. 1-4 step image methods)
to match base quality (Saito et al., ), and have not successfully used layer or GAN-based distillation methods.

Figure: Figure 1: Presto-S. Our goal is to distill the initial “real” score model (grey) $\mu_{\bm{\theta}}$ into a few-step generator (light blue) $G_{\bm{\phi}}$ to minimize the KL divergence between the distribution of $G_{\bm{\phi}}$’s outputs and the real distribution. This requires that we train an auxillary “fake” score model $\mu_{\bm{\psi}}$ (dark blue) that estimates the score of the *generator’s* distribution at each gradient step. Formally: (1) real audio is corrupted with Gaussian noise sampled from the generator noise distribution $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0.75390625,0,0}gen}}}(\sigma^{\text{inf}})$ which is then (2) passed into the generator to get its output. Noise is then added to this generation according to three *different* noise distributions: (3) $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor} {rgb}{0.984375,0.7421875,0}DMD}}}(\sigma^{\text{train}})$, which is (4) passed into both the real and fake score models to calculate the distribution matching gradient $\nabla_{\phi}\mathcal{L}_{\text{DMD}}$; (5) $p_{\text{{\color[rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{ pgfstrokecolor}{rgb}{0.1796875,0.37109375,0.49609375}DSM}}}(\sigma^{\text{ train/inf}})$, which is used to (6) train the fake score model on the *generator’s* distribution with $\mathcal{L}_{\text{fake-DSM}}$; and (7) an adversarial distribution $p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{ pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{train}})$, which along with the real audio is (8) passed into a least-squares discriminator built on the fake score model’s intermediate activations to calculate $\mathcal{L}_{\text{GAN}}$.
Refer to caption: x1.png

We present Presto(^1^11*Presto* is the common musical term denoting fast music from 168-200 beats per minute.), a dual-faceted distillation approach to inference acceleration for score-based diffusion transformers via reducing the number of sampling steps and the cost per step.
Presto includes three distillation methods:
(1) Presto-S, a new distribution matching distillation algorithm for *score-based*, EDM-style diffusion models
(see  Fig. 1) leveraging GAN-based step distillation with the flexibility of *continuous-time* models,
(2) Presto-L, a conditional layer distillation method designed to better preserve hidden state variance during distillation, and
(3) Presto-LS, a combined layer-step distillation method that critically uses layer distillation and *then* step distillation while disentangling layer distillation from real and fake score-based gradient estimation.

To evaluate our approach, we ablate the design space for both distillation processes. First, we show our step distillation method achieves best-in-class acceleration and quality via careful choice of loss noise distributions, GAN design, and continuous-valued inputs, the first such method to match base TTM diffusion sampling quality with 4-step inference.
Second, we show our layer distillation method offers a consistent improvement in both speed *and* performance over SOTA layer dropping methods and base diffusion sampling.
Finally, we show that layer-step distillation accelerates our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than the comparable SOTA model) while notably improving diversity over step-only distillation.

Overall, our core contributions include the development of a holistic approach to accelerating score-based diffusion transformers including:

- •
The development of distribution matching distillation for continuous-time score-based diffusion (i.e. EDM), the first GAN-based distillation method for TTA/TTM.
- •
The development of an improved layer distillation method that consistently improves upon both past layer distillation method and our base diffusion model.
- •
The development of the first combined layer and step distillation method.
- •
Evaluation showing our step, layer, and layer-step distillation methods are all best-in-class and, when combined, can accelerate our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than Stable Audio Open (Evans et al., )), the fastest TTM model to our knowledge. For sound examples, see [https://presto-music.github.io/web/](https://presto-music.github.io/web/).

## 2 Background & Related Work

### 2.1 Music Generation

Audio-domain music generation methods commonly use autoregressive (AR) techniques (Zeghidour et al., ; Agostinelli et al., ; Copet et al., ) or diffusion (Forsgren & Martiros, ; Liu et al., ; ; Schneider et al., ). Diffusion-based TTA/TTM (Forsgren & Martiros, ; Liu et al., ; ; Schneider et al., ; Evans et al., ) has shown the promise of full-text control (Huang et al., ), precise musical attribute control (Novack et al., ; ; Tal et al., ), structured long-form generation (Evans et al., ), and higher overall quality over AR methods (Evans et al., ; ; Novack et al., ; Evans et al., ). The main downside of diffusion, however, is that it is slow and thus not amenable to interactive-rate control.

### 2.2 Score-Based Diffusion Models

Continuous-time diffusion models have shown great promise over discrete-time models both for their improved performance on images (Balaji et al., ; Karras et al., ; Liu et al., ) *and* audio (Nistal et al., ; Zhu et al., ; Saito et al., ), as well as their relationship to the general class of flow-based models (Sauer et al., ; Tal et al., ).
Such models involve a forward noising process that gradually adds Gaussian noise to real audio signals $\bm{x}_{\text{real}}$ and
a reverse process that transforms pure Gaussian noise back into data (Song et al., ; Sohl-Dickstein et al., ). The reverse process is defined by a stochastic differential equation (SDE) with an equivalent ordinary differential equation (ODE) form called the *probability flow* (PF) ODE (Song et al., ):

$$ $\mathrm{d}\bm{x}=-\sigma\nabla_{\bm{x}}\log p(\bm{x}\mid\sigma)\mathrm{d}\sigma,$ (1) $$

where $\nabla_{\bm{x}}\log p(\bm{x}\mid\sigma)$ is the score function of the marginal density of $\bm{x}$ (i.e. the noisy data) at noise level $\sigma$ according to the forward diffusion process.
Thus, the goal of score-based diffusion models is to learn a *denoiser* network $\mu_{\bm{\theta}}$ such that $\mu_{\bm{\theta}}(\bm{x},\sigma)=\mathbb{E}[\bm{x}_{\text{real}}\mid\bm{x},\sigma]$. The score function is:

$$ $\nabla_{\bm{x}}\log p(\bm{x}\mid\sigma)\approx\frac{\bm{x}-\mu_{\bm{\theta}}( \bm{x},\sigma)}{\sigma}.$ (2) $$

Given a trained score model, we can generate samples at inference time by setting a decreasing *noise schedule* of $N$ levels $\sigma_{\text{max}}=\sigma_{N}>\sigma_{N-1}>\dots>\sigma_{0}=\sigma_{\text{min}}$ and iteratively solving the ODE at these levels using our model and any off-the-shelf ODE solver (e.g. Euler, Heun).

The EDM-family (Karras et al., ; ) of score-based diffusion models is of particular interest and unifies several continuous-time model variants within a common framework and improves model parameterization and training process.
The EDM score model is trained by minimizing a reweighted denoising score matching (DSM) loss (Song et al., ):

$$ $\mathcal{L}_{\text{DSM}}=\mathbb{E}_{\bm{x}_{\text{real}}\sim\mathcal{D}, \sigma\sim p(\sigma^{\text{train}}),\epsilon\sim\mathcal{N}(0,\bm{I})}\left[ \lambda(\sigma)\|\bm{x}_{\text{real}}-\mu_{\bm{\theta}}(\bm{x}_{\text{real}}+ \epsilon\sigma,\sigma)\|_{2}^{2}\right],$ (3) $$

where $p(\sigma^{\text{train}})$ denotes the *noise distribution* during training, and $\lambda(\sigma)$ is a noise-level weighting function. Notably, EDM defines a *different* noise distribution to discretize for inference $p(\sigma^{\text{inf}})$ that is distinct from $p(\sigma^{\text{train}})$ (see Fig. 2), as opposed to a noise schedule shared between training and inference.
Additionally, EDMs represent the denoising network using extra noise-dependent preconditioning parameters, training a network $f_{\bm{\theta}}$ with the parameterization:

$$ $\mu_{\bm{\theta}}(\bm{x},\sigma)=c_{\text{skip}}(\sigma)\bm{x}+c_{\text{out}}( \sigma)f_{\bm{\theta}}(c_{\text{in}}(\sigma)\bm{x},c_{\text{noise}}(\sigma)).$ (4) $$

For TTM models, $\mu_{\bm{\theta}}$ is equipped with various condition embeddings (e.g. text) $\mu_{\bm{\theta}}(\bm{x},\sigma,\bm{e})$. To increase text relevance and quality at the cost of diversity, we employ *classifier free guidance* (CFG) (Ho & Salimans, ), converting the denoised output to: $\tilde{\mu}^{w}_{\bm{\theta}}(\bm{x},\sigma,\bm{e})=\mu_{\bm{\theta}}(\bm{x},
\sigma,\bm{\emptyset})+w(\mu_{\bm{\theta}}(\bm{x},\sigma,\bm{e})-\mu_{\bm{
\theta}}(\bm{x},\sigma,\bm{\emptyset}))$, where $w$ is the guidance weight and $\bm{\emptyset}$ is a “null” conditioning.

### 2.3 Diffusion Distillation

Step distillation is the process of reducing diffusion sampling steps by distilling a base model into a few-step generator. Such methods can be organized into two broad categories.
Online consistency approaches such as consistency models (Song et al., ), consistency trajectory models (Kim et al., ), and variants (Ren et al., ; Wang et al., ) distill directly by enforcing consistency across the diffusion trajectory and optionally include an adversarial loss (Kim et al., ).
While such approaches have strong 1-step generation for images, attempts for audio have been less successful and only capable of generating short segment (i.e. $<10$ seconds), applied to lower-quality base models limiting upper-bound performance, needing up to 16 sampling steps to match baseline quality (still slow),
and/or did not successfully leverage adversarial losses which have been found to increase realism for other domains (Bai et al., ; Saito et al., ; Novack et al., ).

In contrast, offline adversarial distillation methods include Diffusion2GAN (Kang et al., ), LADD (Sauer et al., ), and DMD (Yin et al., ).
Such methods work by generating large amounts of offline noise–sample pairs from the base model, and finetuning the model into a conditional GAN for few-step synthesis.
These methods can surpass their adversarial-free counterparts, yet require expensive offline data generation and massive compute infrastructure to be efficient.

Alternatively, improved DMD (DMD2) (Yin et al., ) introduces an online adversarial diffusion distillation method for images.
DMD2 (1) removes the need for expensive offline data generation (2) adds a GAN loss and (3) outperforms consistency-based methods and improves overall quality.
DMD2 primarily works by distilling a one- or few-step generator $G_{\bm{\phi}}$ from a base diffusion model $\mu_{\text{real}}$, while simultaneously learning a score model of the generator’s distribution online $\mu_{\text{fake}}$ in order to approximate a target KL objective (with $\mu_{\text{real}}$) used to train the generator.
To our knowledge, there are no adversarial diffusion distillation methods for TTM or TTA.

Beyond step distillation, layer distillation, or the process of dropping interior layers to reduce the cost per sampling step, has been recently studied (Moon et al., ; Wimbauer et al., ). Layer distillation draws inspiration from transformer early exiting and layer caching (Hou et al., ; Schuster et al., ) and has found success for image diffusion, but has not been compared or combined with step distillation methods and has not been developed for TTA/TTM. In our work, we seek to understand how step and layer distillation interact for accelerating music generation.

## 3 Presto!

We propose a dual-faceted distillation approach for inference acceleration of continuous-time diffusion models.
Continuous-time models have been shown to outperform discrete-time DDPM models (Song et al., ; Karras et al., ; ), but past DMD/DMD2 work focuses on the latter.
Thus, we redefine DMD2 (a step distillation method) in Section 3.1 for continuous-time score models, then present
an improved formulation and study its design space in Section 3.2.
Second, we design a simple, but powerful improvement to the SOTA layer distillation method to understand the impact of reducing inference cost per step in Section 3.3.
Finally, we investigate how to combine step and layer distillation methods together in Section 3.4.

### 3.1 EDM-Style Distribution Matching Distillation

We first redefine DMD2 in the language of continuous-time, score-based diffusion models (i.e. EDM-style).
Our goal is to
distill
our score model $\mu_{\bm{\theta}}$ (which we equivalently denote as $\mu_{\text{real}}$, as it is trained to model the score of real data) into an accelerated
generator $G_{\bm{\phi}}$
that can sample in 1-4 steps.
Formally, we wish to minimize the KL Divergence between the real distribution $p_{\text{real}}$ and the generator $G_{\bm{\phi}}$’s distribution $p_{\text{fake}}$: $\mathcal{L}_{\text{DMD}}=D(p_{\text{real}}\|p_{\text{fake}})$.
The KL term cannot be calculated explicitly, but we can calculate its *gradient* with respect to the generator if we can access the score of the generator’s distribution.
Thus, we also train a “fake” score model $\mu_{\bm{\psi}}$ (or equivalently, $\mu_{\text{fake}}$) to approximate the generator distribution’s *score function* at each gradient step during training.

First, given some real data $\bm{x}_{\text{real}}$, we sample a noise level from a set of predefined levels $\sigma\sim\{\sigma_{i}\}_{\text{gen}}$,
and then pass the corrupted real data through the generator to get the generated output $\hat{\bm{x}}_{\text{gen}}=G_{\bm{\phi}}(\bm{x}_{\text{real}}+\sigma\epsilon,\sigma)$, where $\epsilon\sim\mathcal{N}(0,\bm{I})$ (we omit the conditioning $\bm{e}$ for brevity).
The gradient of the KL divergence between the real and the generator’s distribution can then be calculated as:

$$ $\nabla_{\bm{\phi}}\mathcal{L}_{\text{DMD}}=\mathbb{E}_{\begin{subarray}{c} \sigma\sim\{\sigma_{i}\},\epsilon\sim\mathcal{N}(0,\bm{I})\end{subarray}}\left [\left((\mu_{\text{fake}}(\hat{\bm{x}}_{\text{gen}}+\sigma\epsilon,\sigma)- \tilde{\mu}_{\text{real}}^{w}(\hat{\bm{x}}_{\text{gen}}+\sigma\epsilon,\sigma) \right)\nabla_{\bm{\phi}}\hat{\bm{x}}_{\text{gen}}\right],$ (5) $$

where $\{\sigma_{i}\}$ are the predefined noise levels for all loss calculations, and $\tilde{\mu}_{\text{real}}^{w}$ is the *CFG-augmented* real score model. To ensure that $\mu_{\text{fake}}$ accurately models the score of the generator’s distribution at each gradient update, we train the fake score model with the weighted-DSM loss (i.e. standard diffusion training), but on *the generator outputs*:

$$ $\arg\min_{\bm{\psi}}\mathcal{L}_{\text{fake-DSM}}=\mathbb{E}_{\sigma\sim\{ \sigma_{i}\},\epsilon\sim\mathcal{N}(0,\bm{I})}\left[\lambda(\sigma)\|\hat{\bm {x}}_{\text{gen}}-\mu_{\text{fake}}(\hat{\bm{x}}_{\text{gen}}+\sigma\epsilon, \sigma)\|_{2}^{2}\right]$ (6) $$

To avoid using offline data (Yin et al., ), the fake score model is updated *5 times as often* as the generator to stabilize the estimation of the generator’s distribution.
DMD2 additionally includes an explicit adversarial loss in order to improve quality. Specifically, a discriminator head $D_{\bm{\psi}}$ is attached to the intermediate feature activations
of the fake score network $\mu_{\text{fake}}$, and thus is trained with the nonsaturating GAN loss:

$$ $\arg\min_{\bm{\phi}}\max_{\bm{\psi}}\mathbb{E}_{\begin{subarray}{c}\sigma\sim \{\sigma_{i}\},\\ \epsilon\sim\mathcal{N}(0,\bm{I})\end{subarray}}[\log D_{\bm{\psi}}(\bm{x}_{ \text{real}}+\sigma\epsilon,\sigma)]+\mathbb{E}_{\begin{subarray}{c}\sigma\sim \{\sigma_{i}\},\\ \epsilon\sim\mathcal{N}(0,\bm{I})\end{subarray}}[-\log D_{\bm{\psi}}(\hat{\bm{ x}}_{\text{gen}}+\sigma\epsilon,\sigma)],$ (7) $$

which follows past work on using diffusion model backbones as discriminators (Sauer et al., ). In all, the generator $G_{\bm{\phi}}$ is thus trained with a combination of the distribution matching loss $\mathcal{L}_{\text{DMD}}$ and the adversarial loss $\mathcal{L}_{\text{GAN}}$, while the fake score model (and its discriminator head) is trained with the fake DSM loss $\mathcal{L}_{\text{fake-DSM}}$ and the adversarial loss $\mathcal{L}_{\text{GAN}}$. To sample from the distilled generator, DMD2 uses consistency model-style “ping-pong sampling” (Song et al., ), where the model iteratively denoises (starting at pure noise $\sigma_{\text{max}}$) and *renoises* to progressively smaller noise levels.

Regarding past work, we note Yin et al. () *did* present a small-scale EDM-style experiment,
but treated EDM models as if they were functions of discrete noise timesteps.
This re-discretization runs counterintuitive to using score-based models for distribution matching, since the fake and real score models are meant to be run and trained in continuous-time and can adapt to variable points along the noise process.
Furthermore, this disregards the ability of continuous-time models to capture the *entire* noise process from noise to data and enable *exact* likelihoods rather than ELBOs (Song et al., ).
Additionally, since DMD2 implicitly aims to learn an integrator of the PF ODE
$G_{\bm{\phi}}(\bm{x},\sigma)\approx\bm{x}+\int_{\sigma}^{\sigma_{\text{min}}}-
\delta\nabla\log p(\bm{x}\mid\delta)\textrm{d}\delta$ (like other data-prediction distillation methods (Song et al., )), learning this integral for any small set $\{\sigma_{i}\}$ restricts the generator’s modeling capacity.

### 3.2 Presto-S: Score-based Distribution Matching Distillation

Given the new notation, we proceed to develop our *score-based* distribution matching step distillation, Presto-S. Our algorithm is shown in Fig. 1, in Appendix 1, and discussed below.

#### 3.2.1 Continuous-Time Generator Inputs

In Section 3.1, the noise level and/or timestep is sampled from a *discrete*, hand-chosen set $\{\sigma_{i}\}_{\text{gen}}$.
Discretizing inputs, however, forces the model to 1) be a function of a specific *number* of steps, requiring users to retrain separate models for each desired step budget (Yin et al., ; Kohler et al., ) and 2) be a function of *specific* noise levels, which may not be optimally aligned with where different structural, semantic, and perceptual features arise in the diffusion process (Si et al., ; Kynkäänniemi et al., ; Balaji et al., ; Sabour et al., ).
When extending to continuous-time models, we train the distilled generator $G_{\bm{\phi}}$ as a function of the continuous noise level sampled from the *distribution* $\sigma\sim p(\sigma)$.
This allows our generator to both adapt better to variable budgets and to variable noise levels, as the generator can be trained with all noise levels sampled from $p(\sigma)$.

#### 3.2.2 Perceptual Loss Weighting with Variable Noise Distributions

A key difference between discrete-time and continuous-time diffusion models is the need for *discretization* of the noise process during inference.
In discrete models, a single noise schedule defines a particular mapping between timestep $t$ and its noise level $\sigma$, and is fixed throughout training and inference.
In continuous-time EDM models, however, we use a noise *distribution* $p(\sigma^{\text{train}})$ to sample during training, and a separate noise distribution for inference $p(\sigma^{\text{inf}})$ that is discretized to define the sampling schedule.
In particular, when viewed in terms of the *signal-to-noise ratio* $1/\sigma^{2}$ or SNR as shown in Fig. 2, the *training* noise distribution puts the majority of its mass in the mid-to-high SNR range of the diffusion process. This design choice focuses on semantic and perceptual features, while the *inference* noise distribution is more evenly distributed but with a bias towards the low-SNR region, giving a bias to low-frequency features.

Figure: Figure 2: Training/Inference distributions for EDM models, in decibel SNR space.
Refer to caption: x2.png

However, recall that *every* loss term including (5), (6), and (7) requires an additional re-corruption process that must follow a noise distribution, significantly expanding the design space for score-based models.
Thus, we disentangle these forward diffusion processes and replace the shared discrete noise set with four *separate noise distributions* $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.75390625,0,0}gen}}}$, $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0.984375,0.7421875,0}DMD}}}$, $p_{\text{{\color[rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{
pgfstrokecolor}{rgb}{0.1796875,0.37109375,0.49609375}DSM}}}$, and $p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}$, corresponding to the inputs to the generator and each loss term respectively, with no restrictions on how each weights each noise level
(rather than forcing a particular noise weighting for all computation).

Then, if we apply the original DMD2 method naively to the EDM-style of score-models, we get $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.75390625,0,0}gen}}}(\sigma^{\text{inf}})=p_{\text{{\color[rgb]{
0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.984375,0.7421875,0}DMD}}}(\sigma^{\text{inf}})=p_{\text{{\color[rgb]{
0.1796875,0.37109375,0.49609375}\definecolor[named]{pgfstrokecolor}{rgb}{
0.1796875,0.37109375,0.49609375}DSM}}}(\sigma^{\text{inf}})=p_{\text{{\color[
rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{inf}})$.
This choice of $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.75390625,0,0}gen}}}(\sigma^{\text{inf}})$ reasonably aligns the generator inputs during distillation to the inference process itself, but each loss noise distribution is somewhat misaligned from its role in the distillation process.
In particular:

- •
$p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0.984375,0.7421875,0}DMD}}}$: The distribution matching gradient is the only point that the generator gets a signal from the *CFG-augmented* outputs of the teacher. CFG is critical for text following, but *primarily* within the mid-to-high SNR region of the noise (Kynkäänniemi et al., ).
- •
$p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}$: As in most adversarial distillation methods (Sauer et al., ; Yin et al., ), the adversarial loss’s main strength is to increase the perceptual *realism/quality* of the outputs, which arise in the mid-to-high SNR regions, rather than structural elements.
- •
$p_{\text{{\color[rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{
pgfstrokecolor}{rgb}{0.1796875,0.37109375,0.49609375}DSM}}}$: The score model training should in theory mimic standard diffusion training, and may benefit from the training distribution’s provably faster convergence (Wang et al., ) (as the fake score model is updated *online* to track the generator’s distribution).

Thus, we shift all of the above terms to use the training distribution $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0.984375,0.7421875,0}DMD}}}(\sigma^{\text{train}}),p_{\text{{\color[rgb]
{0.1796875,0.37109375,0.49609375}\definecolor[named]{pgfstrokecolor}{rgb}{
0.1796875,0.37109375,0.49609375}DSM}}}(\sigma^{\text{train}})$ and $p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{train}})$
to force the distillation process to focus on perceptually relevant noise regions.

#### 3.2.3 Audio-Aligned Discriminator Design

The original DMD2 uses a classic non-saturating GAN loss. The discriminator is a series of convolutional blocks downsampling the intermediate features into a *single* probability for real vs. fake. While this approach is standard in image-domain applications, many recent adversarial waveform synthesis works (Kumar et al., ; Zhu et al., ) use a *Least-Squares* GAN loss:

$$ $\arg\min_{\bm{\phi}}\max_{\bm{\psi}}\mathbb{E}_{\begin{subarray}{c}\sigma\sim p _{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{ pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{train}} ),\\ \epsilon\sim\mathcal{N}(0,\bm{I})\end{subarray}}[\|D_{\bm{\psi}}(\bm{x}_{\text {real}}+\sigma\epsilon,\sigma)\|_{2}^{2}]+\mathbb{E}_{\begin{subarray}{c} \sigma\sim p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[ named]{pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{ train}}),\\ \epsilon\sim\mathcal{N}(0,\bm{I})\end{subarray}}[\|1-D_{\bm{\psi}}(\hat{\bm{x} }_{\text{gen}}+\sigma\epsilon,\sigma)\|_{2}^{2}],$ (8) $$

where the outputs of the discriminator $D_{\bm{\psi}}$ are only *partially* downsampled into a lower-resolution version of the input data (in this case, a latent 1-D tensor). This forces the discriminator to attend to more fine-grained, temporally-aligned features for determining realness, as the loss is averaged across the partially downsampled discriminator outputs. Hence, we use this style of discriminator for Presto-S to both improve and stabilize (Mao et al., ) the GAN gradient into our generator.

### 3.3 Presto-L: Variance and Budget-Aware Layer Dropping

Figure: Figure 3: Baseline layer dropping (left) vs. Presto-L (right). Standard layer dropping uses the noise level $\sigma$ to set the budget of layers to drop, starting from the back of the DiT blocks. Presto-L shifts this dropping by one to the second-to-last block and adds explicit budget conditioning.
Refer to caption: x3.png

Given our step distillation approach above, we now seek methods to reduce the cost of individual *steps* themselves here through layer distillation, and then combine both step and layer distillation in Section 3.4.
We begin with the current SOTA method: ASE (Moon et al., ). ASE employs a fixed dropping schedule that monotonically maps noise levels to compute budgets, allocating more layers to lower noise levels. We enhance this method in three key ways: (1) ensuring consistent variance in layer distilled outputs, (2) implementing explicit budget conditioning, and (3) aligning layer-dropped outputs through direct distillation.

Variance Preservation:
First, we inspect the within-layer activation variance of our base model in Fig. 4. We find that while the variance predictably
increases over depth, it notably spikes *on the last layer* up to an order of magnitude higher, indicating that the last layer behaves much differently as it is the direct input to the linear de-embedding layer.
ASE, however, always drops layers starting from the *last* layer and working backwards, thus always removing this behavior.
Hence, we remedy this fact and *shift* the layer dropping schedule by 1 to drop starting at the *second* to last layer, always rerouting back into the final layer to preserve the final layer’s behavior.

Figure: Figure 4: Hidden activation variance vs. layer depth. Each line is a unique noise level.
Refer to caption: x4.png

Budget Conditioning:
We include *explicit* budget conditioning into the model itself so that the model can directly adapt computation to the budget level.
This conditioning comes in two places: (1) a global budget embedding added to the noise level embedding, thus contributing to the internal Adaptive Layer Norm (AdaLN) conditioning inside the DiT blocks, and (2) an additional AdaLN layer on the outset of the DiT blocks conditional only on the budget, in order to effectively rescale the outputs to account for the change in variance. Following (Peebles & Xie, ; Zhang et al., ), we zero-initialize both budget conditioning modules to improve finetuning stability.

Knowledge Distillation: To encourage distillation without holding the base model in memory, we employ a
*self-teacher* loss.
Formally, if $h_{\bm{L}}(\bm{x},\bm{\theta})$ and $h_{\text{full}}(\bm{x},\bm{\theta})$ are the normalized outputs of the final DiT layer with and without layer dropping respectively,
the self-teacher loss is $\mathcal{L}_{\text{st}}=\|h_{\bm{L}}(\bm{x},\bm{\theta})-\texttt{sg}(h_{\text{
full}}(\bm{x},\bm{\theta}))\|_{2}^{2},$
where sg denotes a stop-gradient. This gives additional supervision during the early phases of finetuning so the layer-dropped outputs can match full model performance.

We show the differences between our Presto-L and the baseline approach in Fig. 3. By conditioning directly on the budget, and shifting the dropping schedule to account for the final DiT block behavior, we able to more adapt computation for reduced budgets while preserving performance.

### 3.4 Presto-LS: Layer-Step Distillation

As the act of layer distillation is, in principle, unrelated to the step distillation, there is no reason *a priori* that these methods could not work together.
However, we found combining such methods to be surprisingly non-trivial.
In particular, we empirically find that attempting both performing Presto-L finetuning and Presto-S at the same time OR performing Presto-L finetuning from an initial Presto-S checkpoint results in large instability and model degradation, as the discriminator dominates the optimization process and achieves near-perfect accuracy on real data.

We instead find three key factors in making combined step and layer distillation work:
(1) *Layer-Step Distillation* – we first perform layer distillation then step distillation, which is more stable as the already-finetuned layer dropping prevents generator collapse;
(2) *Full Capacity Score Estimation* – we keep the real and fake score models initialized from the *original* score model rather than the layer-distilled model, as this stabilizes the distribution matching gradient and provides regularization to the discriminator since the fake score model and the generator are initialized with different weights; and
(3) *Reduced Dropping Budget* – we keep more layers during the layer distillation. We discuss more in Section 4.6 and how alternatives fail in Appendix A.4.

## 4 Experiments

We show the efficacy of Presto via a number of experiments. We first ablate the design choices afforded by Presto-S, and separately show how Presto-L flatly improves standard diffusion sampling. We then show how Presto-L and Presto-S stack up against SOTA baselines, and how we can combine such approaches for further acceleration, with both quantitative and subjective metrics. We finish by describing a number of extensions enabled by our accelerated, continuous-time framework.

### 4.1 Setup

Model: We use latent diffusion (Rombach et al., ) with a fully convolutional VAE (Kumar et al., ) to generate mono 44.1kHz audio and convert to stereo using MusicHiFi (Zhu et al., ).
Our latent diffusion model builds upon DiT-XL (Peebles & Xie, ) and takes in three conditioning signals: the noise level, text prompts, and beat per minute (BPM) for each song. We use FlashAttention-2 (Dao, ) for the DiT and torch.compile for the VAE decoder and MusicHiFi.
For more details, see Appendix A.1.

Data: We use a 3.6K hour dataset of mono 44.1 kHz licensed instrumental music, augmented with pitch-shifting and time-stretching. Data includes musical meta-data and synthetic captions.
For evaluation, we use Song Describer (no vocals) (Manco et al., ) split into 32 second chunks.

Baselines: We compare against a number of acceleration algorithms and external models, including Consistency Models (CM) (Song et al., ), SoundCTM (Saito et al., ), DITTO-CTM (Novack et al., ),
DMD-GAN (Yin et al., ),
ASE (Moon et al., ),
MusicGen (Copet et al., ),
and Stable Audio Open (Evans et al., ). See Appendix A.2 for more details.

Metrics: We use Frechet Audio Distance (FAD) (Kilgour et al., ), Maximum Mean Discrepancy (MMD) (Jayasumana et al., ), and Contrastive Language-Audio Pretraining (CLAP) score (Wu et al., ), all with the CLAP-LAION music backbone (Wu et al., ) given its high correlation with human perception (Gui et al., ). FAD and MMD measure audio quality/realness with respect to Song Describer (lower better), and CLAP score measures prompt adherence (higher better).
When comparing to other models, we
also include density (measuring quality), recall and coverage (measuring diversity) (Naeem et al., ), and real-time factor (RTF) for both mono (M) and stereo (S, using MusicHiFi), which measures the total seconds of audio generated divided by the generation time, where higher is better for all.

### 4.2 Exploring the Design Space of Presto-S

**Table 1: (Top) Comparing different choices of noise distribution for the Presto-S process. (Bottom) for best performing noise distributions, performance for standard GAN design vs. proposed least-squares GAN.**
| $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.75390625,0,0}gen}}}$ | $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}<br>{rgb}{0.984375,0.7421875,0}DMD}}}$ | $p_{\text{{\color[rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1796875,0.37109375,0.49609375}DSM}}}$ | $p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}$ | FAD | MMD | CLAP |
| --- | --- | --- | --- | --- | --- | --- |
| Least-Squares GAN |  |  |  |  |  |  |
| Inf. | Inf. | Inf. | Inf. | 0.37 | 1.73 | 27.45 |
| Inf. | Inf. | Tr. | Inf. | 0.37 | 1.58 | 26.45 |
| Inf. | Inf. | Tr. | Tr. | 0.37 | 1.51 | 24.90 |
| Inf. | Tr. | Tr. | Inf. | 0.27 | 1.27 | 33.12 |
| Inf. | Tr. | Inf. | Tr. | 0.23 | 0.86 | 33.29 |
| Inf. | Tr. | Tr. | Tr. | 0.22 | 0.83 | 33.13 |
| Tr. | Tr. | Tr. | Tr. | 0.24 | 0.99 | 30.89 |
| Non-Saturating GAN |  |  |  |  |  |  |
| Inf. | Tr. | Inf. | Tr. | 0.24 | 0.89 | 31.48 |
| Inf. | Tr. | Tr. | Tr. | 0.25 | 0.96 | 31.78 |
| Tr. | Tr. | Tr. | Tr. | 0.26 | 1.04 | 29.46 |

Loss Distribution Choice: In Table 1 (Top), we show the FAD, MMD, and CLAP score for many Presto-S distilled models with different noise distribution choices.
We find that the original DMD2 (Yin et al., ) setup (first row) underperforms compared to adapting the loss distributions.
The largest change is in switching $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0.984375,0.7421875,0}DMD}}}$ to the training distribution, which improves all metrics.
This confirms our hypothesis that by focusing on the region most important for text guidance (Kynkäänniemi et al., ), we improve both audio quality and text adherence.
Switching $p_{\text{{\color[rgb]{0.57421875,0.81640625,0.3125}\definecolor[named]{
pgfstrokecolor}{rgb}{0.57421875,0.81640625,0.3125}GAN}}}$ to the training distribution also helps; in this case, the discriminator is made to focus on higher-frequency features (Si et al., ), benefiting quality.
We also find only a small improvement when using the training distribution for $p_{\text{{\color[rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{
pgfstrokecolor}{rgb}{0.1796875,0.37109375,0.49609375}DSM}}}$.
This suggests that while the training distribution should lead to more stable learning of the online generator’s score (Wang et al., ), this may not be crucial.
For all remaining experiments, we use $p_{\text{{\color[rgb]{0.984375,0.7421875,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0.984375,0.7421875,0}DMD}}}(\sigma^{\text{train}})=p_{\text{{\color[rgb]
{0.57421875,0.81640625,0.3125}\definecolor[named]{pgfstrokecolor}{rgb}{
0.57421875,0.81640625,0.3125}GAN}}}(\sigma^{\text{train}})=p_{\text{{\color[
rgb]{0.1796875,0.37109375,0.49609375}\definecolor[named]{pgfstrokecolor}{rgb}{
0.1796875,0.37109375,0.49609375}DSM}}}(\sigma^{\text{train}})$
and $p_{\text{{\color[rgb]{0.75390625,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0.75390625,0,0}gen}}}(\sigma^{\text{inf}})$.

Discriminator Design: We ablate the effect of switching from the chosen least-squares discriminator to the original softplus non-saturating discriminator, which notable treats the discriminator as a binary classifier and predicts the probability of real/generated. In Table 1 (Bottom), we find that using the least-squares discriminator leads to consistent improvements in audio quality (FAD/MMD) and in particular text relevance (CLAP), owing to the increased stability from the least-squares GAN.

Figure: Figure 5: Continuous generator inputs vs. discrete inputs. Continuous inputs shows more consistent scaling with compute, while generally performing better in both quality and text relevance.
Refer to caption: x5.png

Continuous vs. Discrete Generator Inputs:
We test how *continuous-time* conditioning compares against a discrete and find the former is preferred as shown in Fig. 5.
Continuous noise levels maintain a correlation where more steps improve quality, while discrete time models are more inconsistent.
Additionally, the continuous-time conditioning performs best in text relevance.
While the 1 and 2-step discrete models show slightly better FAD metrics than continuous on 1 and 2-step sampling, these models have a failure mode as shown in Fig. 11:
2-step discrete models drop high-frequency information and render transients (i.e. drum hits) poorly for genres like R&B or hip-hop.

### 4.3 Presto-L Results

Figure: Figure 6: Presto-L results. Presto-L improves both the latency *and* the overall performance across all metrics, against both the leading layer dropping baseline and the base model.
Refer to caption: x6.png

We compare Presto-L with both our baseline diffusion model and ASE (Moon et al., ) using the 2nd order DPM++ sampler (Lu et al., ) with CFG++ (Chung et al., ).
For ASE and Presto-L, we use the optimal “D3” configuration from Moon et al. (), which corresponds to a dropping schedule, in terms of decreasing noise level (in quintiles), of $[14,12,8,4,0]$ (i.e. we drop 14 layers for noise levels in the top quintile, 12 for the next highest quintile, and so on).
Layer distillation results at various sampling budgets are shown in Fig. 6.
Presto-L yields an improvement over the base model on all metrics, speeding up by $\approx$27% *and* improving quality and text relevance.
ASE provides similar acceleration but degrades performance at high sampling steps and scales inconsistently.
The behavior of dropping layers *improving* performance can be viewed via the lens of multi-task learning, where (1) denoising each noise level is a different task (2) later layers only activating for lower noise levels enables specialization for higher frequencies. See Appendix A.7 for further ablations.

**Table 2: Full Results on Song Describer (No vocals).^∗External baseline RTFs are all natively stereo.**
| Model | NFE | RTF-M/S ($\uparrow$) | FAD ($\downarrow$) | MMD ($\downarrow$) | CLAP Score ($\uparrow$) | Density ($\uparrow$) | Recall ($\uparrow$) | Coverage($\uparrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| External Baselines* |  |  |  |  |  |  |  |  |
| MusicGen-Small | 1.6K | 0.77 | 0.31 | 1.60 | 30.61 | 0.36 | 0.16 | 0.43 |
| MusicGen-Medium | 1.6K | 0.39 | 0.27 | 1.30 | 31.85 | 0.43 | 0.19 | 0.54 |
| MusicGen-Large | 1.6K | 0.37 | 0.25 | 1.21 | 32.83 | 0.44 | 0.15 | 0.54 |
| Stable Audio Open | 100 | 4.54 | 0.23 | 1.07 | 35.05 | 0.29 | 0.37 | 0.49 |
| Base Model, Diffusion Sampling |  |  |  |  |  |  |  |  |
| DPM-2S | 80 | 7.72 / 7.34 | 0.24 | 0.82 | 31.56 | 0.31 | 0.20 | 0.41 |
| DPM-2S+ASE | 80 | 9.80 / 9.22 | 0.25 | 1.12 | 30.03 | 0.27 | 0.16 | 0.41 |
| DPM-2S+Presto-L (ours) | 80 | 9.80 / 9.22 | 0.18 | 0.61 | 32.28 | 0.38 | 0.29 | 0.51 |
| Consistency-Based Distillation |  |  |  |  |  |  |  |  |
| CM | 4 | 118.77 / 67.41 | 0.47 | 2.50 | 26.33 | 0.17 | 0.01 | 0.16 |
| SoundCTM | 4 | 105.78 / 63.01 | 0.35 | 1.72 | 29.61 | 0.17 | 0.17 | 0.26 |
| DITTO-CTM | 4 | 118.77 / 67.41 | 0.36 | 1.62 | 28.31 | 0.22 | 0.04 | 0.32 |
| Adversarial Distillation |  |  |  |  |  |  |  |  |
| DMD-GAN | 4 | 118.77 / 67.41 | 0.29 | 1.16 | 27.56 | 0.57 | 0.07 | 0.41 |
| Presto-S (ours) | 4 | 118.77 / 67.41 | 0.22 | 0.83 | 33.13 | 0.60 | 0.10 | 0.50 |
| Presto-LS (ours) | 4 | 138.84 / 73.43 | 0.23 | 0.73 | 32.21 | 0.49 | 0.14 | 0.48 |

### 4.4 Full Comparison

In Table 2, we compare against multiple baselines and external models.
For step distillation, Presto-S is best-in-class and the only distillation method to close to base model quality, while achieving an over 15x speedup in RTF from the base model.
Additionally, Presto-LS improves performance for MMD, beating the base model with further speedups (230/435ms latency for 32 second mono/stereo 44.1kHz on an A100 40 GB). We also find Presto-LS improves *diversity* with higher recall. Overall, Presto-LS is 15x faster than SAO. We investigate latency more in Appendix A.6.

### 4.5 Listening Test

We also conducted a subjective listening test to compare Presto-LS with our base model, the best non-adversarial distillation technique SoundCTM (Saito et al., ) distilled from our base model, and Stable Audio Open (Evans et al., ). Users ($n=16$) were given 20 sets of examples generated from each model
(randomly cut to 10s for brevity) using random prompts from Song Describer and asked to rate the musical quality, taking into account both fidelity and semantic text match between 0-100. We run multiple paired t-tests with Bonferroni correction and find Presto-LS rates highest against all baselines ($p<0.05$). We show additional plots in Fig. 12.

### 4.6 Presto-LS Qualitative Analysis

Figure: Figure 7: Presto-S hidden activation var.
Refer to caption: x7.png

While Presto-LS improves speed and quality/diversity over step-only distillation, the increases are modest,
as the dropping schedule for Presto-L
was reduced ($[12,8,8,0,0]$) for step distillation stability.
To investigate more, we analyze the hidden state activation variance of our step-distilled model in Fig. 7.
The behavior is quite different than the base model,
as the “spike” in the final layer is more amortized across the last 10 layers and never reaches the base model’s magnitude.
We hypothesize step-distilled models have more unique computation *throughout* each DiT block, making layer dropping difficult.

### 4.7 Extensions

Adaptive Step Schedule: A benefit of our continuous-time distillation is that besides setting how many steps (e.g., 1-4), we can set *where* those steps occur along the diffusion process by tuning the $\rho$ parameter in the EDM inference schedule, which is normally set to $\rho=7$.
In particular, decreasing $\rho$ (lower bounded by 1) puts more weight on low-SNR features and increasing $\rho$ on higher-SNR features (Karras et al., ).
Qualitatively, we find that this process enables increased diversity of outputs, even from the same latent code (see Appendix A.5).

CPU Runtime:
We benchmark Presto-LS’s speed performance for CPU inference.
On an Intel Xeon Platinum 8275CL CPU, we achieve a mono RTF of 0.74, generating 32 seconds of audio in 43.34 seconds. We hope to explore further CPU acceleration in future work.

Fast Inference-Time Rejection Sampling: Given Presto-LS’s speed,
we investigated using *inference-time* compute to improve performance.
Formally, we test the idea of *rejection sampling*, inspired by Kim et al. (), where we generate a batch of samples and reject $r$ fraction of them according to some ranking function. We use
the CLAP score to discard samples that have poor text relevance. Over a number of rejection ratios (see Fig. 13), we find that CLAP rejection sampling strongly improves text relevance while maintaining or *improving* quality at the cost of diversity.

## 5 Conclusion

We proposed Presto, a dual-faceted approach to accelerating latent diffusion transformers by reducing sampling steps and cost per step via distillation. Our core contributions include the development of score-based distribution matching distillation (the first GAN-based distillation for TTM), a new layer distillation method, the first combined layer-step distillation, and evaluation showing each method are independently best-in-class and, when combined, can accelerate our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than the comparable SOTA model), resulting in the fastest TTM model to our knowledge. We hope our work will motivate continued work on (1) fusing step and layer distillation and (2) new distillation of methods for continuous-time score models across media modalities such as image and video.

## Acknowledgements

We would like to thank Juan-Pablo Caceres, Hanieh Deilamsalehy, and
Chinmay Talegaonkar.

## Appendix A Appendix

### A.1 Model Design Details

As we perform latent diffusion, we first train a variational autoencoder. We build on the Improved RVQGAN (Kumar et al., ) architecture and training scheme by using a KL-bottleneck with a dimension of 32 and an effective hop of 960 samples, resulting in an approximately 45 Hz VAE. We train to convergence using the recommended mel-reconstruction loss and the least-squares GAN formulation with L1 feature matching on multi-period and multi-band discriminators.

Our proposed base score model backbone builds upon DiT-XL (Peebles & Xie, ), with modifications aimed at optimizing computational efficiency.
Specifically, we use a streamlined transformer block design, consisting of a single attention layer followed by a single feed-forward layer, similar to Llama (Dubey et al., ).
Our model utilizes three types of conditions including noise levels (timesteps) for score estimation, beat per minute (BPM) values of the song, and text descriptions.
Following EDM, we apply a logarithmic transformation to the noise levels, followed by sinusoidal embeddings.
Similarly, BPM values are input as scalars then go through sinusoidal embeddings to generate BPM embeddings.
These processed noise-level embeddings and BPM embeddings are then combined and integrated into the DiT block through an adaptive layer normalization block.
For text conditioning, we compute text embedding tokens with T5-based encoders and concatenate with audio tokens at each attention layer.
As a result, the audio token query attends to a concatenated sequence of audio and text keys, enabling the model to jointly extract relevant information from both modalities. To provide baseline architectural speedups, we use FlashAttention-2 (Dao, ) for the DiT and Pytorch 2.0’s built in graph compilation (Ansel et al., ) for the VAE decoder and MusicHifi mono-to-stereo.

For the diffusion model hyparameter design, we follow Karras et al. (). Specifically, we set $\sigma_{\text{data}}=0.5$, $P_{\text{mean}}=-0.4$, $P_{\text{std}}=1.0$, $\sigma_{\text{max}}=80$, $\sigma_{\text{min}}=0.002$. We train the base model with $10\%$ condition dropout to enable CFG. The base model was trained for 5 days across 32 A100 GPUs with a batch size of 14 and learning rate of 1e-4 with Adam. For all score model experiments, we use CFG++ (Chung et al., ) with $w=0.8$.

For Presto-S, following Yin et al. () we use a fixed guidance scale of $w=4.5$ throughout distillation for the teacher model as CFG++ is not applicable for the distribution matching gradient. Additionally, we use a learning rate of 5e-7 with Adam. For all step distillation methods, we distill each model with a batch size of 80 across 16 Nvidia A100 GPUs for 32K iterations. We train all layer distillation methods for 60K iterations with a batch size of 12 across 16 A100 GPUs.

### A.2 Baseline Description

We benchmark against multiple diffusion acceleration algorithms and open-source models:

- •
Consistency Models (CM) (Song et al., ; Bai et al., ):
This distillation technique learns a mapping from anywhere on the diffusion process to the data distribution (i.e. $\bm{x}_{t}\rightarrow\bm{x}_{0}$) by enforcing the self-consistency property that $G_{\bm{\phi}}(\bm{x}_{t},t)=G_{\bm{\phi}}(\bm{x}_{t^{\prime}},t^{\prime})\quad
\forall t,t^{\prime}$. We follow the parameterization used in past audio works (Bai et al., ; Novack et al., ) that additionally distills the CFG parameter into the model directly.
- •
SoundCTM (Saito et al., ):
This approach distills a model into a consistency *trajectory* model (Kim et al., ) that enforces the self-consistency property for the entire diffusion process, learning an anywhere-to-anwhere mapping. SoundCTM forgoes the original CTM adversarial loss and calculates the consistency loss using intermediate features of the base model.
- •
DITTO-CTM (Novack et al., ),
This audio approach is also based off of (Kim et al., ), yet brings the consistency loss back into the raw outputs and instead replaces CTM’s multi-step teacher distillation with single-step teacher (like CMs) and removes the learned target timestep embedding, thus more efficient (though less complete) than SoundCTM.
- •
DMD-GAN (Yin et al., ):
This approach removes the distribution matching loss from DMD2, making it a fully GAN-based finetuning method, which is in line with past adversarial distillation methods (Sauer et al., )).
- •
ASE (Moon et al., ),
This funetuning approach for diffusion models, as discussed in Sec. 3.3, finetunes the base model with the standard DSM loss, but for each noise level drops a fixed number of layers, starting at the back of the diffusion model’s DiT blocks.
- •
MusicGen (Copet et al., ): MusicGen is a non-diffusion based music generation model that uses an autoregressive model to predict discrete audio tokens (Défossez et al., ) at each timestep in sequence, and comes in small, medium, and large variants (all stereo).
- •
Stable Audio Open (Evans et al., ): Stable Audio Open is a SOTA open-source audio diffusion model, which can generate variable lengths up to 45s in duration. Stable Audio Open follows a similar design to our base model, yet uses cross-attention for conditioning rather than AdaLN which we use, which increases runtime.

### A.3 Presto-S Algorithm

Our complete Presto-S algorithm is outlined below. For a visual representation, please see Fig. 1.

Figure: Algorithm 1 Presto-S

### A.4 Analyzing Failure Modes of Combined Layer and Step Distillation

We empirically discovered a number of failure modes when trying to combine step and layer distillation. As noted in Section 4.6, the heavier per-layer requirements of distilled few-step generation made all standard dropping schedules (Moon et al., ) intractable and prone to quick generator collapse, necessitating a more conservative dropping schedule. In Fig. 8, we show the generator loss, discriminator loss, distribution matching gradient, and the discriminator’s accuracy for the *real* inputs over distillation, for a number of different setups:

Figure: Figure 8: Step distillation losses for early distillation for multiple combination methods. Presto-LS is the only setup that avoids generator degradation and high variance distribution matching gradients.
Refer to caption: x8.png

- •
Presto-S, pure step distillation mechanism (blue).
- •
Presto-LS, optimal combined setup where we pretrain the model with Presto-L and then perform Presto-S, but with keeping the real and fake score models initialized from the original score model (orange).
- •
LS with L-Fake/Real, which mimics Presto-LS but uses the Presto-L model for the fake and real score models as well (green).
- •
Step then Layer, where we first perform Presto-S distillation and then continue distillation with Presto-L layer dropping on the generator (red).
- •
Step and Layer jointly, where we perform Presto-S and Presto-L at the same time initialized from the original score model (purple),

We see that the runs which do not initialize with pretrained Presto-L (Step then Layer, Step and Layer) show clear signs of generator degradation, with increased generator loss, decreased discriminator loss, and notably near perfect accuracy on real samples, as attempting to learn to drop layers from scratch during step distillation gives strong signal to the discriminator. Additionally, LS with L-Fake/Real inherits similar collapse issues but has a higher variance distribution matching gradient as the layer-distilled real and fake score models are poor estimators of the gradient.

### A.5 Inference-time noise schedule Sensitivity analysis

Given our final Presto-LS distilled 4-step generator, we show how changing the inference-time noise schedule can noticeably alter the outputs, motivating our idea of a continuous-time conditioning.

The EDM inference schedule follows the form of:

$$ $\sigma_{i<N}=\left(\sigma_{\text{max}}^{1/\rho}+\frac{i}{N-1}(\sigma_{\text{ min}}^{1/\rho}-\sigma_{\text{max}}^{1/\rho})\right)^{\rho},$ (9) $$

where increasing the $\rho$ parameter puts more weight on the low-noise, high-SNR regions of the diffusion process. In Fig. 9, we show a number of samples generated from Presto-LS with identical conditions and latent codes (i.e. starting noise and all other added gaussian noise during sampling), only changing $\rho$, from the standard of 7 to 1000 (high weight in low-noise region). We expect further inference-time tuning of the noise schedule to be beneficial.

Figure: Figure 9: Generations from Presto-LS from the *same* text prompt and latent code (i.e. starting noise and added noise during sampling), only varying the $\rho$ parameter between (7 and 1000). Purely shifting the noise schedule for 4-step sampling allows for perceptually distinct outputs.
Refer to caption: x9.png

### A.6 RTF Analysis

We define the RTF for a model $\bm{\theta}$ as:
$\text{RTF}_{b}(\bm{\theta})=\frac{bT_{\bm{\theta}}}{\text{latency}_{\bm{\theta
}}(b)}$, where $T_{\bm{\theta}}$ is the generation duration or how much *contiguous* audio the model can generate at once
and $\text{latency}_{\bm{\theta}}(b)$ is the time it takes for generating a batch of $b$ samples, following (Evans et al., ; Zhu et al., ).
This is different from the fixed-duration batched RTF used in Nistal et al. ().
We test $b=1$ as well as the *maximum* batch size we could attain for each model on a single A100 40GB to get a sense of maximum throughput.
We show results in Table 3 and Table 4 for all components of our generative process, including latency metrics for generation (i.e. the diffusion model or distilled generator), decoding (i.e. VAE decoder from latents to audio) and the optional mono-to-stereo (M2S), as well as overall RTF/latency for mono and stereo inference.
We omit the MusicGen models and the other step-distillation methods which share the same RTF as Presto-S.
For the fastest model Presto-LS, the biggest latency bottleneck is the mono-to-stereo model (Zhu et al., ) and VAE decoder.
In future work, we hope to optimize the VAE and mono-to-stereo modules for faster inference.

**Table 3: Latency (ms) and real-time factor for a batch size of one on an A100 40GB GPU.**
| Model | Generation | Decoding | Mono | Mono | M2S | Stereo | Stereo |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Latency | Latency | RTF | Latency | Latency | RTF | Latency |  |
| Stable Audio Open | 6159.01 | 887.99 | N/A | N/A | 0 | 4.54 | 7047 |
| Base DM | 4079.81 | 64.45 | 7.72 | 4144.27 | 205.31 | 7.36 | 4349.58 |
| ASE | 3200.73 | 64.45 | 9.80 | 3265.19 | 205.31 | 9.22 | 3470.50 |
| Presto-L | 3201.19 | 64.45 | 9.80 | 3265.64 | 205.31 | 9.22 | 3470.95 |
| SoundCTM | 238.06 | 64.45 | 105.78 | 302.51 | 205.31 | 63.01 | 507.83 |
| Presto-S | 204.98 | 64.45 | 118.77 | 269.43 | 205.31 | 67.41 | 474.74 |
| Presto-LS | 166.04 | 64.45 | 138.84 | 230.49 | 205.31 | 73.43 | 435.8 |

**Table 4: Latency (ms) and real-time factor for max batch size on an A100 40GB GPU.**
| Model | Generation | Decoding | Mono | Mono | M2S | Stereo | Stereo |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Latency | Latency | RTF | Latency | Latency | RTF | Latency |  |
| Stable Audio Open | 34602.86 | 4227.54 | N/A | N/A | 0 | 7.42 | 38830.4 |
| Base | 18935.26 | 1198.21 | 14.3 | 20133.46 | 1775.73 | 96.38 | 21909.19 |
| ASE | 14584.85 | 1198.21 | 18.25 | 15783.05 | 1775.73 | 96.25 | 17558.78 |
| Presto-L | 14655.02 | 1198.21 | 18.17 | 15853.23 | 1775.73 | 96.25 | 17628.96 |
| SoundCTM | 1135.65 | 1198.21 | 123.4 | 2333.86 | 1775.73 | 92.98 | 4109.58 |
| Presto-S | 715.41 | 1198.21 | 150.5 | 1913.62 | 1775.73 | 92.18 | 3689.34 |
| Presto-LS | 695.19 | 1198.21 | 152.11 | 1893.4 | 1775.73 | 92.13 | 3669.13 |

### A.7 Presto-L Design Ablation

Figure: Figure 10: Presto-L ablation. Each individual change of our layer distillation vs ASE is beneficial.
Refer to caption: x10.png

To investigate how each facet of our Presto-L method contributes to its strong performance vs. ASE, we ran an additional ablation combining ASE with each component (i.e. the shifted dropping schedule, explicit budget conditioning, and the self-teacher loss).
In Fig. 10, we see that the core of Presto-L’s improvements come from the shifted dropping schedule (which preserves final layer behavior), as the ASE+shift performs similarly to Presto-L on high-step FAD and MMD. Additionally, we find that the budget conditioning and self-teacher loss help text relevance more so than the shifted schedule does. All together, the combination of Presto-L’s design decisions leads to SOTA audio quality (FAD/MMD/Density) and text relevance compared to any one facet combined with ASE.

### A.8 Discrete-Time Failure Modes

In Fig. 11, we visualize the poor performance of distilled models that use 1-2 step discrete-time conditioning signals. Notice that for the same random seed, the high-frequency performance is visually worse for discrete-time vs. continuous-time conditioning, motivating our proposed methods.

Figure: Figure 11: Failure mode of 1-2 step discrete models vs. continuous models (each row is same random seed and text prompt), with 2-step generation. Hip-Hop adjacent generations noticeably drop high frequency information, and render percussive transients (hi-hats, snare drums) poorly.
Refer to caption: x11.png

### A.9 Listening Test Results

We visualize our listening test results from Section 4.5 using a violin plot.

Figure: Figure 12: Violin plot from our listening test. Presto-LS is preferred over other baselines ($p<0.05$).
Refer to caption: x12.png

### A.10 Rejection Sampling Results

We show rejection sampling results where we generate a batch during inference and then use CLAP to reject the $r$ least similar generations to the input text prompt. CLAP rejection sampling noticeably improves CLAP Score and maintains (and sometimes *improves*) FAD and MMD, but reduces diversity as expected.

Figure: Figure 13: Rejection sampling eval metrics vs. rejection ratio. Base Presto-LS in red. CLAP rejection sampling improves both CLAP score and overall quality, while reducing diversity.
Refer to caption: x13.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Is it possible to open the implementation without sharing pretrained model weights? Or open a part of key tricks or components in the work?
soundness: 4
strengths: 1. Very solid target: enhancing efficiency from both the number of steps and the runtime of a single ste.
2. novel to audio community to combine both step distillation and layer distillation, on top of the successful usage of an adversarial loss.
3. Clear presentation. The origin of the proposed techniques, and the differences between Presto! and prior arts are well explained. Related works are very up-to-date hence informative.
4. Great objective measurements. Presto! could outperform or keep on-par with the teacher model, while reducing the sampling steps to as few as 4, for the first time in audio community. The proposed continuous step distillation offers a more predictable performance gain when more inference budget is given.
5. Excellent subjective quality. The violin plot shows that Presto! outperforms Stable Audio Open and SoundCTM, two works published earlier this year, which is impressive. The samples in demo page sound great.
summary: The paper proposes "Presto!" for effective and efficient music generation. More specificly, Presto! is a set of model distillation techniques that aims at improving the inference efficiency of continuous diffusion models from two aspects:
1. overall inference steps,
2. runtime of each inference step.

The motivations and approaches are sound, and the results on the demo page are extremely convincing. Huge efforts have been done to applying latest SOTA methods to their own DNN architecture. Experiments are well designed to support the claims and motivations.

It is unfortunate that the authors do not plan to open their method, as I feel the framework is a little bit complex: it invovles extensive grid seearch for a better stability in adversarial training.
weaknesses: The ablation study indicates quite high complexity if we want to stably and effectively use Presto!. I am interested in how well the framework could be when applied to other VAEs or generator backbones, especially if no resource is open.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In line 35, does 5-20 seconds latency refer to 32 seconds audio? If yes, please clarify because it's mentioned in the abstract but not in the introduction.
soundness: 3
strengths: This paper marks an important step toward a unified approach for faster generative models by bridging adversarial and diffusion methods for text-to-music tasks. In particular, Section 3.1 demonstrates this by applying GAN-based distillation to continuous-time score-based diffusion models. 

Aligning most distributions with the training distribution, as shown in Table 1, empirically improves all metrics, underscoring the robustness of their approach.

The authors also navigate the task of combining layer and step distillation, finding a highly effective recipe that substantially enhances generation speed. 

Achieving a 10-18x speedup in audio generation is a noteworthy accomplishment, positioning this collection of methods as a valuable resource for practical applications in the field.
summary: This paper introduces an innovative approach to improve text-to-music generative models by significantly reducing inference time. Focusing on score-based diffusion models, the authors present two main contributions: first, a reformulated few-step synthesis approach utilizing online GAN-based adversarial distillation, adapted specifically for continuous time diffusion through the DMD2 framework; and second, a layer distillation technique, inspired by ASE, which uses a layer-budgeting module to prioritize layers based on noise levels.

They further explore the combination of step and layer distillation, demonstrating cases where the methods complement each other effectively. Their experiments, conducted on latent diffusion models with VAE and DiT blocks, use internal datasets for training and Song Describer for evaluation. Comparative analysis on the CLAP-LAION dataset with FAD scores shows that each method, independently and in combination, improves upon alternatives while maintaining real-time factor (RTF) performance.
weaknesses: While the paper is well-structured in a general sense, certain sections are hard to follow due to the level of detail. For example, Figure 1 is dense and, despite a detailed caption, remains challenging to interpret. The figure also uses various notations that lack prior explanation, making it difficult for readers to follow the intended process.

Additionally, the models are trained on internally licensed data, so the results cannot be easily referenced in future work. With no indication that the code will be released, reproducing this complex setup—including multiple models, distributions, and training phases—would be exceptionally difficult. This limitation means future researchers may be restricted to evaluating audio examples alone rather than building upon this work directly.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I would appreciate the authors' response to my comment in "Weaknesses".
soundness: 4
strengths: 1. The paper is well written. I could understand their motivation, the proposed method, and the experimental results.
2. The authors modified the existing DMD2 and ASE frameworks while conducting ablation studies. As ablation studies are comprehensive, readers can understand the reasons for the modifications/choices.
3. The authors demonstrated that the proposed distillation method works well in several aspects (AD, MMD, CLAP scores, and subjective evaluation) despite its fast generation.
summary: This paper presents a distillation method to develop a fast music generation model. The authors modify the [DMD2](https://arxiv.org/abs/2405.14867) and [ASE](https://proceedings.mlr.press/v235/moon24a.html) frameworks to improve the model performance. The proposed method not only enables a model to generate music signals with a few NFEs but also makes the model size smaller. The authors demonstrated that their trained model outperforms models trained/equipped with previous methods in terms of multiple evaluation metrics, using their in-house data for training and the Song Describer dataset for evaluation. They also conducted comprehensive ablation studies to show that their choices are reasonable.
weaknesses: I do not think that the paper has a critical weak point, but let me mention some weaknesses.

- Although the paper is well-written, some readers would struggle to reproduce the results without their code public. (I understand some institutes/companies do not make their code publicly available due to their policy.)
- Since the proposed training framework is general, I thought I would like to see experimental results in the text-to-audio generation task evaluated on AudioCaps. Furthermore, if we have a DiT-based baseline model for text-to-image or text-to-video generation, we can apply the proposed method to them. I thought I would like to see the performance in those tasks as well.
- The explanation of baselines in Section 4.1 is a little unclear to me. My understanding is that the authors trained the teacher model and applied the proposed technique and previous ones (CM, SoundCTM, DITTO-CTM, DMD-GAN, and ASE). For MusicGen and Stable Audio Open, they just downloaded and evaluated the official distributed models. Whether my understanding is correct or not, a clearer description would be appreciated.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - Around L90–L91 in Introduction, the authors mention TTA. Are there plans to conduct text-to-audio generation experiments during the rebuttal period?
   - If not, claiming it as a contribution in the Introduction is an overstatement.
- How exactly is the CFG-augmented real score, represented by $\tilde{\mu}^{w}_{\text{real}}$ in Eq (5), formulated?
     - Is it something like $\mu_{\text{real}}( \hat{x}_{\text{gen}}+\sigma\epsilon, \sigma, \varnothing) + w (\mu_{\text{real}}( \hat{x}_{\text{gen}}+\sigma\epsilon, \sigma, \bf{c}) - \mu_{\text{real}}( \hat{x}_{\text{gen}}+\sigma\epsilon, \sigma, \varnothing)$, $where $\bf{c}$ is condition?
- Section 3.1, L210–L211: Regarding “the fake score model is updated 5 times as often as the generator to stabilize the estimation of the generator’s distribution,” how robust is this to different model training setups, architectures, and datasets? For example, if the number “5” were changed to “4” or “6,” how significantly would it affect training stability and generation quality in those sense?
- Section 3.1, L212–L214: In “Specifically, a discriminator head $D_{\psi}$ is attached to the intermediate feature activations of the fake score network,” which specific part of the DiT block do the “intermediate feature activations” refer to? Also, how is the architecture of the discriminator head $D_{\psi}$ actually configured? The explanation in Section 3.2.3 is still high-level for both points. Could you provide further detail using pseudo-code or a figure illustrating the architecture?
- In Algorithm 1, $\nu_1, \nu_2, g_1, g_2$ are mentioned. What values do the authors set as $\nu_1, \nu_2$? Are both $g_1, g_2$ optimized with Adam at a learning rate of 1e-4? Also, do the authors use learning rate schedulers?
- How is Presto-L distilled? The explanation in Section 3.3 and $\mathcal{L}_{\text{st}}$ alone seem insufficient to clarify the distillation process. Could you provide a more detailed explanation using an algorithm table, pseudo-code, or a figure illustrating the distillation process?
- Since multi-step sampling uses stochastic sampling like CM, I would expect that as the number of sampling steps increases, sample quality will eventually plateau and then degrade if increased further. Have you observed such a phenomenon when increasing the number of sampling steps to around 15, for example?
soundness: 4
strengths: - The authors propose the first diffusion-based text-to-music distillation model to successfully incorporate GAN loss.
- They also introduce a layer-distillation method to reduce computational cost at each sampling step, which has not yet been explored in the text-to-music task.
- Additionally, they present a methodology for integrating these two orthogonal distillation techniques.
- The effectiveness of these three methods is demonstrated through extensive evaluation.
summary: The authors propose Presto, a distilled diffusion-based text-to-music model. Specifically, they introduce Presto-S, a model that applies DMD2 to a text-to-music generative model, and Presto-L, a model that performs layer-based distillation by extending ASE. Additionally, they propose Presto-LS, which combines these two distillation techniques. Experimental evaluations demonstrate that all three models achieve high-quality and efficient text-to-music generation.
weaknesses: - The explanation of Presto-L seems to assume prior knowledge of ASE, which is unfriendly for readers. For better readability and self-containment, it would be helpful to explain Presto-L more thoroughly by including an algorithm, equations, or an illustrative figure of the distillation process, as done for Presto-S. At the very least, a more detailed literature review of ASE should be provided. (ASE is a relatively new method presented at ICML 2024, and as the authors note in the last paragraph of Section 2, such layer-distillation methods have not yet been explored in text-to-music, underscoring the need for a detailed explanation)
- The contribution to the research community in terms of reproducibility is limited. Therefore, I strongly recommend either releasing the codebase or providing sufficient implementation details to facilitate straightforward replication, at least to allow future research to conduct follow-up studies (see also Questions).
    - While I understand from the reproducibility statement that the authors chose not to release checkpoints, it should still be feasible to release the codebase itself (as done in some prior works [1][2]) or to include sufficient implementation details for reproduction in the Appendix.

[1] Evans, Z., Carr, C.J., Taylor, J., Hawley, S.H. and Pons, J., 2024. Fast timing-conditioned latent audio diffusion. arXiv preprint arXiv:2402.04825.

[2] Evans, Z., Parker, J.D., Carr, C.J., Zukowski, Z., Taylor, J. and Pons, J., 2024. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301.

Meta-review: **Paper Summary:**

This paper improves the inference speed of text-to-music diffusion models, adapting techniques for model distillation (DMD2) and inference acceleration (ASE) from prior work on vision. These adaptations are well-motivated, and experiments convincingly validate their high quality and low inference costs of the resulting models.

**Strengths:**

The reviewers praise the paper’s clarity, thoroughness, and convincing experiments. Reviewer eJot states “It was a pleasure to review this great paper” and reviewer pB9s similarly comments “it was truly a pleasure to review your work.” Even the most critical review, provided by reviewer ZS9n states "This paper marks an important step toward a unified approach for faster generative models by bridging adversarial and diffusion methods for text-to-music tasks.”

**Weaknesses:**

Every reviewer criticized the authors’ choice to withhold code and/or model checkpoints. I share this criticism. That said, the authors have taken significant steps during the rebuttal period to provide additional details (short of code) on the implementation of their methods.

**Conclusion:**

This work represents a substantial methodological contribution to the text-to-music literature. In my opinion, the concerns about reproducibility due to lack of code release have been adequately addressed by the additional descriptive material added to the appendix during the discussion period. I recommend acceptance.

Decision: Accept (Spotlight)