Presto! Distilling Steps and Layers for Accelerating Music Generation
Authors: Zachary Novack
Ge Zhu
Jonah Casebeer
Julian McAuley
Taylor Berg-Kirkpatrick
Nicholas J. Bryan
Venue: ICLR 2025 Spotlight (2025)
Keywords: music generation
diffusion distillation
diffusion
diffusion acceleration
text-to-music generation
layer dropping
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Despite advances in diffusion-based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach to inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, we develop a new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step, we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted approach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44.1kHz, 15x faster than the comparable SOTA model) — the fastest TTM to our knowledge.

Published as a conference paper at ICLR 2025
Presto! DISTILLING STEPS AND LAYERS FOR ACCELERATING MUSIC GENERATION
Zachary Novack" UC - San Diego
Ge Zhu & Jonah Casebeer Adohe Research
Julian McAuley & Taylor Berg-Kirkpatrick UC- San Diego
Nicholas J. Bryan
Adobe Research
ABSTRACT
Despite advances in diffusion- based text-to-music (TTM) methods, efficient, high-quality generation remains a challenge. We introduce Presto!, an approach inference acceleration for score-based diffusion transformers via reducing both sampling steps and cost per step. To reduce steps, develop new score-based distribution matching distillation (DMD) method for the EDM-family of diffusion models, the first GAN-based distillation method for TTM. To reduce the cost per step. we develop a simple, but powerful improvement to a recent layer distillation method that improves learning via better preserving hidden state variance. Finally, we combine our step and layer distillation methods together for a dual-faceted ap- proach. We evaluate our step and layer distillation methods independently and show each yield best-in-class performance. Our combined distillation method can generate high-quality outputs with improved diversity, accelerating our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44. 1Kz,, 15x faster than the comparable SOTA model) the fastest TTM to our knowledge.
INTRODUCTION
We have seen renaissance of dodain generative media (Chen et al. 2024 Agostinelli et al., 2023 Liu et al., 2023] Copet et al. [0033). with increasing capabilities for both text-to-Audin (TTA) and Text-to-Music (TTM) generation. This work has been driven in-pat by audio-domain diffu- sion models (Song et al. 2020 Ho et al., 2020, Song et al., 2021), enabling considerably better audio modeling than generative adversarial network (GAN) or variational autoencoder (VAE) meth- ods (Dhariwal & Nichol 2021). Diffusion models, however, suffer from long inference times due to their iterative denoising process, requiring substantial number of function evaluations (NFE) dur- ing inference (i.e. sampling) and resulting in =5-20 seconds at best for non-batched ==32s outputs.
Accelerating diffusion inference typically focuses on step distillation, i.c. the process of reducing the number of sampling steps by distilling the diffusion model into a few-step generator. Methods include consistency- based (Salimans & Ho. 2022 [Song al. 2023 Kim al.
2023) and adver-
sarial (Sauer et al., 2023) Yin et al. 2023) 2024, Kang et al. 2022) approaches.
Others have also
investigated distiddistillation (Ma et al. [2024] Wimbauer
20244 Moon et al. 2024). which
draws from transformer early exiting (Hou et al. 2020)] Schuster et al. 2021) by dropping interior layers reduce the cost per sampling step for image generation. For T TTATTMM models, however, distillation techniques have only been applied to shorter or lower-quality audio (Bai et 2024 No- vack al. 2024a), necessitate 110 steps (vs. 1-4 step image methods) match base quality (Saito et al. 2024). and have not successfully used layer GAN-based distillation methods.
We present Presto a dual-faceted distillation approach to inference acceleration for score-based diffusion transformers via reducing the number of sampling steps and the cost per step. Presto includes three distillation methods: (1) Presto-S, a new distribution matching distillation algorithm for score-based, EDM-style diffusion models (see Fig. I leveraging GAN-based step distillation
Work done while an intern at Adohe.
Presto is the common musical term denoting fast music from 168-200 heats per minute.

---

Published as a conference paper at ICLR 2025
Distillation Gradient Computation
POAN
Leart-Squares
take score model tracks score of generotor
Lean
(a
Pussufe frattflett
LSK
Generator
Fake Score Model
V LDMD
forward diiuson
Posan trainy
generitoon /score
Generated Music
Real Score Model
Figure Presto-S. Our goal to distill the initial "real" score model (grey) μg into few-step generator (light blue) G minimize the KI. divergence between the distribution of G outputs and the real distribution. This requires that we train an auxillary "fake" score model fty (dark blue) that estimates the score of the generator's distribution at each gradient step. Formally: (1) real audio
corrupted with Gaussian noise sampled from the generator noise distribution Pgen Pgen(o"n)) (a which is then (2) passed into the generator get its output. Noise is then added to this generation according to three different noise distributions: (3) POMD (atmin), which is (4) passed into both the real and fake score models to calculate the distribution matching gradient VoLDMD: (5) posv(lottaninn
which is used to (6) train the fake score model on the generator's 's distribution with DSae- and (7) an adversarial distribution PGAN(ottinn) which along with the real audio (8) passed into least- squares discriminator built on the fake score model's intermediate activations to calculate LGAN.
with the flexibility continuous-time models, (2) Presto-L, a conditional layer distillation method designed to better preserve hidden state variance during distillation, and (3) Presto-LS, a combined layer-step distillation method that critically uses layer distillation and then step distillation while disentangling layer distillation from real and fake score-based gradient estimation.
To evaluate our approach, we ablate the design space for both distillation processes. First, we show step distillation method achieves best-in-class acceleration and quality via careful choice of loss noise distributions, GAN design, and continuous-valuee inputs, the first such method to match base TTM diffusion sampling quality with 4-step inference. Second, we show our layer distillation method offers a consistent improvement in both speed and performance over SOTA layer dropping methods and base diffusion sampling. Finally, we show that layer-step distillation accelerates our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44. 1kHz, 15x faster than the comparable SOTA model) while notably improving diversity over step-only distillation.
Overall, our core contributions include the development of a holistic approach to accelerating score- based diffusion transformers including (1) The development of distribution matching distillation for continuous-time score-based diffusion (i.e. EDM), the first GAN-based distillation method for TTM. (2) The development of an improved layer distillation method that consistently improves upon both past layer distillation method and our base diffusion model. (3) The development of the first combined layer and step distillation method. (4) Evaluation showing our step, layer, and layer-step distillation methods are all best-in-class and, when combined, can accelerate
base model by 10-18x (230/435ms latency for 32 second mono/stereo 44. kKZz, 15x faster than Stable Audio Open (Evans et al. 2024c)), the fastest TTM model to our knowledge. For sound examples (anonymous link), see ht t / http:pprsttommsscc..... gi thub. ic /
2 BACKGROUND & RELATED WORK
2.1 MUSIC GENERATION
Audio-domain domain music generation methods commonly autoregressive (AR) techniques Zeghidour
et al. 2021 Agostinelli et al. 2023) Copet et al. 2023) or diffusion (Forsgren & Martiros 2022
Liu et al. 2023 2024b Schneider et al. 2023). Diffusion-hased TTA/TTM (Forsgren & Martiros.
2

---

Published as a conference paper at ICLR 2025
2022 Liu et al. 2023 2024b;
Schneider al. 2023 Evans et al. 2024a has shown the promise of 2023). precise musical attribute control (Novack et al. 2024b a Tal
full full-text control (Hung et al.
al. 2024), structured long-for form generation (Evans et al. 2024b), and higher overall quality over AR methods (Evans et al. 2024a b. Novack et al.] [024b) Hvans et al. 2024c). The main downside
of diffusion, however, that it is slow and thus not amenable to interactive-rate control.
2.2 SCORE-BASED DIFFUSION MODELS
Continuous-time diffusion models have shown great promise over discrete-time models both for
their improved performance on images (Balaji et al., 2022 Karras et al. 2023] Liu et al., 2024a) and audio (Nistal et al. 2024 Zhu et al.) 2023 Saito et al. 2024). as well their relationship to the general class of flow-based models (Sauer et al. 2024 Tal et al, 2024). Such models involve a
forward noising process that gradually adds Gaussian noise real audio signals Zrea and reverse process that transforms pure Gaussian noise back into data (Song et al. [021 Sohl-Dicksteen et al., 2013). The reverse process is defined by a stochastic differential equation (SDE) with an equivalent ordinary differential equation (ODE) form called the probability flow (PF) ODE (Song et al. 2021):
dæ = - p(x o)do,
where a. p(æ (r) the score function of the marginal density (i.e. the noisy data) at noise level according to the forward diffusion process. Thus, the goal of score-based diffusion models
is to learn a denoiser network μi such that uo (x, σ)) = [raal x o]. The score function is:
(o(
V loz |o)
(2)
Given a trained score model, we can generate samples at inference time by setting a decreasing noise schedule of N levels Tma ON > xx-1 these levels using our model and any off-the-shelf ODE solver (e.g. Euler, Heun).
do m and iteratively solving the ODE at
The EDM famillmyly (Karras al. 2022 2023) of score-based diffusion models is of particular inter- est and unifies several continuous time model variants within a common framework and improves model parameterization and training process. The EDM score model is trained by minimizing
reweighted denoising score matching (DSM) loss (Song et al. 2021):
CDMM = Ex Depaar-,aa-piie D.a-plo""" s (
He (Xreel + EO, o >)1
(3)
where p(gltain) denotes the noise distribution during training, and A(o) is noise-level weighting function. Notably, EDM defines a different noise distribution to discretize for inference p(o""f) that is distinct from trantaii) ) (see Fig. 2). as opposed to noise schedule shared between training and inference. Additionally, EDMs represent the denoising network using extra noise-dependent preconditioning parameters, training a network fo with the parameterization:
(4) For TTM models, μe is equipped with various condition embeddings (e.g. text) u ar, e). To increase text relevance and quality at the cost of diversity, we employ classifier free guidance (CFG) (Ho & Salimans 2021), converting the denoised output to: 1i (ar, (,, (pe (x, o, e) p o, 0)), where w is the guidance weight and 0 is a "null" conditioning.
(x( Caip(()) (o) fo (cm (r)x Croise (o)).
( o, 0) +
2.3 DIFFUSION DISTILLATION
Step distillation the process of reducing diffusion sampling steps by distilling base model into few-step generator. Such methods can be organized into two broad categories. Online consistency approaches such as consistency models (Song et al.) 2023), consistency trajectory models (Kim et al.) 2023), and variants (Ren et al. 2024] Wang et al., 2024a) distill directly by enforcing consistency across the diffusion trajectory and optionally include an adversarial loss (Kim et al., 2023). While such approaches have strong 1-step generation for images, attempts for audio have been less suc- cessful and only capable of generating short segment (i.e. < 10 seconds), applied to lower-quality base models limiting upper-bound performance, needing up to 16 sampling steps to match baseline quality (still slow), and/or did not successfully leverage adversarial losses which have been found to increase realism for other domains (Bai et al. 2024 Saito al. 2024 Novack al. 2024a).
3

---

Published as a conference paper at ICLR 2025
In contrast, offline adversarial distillation methods include Diffusion2GAN (Kang al. 2024). LADD (Sauer et al. 2024), and DMD (Yin et al. 2023). Such methods work by generating large amounts of offline solse-sample pairs from the base model, and finetuming the model into condi- tional GAN for few-step synthesis. These methods can surpass their adversarial-fres counterparts, yet require expensive offline data generation and massive compute infrastructure to be efficient.
Alternatively, improved DMD (DMD2) (Yin et al. 2024) introduces an online adversarial diffusion distillation method for images. DMD2 (I) removes the need for expensive offline data generation (2) adds a GAN loss and (3) outperforms consistency-bant
methods and improves overall quality. DMD2 primarily works by distilling one- or few-step generator G from base diffusion model Mreal- while simultaneously learning score model of the generator's ' distribution online Plake in order to approximate a target KI. objective (with (rraa)) used train the generator. To our knowledge, there are no adversarial diffusion distillation methods for TTM or TTA.
Beyond step distillation, layer distillation, or the process of dropping interior layers to reduce the cost per sampling step, has been recently studied (Moon et al. (0224] Wimbauer et al. (2024). Layer distillation draws inspiration from transformer early exiting and layer caching (Hou et al., 2020] Schuster et al.] (02T) and has found success for image diffusion, but has not been compared or combined with step distillation methods and has not been developed for TTA/TTM. In our work, we seek to understand how step and layer distillation interact for accelerating music generation.
Presto!
We propose a dual-faceted distillation approach for inference acceleration of continuous-time diffu- sion models. Continuous-time models have been shown to outperform discrete-time DDPM mod- els (Song et al. 2020 Karras et al,, 2022 2024), but past DMD/DMD2 work focuses on the latter. Thus, redefine DMD2 (a step distillation method) in Section 3.1 for vontinuous-time score mod- els, then present an improved formulation and study its design space in Section 3.2 Second, we design simple, but powerful improvement the SOTA layer distillation method understand the impact of reducing inference cost per step in Section 3.3 Finally, we investigate how to combine step and layer distillation methods together in Section
3.4
3.1 EDM-STYLE DISTRIBUTION MATCHING DISTILLATION
We first redefine DMD2 in the language of continuous-time score-based diffusion models (i.e. EDM-style). Our goal is to distill our score model μe (which we equivalently denote as Preat. as it is trained to model the score of real data) into an accelerated generator G. that can sample in 1-4 steps. Formally, we wish to minimize the reverse KL Divergence between the real distribution Preal and the generator G 's distribution Plak: LDND = (Piakel Preal). l The KL term cannot be calculated explicitly, but we can calculate its gradient with respect to the generator if of the generator's ' distribution. Thus, we also train a "fake" score model ue (or equivalently, Ptake) to approximate the generator distribution's 's score function at each gradient step during training.
access the score
First, given some real data Freal, we sample a noise level from a set of predefined levels a ~ { and then pass the corrupted real data through the generator get the generated output &gen G (Ireal + ae, J), where ~ NO, I) (we omit the conditioning e for brevity). The gradient of the KL divergence between the real and the generator's distribution can then be calculated as:
VoLDMD Eo-(ar)
[[ + oe, σ) Preat (Āgen + oe, σ)) V Vesgell
(5)
where }o,} are the predefined noise levels for all loss calculations, and a is the CFG-augmented real score model. To ensure that Pake accurately models the score of the generator's 's distribution at each gradient update, we train the fake score model with the weighted-DSSM loss (i.e. standard diffusion training), but on the generator outputs:
arg min Laake-DMM Bow [A(a)]l@ggo- IMakko (frco ae, a) I +
(6)
To avoid using offline data (Yin al. 2023), the fake score model is updated 5 times often as the generator to stabilize the estimation of the generator's distribution. DMD2 additionally includes an explicit adversarial loss in order improve quality. Specifically, discriminator head De is
4

---

Published as a conference paper at ICLR 2025
attached to the intermediate feature activations of the fake score network Prake and thus is trained with the nonsaturating GAN loss:
arg min max E (n,),, [log Du (Zreal oe, + a~ (r:). [- log Do e + oc, (0,)
(0,1)
which follows past work on using diffusion model backbones as discriminators (Sauer et al. 2024). In all, the generator G is thus trained with a combination of the distribution matching loss LDMD and the adversarial loss LGAN, while the fake score model (and its discriminator head) is trained with the fake DSM loss LSM and the adversarial loss LGAN. To sample from the distilled generator,
DMD2 uses consistency model-style "ping-pong sampling" (Song et al. 2023). where the model iteratively denoises (starting at pure noise max and renoises to progressively smaller noise levels.
Regarding past work, we note Yin et al. (2024) did present a small-scale EDM-style experiment, but treated EDM models as if they were functions of discrete noise timesteps. This re-discretization runs counterintuitive to using score-based models for distribution matching, since the fake and real score models meant to be and trained in continuous-tims and adapt to variable points along the noise process. Furthermore, this disregards the ability of continuous-time models to capture the entire noise process from noise data and enable exact likelihoods rather than ELBOs (Song al. 2021 Additionally, since DMD2 implicitly aims to learn an integrator of the PF ODE Go 2 0 R
+ Caann -5V log p( 8)d6 (like other data-prediction distillation methods (Song et al. 2023
learning this integral for any small set {oi} restricts the generator's modeling capacity,
3.2 PRESTO-S: SCORE-BAR BASED DISTRIBUTION MATCHING DISTILLATION
We develop our score-based distribution matching step distillation, Presto-S below and in Fig. aas well as the algorithm in Appendix A.3 a pseudo-code walkthrough in Appendix A.4, and expanded
visualization in Appendix A.5
3.2.1 CONTINUOUS- TIME GENERATOR INPUTS
In Section 3.1 the noise level and/or timestep is sampled from a discrete, hand-chosen set (oi)pen- Discretizing inputs, however, forces the model 1) be function of specific number of steps. requiring users to retrain separate models for each desired step budget (Yin et al. 2024 Kohler et al.) 2024) and 2) be function of specific noise levels, which may not be optimally aligned with where different structural, semantic, and perceptual features arise in the diffusion process (Si et al. [024]] Kynkäänniemi et al., 2024 BBalji et al. 2022, Sabour et al. 2024). When extending to continuous- time models, we train the distilled generator G& as a function of the continuous noise level sampled from the distribution a ~ p(a). This allows our generator to both adapt better to variable budgets and to variable noise levels, as the generator can be trained with all noise levels sampled from p(or).
3.2.2 PERCEPTUAL LOSS WEIGHTING WITH VARIABLE NOISE DISTRIBUTIONS
A key difference between discrete-time and continuous-timg diffusion models is the need for discretization of the noise process during inference. In discrete models, single noise schedule defines
Training Noise Distribution
Inference Noise Distribution
pi00-SNRR
particular mapping between timestep & and its noise level a, and is fixed throughout training and inference. In continuous time EDM models. how- ever, we use a noise distribution p(o"tain) to sample during training, and a separate noise distribution for inference p(arint) that is discretized to define the sampling schedule. In particular, when viewed in terms of the signal-to-noise ratio 1/a" or SNR as shown in Fig. 2 the training noise distribution puts the majority of its mass in the mid-to-high high SNR range of the diffusion process. This design choice focuses on semantic and perceptual features, while the inference noise distribution is more evenly distributed but with a bias towards the low-SNR region, giving bias to low- www-qreuency features.
0.02
-20
-10
10
20
log.SNR (dB)
Figure 2: Training/Inference distributions for EDM models, in decibel SNR space.
5

---

Published as a conference paper at ICLR 2025
However, recall that every loss term including S. O and requires additional re-cormuption process that must follow a noise distribution, significanll expanding the design space for score- based models. Thus, disentangle these forward diffusion processes and replace the shared dis- crete noise set with four separate noise distributions Pgen= PoMD, PDSM, and PGAN, corresponding to the inputs to the generator and each loss term respectively, with no restrictions on how each weights each noise level (rather than forcing a particular noise weighting for all computation).
Then, if we apply the original DMD2 method naively to the EDM-style of score-models, get Pgen (o pDMD ( PDSM (o )ol This choice of Pgen (o" reasonably aligns the generator inputs during distillation to the inference process itself, but each loss noise distribution is hat misaligned from its role in the distillation process. In particular:
POMD: The distribution matching gradient is the only point that the generator gets a signal from the CFG-augnt #FG-augmented outputs of the teacher. CFG is critical for text following, but primurily within the mid-to-high SNR region of the noise (Kynkäänniemi et al. 2024).
POAN: As in most adversarial distillation methods (Sauer et 2023 Yin al.
2023 the
adversarial loss's ' main strength is to increase the perceptual realism qualiit of the outputs, which arise in the mid-to-high SNR regions, rather than structural elements.
PDSM: The score model training should in theory mimic standard diffusion training, and may benefit from the training distribution's 's provably faster convergence (Wang et al. 2024b) (as the fake score model is updated online to track the generator's distribution).
Thus, we shift all of the above terms to use the training distribution POMD (ottin) (ossa(""ain) and PGAN main) to force the distillation process focus on perceptually relevant noise regions.
3.2.3 AUDIO-ALIGNED DISCRIMINATOR DESIGN
The original DMD2 uses a classic saturatta GAN loss. The discriminator is a series of con- volutional blocks downsampling the intermediate features into single probability for real vs. fake. While this approach is standard in image-domain applications, many recent adversarial waveform synthesis works (Kumar et al. 2023 Zhu al. 2024) use Least-Squares GAN loss:
arg min max 1
compuni ) ! D toe, 1 poppan (ot)) (o.1
[|| 1al1--wah Dxp gen + t0 (8)
c0,
where the outputs of the discriminator D& are only partially downsampled into lower-resolution version of the imput data (in this case, latent 1-D tensor). This forces the discriminator to attend to more fine graineddd,, temporally-alignes across the partially downsampled discriminator outputs. Hence, we use this style of discriminator for Presto-S to both improve and stabilize (Mao et al. 2017) the GAN gradient into our generator.
features for determining realness, as the loss is averaged
3.3 PRESTO-L: VARIANCE AND BUDGET-AWARD LAYER DROPPING
Standard Layer Dropping
Presto-L
Latent
Pattch
Latent
Patch
DIT Blocs
DIT Block
DKI
DIT Block
Lineor Partooh
DIT Block
DIT Block
DIT Blook
DIT Block
Busiped AddiN
Conditionngg
Blook
Conditioning
Emnedd
Embed
Noise teveto
Noise lovellr
Layer Sudget
Loyer Budget
Figure 3: Bascline layer dropping (left) vs. Presto-L (right). Standard layer dropping uses the noise level a to set the budget layers to drop, starting from the back of the DiT blocks. Presto-L shifts this dropping by one to the second-to-last block and adds explicit budget conditioning.
Given our step distillation approach above, we now seek to reduce the cost of individual steps them- selves through layer distillation, and then combine both step and layer distillation Section 3.4 We begin with the current SOTA method: ASE (Moon et al. 2024). ASE employs a fixed dropping schedule that monotonically maps noise levels to compute budgets, allocating more layers to lower noise levels. We enhance this method in three key ways: (1) ensuring consistent variance in layer distilled outputs, (2) implementing explicit budget conditioning, and (3) aligning layer-dropped out- puts through direct distillation. See Appendix A.6 more details.

---

Published as a conference paper at ICLR 2025
First, inspect the within-layer activation variance of our base model in Fig. 4 We find that while the variance predictably increases over depth, it notably spikes on the last layer up an order of magnitude higher, indicating that the last layer behaves much differently as it is the direct input to the linear de-embedding layer. ASE, however, always drops layers starting from the last layer and working backwards, thus always removing this behavior. Hence, we remedy this fact and shift the layer dropping schedule by I to drop starting at the second to last layer, always rerouting back into the final layer preserve the final layer's behavior.
Variance Preservation:
Budget Conditioning: We include explicif bud- get conditioning into the model itself so that the model can directly adapt computation to the budget level. This conditioning comes two places: (1)
Standard Deviation
10²
global budget embedding added to the noise level embedding, thus contributing to the internal Adap- tive Layer Norm (AdaLN) conditioning inside the DiT blocks, and (2) an additional AdaLN layer on the outset of the DiT blocks conditional only on the budget, in order to effectively rescale the outputs to account for the change in variance. Following (Pee-)
10
8=00000
0=0.00
0
5
10
20
25
Layer
Figure 4: Hidden activation variance vs.
layer depth. Each line a unique noise level. bles & Xie 2023; ZZang et al., 2023), we zero-initialize both budget conditioning modules to im- prove finetuning stability.
To encourage distillation without holding the base model in memory, we employ a self-teacher loss. Formally, if fyz (z, 0) and has (æ: 0) are the normalized outputs of the final DiT layer with and without layer dropping respectively. the self-teacher loss is Ly ||h (2x, 0) sq sq (hmm (x. 0)) where sq denotes a stop-ge nop-gradient. This gives additional supervision during the early phases of finetuning so the layer-dropper outputs can match full model performance.
Knowledge Distillation:
We show the differences between our Presto-L and the baseline approach in Fig. 3 By conditioning directly on the budget, and shifting the dropping schedule to account for the final DIT block behavior, we able more adapt computation for reduced budgets while preserving performance.
3.4 PRESTO-LS: LAYER-STEP DISTILLATION
As the act of layer distillation is, in principle, unrelated to the step distillation, there is no reason a priori that these methods could not work together. However, we found combining such methods to be surprisingly non-trivial. particular, we empirically find that attempting both performing Presto-L finetuning and Presto-S at the same time OR performing Presto-L finetuning from an initial Presto-S checkpoint results in large instability and model degradation, as the discriminator dominates the optimization process and achieves near-perfect accuracy on real data.
We instead find three key factors in making combined step and layer distillation work: (1) Layer- Step Distillation we first perform layer distillation then step distillation, which is more stable as the already- finetl--f layer dropping prevents generator collapse; (2) Full Capacity Score Estimation we keep the real and fake score models initialized from the original score model rather than the layer- distilled model, as this stabilizes the distribution matching gradient and provides regularization to the discriminator since the fake score model and the generator are initialized with different weights; and (3) Reduced Dropping Budget - we keep more layers during the layer distillation. We discuss more in Section 4.6 and how alternatives fail in Appendix A.7
4 EXPERIMENTS
We show the efficacy of Presto via a number of experiments. We first ablate the design choices afforded by Presto-S, and separately show how Presto-L flatly improves standard diffusion sam- pling. We then show how Presto-L and Presto-S stack up against SOTA baselines, and how we can combine such approaches for further acceleration, with both quantitative and subjective metrics. We finish by describing a number of extensions enabled by our accelerated, continuous-ting framework.
7

---

Published as a conference paper at ICLR 2025
4.1 SETUP
Model:
We use latent diffusion (Rombach et al. 2022 with a fully convolutional VAE (Kumar
al. 2023) to generate mono 4k.T audio and convert stereo using MusicHiFi (Zhu et al, Our latent diffusion model builds upon DiT-XL (Peebles & Xie 2023) and takes in three conditioning signals: the noise level, text prompts, and beat per minute (BPM) for each song. We use FlashAttention-2 (Dao 2023) for the DiT and torch. compile for the VAE decoder and MusicHiFi. For more details, see Appendix [A.I
2024..
Data: We 3.6K hour dataset of mono 44.1 kHz licensed instrumental music, augmented with pitch-shifting and time-stretching. Data includes musical meta-data and synthetic captions. For evaluation, use Song Describer (no vocals) (Manco et al. 2023) split into 32 second chunks.
Baselines: We compare against a number of acceleration algorithms using our base model: Consis- tency Models (CM) (Song al. 2023), SoundCTM (Saito et al.
2024), DITTO-CTM (Novack et al., 2024a). DMD-GAN (Yin et al. 2024), and ASE (Moon et al.
2024 as well as MusicGie (Copet and Stable Audio Open (Evans al. 2024c). See Appendix A.2 for more details.
al. 2023
Metrics: We use a number of common evaluation metrics for text-to-music generation, includ- ing distributional quality/diversity metrics PRADMMDDDDenitt/ecaalllCCCCeee ence (CLAP Score), and latency (RTF). See Appendix A.2 for more details.
prompt adher-
4.2 EXPLORING THE DESIGN SPACE OF PRESTO-S
Loss Distribution Choice: Table [ (Top), show the FAD, MMD, and CLAP score for many Presto-S dis- tilled models with different noise distri- bution choices. We find that the orig- inal DMD2 (Yin et al.) 2024) setup (first row) underperforms compared to adapting the loss distributions. The largest change is in switching POMD the training distribution, which im- proves all metrics. This confirms our hypothesis that by focusing on the region most important for text guid- ance (Kynkäänniemi et al. 2024), we improve both audio quality and text ad- herence. Switching PGAN to the training distribution also helps; in this case, the discriminator is made to focus higher- frehur-freyuecy fea- tures (Si et al., (024)), benefiting quality. We also find only small improvement when using the training distribution for PDSM- This suggests that while the training distribution should lead to more stable learning of the online generator's score (Wang et al. 2024b), this may not be crucial. For all remaining experiments, we use PDMD (raim)))
Pown
PDSM
POAN
RAD
MMD
CLAP
Lenst-Squares
Inf.
Inf.
0.37
1.73
27.45
Inf.
Inf.
Tr.
0.37
1.58
26.45
Inf.
Inf.
Tr.
Tr.
0.37
1.51
24.90
Inf.
Tr.
Tr.
0.27
1.27
33.12
Tr.
0.23
0.36
Inf.
Tr.
Tr.
Tr.
0.22
0.83
33.13
Tr.
TC:
Tt.
0..24
0.99
30.89
Non-Saturating GAN
Inf.
Tr.
Inf.
Tr.
0.26
0.39
31.48
Inf.
Tr.
Tr.
Tr.
0.25
0.96
31.78
Tt.
0.26
1.04
29.46
Table 1: (Top) Comparing different choices of noise dis- tribution for the Presto-S process. (Bottom) for best performing noise distributions, performance for standard GAN design vs. proposed least-squares GAN.
PGAN
( Irain
PDSM train)) and Peen(ai"l).
We ablate the effect of switching from the chosen least-squares discrimina- to the original softplus non-saturating discriminator, which notable treats the discriminator binary classifier and predicts the probability of real/generated. In Table (Bottom), we find that us- ing the squares discriminator leads consistent improvements in audio quality (FAD/MMD) and in particular text relevance (CLAP), owing to the increased stability from the least-squares GAN.
Discriminator Design:
We test how continuous-time conditioning compares against a discrete and find the former is preferred as shown in Fig. Continuous noise levels main- tain a correlation where more steps improve quality, while discrete time models are more inconsis- tent. Additionally, the continuous-time conditioning performs best in text relevance. While the and 2-step discrete models show slightly better FAD metrics than continuous these models have a failure mode as shown in Fig. [3 2-step discrete models drop high-frequent information and render transients (i.e. drum hits) poorly for genres like R&B hip-hop.
Continuous vs. Discrete Generator Inputs:
I and 2-step sampling,
8

---

Published as a conference paper at ICLR 2025
FAI
MMD
CL
0.40
Cord muuus Prestod
Continusus Prestod
0.34
Presto
Presto
0.35
2-step
Presto
2.0
Prisoo
0.32
0.30
0.30
Continutus as Presto-
1 sistop Presto 5
0.25
1.0
0.25
Prestoi5
Prested
Sample Step
Sumple Step
Sample Step
Figure 5: Continuous generator inputs vs. discrete inputs. Continuous inputs shows more consistent scaling with compute, while generally performing better in both quality and text relevance.
0.350
ASE
1.6
ASE
Presto-L
Presto-L
1.4
0.275
1.2
0.500
0.25
DM
ASE
0.75
10
1.5
2.0
2.5
5.0
3.5
4.0
1.0
15
2.5
2.5
3.0
3.5
4.0
1.0
1.5
2.0
2.5
3.5
3.5
4.0
Runtime (8)
Runtime (N)
Runtime (s
Figure 6: Presto-L results. Presto-L improves both the latency and the overall performance across all metrics, against both the leading layer dropping baseline and the base model.
4.3 PRESTO-L RESULTS
We compare Presto-L with both our baseline diffusion model and ASE (Moon et al. 2024 using the 2nd order DPM++ sampler (Lu et al. 2022) with CFG++ (Chung et al. 2024). For ASE and Presto-L, we use the optimal "D3" configuration from Moon et al. (2024), which corresponds to a dropping schedule, in terms of decreasing noise level (in quintiles), of 14, 12, 8, 4, (i.e. we drop 14 layers for noise levels in the top quintile, 12 for the next highest quintile, and so on). Layer distil- lation results at various sampling budgets are shown in Fig. 6 Presto-L yields an improvement over the base model on all metrics, speeding up by x27% and improving quality and text relevance. ASE provides similar acceleration but degrades performance at high sampling steps and scales inconsis- tently. Dropping layers improving performance can be viewed via the lens of multi-task learning, where (I) denoising each noise level is a different task (2) later layers only activating for lower noise levels enables specialization for higher frequencies. See Appendix A.10 for further ablations.
4.4 FULL COMPARISON
In Table we compare against multiple baselines and external models. For step distillation, Presto- S is best-in-class and the only distillation method to close base model quality, while achieving an over 1.5x speedup in RTF from the base model. Additionally, Presto-LS improves performance for MMD. beating the base model with further speedups (230/435ms latency for 32 second mono/stereo 44. 1kHz on A100 40 GB). We also find Presto-LS improves diversity with higher recall. Overall, Presto-LS is 15x faster than SAO. We investigate latency more in Appendix A.9
4.5 LISTENING TEST
We also conducted a subjective listening test to compare Presto-LS with our base model, the best non-adversarial distillation technique SoundCTM (Saito et al. 2024) distilled from our base model, and Stable Audio Open (Evans et al. 2024c). Users (n = 16) were given 20 sets of examples generated from each model (randomly cut to 10s for brevity) using random prompts from Song Describer and asked to rate the musical quality, taking into account both fidelity and semantic text match between 0- 0-0 We run multiple paired t-tests with Bonferroni correction and find Presto-LS rates highest against all baselines (p < 0.05). We show additional plots in Fig. 14
4.6 PRESTO-LS QUALITATIVE ANALYSTS
While Presto-LS improves speed and quality/diversity over step-only distillation, the increases are modest, the dropping schedule for Presto-I. reduced ([22, 8, 0) for step distillation

---

Published as a conference paper at ICLR 2025
Model
NPC
RTF-MIS (t)
(AD
(M)
CLAP Score (1)
Density (1)
Recall (1)
Coverage(1)
External Baselines*
Susillien-Smaal
16K
0.77
0.31
1.60
30.61
0.36
0.16
0.43
Musietien-Mediumt
0.27
MusieClen-Large Stable Andio Open
166K
0.37
0.25
1.21
32,83
0.44
0.15
0.54
100
4.54
0.23
1.07
35.05
0.20
0.37
0.49
Base Model, Dillasion Sumpling
DPM--2
80
1:72/734
0.24
12,82
31.56
0.51
0.20
0.41
DPM-25 25+ASE
80
9.80/9.22
0.25
30.03
0.27
0.41
Presto-L. (ours)
9,80
Consistency Distillation
0.47
SonndCTM
4
105.78
63.01
0.35
1.72
29.61
0.17
0.17
0.26
DITTO-CTM
4
118.77/65 /67.41
0.36
1.62
28.31
0.22
0.04
0.32
Adversarial Distillation
DMD-GAN
.1
0.29
Presto-S (ours)
Presto-LS (ours)
138.84 /7.44
0.23
0.73
32.21
0.49
0.14
0.48
Table Full Results on Song Describer (No vocals). External baseline RTFs are all natively stereo.
stability. To investigate more, analyze the hidden state activation variance of our step-distilled model in Fig. 7 The behavior is quite different than the base model, as the "spike" in the final layer is more amortized across the last 10 layers and never reaches the base model's magnitude. We hypothesize step-distilled models have more unique computation throughout each DiT block, making layer dropping difficult.
4.7 EXTENSIONS
Adaptive Step Schedule: A benefit of our continuous-time distillation is that besides setting how many steps (e.g., 1-4), we can set where those steps occur along the diffusion process by tuning the P parameter in the EDM inference schedule, which normally set to = 7. In particular, decreasing (lower bounded by 1) puts more weight on low- SNR features and increasing p higher-SNR fea- tures (Karras et al. 2022). Qualitatively, we find that this process enables increased diversity of outputs, even from the same latent code (see Appendix (A.))
Standard Deviation
a=80.00 00000
o=2.52
10a
d=17.53
o=0.17
0
5
25
Layer
Figure 7: Presto-S hidden activation var.
CPU Runtime: We benchmark Presto-LSS's speed performance for CPU inference. On Intel Xeon Platinum 8275CL CPU, we achieve a mono RTF of 0.74, generating 32 seconds audio in 43.34 seconds. We hope to explore further CPU acceleration future work.
Fast Inference-Time Rejection Sampling: Given Presto-LS's speed, we investigated using inference-time compute to improve performance. Formally, test the idea of rejection sampling, inspired by Kim et al. (2023), where we generate batch of samples and reject fraction of them according to some ranking function. We use the CLAP score to discard samples that have poor text relevance. Over a number of rejection ratios (see Fig. [5). we find that CLAP rejection sampling strongly improves text relevance while maintaining or improving quality the cost of diversity.
CONCLUSION
We proposed Presto, a dual-faceted approach to accelerating latent diffusion transformers by reduc- ing sampling steps and cost per step via distillation. Our core contributions include the develop- ment of score-based distribution matching distillation (the first GAN-based distillation for TTM), a new layer distillation method, the first combined layer-step distillation, and evaluation showing each method are independently best-in-class and, when combined, can accelerate our base model by 10-18x (230/435ms latency for 32 second mono/stereo 44. 14.1 15x faster than the comparable SOTA model), resulting in the fastest TTM model to knowledge. We hope our work will moti- vate continued work on (1) fusing step and layer distillation and (2) new distillation of methods for continuous-time models across media modalities such image and video.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Juan-Pablo Caceres, Hanich Deilamsalchy, and Chinmay Talegaonkar
ETHICS STATEMENT AND REPRODUCIBILIIT
As TTM systems become more powerful, there is both the opportunity to increase accessibility of musical expression, but also concern such systems may compete with creators. To reduce risk, we train our TTM work only on instrumental licensed music. Additionally, we hope that our focus on efficiency useful to eventually make interactive-rate co-creation tools, allowing for greater flexibility and faster idcation. Following these concerns, do not plan to release model, but have done our best to compare against multiple open source baselines and/or re-train alternative methods for comparison and in-depth understanding of the reproducible insights of our work.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv:2301. 11000.11135... 1355, 2023.
Jason Ansel, Edward Yang. Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito,o Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk. Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang. Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024.
Yatong Bai, Trung Dang, Dung Tran, Kazuhito Koishida, and Somayeh Sojoudi. Accelerating diffusion-based text-to-audio generation with consistency distillation. In Interspeech, 2024.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. eDiff-II Text-to-image diffusion models with ensemble of expert denoisers. arXiv:2211.01322 2022.
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor nov. MusicLDM: Enhancing novelty in text-to-music generation using reat-synchronous mixup strategies. In IEEE International Conference on Audio, Speech and Signal Processing (ICASSP), 2024.
and Shlomo Dub-
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park. Hyelin Nam, and Jong Chul Ye. CFG++: Manifold-conssrained classifier free guidance for diffusion models. arXiv:2406.0800 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défosse7. Simple and controllable music generation. In Neural Information Processing Sys- tems (NeurIPS), 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08661, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Neural Information Processing Systems (NeurIPS). 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv:2407.2/ 2024.
11

---

Published as a conference paper at ICLR 2025
Alexandre Dfossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio com- pression. arXiv:2210.13438 2022.
Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast audio diffusion. International Conference on Machine Learning (ICML), 2024a.
latent
Zach Evans, Julian Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv:2404.13001, 2024b.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv:2407.14358, 2024c.
Seth Forsgren and Hayk Martiros. Riffusion: Stable diffusion for real-time music generation, 2022.
URL httts::/r.ffuusion. wtt.:ffssiin.co/ooooooooaooo about
Azalea Gui, Hannes Gamper, Sebastian Braun, and Dimitra Fmmanouilidou. Adapting Frechet
Audio Distance for generative music evaluation. In IEEE International Conference on Audio, Speech and Signal Processing (ICASSP), 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS Workshop on Deep Gen. Models and Downstream Application,,, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffussion probabilistic models. Neural Infor- mation Processing Systems (NeurIPS), 2020.
Lu Hou, Zhiqi Huang. Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Neural Information Processing Systems (NeurIPS), 2020.
Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2Music: Text-conditioned music generation with diffusion models. arXiv:2302.03917, 2023.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Yan Zhu, and Taesung Park. Distilling diffusion models into conditional gans. arXiv:2405.05967. 2024.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurIPS, 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. arXiv:2312.026960 2023.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Ana- lyzing and improving the training dynamics of diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. Frechet audio distance: A metric for evaluating music enhancement algorithms, arXiv:1812 08466, 2018.
Dongjun Kim, Chich-Hsing Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufujii and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ODE trajectory of diffusion. In International Conference on Learning Representations (ICLR), 2023.
Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Va- jda, and Ali K. Thabet. Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation. arXiv:2405.08: 005522244 2024.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved RVQGAN. In Neural Information Processing Systems (NeurIPS). 2023.
12

---

Published as a conference paper at ICLR 2025
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXhv:2404.07724 2024.
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3; Improving text-to-image alignment with deep-fusion large language models, 2024a.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Inter- national Conference on Machine Learning (ICML), 2023.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm 2: Learning holistic audio generation with superviid pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cexsing, 2024b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.01095, 2022.
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Seamiingtt-cccche Accelerating diffusion transformer via layer caching. arXiv:2406.01777 017333 2024.
Ilaria Manco, Benno Week, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bodganov, Yusong
Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. The song describer dataset: corpus of audio captions for music-and-language evaluation. arXiv:2311.1005. 2023.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. IEEE/CVF International Conference on Com- puter Vision (ICCV), 2017.
Taehong Moon, Moonseok Choi, Eunggu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, and Juho Lee. A simple early exiting framework for accelerated sampling in diffusion models. In International Conference Machine Learning (ICML), 2024.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference Machine Learning. PMLR, 2020.
Javier Nistal, Marco Pasini, Cyran Aouameur, Maarten Grachten, and Stefan Lattner. Diff-a-riff: Musical accompaniment co-creation via latent diffusion models. arXiv:2406. 08384, 2024.
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrice and Nicholas J. Bryan. DITTO-2: Dis- tilled difffusion inference-time t-optimization for music generation. In International Society for Music Information Retrieval (ISMIR), 2024a.
Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, and Nicholas J. Bryan. DITTO: Diffu- sion inference-ting time T-optimization for music generation. In International Conference on Machine Learning (ICML), 2024b.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEF/CVF International Conference on Computer Visio (ICCV), 2023.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xue- feng Xiao. Hyper-SD: Trajectory segmented consistency model for efficient image synthesis. MrXiv:2404113666, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffuuion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models, 2024.
13

---

Published as a conference paper at ICLR 2025
Koichi Saito, Dongjun Kim, Takashi Shibuya, Chieh-Hsin Lai, Zhi-Wei Zhong, Yuhta Takida, and Yuki Mitsufuji. Soundctm: Uniting score-based and consistency models for text-to-sound gener- ation. arXiv:240..88500 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
arXiv:2202.00511. 2022.
Axel Sauer, Dominik Lorenz, A. Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv:2311.1704. 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv:2403.12011 2024.
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Mo), usai: Text-to-music generation with long-context latent diffusion. arXiv:2301.11757. 2023.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. arXiv:2104.0880 2021.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. FreeU: Free lunch in diffusion U-Net. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICML), 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations (ICLR), 2020.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. International Conference on Machine Learning (ICML), 2023.
Or Tal, Alon Ziv, Itai Gat, Felix Kreuk, and Yossi Adi. Joint audio and symbolic conditioning for temporally controlled text-to-music generation. arXiv:2406. 10970, 2024.
Yu-Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song. Yu Liu, Hongsheng Li, and Xiaogang Wang. Phased consistency model. arXiv:2405. 18407, 2024a.
Yuqing Wang. Ye He. and Molei Tao. Evaluating the design space of diffusion-based generative models. arXiv:2406. /2839, 2024b.
Felix Wimbauer. Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu. Peizhao Zhang. Sam Tsai, Jonas Kohler, et al. Cache me if you Accelerat- ing diffuusinn models through block caching. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dub- nov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to- caption augmentation. In IEEE International Conference Audio, Speech and Signal Processing (ICASSP), 2023.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. (.Ni::33/1.1880 2023.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. 2405v:2405. 14867, 2024.
14

---

Published as a conference paper at ICLR 2025
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- Stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP),
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control texl-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Ge Zhu, Yutong Wen, Marc-André Carbonneau, and Zhiyao Duan. Edmsound: Spectrogram based diffusion models for efficient and high-quality audio synthesis. arXiv:2311.08667 2023.
Ge Zhu, Juan-Pable Caceres, Zhiyao Duan, and Nicholas Bryan. MusicHiFi: Fast high-fidelity stereo vocoding. IEEE Signal Processing Letters (SPL), 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MODEL DESIGN DETAILS
As we perform latent diffusion, we first train a variational autoencoder. We build on the Improved RVQGAN (Kumar et al., 2023) architecture and training scheme by using a KL-bottleneck with dimension of 32 and an effective hop of 960 samples, resulting in an approximately 45 Hz VAE. We train to convergence using the recommended mel-reconstruction loss and the least-squares GAN formulation with L1 feature matching on multi-period and multi-band discriminators.
Our proposed base score model backbone builds upon DiT-XL (Peebles & Xie 2023). with modifi- cations aimed at optimizing computational efficiency. Specifically, we use streamlined transformer block design, consisting of a single attention layer followed by single feed-forward layer, similar to Ilama (Dubey et al., 2024). Our model utilizes three types of conditions including noise levels (timesteps) for score estimation, beat per minute (BPM) values of the song, and text descriptions. Following EDM, we apply logarithmic transformation to the noise levels, followed by sinusoidal embeddings. Similarly, BPM values are input as scalars then go through sinusoidal embeddings to generate BPM embeddings. These processed noise-level embeddings and BPM embeddings then combined and integrated into the DiT block through an adaptive layer normalization block. For text conditioning, we compute text embedding tokens with T5-based encoders and concatenate with audio tokens at each attention layer. As a result, the audio token query attends to a concatenated sequence of audio and text keys, enabling the model to jointly extract relevant information from both modalities. To provide baseline architectural speedups, we use FlashAttention-22 (Dao) 2023) for the DiT and Pytorch 2.0's built in graph compilation (Ansel et al., 2024) for the VAE decoder and MusicHifi mono-to-stereo.
Our discriminator design follows Yin et al. (2024) with a number of small modifications. D& con- sists of 4 blocks of ID convolutions interleaved with GroupNorm and SiLU activations, and a final linear layer to collapse the channel dimension. The discriminator thus does not use any final linear layer to project
single value, and instead its' output is also ID sequence but at even heavier downsampling than the input representation at =2.8 Hz. The discriminator receives its' input from the output of the 14th DiT Block (i.e. the halfway point through our 28 block DiT), as DiT's lack clear "bottleneck" layer to place the discriminator like in UNets. We leave further investigation into discriminator design and placement inside the model for future work.
For the diffusion model hyparameter design, we follow Karras et al. (2024). Specifically, we set Admaa 0.5, Pmean -0.4, Pud = 1.0, mmx 80, m 0.002. We train the base model with 10% condition dropout to enable CFG. The base model was trained for 5 days across 32 A 100 GPUs with batch size of 14 and learning rate of Ic-4 with Adam. For all score model experiments, we use
CFG+ (Chung et al. 2024) with w = 0.8.
For Presto-S, following Yin et al. (0024) we use fixed guidance scale of w 4.5 throughout distillation for the teacher model as CFG++ is not applicable for the distribution matching gradient. We use 5 fake score model (and discriminator) updates per generator update, following YYn et al.) (2024), as we found little change in performance when varying the quantity around 5 (though using S 3 updates resulted large training instability). Note that throughout Presto-S, the fake score model and the discriminator share an optimizer state. Additionally, we use a learning rate of 5e-7 with Adam for both the generator and fake score model / discriminator. We set 11 = 0.01 and 1/2 0.005 following Yin et al. (2024). For all step distillation methods, we distill each model with a batch size of 80 across 16 Nvidia A 100 GPUs for 32K iterations. We train all layer distillation methods for 60K iterations with batch size of 12 across 16 A100 GPUs with learning rate of 8e-5. For Presto-L, we set v 0.1.
A.2 EXPERIMENTAL DETAILS
A.2.1 BASELINE DETAILS
Our benchmarks are divided into two main classes: acceleration algorithms and external ooun-souree models. For acceleration algorithms, we distill our internal base model per method, utilizing pub-
licly available code as reference when available (Song et al. 2023 Saito et al. 2024 Yin al.)
16

---

Published as a conference paper at ICLR 2025
2024 For the open-source external models, we use the models directly in their default setups as recommended by Copet et al. (2023): Evans et al. (024c).
Consisteney Models (CM) (Song et al.) 2023 Bai et al. 2024): This distillation technique
learns a mapping from anywhere on the diffusion process to the data distribution (i.e. ->
To) by enforcing the self-consistency property that G (my, t) = (a(arrt t') Vi, We follow the parameterization used in past audio works (Bai et al. 2024 Novack et al. (024a)
that additionally distills the CFG parameter into the model directly.
SoundCTM (Saito al. 2024): This approach distills a model into consistency tra- jectory model (Kim et al. (023) that enforces the self-consistency property, learning an anywhere- anywhere-to-anyywerr toeyhwhe mapping. SoundCTM forgoes the original CTM adversarial loss
and calculates the consistency loss via intermediate base model features.
DITTO-CTM (Novack et al. 2024a), This audio approach is also based off of (Kim et al., )023), yet brings the consistency loss back into the raw outputs and instead replaces CTM's multi-step teacher distillation with single-step teacher (like CMs) and removes the learned target timestep embedding, thus more efficient (though less complete) than SoundCTM.
DMD-GAN (Yin et
2024): This approach removes the distribution matching loss from DMD2, making it fully GAN-based finetuning method, which is in line with past adver- sarial distillation methods (Sauer al. 2023
ASE (Moon et al. 2024), This funetuning approach for diffusion models, discussed in Sec. [.3] finetunes the base model with the standard DSM loss, but for each noise level drops fixed number of layers, starting the back of the diffusion model's DiT blocks.
MusicGen (Copet et al. 2023): MusicGen is non-diffusion based music generation model that uses an autoregressive model to predict discrete audio tokens (Dlossez et al. 2022 at each timestep in sequence, and comes in small, medium, and large variants (all stereo).
Stable Audio Open (Fvann et al. 2024c): Stable Audio Open is SOTA open-source audio diffusion model, which can generate variable lengths up to 45s in duration. Stable Audio Open follows a similar design to our base model, yet uses cross-attention for conditioning rather than AdaLN which we use, which increases runtime.
A.2.2 METRICS DETAILS
We use Frechet Audio Distance (FAD) Kilgour et al., 2018), Maximum Mean Discrepancy (MMD) (Jayasuanna et al. 2024). and Contrastive Language- Audio Pretraining (CLAP) score (Wu et al. 2023), all with the CLAP-LAION music backbone (Wu al., 2023) given its high correlation with human perception GGui et al 2024). FAD and MMD measure audio quality/realness with respect to Song Describer (lower better). and CLAP score measures prompt adherence (higher better). When comparing other models,
also include density (measuring quality), recall and coverage (mea- suring diversity) (Naeem et al. 2020), and real-time factor (RTF) for both mono (M) and stereo (S, using MusicHiFi), which measures the total seconds of audio generated divided by the generation time, where higher is better for all.
17

---

Published as a conference paper at ICLR 2025
A.3 PRESTO-S ALGORITHM
Algorithm/ Presto-S
input generator Gø. real score model Preal, fake score model Map, discriminator Di CFG weight w, Pgen Po Prmp (auala,, PDsM (o ), PGAN(ollan) real sample Treals GAN weights vi, 42,
optimizers 91,92, weighting function
1: σ ~ Pgen (a (a
2: Egen (0,
3: 2 gen (Trrlll
rreal + E'genn σ)
4: if generator turn then
5: a POMD (otrain)
6: Edmd ~ (),
7: VΦLDMD (CHap (ággn + Eddmd, o) Prreal (# + Fedmd o)) V L
8:
PGAN (Gaan(a
9:
Efake (O
10: LGAN ||| Dep ( gen Elake 0)1
11:
Φ + Φ - 91 &V + 21 V LGAN)
12:
else
13: T (DSM
Edsm (0,
15:
DSM = X(or)l(@gen
Me (āgon + Edsms o)1
16: Treal, fake ~ OOCA((o
Greals Elake 10,
18:
LGAN | [ (2een + Ttake Take; Taaee) + +111- (
(Freal + Cealt (reat)
3 26 92 V DSN + 212 LGAN)
20:
end if
output &.
We outline condensed algorithm of Presto-S in math notation in Algorithm! |
A.4 PRESTO-S PSEUDO-CODE WALKTHROUGHI
We provide comprehensive algorithm walkthrough using PyTorch psuedo-code of our Presto-S training loop below. To perform Presto-S, we first define the corruption process for any given clean sample, according to either the training p(otmin) or the inference p(oi"() noise distribution:
def diffuse (x, dist):
2 eps = noise saise_normall like (x)
if dist == 'training':
sigma- training_dist_In like (x)
5 elif ćist 'inference':
6 sigma- = inference_dist_1ikg (x)
7
return x sigma eps, sigma
We then define each of the component loss functions for the Presto-S continuous-time DMD2 dis- tillation process. This corresponds to the three loss types: the distribution matching loss, the least- squares GAN loss, and the fake denoising score matching loss. For the distribution matching loss, we corrupt some generated sample according to the training distribution and then pass that into both the fake and real score models (where the real score model uses classifier-free guidance). The difference in these scores forms the distribution matching gradient:
def dmd (x, real_score scoeeeeerreeeee soke_score_model. efg):
2 x_noise, sigma diffuse (x, 'training')
3 fake_dencised =
(x_noise, sigma)
4 real_dencised = real leal_score_mmett (x -__noise, sigma, efg) return fake_ dekoisded real_ denciseed
For the least-squares GAN loss, we corrupt some sample (either real or generated) according to the training distribution and pass this through the discriminator (which itself involves first passing
18

---

Published as a conference paper at ICLR 2025
through some of the fake score model to extract intermediate features). The output of the discrim- inator is then passed into the least-squares loss against some target value (i.e. the generator wants to push the discriminator outputs on generated samples towards 1, while the discriminator aims to push generated samples towards and real samples towards 1):
def gan (x, discriminator, tgt=1);
noise, sigma diffuse (x, 'training") sigma)
3 d out discriminator (x (x_noise, noise, return mae (tgt, d_out)
Finally, we have the fake DSM loss. This loss is identical to the normal diffusion loss (with weighted MSE between the outputs of the score model and the clean data), yet will be calculated treating generator outputs as the ground truth clean data and using the fake score model:
def dam (x, fake
model)
2 x noiss,, sigma diffuse (x, 'training')
3 x x_denoised fake_ cecee_scre__mmme @ecor_mmoolloo (x_ncise, sigma)
4 return mss (x, x_dencised, sigma)
Given these helper loss functions, as follows. For both the generator and discriminator turns, we first corrupt some real input data according the inference distribution, and pass this through our generator get the generator outputs x.denoised (steps (1) and (4) in Fig. 88. If is a generator turn (which happens once for every fake score turns), we calculate the distribution matching loss (step (2)) and the generator adversarial loss (step (3)) on x.dencisen and update the generator. If it is fake score turn, we calculate and the discriminator' adversarial loss (step (5)) on both the generated x x_denoi sed and real samples x and the fake DSM loss (step (6)) x x_denoiss sed, thus updating the fake score model and the discriminator:
proceed with the main distillation loop, which is
def forward generator, discriminator, fake_score score_ score_model, real www.sscooreccee model_ generator_turo, mu_l, n
3 ):
4 # step (1) and (4)
5 x_noise, sigma = diffuse (x, 'inference')
x_denoised = generator (x_neise, sigma)
8 generator_turn: # GENERATOR TURN
9
# Distribution Matching Loss, step (2)
dma_ = dxdoissd, real_score_modde take_score_model, ctg)
Generator Adversarial Loss, step (3) loss gan (x_denoi sed, discriminator, 1)
13
loss dos_loss + mu_1 g_less
else: FAKE SCORE TURN Discriminator Adversarial Less, step (5)
18
loss gan gan discriminator, 1) + gan (x_ (x_denoised,, discriminator, 0)
fake DSM loss, step (6) dem dsm dx_dnoisedd, fake_score_model
loss -
+ nu_2 d loss
return css
This constitutes one full update of the Presto-S process, alternating between the generator and fake score model / discriminator updates. At inference time, we can feed in pure noise and alternate between generating clean data with our generator and adding progressively smaller noise back to the generation (for some pre-defined list of noise levels), allowing for multi-step sampling:
19

---

Published as a conference paper at ICLR 2025
def inference (generator, sigmas, start_noise).
2 start_ncise
3 for sigma in sigmas:
4 X = x nolse_normal_iik (x) sigma
generator (x, sigma)
return
A.5 PRESTO-S EXPANDED DIAGRAM
For an in-depth visual illustration of Presto-S, please see Fig. 8 and Fig. for expanded training and inference diagrams.
Training:
Real Music
Generator
Generated Music
Generator Turn:
G
PDMD Arain)
POAN train)
Fake Score Model
Ala
LGAN
Fake Score Model
DDDD
Preal
DMD Loss:
Real Score Model
Adversarial Loss:
Discriminator
d Φ Gi(VaCDDD + in V&VCCa))
Real Music
Generator
Generated Music
Fake Score
Pon (
Turn:
G
Fake Score Model
FCAN srAin)
Poss (mr traln//nf
Fake Score Model
Cake FSk DSM
CGAN
Disoriminator
Adversarial Loss:
Fake-DSM
g2(V ga Daak CoNN)
Figure 8: Presto-S training process.
Inference:
<~ Niio, I
Generator
G s
Figure 9: Presto-S inference. For multi-step sampling, use sing-pong-liee sampling.
20

---

Published as a conference paper at ICLR 2025
A.6 PRESTO-L ALGORITHM
Algorithm Presto-L
input pre-trained score model μo. real sample Preal, self-teacher weight v, optimizer g: 92, weight-
ing function X, of DiT blocks B, budget mapping E, layer drop function LD
1: σ ~ p(oltinn
2: E(Or)
3: 6 NN I)
4: &r. hr. LD( poo Treal + aee,
5: Prail, haill = LD Freal + oe, B)
6: LDSM >(o))ll
& 2
7: Ls = |hz sg (hrm)) 11
8: 0+0 g eDV + VVlLa)
output
Algorithm LD: Modified DiT forward pass with layer dropping and budget conditioning.
input score model noise embedder / score model budget embedder Me
budget score model DiT final imput σ, noise
blocks {ub 1. score score model budget AdaLN IN score model output layer /4 level Ø, budget
poise (r) 1: e, 140 budget
// embed noise level // embed budget
2: Ve (b)
3: e = e, + Cb
4: for i == 1 to b 1 do
5: // apply first b-1 DiT blocks
6: x = Ho (2r e)
7: end for
8: x ug (I, e)
apply final DiT block
9: x Po (x, eb)
// apply budget-based AdaLN 11 get normalized hidden state for Cat
10: h = x |
output Po (
inal
We show the full algorithm in detail for Presto-L in Algorithm
2 which proceeds modified
version of standard diffusion training like in Moon et al. (2024). We first sample some noise level a, and then map the noise level its corresponding budget b given some mapping function E(-)- Following Moon et al. 20044), f - {i) noise level according the training noise distribution F(oo) (where F is the cumulative distribution function) to some budget amount, which we write as [91. 92, 4s Qs] for a mapping based on descending quintiles (e.g. 91 14 means that all noise levels in the largest quintile drop layers).
deterministic map from the percentile of the
We then call the modified forward function of the model LD (see Algorithm 3 on the noisy inputs with both the given budget and the full budget (i.e. using all DiT blocks). LD modifies the forward pass of the model by (1) adding global budget embedding that is added to the noise embedding (2) only iterating through the first 1 DiT blocks followed by the final DiT block (to preserve final block behavior, see Section 3.3 (3) adding an additional AdaLN conditional only on the budget after the final DiT block and (4) also returning the normalized hidden state the model (i.e. the input to the final layer of the DiT, normalized along the channel dimension). We calculate the standard denoising score matching loss LDSM as normal but with our layer-dropped outputs, and additionally calculate La the MSE between the layer-dropped hidden state and the full budget hidden state (with a stop-gradient operation on the full budget pass.
A.7 ANALYZING FAILURE MODES OF COMBINED LAYER AND STEP DISTILLATION
We empirically discovered number of failure modes when trying to combine step and layer distil- lation. As noted in Section 4.6 the heavier per-layer requirements of distilled few-step generation made all standard dropping schedules (Moon et al.) 2024) intractable and prone to quick generator collapse, necessitating more conservative dropping schedule. In Fig. 10 show the generator
21

---

Published as a conference paper at ICLR 2025
loss, discriminator loss, distribution matching gradient, and the discriminator's accuracy for the real inputs over distillation, for a number of different setups:
Dactiiiete East
Geventor
Daribstte Vachiy Goddrtt
Step distillation losses for early distillation for multiple combination methods. Presto-LS is the only setup that avoids generator degradation and high variance distribution matching gradients.
Figure 10:
Presto-S, pure step distillation mechanism (blue).
Presto-LS, optimal combined setup where we pretrain the model with Presto-L and then perform Presto-S, but with keeping the real and fake score models initialized from the original score model (orange).
LS with L-Fake/Real, which mimics Presto-LS but uses the Presto-L. model for the fake and real score models well (green).
Step then Layer, where we first perform Presto-S distillation and then continue distillation with Presto-L layer dropping on the generator (red).
Step and Layer jointly, where we perform Presto-S and Presto-L at the same time initial- ized from the original score model (purple),
We see that the runs which do not initialize with pretrained Presto-L (Step then Layer, Step and Layer) show clear signs of generator degradation, with increased generator loss, decreased discrim- inator loss, and notably near perfect accuracy on real samples, as attempting to learn to drop layers from scratch during step distillation gives strong signal to the discriminator. Additionally, LS with L-Fake/Real inherits similar collapse issues but has higher variance distribution matching gradient the layer-distilled real and fake score models poor estimators of the gradient.
A.8 INFERENCE-TIME NOISE SCHEDULE SENSITIVITY ANALYSIS
Given our final Presto-LS distilled 4tsp generator, we show how changing the inference-tie time noise schedule can noticeably alter the outputs, motivating our idea of continuous- timmeoeuss-teeeeeee conditioning.
The EDM inference schedule follows the form of:
dieN
L1/p + N 1 (o
max
(9)
where increasing the P parameter puts more weight on the low-noise, high-SNR regions of the diffusion process. In Fig. [1] show number of samples generated from Presto-IS with identical
22

---

Published as a conference paper at ICLR 2025
conditions and latent codes (i.e. starting noise and all other added gaussian noise during sampling), only changing p. from the standard of to 1000 (high weight low-noise region). We expect further inference-tine time tuning of the noise schedule to be beneficial.
p 7
p= 1000
"A squirrel dancing in the backyard, uplifting" BPM=120
"song for my departed goldfish"
goldfissh" BPM=80
"active winter on the mountains" BPM=100
"Epic videogame boss battle OST" BPM=140
"sea shanty for a drunken sailor" BPM=120
Figure 11:
Generations from Presto-LS from the same text prompt and latent code (i.e. starting noise and added noise during sampling), only varying the p parameter between (7 and 1000). Purely shifting the noise schedule for 4-step sampling allows for perceptually distinct outputs.
A.9 RTF ANALYSIS
latencya (b) where Tg is the generation duration or how much contiguous audio the model can generate at once and latency, is the time takes for generation following (Evans et al., 2024b) Zhu et al. 2024). This is different from the fixed-duration batched RTF used in Nistal et al. (0024). We test
We define the RTF for a model 0 as: RTF;(0)
bI'e
as well as the maximum batch size we could attain for each model on single A10 40GB to get sense of maximum throughput. We show results in Table 3 and Table for all components of our generative process, including latency metrics for generation (i.c. the difrusion model or distilled generator), decoding (i.e. VAE decoder from latents to audio) and the optional mono-to-stereo (M2S), as well as overall RTF/latency for mono and stereo inference. We omit the MusieGen models and the other step-distillation methods which
23

---

Published as a conference paper at ICLR 2025
Generation Latency
Decoding Latency
Mono
Mono Lateney
M2S
Sterco RTF
Stereo
Model
RTF
Latency
Lateney
Stable Audio Open
6159.01
887.99
N/A
N/A
4.54
7047
Basc DM
4079.81
64.45
7.72
4144.27
205.31
7.36
4349.58
ASE
3200.73
64,45
9.80
3265.19
205.31
9.22
3470,50
Presto-L
3201.19
64.45
9.80
3265.64
205.31
9.22
3470,95
SoundCTM
238.06
64.45
105.78
302.51
205.31
63.01
507.83
Presto-S
204.98
64.45
118.77
269.43
205.31
67.41
474.74
Presto-LS
166.04
64.45
138.84
230.49
205.31
73.43
435.8
Table 3: Latency (ms) and real-time factor for batch size of one A100 40GB GPU.
Generation Latency
Decoding Latency
Mono RTF
Mono Latency
M2S
Stereo RTF
Stereo
Model
Latency
Latency
Stable Audio Open
34602.86
4227.54
N/A
N/A
7.42
38830,4
Base
18935.26
1198.21
14.3
20133.46
1775.73
96.38
21909.19
ASE
14584.85
1198.21
18.25
15783.05
1775.73
96.25
17558.78
Presto-I.
14655.02
1198.21
18.17
15853.23
1775.73
96.25
17628.96
SoundCTM
1135.65
1198.21
123.4
2333.86
1775.73
92.98
4109.58
Presto-S
715.41
1198.21
150.5
1913.62
1775.73
92.18
3689. 334
Presto-LS
695.19
1198.21
152.11
1893.4
1775.73
92.13
3669.13
Table 4: Latency (ms) and real-time factor for max batch size on an A100 40GB GPU.
share the same RTF as Presto-S. For the fastest model Presto-IS, the biggest latency bottleneck is the mono-to-stereo model (Zhu et al. 2024) and VAE decoder. In future work, we hope to optimize the VAE and mono-to-stereo modules for faster inference.
A. PRESTO-L DESIGN ABLATION
6.00
Madoos
1.0
bugee Su
220
223
0200
6.10
ACC Sell heche Lo0
Auroine
0.4-
1.2
0.54
035
525
Bedae Corali Coralarity
1.5
28
30
45
30
20
25
2.5
10
23
20
Figure 12: Presto-L ablation. Each individual change of our layer distillation vs ASE beneficial.
To investigate how each facet of our Presto-I. method contributes to its strong performance ASE, we ran an additional ablation combining ASE with each component (i.e. the shifted dropping schedule, explicit budget conditioning, and the self-teacher loss). In Fig. 12] we see that the core of Presto-L's improvements come from the shifted dropping schedule (which preserves final layer behavior), the ASE+shift performs similarly to Presto-L on high-step FAD and MMD. Addition- ally, we find that the budget conditioning and self-teacher loss help text relevance more so than the shifted schedule does. All together, the combination of Presto-I. 's design decisions leads to SOTA
24

---

Published as a conference paper at ICLR 2025
audio quality (FAD/MMD/Density)) and text relevance compared to any one facet combined with ASE.
A.11 DISCRETE- TIME FAILURE MODES
In Fig. visualize the poor performance of distilled models that 1-2 step discrete-time con- ditioning signals. Notice that for the same random seed, the high-freque frequeny performance is visually worse for discrete-time vs. continuous- time conditioning, motivating our proposed methods.
Discrete (1-2) DMD
Continuous DMD
Text
Fugeery
Oupprnn
Mocelnn
Text Prompt: Grocy hi hop,
Slurry Rransietta
Clear Trarsienta
Figure 13: Failure mode of 1-2 step discrete models vs. continuous models (each row is same random seed and text prompt), with 2-step generation. Hip-Hop adjacent generations noticeably drop high frequency information, and render percussive transients (hi-hats, snare drums) poorly.
A. LISTENING TEST RESULTS
We visualize our listening test results from Section 4.5 using a violin plot.
Meaa Opinion Score
75
50
25
Figure 14: Violin plot from our listening test. Presto-LS is preferred over other baselines (p < 0.05).
A.13 REJECTION SAMPLING RESULTS
We show rejection sampling results where we generate a batch during inference and then use CLAP to reject the r least similar generations to the input text prompt. CLAP rejection sampling improves CLAP Score and maintains (and sometimes improves) FAD and MMD, but reduces diversity.
ND
P40
0.50
010
6:30
0.2
420
415
2.3
2.5
3.50
2.0
230
013
Figure 15: Rejection sampling eval metrics vs. rejection ratio. Base Presto-LS red. CLAP rejection sampling improves both CLAP score and overall quality, while reducing diversity.
25


Reviews:
[{"review_id": "Z9NflfrKww", "reviewer": "Reviewer_Dv9d", "summary": "The paper proposes \"Presto!\" for effective and efficient music generation. More specificly, Presto! is a set of model distillation techniques that aims at improving the inference efficiency of continuous diffusion models from two aspects:\n1. overall inference steps,\n2. runtime of each inference step.\n\nThe motivations and approaches are sound, and the results on the demo page are extremely convincing. Huge efforts have been done to applying latest SOTA methods to their own DNN architecture. Experiments are well designed to support the claims and motivations.\n\nIt is unfortunate that the authors do not plan to open their method, as I feel the framework is a little bit complex: it invovles extensive grid seearch for a better stability in adversarial training.", "questions": "Is it possible to open the implementation without sharing pretrained model weights? Or open a part of key tricks or components in the work?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Very solid target: enhancing efficiency from both the number of steps and the runtime of a single ste.\n2. novel to audio community to combine both step distillation and layer distillation, on top of the successful usage of an adversarial loss.\n3. Clear presentation. The origin of the proposed techniques, and the differences between Presto! and prior arts are well explained. Related works are very up-to-date hence informative.\n4. Great objective measurements. Presto! could outperform or keep on-par with the teacher model, while reducing the sampling steps to as few as 4, for the first time in audio community. The proposed continuous step distillation offers a more predictable performance gain when more inference budget is given.\n5. Excellent subjective quality. The violin plot shows that Presto! outperforms Stable Audio Open and SoundCTM, two works published earlier this year, which is impressive. The samples in demo page sound great.", "weaknesses": "The ablation study indicates quite high complexity if we want to stably and effectively use Presto!. I am interested in how well the framework could be when applied to other VAEs or generator backbones, especially if no resource is open.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WAMLjQD8u", "reviewer": "Reviewer_ZS9n", "summary": "This paper introduces an innovative approach to improve text-to-music generative models by significantly reducing inference time. Focusing on score-based diffusion models, the authors present two main contributions: first, a reformulated few-step synthesis approach utilizing online GAN-based adversarial distillation, adapted specifically for continuous time diffusion through the DMD2 framework; and second, a layer distillation technique, inspired by ASE, which uses a layer-budgeting module to prioritize layers based on noise levels.\n\nThey further explore the combination of step and layer distillation, demonstrating cases where the methods complement each other effectively. Their experiments, conducted on latent diffusion models with VAE and DiT blocks, use internal datasets for training and Song Describer for evaluation. Comparative analysis on the CLAP-LAION dataset with FAD scores shows that each method, independently and in combination, improves upon alternatives while maintaining real-time factor (RTF) performance.", "questions": "In line 35, does 5-20 seconds latency refer to 32 seconds audio? If yes, please clarify because it's mentioned in the abstract but not in the introduction.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper marks an important step toward a unified approach for faster generative models by bridging adversarial and diffusion methods for text-to-music tasks. In particular, Section 3.1 demonstrates this by applying GAN-based distillation to continuous-time score-based diffusion models. \n\nAligning most distributions with the training distribution, as shown in Table 1, empirically improves all metrics, underscoring the robustness of their approach.\n\nThe authors also navigate the task of combining layer and step distillation, finding a highly effective recipe that substantially enhances generation speed. \n\nAchieving a 10-18x speedup in audio generation is a noteworthy accomplishment, positioning this collection of methods as a valuable resource for practical applications in the field.", "weaknesses": "While the paper is well-structured in a general sense, certain sections are hard to follow due to the level of detail. For example, Figure 1 is dense and, despite a detailed caption, remains challenging to interpret. The figure also uses various notations that lack prior explanation, making it difficult for readers to follow the intended process.\n\nAdditionally, the models are trained on internally licensed data, so the results cannot be easily referenced in future work. With no indication that the code will be released, reproducing this complex setup—including multiple models, distributions, and training phases—would be exceptionally difficult. This limitation means future researchers may be restricted to evaluating audio examples alone rather than building upon this work directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sSn4ZFUuPv", "reviewer": "Reviewer_eJot", "summary": "This paper presents a distillation method to develop a fast music generation model. The authors modify the [DMD2](https://arxiv.org/abs/2405.14867) and [ASE](https://proceedings.mlr.press/v235/moon24a.html) frameworks to improve the model performance. The proposed method not only enables a model to generate music signals with a few NFEs but also makes the model size smaller. The authors demonstrated that their trained model outperforms models trained/equipped with previous methods in terms of multiple evaluation metrics, using their in-house data for training and the Song Describer dataset for evaluation. They also conducted comprehensive ablation studies to show that their choices are reasonable.", "questions": "I would appreciate the authors' response to my comment in \"Weaknesses\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written. I could understand their motivation, the proposed method, and the experimental results.\n2. The authors modified the existing DMD2 and ASE frameworks while conducting ablation studies. As ablation studies are comprehensive, readers can understand the reasons for the modifications/choices.\n3. The authors demonstrated that the proposed distillation method works well in several aspects (AD, MMD, CLAP scores, and subjective evaluation) despite its fast generation.", "weaknesses": "I do not think that the paper has a critical weak point, but let me mention some weaknesses.\n\n- Although the paper is well-written, some readers would struggle to reproduce the results without their code public. (I understand some institutes/companies do not make their code publicly available due to their policy.)\n- Since the proposed training framework is general, I thought I would like to see experimental results in the text-to-audio generation task evaluated on AudioCaps. Furthermore, if we have a DiT-based baseline model for text-to-image or text-to-video generation, we can apply the proposed method to them. I thought I would like to see the performance in those tasks as well.\n- The explanation of baselines in Section 4.1 is a little unclear to me. My understanding is that the authors trained the teacher model and applied the proposed technique and previous ones (CM, SoundCTM, DITTO-CTM, DMD-GAN, and ASE). For MusicGen and Stable Audio Open, they just downloaded and evaluated the official distributed models. Whether my understanding is correct or not, a clearer description would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWYW9JaT0Q", "reviewer": "Reviewer_pB9s", "summary": "The authors propose Presto, a distilled diffusion-based text-to-music model. Specifically, they introduce Presto-S, a model that applies DMD2 to a text-to-music generative model, and Presto-L, a model that performs layer-based distillation by extending ASE. Additionally, they propose Presto-LS, which combines these two distillation techniques. Experimental evaluations demonstrate that all three models achieve high-quality and efficient text-to-music generation.", "questions": "- Around L90–L91 in Introduction, the authors mention TTA. Are there plans to conduct text-to-audio generation experiments during the rebuttal period?\n   - If not, claiming it as a contribution in the Introduction is an overstatement.\n- How exactly is the CFG-augmented real score, represented by $\\tilde{\\mu}^{w}_{\\text{real}}$ in Eq (5), formulated?\n     - Is it something like $\\mu_{\\text{real}}( \\hat{x}_{\\text{gen}}+\\sigma\\epsilon, \\sigma, \\varnothing) + w (\\mu_{\\text{real}}( \\hat{x}_{\\text{gen}}+\\sigma\\epsilon, \\sigma, \\bf{c}) - \\mu_{\\text{real}}( \\hat{x}_{\\text{gen}}+\\sigma\\epsilon, \\sigma, \\varnothing)$, $where $\\bf{c}$ is condition?\n- Section 3.1, L210–L211: Regarding “the fake score model is updated 5 times as often as the generator to stabilize the estimation of the generator’s distribution,” how robust is this to different model training setups, architectures, and datasets? For example, if the number “5” were changed to “4” or “6,” how significantly would it affect training stability and generation quality in those sense?\n- Section 3.1, L212–L214: In “Specifically, a discriminator head $D_{\\psi}$ is attached to the intermediate feature activations of the fake score network,” which specific part of the DiT block do the “intermediate feature activations” refer to? Also, how is the architecture of the discriminator head $D_{\\psi}$ actually configured? The explanation in Section 3.2.3 is still high-level for both points. Could you provide further detail using pseudo-code or a figure illustrating the architecture?\n- In Algorithm 1, $\\nu_1, \\nu_2, g_1, g_2$ are mentioned. What values do the authors set as $\\nu_1, \\nu_2$? Are both $g_1, g_2$ optimized with Adam at a learning rate of 1e-4? Also, do the authors use learning rate schedulers?\n- How is Presto-L distilled? The explanation in Section 3.3 and $\\mathcal{L}_{\\text{st}}$ alone seem insufficient to clarify the distillation process. Could you provide a more detailed explanation using an algorithm table, pseudo-code, or a figure illustrating the distillation process?\n- Since multi-step sampling uses stochastic sampling like CM, I would expect that as the number of sampling steps increases, sample quality will eventually plateau and then degrade if increased further. Have you observed such a phenomenon when increasing the number of sampling steps to around 15, for example?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The authors propose the first diffusion-based text-to-music distillation model to successfully incorporate GAN loss.\n- They also introduce a layer-distillation method to reduce computational cost at each sampling step, which has not yet been explored in the text-to-music task.\n- Additionally, they present a methodology for integrating these two orthogonal distillation techniques.\n- The effectiveness of these three methods is demonstrated through extensive evaluation.", "weaknesses": "- The explanation of Presto-L seems to assume prior knowledge of ASE, which is unfriendly for readers. For better readability and self-containment, it would be helpful to explain Presto-L more thoroughly by including an algorithm, equations, or an illustrative figure of the distillation process, as done for Presto-S. At the very least, a more detailed literature review of ASE should be provided. (ASE is a relatively new method presented at ICML 2024, and as the authors note in the last paragraph of Section 2, such layer-distillation methods have not yet been explored in text-to-music, underscoring the need for a detailed explanation)\n- The contribution to the research community in terms of reproducibility is limited. Therefore, I strongly recommend either releasing the codebase or providing sufficient implementation details to facilitate straightforward replication, at least to allow future research to conduct follow-up studies (see also Questions).\n    - While I understand from the reproducibility statement that the authors chose not to release checkpoints, it should still be feasible to release the codebase itself (as done in some prior works [1][2]) or to include sufficient implementation details for reproduction in the Appendix.\n\n[1] Evans, Z., Carr, C.J., Taylor, J., Hawley, S.H. and Pons, J., 2024. Fast timing-conditioned latent audio diffusion. arXiv preprint arXiv:2402.04825.\n\n[2] Evans, Z., Parker, J.D., Carr, C.J., Zukowski, Z., Taylor, J. and Pons, J., 2024. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404.10301.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)