Efficient Diffusion Models: A Survey
Authors: Hui Shen
Jingxuan Zhang
Boning Xiong
Rui Hu
Shoufa Chen
Zhongwei Wan
Xin Wang
Yu Zhang
Zixuan Gong
Guangyin Bao
Chaofan Tao
Yongfeng Huang
Ye Yuan
Mi Zhang
Venue: Accepted by TMLR ()

Abstract: Diffusion models have emerged as powerful generative models capable of producing high-quality contents such as images, videos, and audio, demonstrating their potential to revolutionize digital content creation. However, these capabilities come at the cost of significant computational resources and lengthy generation time, underscoring the critical need to develop efficient techniques for practical deployment. In this survey, we provide a systematic and comprehensive review of research on efficient diffusion models. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient diffusion model topics from algorithm-level, system-level, and framework perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at github.com/AIoT-MLSys-Lab/Efficient-Diffusion-Model-Survey. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient diffusion model research and inspire them to contribute to this important and exciting field.

Published in Transactions on Machine Learning Research (05/2025)
Efficient Diffusion Models: A Survey
Hui Shen
shen. / 17800osu..ooo edu
Jingxuan Zhang
jz97@iu. edu
Boning Xiong
bnaiong24 fom.fudon..c. edu.cr
Rui Hu
marshallruigmaailcoomm com
Shoufa Chen
shoufuchend60@@maill.com
Zhongwei Wan
wan.51224ksn.edh
Xin Wang
weng. wong.1559000ss..ell
Yu Zhang
shangyu. wtw.gyyu.onsel@@mms...oo
Zixuan Gong"
gongass@onggji.duu. edw..
Guangyin Bao
haogyStonggii.dd edu.
Chaofan Tao
teftrees@gmaii. com
Yongfeng Huang?
1153187959 culk. edu.
Ye Yuan
yuonye -lt_pkk@@pppkk edu.co
Mi Zhang'
mizhang Bosu edu'
The Ohio State University
Indiana University
Fudan University Hangzhou City University
The University of Hong Kong
6 Tongji University
The Chinese University of Hong Kong PPeking University
Reviewed on OpenReview: ht tp // en ew n nt /f um d= EC Ow
Abstract
Diffusson models have emerged as powerful generative models capable of producing high- quality contents such as images, videos, audio, and text, demonstrating their potential to revolutionize digital content generation. However, these capabilities come at the cost of their significant resource demands and lengthy generation time, underscoring the need to develop efficient techniques for practical deployment. In this survey, provide system- atic and comprehensive review of research on efficient diffusion models. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet intercon- nected efficient diffusion model topics from algorithm-level, system-level, and framework perspoctive, respectively. We have also created a GitHub repository where we organize the papers featured this survey at hhttps:/ hhtps://giuhubbccom Model-Survey) We hope our survey can serve as valuable resource to help researchers and practitioners gain systematic understanding of efficient diffusion model research and inspire them to contribute to this important and exciting field.
"The marker denotes Co-first authors.

---

Published in Transactions on Machine Learning Research (05/2025)
1 Introduction
Diffusion models kickstart a new era in the field of artificial intelligence generative content (AIGC), garnering umprecodented attention (Yang et al., 2023b Croitoru et al. 2023). Espocially in the context of image synthesis tasks, diffusion models have demonstrated impressive and diverse generative capabilities. The powerful cross-modal capabilities of diffusion models have also further fueled the vigorous development of downstream tasks (Chen et al. 2023b). Despite the increasing maturity of diffusion model variants after mumerous iterations (Zhang et al. 2023d et al. 2023a), generating high-resolution complex natural scenes remains both time-consuming and computationally intensive, whether the initial pixel-level approach (Ho ct al. 2020) or the latent space variant (Rombach ct al. 2022). Therefore, order to optimize user-level deployment of diffusion models, researchers have never ceased their pursuit of elficient diffusion models.
Despite the growing popularity of diffussin models in recent years, one of the significant issues with diffusion model is that its multi-step denoising procedure requires numerous timesteps to reconstruct a high-quality sample from noise. This multi-step process is not only lime-consuning but also computationally intensive, resulting in a heavy workload. Therefore, improving the efficiency of diffusion models is crucial. In recent years, various studies have been presented to address this problem, such as controlling the noise added during training (Hang& & Gu
2024 Chen et al. 2023a) and selecting appropriate sampling timesteps [Watson al. 2024 among others.
2021
Sabour al.
While there numerous comprehensive surveys diffusion models (Yang et al. 2023b Chen et al. 2024
Croitoru et al. 2023 Cao et al. 2024) and those focused on specific fields and tasks (Ulhaq et al. 2022 Lin
et al. 2024c Kazerouni et al. 2023 Lin et al.) 2024b Peng et al.] 2024b Daras et al.) 2024), discussions
on the efficiency of diffusion models are notably scarce. The only existing survey addressing efficiency (Ma)
et al. 2024c) serves as an initial exploration in this area. In our work, we provide a more comprehensive and detailed taxonomy of efficient techniques, covering broader and more recent collection of research papers.
The overarching goal of this survey is to provide holistic view of the technological advances in efficient diffu- sion models from algorithm-level, system-level, and framework perspectives, as illustrated in Figure] 1 These four categories cover distinct yet interconnected research topics, collectively providing a systematic and comprchensive review of efficient diffusion models rescarch. Specifically,
Algorithm-Level Methods: Algorithm-level methods are critical for improving the computational efficiency and scalability of diffusion models, as their training and inference processes are often resource-intensvve In $3 we survey efficient techniques that cover research directions related to efficient training, efficient fine-tuning, efficient sampling, and model compression.
System-Level Methods: System-level methods aim optimize the infrastructure and compu- tational resources required for training and deploying diffusion models. In $ we survey efficient techniques that cover research directions related to optimized hardwarc-software co-design, parallel computing, and caching techniques.
Frameworks: The advent of diffussio models necessitates the development of specialized frame- works to efficiently handle their training, fine-tuning, inference, and serving. While mainstream AI frameworks such as TensorFlow and PyTorch provide the foundations, they lack built-in support for specific optimizations and features crucial for diffusion models. In $5 we survey existing frame- works specifically designed for efficient diffusion models, covering their unique features, underlying libraries, and specializations.
In addition to the survey, we have established a GitHub repository where we compile the papers featured in this survey at https://github..om//// wtwpa/aaoooooatAMSSSSSsSSaSSS Labfffiiiaa/ Mrifussion-Mode-SSrreeee We will actively
maintain it and incorporate new research it emerges.
2

---

Published in Transactions on Machine Learning Research (05/2025)
Lateut Diffusiou
Score Matching
Loss Formulation
Efficient Training ($3.1)
Rectified Flow
Data- Datz-Dependont Adaptive Priors
Training Tricks
Supervision
LoRA
Noise Schedule
Edficient Fine-tuning 3$3.2
Adapter
ControlNet
8116 Solver
Efficlent Solver
ODE Solver
Algorithm ($3
Parallel Sampling
Sampling Scheduling
Timestep Schedule
Efficient Sampling (.3
Early Exit
Truncated Sampling
Retrieval-Based Diffusion
Efficient Diffusion
Voctor Field Distilationn
Knowlodge Distillation
Generator Distillation
Post- Poet-Trainig Quantization
Quantization
Compression 39.4
Quantization-Awan Aware Training
Pruning
Surdware-Suftware Co-Design
System (64
Parallel Computing ($ 4.2
Caching Tochnique ($4.3
Frameworka 5
Flash Attention, xFocmers, DeepSpeed, Ouet'low, Staible-Fasi,, Oueditf, DeepCocche, TGATE, xDT
Figure 1: Taxonomy of efficient diffusion model literature.
2 Background and Applications
2.1 Basic formulas for Diffusion models
Diffusion models have emerged as powerful generative models capable of producing high-quality samples across various domains. This section explores the thcoretical foundations and recent advancements in the diffusion model framework. We begin by examining the fundamental formulation of Denoising Diffusion Probabilistic Models (DDPMs) (oo et al. 2020), which leverage forward and backward stochastic processes to gradually transform data into noise and then reverse this process for generation. We then explore Score Matching (Hyvärinen & Dayan (005) as an alternative formulation that directly optimizes gradient fields of probability densities. Continuing our discussion, we analyze how Stochastic Differential Equations (SDFs) and Ordinary Differential Equations (ODEs) (Song et al. 2020c) provide continuous perspectives on diffusion. Finally, we introduce Flow Matching (Lipman el al. 2022 as a recent paradigun that offers improved efficiency by directly learning vector fields that transform distributions, utilizing ODEs to provide deterministic process.
2.1.1 DDPM
To better understand the directions for improving eflicient diffusion models, is essential first to comprehend the fundamental framework of diffusion models. Denoising Diffusion Probabilistic Models (DDPMs) (Ho
et al. 2020) generate data through a process analogous to thermodynamic diffusion, consisting of two key components: forward process and a reverse process. These processes work in concert to enable high-quality generative modeling.
The forward process in DDPM a fixed Markov chain involving gradually adding Gaussian noise to the data until it becomes pure noise. q(xx) is denoted as the true data distribution, and assuming that Xg- g(xo) represents sampled data from this distribution. The forward process noted as q(x1: (xo), adds Gaussian
3

---

Published in Transactions on Machine Learning Research (05/2025)
noise step by step, transforming the data from Xo to Xy:
(x)) II (()((x=-1); ((xx|xe-1) == N (xx 1,
(1)
3, is defined the variance of the noise added each limestep. We then convert this to 0t, 1 Bc. Additionally, a₁ = IT-- ag is defined as the cumulative product of at, following the formulation by Sohl- Dickstein et al. Dickssei et al. 2015). This cumulative product allows for modeling the transition from the original data Xu to X₁ as a Gaussian distribution:
(xx)(x0) [xxx; va,xo, (1-a,)
(2)
This expression describes the distribution of x; given the initial data xo- It indicates that x, can be expressed linear combination of Xy and noise, where ~ NOO,, I) represents standard Gaussian noise:
x Va vI ayE
(3)
The reverse process, in contrast, aims to gradually denoise and reconstruct the original data by reversing the noise addition performed the forward process. This reverse process modcled as Markov chain where each step transitions from X, X using learned conditional probability distribution po(x (xx).. The overall process is expressed as:
pe (xx:T) == p(x))( II po(xx-1(x))) (x (1 t), 2e Ex(e (se,t)) t))
(4)
where p(xr) is the initial Gaussian distribution the final time step T, and px(xx-1(xx) represents the conditional probability distribution learned by the model transition between states. The mean (x(,),)) and covariance Eg(x,,) are parameterized functions of the state x,, the time time I, and the model parameters 0. In the training process, the optimization objective is to minimize the negative log-likelihood using the variational bound to approximate the true data distribution:
Po (xx:T)
Po (Xx 1xx) (x: x1- =: L
E| log pa (xo)] S E, log
= Ey
log p(XT)
Σ 121
log
(5)
( (xo)
This objective function decomposes the optimization problem into KL divergences for each timestep, pro- gressively optimizing the reverse process. Expanding the KL terms and using the conditional Gaussian form evaluates the difference between the forward and reverse processes, ultimately simplifying the process into a mean squared error form:
Lssmplle (0) = Rx,xo. [lle c0(Vaix + vI - ā,6, [)]
(6)
2.1.2 Score Matching
Score matching, introduced by Hyvärinen & Dayan 2005
serves effective approach for estimating unnormalized models by minimizing the Fisher divergence between the score function of data distribution
s2(x)= = Vx Pa(x) and the score function of model distribution sm (x; 0) = Vx log Pm (x; 0). This approach
avoids the need to compute the intractable partition function Zo, a common problem in Maximum Likelihood Estimation (MLE).
While DDPM directly optimizes the noise prediction in Eq. 0 score matching objectives can directly be estimated on dataset and optimized with stochastic gradient descent. The loss function for score matching takes different approach, formulated as follows:
L(Ø) = Epu(x) [x:sm():)) - sx(x)]1?)
(7)
2

---

Published in Transactions on Machine Learning Research (05/2025)
Since it typically does not have access to the true score function of the data sx(x), Hyvärinen & Dayan (2005) introduced integration by parts as I(0) J(0) + C' to derive an alternative expression that does not require direct access to Xx(x):
(( Baa(x) tr(V (x: 0)) 8m (x: 0)12
In this expression, tr(-) denotes the trace of the Hessian matrix of 8m (x: 0). The constant is independent of f and can be ignored for optimization purposes. The final form of the unbiasod estimator used for training is:
](0:x()) = N 1 Σ tr(V 8m (xVx)m(x;;))) 0))+ ||sm (x;: 0)1
(9)
2.1.3 Solvers
Given that the cost sampling escalates proportionally with the number of discretized time sleps, many researchers have concentrated on devising discretization schemes that reduce the mumber of time steps. A key insight emerges from recxamining the discrcte forward process in the original DDPM formulation Eq. as reduce the step size between consecutive steps, the process naturally approaches continuous transformation. Consequently, adopting learning-fit free methods using SDE or ODE solvers (Sng al. 2020c) has been proposed.
Song et al. (020cc) firstly presents a stochastic differential equation (SDE) that smoothly transforms a complex data distribution a known prior distribution by slowly injecting noise and a corre- sponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. The discrete noise addition steps in Eq. reformulated into continuous process:
SDE Solver.
SDE accomplishes the transformation from data to noise in the diffusion training process through the fol- lowing equation:
dx = f(x, t) df + g(t) dw
(10)
where w denotes the standard Wiener process, also known Brownian motion. f(x, is vector-valued function called the drift coefficient of x(t), and g(() is scalar function.
Similarly, the reverse process Eq. 4 can be generalized to a continuous-time formulation:
(1)3 dx [f(x,t) (((xx)))g(('Vvv1) log (x)) di + g()dd
(11)
w is a standard Wiener process when time flows backward from T to 0, dt is an infinitosimal negative timestep and Vx log ge (x) represent the score function that we mentioned in Eq.(1) In the diffusion process, reverse- time SDE converts noise into data gradually. The complete SDE process shown in Figure 2
ODE Solver. Unlike SDE solvers, the trajectories generated by ordinary differential equation (ODE) solvers are deterministic (Song et. al. 2020c), remaining unaffected by stochastic variations. Consequently, these
Forward SDE (data noise)
x()
f(x, t) + g(t)dw
x x(T)
score function
x(0)
dx [(x,t) g² (1) v, log pi (x) g())d
x(T)
Reverse SDE (noise - data)
Figure 2: Overview of forward SDE process and reverse SDE process (ong et al.
2020ch
5

---

Published in Transactions on Machine Learning Research (05/2025)
deterministic ODE solvers tend to achieve convergence more rapidly compared to their stochastic counter- parts, although this often comes at the expense of a marginal reduction in sample quality. The corresponding deterministic process Eq. (12( can be derived from the reverse-time SDE Eq. (11 by removing the stochastic
term (())d resulting in a deterministic process that shares the same marginal probability densitics as the reverse-SDE:
dx
2222222 [(xxt) gt) log (x)) dt
(12)
The lorward process also exhibits similar distinction between SDE and ODE approaches, yielding a deter- ministic process that preserves the same marginal distributions:
dx f(x, t)
(13)
2.1.4 Flow Matching
Flow Matching (FM) (Lipman et al. 2022) is a new paradigm for generative modeling based on Continuous Normalizing Flows (CNFs). This approach allows us train CNFs at an unprecedented scale and offers greater efficiency compared traditional diffusion models.
To better understand the process of flow matching, need to dive into the preliminaries. First of all, we make I denote the data space with data points
.rr E IRd. Then, we need to understand several key concepts: Probability density path p: : [0. 1] x Rd - Rso, which is a time-dependent probability density function, and Time-dependenl vector field [0, 1] x Rd - Rd, which describes how data points change over time. A vector field vy can be used to construct a time-dependent diffeomorphic map, defined by an ordinary differential equation (ODE):
d
de - = (x)(
(14)
(x(x) = x
(15)
A Continuous Normalizing Flow (CNF) is a generative model that parametrizes the time-dependent vector field vz(x; f)) using a neural network CCen et al., 2018), where 0 represents the learnable parameters. This vector field defines a flow O that transforms the probability distribution from a simple prior density Po to a more complex target density P1 through the push-forward equation. The core of CNF is using this flow reshape probability distributions:
Pt = [dc]-Po
(16)
The push-forward operator is defined as:
10051 Ox (x)
(x:) del
(17)
The FM loss is formally defined as:
(FM(()= E4-240,,11 z>p((t) [lve(x) - -u((=)))))) 12
(18)
where: (x))) gemerates predefined probability density path px()), Pa(œ) (=x 7) is simple prior distribution (e.g., Gaussian noise), and pi (z) x q(x) approximates the data distribution.
While FM is conceptually straightforward, directly optimizing this objective is intractable due to the lack of closed-form expressions for Pe and u, To address this, FM constructs conditional probability paths (x((a)) and vector fields uy (x)(x) per data sample x1 ~ q(x1), where Po(z[21) = p(x) at time = O, and pr (x(21) = N(x at /. (a normal distribution with æ1 mean and a sufficiently small standard deviation a > 0). These conditional paths are then aggregated into global counterparts through marginalization:
px(x) f pr (1(2211)9(11)))
(19)
6

---

Published in Transactions on Machine Learning Research (05/2025)
This approach avoids explicit modeling of the intractable marginal distributions and enables scalable training, while ensuring that at / 1, the marginal distribution pi approximates the data distribution g(x:).
The marginal vector field can then be defined as:
ux(x)= / (2(22211 Pe(x)(r1)g (((2)) dr (c(x)
(20)
A key theorem in the paper shows that this constructed marginal vector field u, in (2) indeed generates the marginal probability path ps in Eq. 19h.
However, computing the FM objective is still infeasible because it involves complex integrals. To address this issue, the paper proposes the Conditional Flow Matching (CFM) objective:
(CF Br.g(z)) a.) |1 (x) ( un (x((x1)11?
(21)
The second key theorem in the paper proves that the FM and CFM objectives have identical gradients with respect to 0, so optimizing CFM is equivalent to optimizing FM, but computationally simpler. Compared to score matching, flow matching is faster because score matching typically requires computing Hessian matrices in Eq while CFM cleverly avoids such complex calculations by only computing the L2 distance between vector fields. In conclusion, it's worth noting that Flow Matching can be theoretically connected with other diffusion model formulations. As summarized by Gao al. (2024). flow matching, score-basod models, and traditional diffusion models can be understood within a unilied mathematical framework. These approaches essentially represent different parameterizations of the same continuous time process. While traditional diffusion models like DDPM (Ho et al.] 2020) learn to estimate noise or score functions, and score-based models directly optimize gradient fields (Hyvärinen & Dayan 2005), flow matching learns the vector field
itself, directly capturing the transformation map between distributions.
2.2 Applications
Building on the foundational principles of diffusion models outlined in $2.1 this section surveys their practi- cal deployment across diverse generative tasks, with a specific emphasis on efficiency-driven innovations. As computational demands and real-world applicability become increasingly critical, researchers have adapted diffusion models to optimize resource usage while preserving or enhancing output quality. Here, we explore key application domains -image, video, text, audio, and 3D generation highlighting techniques that reduce inference time, memory footprint, and training complexity. These advancements underscore the adaptability of diffusion models and their growing impact in addressing the challenges of scalable, high-fidelity content generation.
2.2.1 Image Generation
Image generation is the primary application domain for efficient diffusion models. Researchers have been de- veloping various approaches to optimize both computational resources and generation quality. The efficiency improvements in this field are well exemplified by several influential works. For example, Stable Diffu- sion RRobach et al.) 2022) pioneered the concept of efficient image generation by operating in a compressed latent space rather than pixel space, significantly reducing memory and computational requirements while maintaining high-quality outputs. Latent Consistency Models (L.C)) (Luo et al. 2023a) further pushed the boundaries by enabling high-quality image generation in just steps through careful design of the consistency loss and distillation process. Progressive distillation (Salimans & Ho 2022 demonstrated that through a student-teacher framework, diffusion models could achieve comparable quality to 50-step sampling using only 2-8 inference steps. ControlNet (Zhang et al.) 2023d) introduced an efficient architecture for adding spatial conditioning controls to raeted dillusion models through pero-initialized convolutions, enabling diverse control capabilities without compromising model efficiency. More recently, Efficient Diffusion (EDM) (Karras) ctt al. 2022) presented a comprehensive framework for training and sampling diffusion models more efficiently, introducing improvements in both training stability and inference speed while maintaining state-of the-art generation quality.
7

---

Published in Transactions on Machine Learning Research (05/2025)
2.2.2 Video Generation
Following the rapid escalation in image generation, video generation similarly garnered widespread atten- tion (Melnik et al.) 2024 Ho et al. 2022b Xing et al. 2023). The heavy model size and the substantial computational costs have further intensified the focus on developing more efficient methods for video gener- ation (Zhang et al.; 2023b Liu al. 2023b Xing et al. 2024 Wanng et al., 2023 Lee et al., 2024b). For example, [Zhang et al. (2023b) introduced AdaDiff, lightweight framework designed to optimize specialized policy gradient method tailored to individual text prompts. This approach facilitates the design of reward functions and enables an effective tradc-off between inference time and generation quality. Specifically to the training process, Liu et al. (2023b) proposed an efficient. training framework ED-T2V to freeze the pre-
training model (Rombach et al. 2022 training additional temporal modules. Similarly, Xing et al. (2024) suggested using spatial and temporal adapters. In their approach, the original T2I model remains frozen during training, and only the newly added adapter modules are updated. Unlike the works above, Wang et al. (223)) presented VideoI.CM, incorporating consistency distillation in the latent space. Vi efficiently distills knowledge from pretraining model, maintaining fidelity and temporal coherence while improving inference speed. Lee et al. (2024b) introduces a grid diffusion model by representing video as a grid of images. It. employs key grid image generation and autoregressive grid interpolation to maintain temporal consistency. Moreover, Ceylan et al.
(023)) leverages self-attention feature injection and guided latent updates, efficiently repurposing image models for video editing, enabling high-quality, consistent edits at minimal computational overhead. YYi et al.] (2023) proposes NUMA-XLL -LL, using a hierarchical coarse-to- fine approach to enable parallelizable, exponential scaling of video length. It achieves great reduction in inference time and is trained directly on long sequences, ensuring long-term consistency.
2.2.3 Text Generation
Efficient diffusion models offer a fresh perspective in text generation through their stochastic and iterative processes. However, they encounter several challenges when applied to discrete data types such as lext. For instance, the common use of Gaussian noise is less effective for discrete corruption, and the objectives designed for continuous spaces become unstable in the text diffusion process, particularly at higher dimensions. With these challenges, Chen et al. (2023a) proposed a diffusion model called Baakedd-Diffuse LM. In the diffusson process, a cross-entropy loss function at each diffusson step is utilized to efficiently bridge the gap ween the continuous representations in the model and the discrete textual outputs. SeqDiffuSeq (Yuan et al, 2024) incorporates an encoder-decoder Transformer architecture, achieving efficient text generation through adaptive noise schedule and self-conditioning (Chen et al. 2022a) techniques. Using the same encoder- decoder architecture, Lovelace et al. (2024) presents a methodology where text is encoded into a continuous latent space. Subsequently, continuous diffusion models are employed for sampling within this space.
2.2.4 Audio Generation
In the field of audio generation, the application of diffusion models presents several unique challenges. First, audio data exhibits strong temporal continuity, especially in high-resolution audio generation tasks, where the model must accurately reconstruct both time-domain and Frequency-domain information. Compared to images or text, even subtle distortions or noise in audio are easily perceptible by humans, directly affecting the listening experience, particularly in speech and music generation tasks. Ensuring high fidelity and maintaining the consistency of details in the generated audio is therefore crucial. Moreover, many audio generation tasks require low-latency foedback, especially in applications like speech synthesis and real-time dialogue, which makes acceleration of diffusion models essential. The multi-dimensionll nature of audio data, such as time-domain, trequency-domain stereo, and spatial audio, further complicates the generation process, requiring the model to handle these dimensions while maintaining consistency during the accelerated generation. To address these challenges, rescarchers have proposed various methods to accelerate diffusion models in audio generation. Some works focus on reducing the number of diffusion steps to speed up the generation process, such as Chen et al. (2020) in WaveGrad and Kong et al. (2020) in DiffWave, which optimize the network structure to reduce generation time while maintaining high audio quality. Further optimization comes from the FastDPM framework (Kong & Ping 2021), which generalizes discrete diffusion steps to continuous ones, using bijective mapping between noise levels to accelerate sampling without
8

---

Published in Transactions on Machine Learning Research (05/2025)
compromising quality. FastDPM's flexibility allows it to adapt to different domains, and in the case of audio synthesis, where stochasticity plays crucial role, it demonstrates superior performance in high-stochasticity tasks like speech generation. Through these approaches, diffusion models not only accelerate the generation process but also reduce computational costs while ensuring that audio quality remains high, meeting the demands of real-time audio generation applications.
2.2.5 3D Generation
As a technique closely aligned with real-world representation, 3D generation holds substantial promise across various sectors, including medical imaging, motion capture, asset production, and scene reconstruction, etc. However, when compared to 2D image generation, distinctive high-resolution elements such volumetric data or point clouds present unique challenges, significantly escalating computational demands. Several
efficient methodologies [Bieder et al.] 2023 Zhou et al.) (Tang et al. 2023b Park et al. 2023 Du et al.
2024 Wu et al. 2024) have been proposed, particularly concentrating on enhancing the sampling process and optimizing the architectural framework, which further handles the computational complexity inherent. One of the most prevalent approaches involves designing more efficient sampling schedules (Bieder et al. 2023
Li al., 2024c Yu ct al. 2024b Zhou et al.p. By utilizing larger sampling step sizes, modifying the sampling strategy belween 2D and 3D, incorporating multi-view parallelism, these techniques address the key bottlenecks in the sampling process, thereby improving sampling efficiency. Moreover, the incorporation of novel architectures, such as state-space models and lightweight feature extractors (Mol 2024 Tang et al. 2023b alleviates the computational burden of processing 3D data, significantly enhancing model elficiency.
Table 1: Representative applications of diffussin models.
Task
Datasets
Metrics
Articles
Image
ImageNet, CIFAR,, MetFace, CelebA HQ, MS COCO, UCI, FFHQ, DiffusionDB, AFHQ, LSUN, SYSTEM-X, LAION
FID, sFID, 1S, NLL, MSE, CLIP Score, PSNR, LPIPS, MACs, CS, PickScore, SA, Score Matching Loss
Liu al.
2002b
Generation
Liu et al.
2023c
Yan et al.
[2024).
Lee ct al.
2024a))
Zhu et al. (2024). elc.
Video
MSR-VTT, InternVid, WebVid-10M, LAION, UCF-101, CGCaption, DAVIS, FlintstonesHD
FID, IS, FVD, IQS, NIQE, CLIPSIM, B-FVD-16, Pixel-MSE
Zhang et al. (2023b).
Generation
Liu et al. (002bb),
Xing et al. (0024).
Wang al.
(2023).
Lee et al.
(2024b
Ceylan et al.
(2023
Yin et al.
[2023]
etc.
Audio
SC09, LJSpeech,, Speech Commands
MOS, FID, IS, mIS, AM Score
Chen et al. [2020]
Generation
Kong et al.
2020
Kong & Ping
2021
etc.
Text
XSUM, Semantic Content, CCD, IWSLT14, WMT14, ROCStories, E2E, QQP, Wiki-Auto, Quasar-T,
Rouge, Semantic Acc, Mem, BLEU, Div, BERTScore, SacreBLEU, MAUVE Score, Content Fluency, POS
Chen et al.
(022a
Generation
Yuan et al.
(024).
Chen et al.
2022a
Lovelace et al. (2024
AG News Topic
elc.
3D Generation
BraTS2020, ShapeNet, Objaverse, Cap3D, LLFF,
Dice, HD95, CD, EMD, 1-NNA, COV, CLIP, Aesthetic, Similarity, R- R-Precision, FID, DIV, MM-Dist, ACC, Diversity, MModality
Bieder et al. (023))
Mo (2024),
HumanML3D, AMASS,
Li et al. (002cc),
KIT, HumanAct12. 12, IBRNet, Instriction-NoRF2NRRRR
Park et al. (0023
Yu 2024b)
etc.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Video LDM [Blttmmann ds el. 2025
MedieVidoo
Zhou ds ml. 2022 PVDM et et ml.
2921
LVDM (ffe or Al. 2012
Latent Diffussin
Latens 3D Graph Mimmooo
2021
Motoon
bosed Diffusion
hen
AudiLDDM
L3DG
Roessle
Tuxsy
2023a
PolceyDifusion
Hegil
HNAdifustoo
Elficient Training
Saeee Matching, Hyvárinon &e Dayan 2005
Slieel Somr Matching Sang ot
2723
Score Matching
NCSN Sooay Prinon 2019 Fnte hoot Weighting
Song et
2021
CLD
(Dockaorn of ah 2027
Loes Formulation
Rectified Flow
Rectifiel Flow [Lir PeltFlow
Yan vt a
ATM
2012
InstaFlow
[Lve at 2020
Liu
StriPlow Zhn et all
2024
Data-Dependent Adaptive Priors
PriutGrul
Laeget al.
Dirress Vignac et al.
2022
DesompDiiff
Guae
2023
Leapfrog DMI Maa et al 2123
Training Tricks
Extrr Loss Supervision
RFPA
Yu ml
2024a]
VA-VAR
Yan ot
2026
DDPM
Ho et ml.
2020
IDDPM
Nichol Dharieal 2001 Laplace Hang Go
2224
Noise Schedule
ResSlaft
Masked- minker
YLe et al.
Chen er.
lamas lol Difuusooe
2023e mara (Snin
[ d' al
237M
27006
VilluS8g (Gong
Seghtfff8en Yean
Figure Summary of efficient training techniques for diffusion models.
3
Algorithm-Level Efficiency Optimization
3.1 Efficient Training
Efficient training aims to optimize the training process of DMs, reducing computational costs while accel- erating convergence. As summarized in Figure g enhancing the efficiency of pro-training can be achieved through different and complementary techniques, including latent diffusion, loss formulation, and specialized training tricks. Latent diffusion models compress the optimization process by operating in lower-dimensional latent spaces, though they occasionally struggle with fine-grained detail reconstruction. Loss formulation methods enhance gradient estimation and stability, though sometimes facing numerical challenges. Mcan- while, various training tricks, including data-dependent adaptive priors, extra loss supervision, and optimized noise schedules, further enhance efficiency by leveraging problem-specific knowledge, though each introduces additional hyperparameters requiring careful tuning.
3.1.1 Latent Space
operate directly in pixel space, where images are generated through iterative noise addition and denoising. While these models achieve high-quality results, their reliance on pixel space introduces significant inefficiencies. Each diffusion step involves operations in a high-dimensional space, leading to substantial computational and memory overhead, especially for high- resolution images (c.g., 512x512). Additionally, the requirement for hundreds to thousands of itcrative steps hinders real-time applications. Although methods like DDIM (Song el al.] 2020aa have been proposed to speed up sampling, the pixel-level processing remains fundamental bottleneck. In contrast, leveraging latent space significantly enhances training efficiency by operating in a nower-dimensional, compact represen- tation, reducing both computational complexity and memory consumption while maintaining high-quality generation.
Early diffusion models, such as DDPM (Ho et al. 2020
As research progresses, researchers have begun exploring compromise approaches to circumvent the high- dimensional pixel space. Consequently, various methods for introducing the latent space have increasingly gained attention from the academic community. To reduce computational complexity, researchers have in- troduced autoencoders (AEs) and their variants compress images into a lower-dimensional latent space. Among them, Variational Antoencoders (VAEs Kinggma et al. (2013)) map images to Gaussian dis- tribution in the latent space via an encoder, while decoder reconstructs images from latent variables. Compared to traditional AEs, VAEs learn the data distribution rather than specific samples, mitigating overfitling. However, the generated images often appear blurry due to insufficient latent space constraints, as the KL regularization weight is relatively low. Vector Quantized Variational Autoencoders (VQ- VAEs (Van Den Oord et al. (2017)) enhance compression efficiency by introducing a discrete latent space through vector quantization. The integration of compression techniques with diffusion models has given rise to var- ious latent space-based diffusion models, with Latent Diffusion Models (LDMs) being the most prominent.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Benefiting from the efficiency advantages of the latent space and the cross-modal cross-attention mechanism of LDMs, a diverse range of image generation(eg, Stable Diffusion( [Rombach et al. (2022) and image editing models(eg, DiffEdit( Couairon et al. (2022))) has emerged.
Beyond traditional image synthesis and editing, various other diffusson model- tasks achieve computa- tional efficiency through the introduction of latent spaces, too. For videos with high-dimensionalltyy, complex temporal dynamics and large spatial variations, learning video distributions within a low-dimensional latent space has proven to be an effective method for efficiently generating high-quality vidcos. For instance, Video LDM Blattmann et al., 2023), MagicVideo (Zhou et al.) 2022), PVDM (Yu et al.) 2023) and LVDM (He) ct al. 2022) apply the latent diffusion model paradigm to video generation. Video LDM focuses on high- resolution video generation. Initially, the LDM is pretrained exclusively on images; subsequently, turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image seeeeseseueeesee sequences, i.e., videos. LVDM proposes hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. To further overcome the performance degradation issue for long video generation, LVDM introduces conditional latent perturbation and unconditional guidance. You et al. (You et al.) 2024) perform 3D graph diffusion in a low-dimensional latent space, which is learned through cascaded 2D-3D graph autocncoders for low-error reconstruction and symmetry-group invariance, resulting in training that is an order of magnitude faster. Motion Latent-based Diffusion (Chen et al. 2023c) is able to produce vivid motion sequences conforming to the given conditional inputs and substantially reduce the computational overhead in both the training and inference stages by performing diffusion process on the motion latent space. AudioLDM (Liu et all, 2023a), text-to-audio system built. on a latent. space that learns continuous audio representations from contrastive languago-audio pretraining latents, offers advantages in both generation quality and computa- tional efficiency. L3DG (Roessle et al.] 2024) utilizes a compressed latent space learned by a veclor-quantized variational autoencoder, coupled with a sparse convolutional architecture to efficiently operate on room-scale scenes. As a result, the complexity of the costly 3D Gaussians generation process through diffusion is signif- icantly reduced. TabSyn (Zhang et al., 2023a) attempt to utilize a diffusion model with a carefully crafted latent space in tabular data synthesis lasks, discovering that generation quality significantly improves while synthesis speed also increases. In reinforcement learning, condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. Hegde et al. (Hegde et al. 2023) propose using latent diffusion models to distill the archive into a single generative model over policy parameters, achieving a compression ratio of 13x. RNAdiffusion (Huang et al.] 2024) compresses token-level, biologically meaningful representations of RNA sequences into a set of fixed-length latent vectors and reconstructs RNA sequences from these latent variables. It utilizes a latent diffusion model to achieve controllable and efficiently translated RNA sequence generation.
3.1.2
Loss Formulation
In this section, we examine methods that enhance the efficiency of different loss formulations diffusion models. For score matching (Hyvärinen & Dayan 2005), we present approaches to reduce its computational costs. For rectified flow (Liu et al. 2022b), ; we explore how its carefully designed formulation enables straight- line sampling trajectories and subsequent improvements that further increase efficiency. We also include Flow Matching's Optimal Transport approach (Lipman et al. 2022) in the rectified flow discussion, as it similarly achieves direct trajectory learning through linear parameterization of probability paths.
Score Matching. Compared to DDPM's straightforward optimization in Eq. 6), although score matching Eq.99 avoids the computation of the partition function Ze, it still faces computational challenges, particu- larly in high-dimensional data. The computation of the trace of the Hessian matrix substantially increases the complexity as the dimensionality grows. Specifically, computing the trace requires many more backward passes than the gradient, making score matching computationally expensive for high-dimensional data.
Therefore, to address the computational inefficient issue of training process, Song et al. (200bb) observed that one-dimensional problems are typically much easier to solve than high-dimensional ones. Inspired by the idea of the Sliced Wasserstein Distance (Rabin et al. 2012), they proposed Sliced Score Matching. The core idea of sliced score matching to project both the score function of the model sm (x; 9) and the data sx(x) onto random direction v, and compare the differences along that direction. The sliced score matching
11

---

Published in Transactions on Machine Learning Research (05/2025)
objective is defined as:
L(0;Pv) 2 Br Bxxx [vv 8m (x: - {x(
(22)
To climinate the dependence on sx(x), integration is appliod by parts, similar to traditional score matching, resulting in the following form:
J(0:pv) = Ep v T 0jv- (v 8. (x: 0))²
(23)
which achieves scalability by reducing the complexity of the problem by projecting high-dimensional score functions onto low-dimensional random directions, thereby avoiding the full Hessian computation.
While effective for dimensionality reduction, score estimation still faces challenges in low data density regions
where data samples are sparse.
Building upon sliced score matching, to address the issue of inaccurate score estimation in low data density regions, Song & Ermon (2099) in-
step
troduces a novel generative framework that employs Langevin dynamics to produce samples basod on estimated gradients of the data, distribu- tion (x)ta(x). They proposed Noise Conditional Score Networks (NCSN) s)(x,o), which jointly estimate scores across multiple noise-perturbed data distributions. By conditioning
step
step
Plurskkeeut OCE Puth
geometric sequence of noise levels > 02 > 01, a single network learns to estimate scores for distributions ranging from highly smoothed Prş (x) that fill low-density regions to con- centrated Po, (x) that preserve the structure of the original data manifold. This unified training approach enables robust score estimation across the entire data space. Following similar derivation, as [Song et. al. (2021), Figure 4: Illustration of the rec- Dockhorn et al. (2021) introduces Coupled Langevin Dynamics (CLD), tified flow.
Rectitou Flow
step 1
T
redefining the score matching objective within the CLD framework. Unlike traditional score matching meth- ods that inject noise directly into the data space, CLD simplifies the task by only requiring the model to learn the score of the conditional distribution Pt (v, xt), where noise is injected into an auxiliary variable v, coupled with the data.
Rectified Flow. As illustrated in Figure Rectified Flow, proposed by (Liu et al.. 2022b Liu 2022), introduces a method for training ordinary differential equation (ODE) models by learning straight transport paths between two distributions, To and mi
2
Output
The key idea is to minimize the transport cost by ensuring that the learned trajectory between these two distributions follows the most direct routc, which can be computationall efficient to simulate. Unlike traditional dif- fusion models, which may follow roundabout paths, Rectified Flow lever- ages a simpler optimization problem to create straight flow, thereby improving both training efficiency and the quality of the generated sam- ples. Flow Matching (Lipman et al. 2022). as discussed in $2.1.4 in- troduces the Optimal Transport (OT) approach as a significant advance- ment in generative modeling. The key innovation of the OT method lies in how parameterizes conditional probability paths. It represents con- ditional probability paths in Eq using Gaussian distributions with means (xxx) toy and standard deviations (x) = 1 (1 mlin)t that change linearly with time. This linear parameterization produces vector field with constant direction, enabling transitions from noise to data along
Diffusion Process
Efficient Enhancement
Priors Initialization
u/ll| Audio
F Trajectory
Graph
Prior Datasets
Figure 5: Illustration of data- dependent adaptive priors for diffusion processes across dif- ferent modalities, demonstrating how tailored priors improve gen- eration on quality.
straight-line trajectories as well.
Building upon the foundation of Rectified Flow, InstaFlow (Liu et al. 2023cp applics the Rectified Flow concopt text-to-image generation, achieving a significant breakthrough. InstaFlow represents a major ad-
vancement. in efficient diffusion models, which are capable of high-quality image generation in just one step.
12

---

Published in Transactions on Machine Learning Research (05/2025)
It applied Rectified Flow to large-scale datasets and complex models like Stable Diffusion, introduced a novel text-conditioned pipeline for one-step image generation, and achieved an FID score of 23.3 on MS COCO 2017-5k.
InstaFlow's success highlights the potential of Rectified Flow in dramatically reducing the computational cost of diffusion models while maintaining high output quality.
Following InstaFlow, Yan et al. (2024) proposed PeRFlow, further extending the Rectified Flow concept to create more flexible and universally applicable acceleration method. PeRFlow divides the sampling process into multiple time windows, applying the reflow operation to each interval, creating piecewise linear flows that allow for more nuanced trajectory optimization. Through carefully designed parameterizations, PeRFlow models can inherit knowledge from pretrained diffusion models, achieving fast convergence and superior transfer ability. This approach positions PeRFlow as a universal plug-and-plan accelerator com- patible with various workflows basod on pretrained diffusion modcls. While Rectified Flow showed great promise, there was still room for improvement, especially in low Number of Funetion Evaluations (NFE) settings. Addressing this, Lee et al. (2024a) focused on enhancing the training process of Rectified Flows. They discovered that single iteration of the Reflow algorithm is often sufficient to learn nearly straight trajectories and introduced a U-shaped timestep distribution and LPIPS-Huber premetric to improve one- round training. These improvements led to significant enhancements in FID scores, particularly in low NFE, settings, outperforming state-of-the-an distillation methods on various datasets. Most recently, [hhu et al. (2024) proposed SlimFlow, a method designed to address the joint compression of inference steps and model size within the Rectified Flow framework, introducing Annealing Reflow address initialization mismatches between large teacher models and small student models, and developing Flow-Guided Distillation to improve performance on smaller student models.
3.1.3 Training Tricks
Training tricks encompass a range of practical strategics aimed at improving the efficiency, convergence, and sample quality of diffusion models by optimizing various aspects of the learning process. In this section, we explore three key approaches: data-dependent adaptive priors, which tailor initial distributions to specific data characteristics for faster convergence; extra loss supervision, which introduces additional loss terms to better align the model with data distributions and enhance generation quality; and noise schedule design, which governs the addition and removal of noise streamline the diffusion process and accelerate sampling.
To enhance the training efficiency of diffusion models and improve the quality of generated samples, data-dependent adaptive priors can be utilized to tailor the prior distribution to specific tasks and datasets. This approach leverages priors that better align with the data distribution, thereby accelerating the training process and ensuring that generated samples more closely match the truc data distribution. Recent studies have explored how dala-dependent adaptive priors can improve the training of diffusion models.
Data-Dependent Adaptive Priors.
As method under efficient training (sce Figure 5], data-dependent adaptive priors can be applied across various modalities, such as speech, graphs, and trajectories. By aligning the prior with the data distribution specific to each modality, the model can achieve faster convergence during training while producing outputs that better reflect the underlying data structure. In traditional diffusson models, the prior is typically assumed to be a standard Gaussian distribution p(z) = I). However, this assumption may not align well with the actual data distribution, potentially leading to inefficiencies in training. By constructing data- dependent adaptive priors based on the data X, the model can achieve better initialization during training, accelerating convergence without relying solely the standard Gaussian assumption.
Lee et al. 2021 introduced PriorGrad, which enhances the training of diffusion models for speech synthesis by using an adaptive prior derived from conditional data statistics. This method siguificantly improves the efficiency of the denoising process during training, leading to faster convergence while enhancing the perceptual quality and robustness of generated samples, even with smaller network capacities.
Vignac et al. (002)) proposed DiGress, a discrete denoising diffusion model for graph generation. By leverag- ing data-dependent priors, this model better captures the discrete nature of graph data, improving training
13

---

Published in Transactions on Machine Learning Research (05/2025)
Proposed leapfrog initializer
Diffusion step
Denaising step
Skipped denoising step
Mean estimation
Variance estimation
Sample prediction
O 1 O O
Traditional denoising steps
x
Ground-truth future trajectory
Random initialization
Predicoed
trajectories
Figure 6: The Leapfrog diffusiin model (Mao et al. 2023) accelerates inference by using a leapfrog initializer to approximate the denoised distribution, replacing extended denoising sequences while preserving represen- tation capacity.
efficiency and the quality of generated graphs, particularly for applications like chemical molecular structures and social networks.
In drug design, Guan et al. 2023 introduced DecompDiff, which improves the training of diffusion models by using decomposed priors to separately model different structural components of drug molecules. This approach enhances the model's ability to capture molecular structure information during training, Icading to the generation of more accurate drug candidates.
As shown in Figure 6 Mao et al. (0023) proposed the Leapfrog Diffusion Model for stochastic trajectory prodiction, introducing a leapfrog initializer based on adaptive priors to skip multiple denoising steps, accel- erating training while maintaining accuracy for real-time applications like autonomous driving. Building on this, Fu et al. (2025) developed MoFlow, one-step flow matching model with MLLEbased distillation for human trajectory forecasting. MoFlow employs novel flow matching loss to ensure accuracy and diversity in predicted trajectories, using data-dependent adaptive priors based on past trajectories and interactions to enhance alignment with the data distribution. Tts IMLE distillation achieves a 100x faster one-step student model with comparable performance. Similarly, Jiang et al. (2024) introduced SceneDiffuser, a scene-level diffusion model using amortized diffusion optimize efficiency for driving simulation, supporting both ini- tialization and rollout. These advances highlight the role of adaptive priors in boosting training efficiency for real-time multimodal applications.
Extra Loss Supervision. To further enhance the training elficiency and sample quality of diffusion models, incorporating additional loss supervision has proven to be an effective strategy. Beyond standard denoising objectives, extra loss terms can guide the model toward better alignment with data distributions, accelerate convergence, and improve the robustness of generated outputs. Recent advancements have demonstrated the power of tailoring loss functions to leverage external knowledge or address specific optimization challenges.
One notable approach introduces REPresentation Alignment (REPA) (Yu et al. 2024a regularization technique that aligns the hidden states of the denoising network with representations from pre-trained visual encoder, such as DINOv2. By adding loss term that enforces this aligmment, REPA ensures that the diffusion Transformer (e.g., DiT or SIT) leverages high-quality external visual priors rather than learning representations from scratch. This supervision accelerates training achieving over 17.5x speedup in some cases and boosts generation quality, reaching a state-of-the-art FID of 1.42 on ImageNet. 256x256 with classifier-free guidance. The additional loss acts as a bridge between the model's internal learning and established visual knowledge, making training both faster and more effective.
Similarly, another method tackles the optimization dilemma in latent diffusion models (LDMs) by introduc- ing extra loss supervision through the VA-VAE framework (Yao et al. 2025). This approach aligns the latent space of a variational autoencoder with a pre-trained vision foundation model, using a loss term to ensure the latent representations capture the data distribution more effectively. This supervision mitigates the trade-of off between reconstruction fidelity and generation quality, common challenge in LDMs where incrcasing latent dimensionality often hampers generative performance. Combined with an optimized LightningDiT architec- ture, this method achieves an FID of 1.35 on ImageNet 256x256, with training convergence accelerated by
14

---

Published in Transactions on Machine Learning Research (05/2025)
over 21x compared to baseline DiT models. The extra loss supervision in VA-VAB enhances the model's ability to balance reconstruction and generation tasks, leading to both efficiency gains and superior sample quality.
Noise Schedule. Noise schedule a crucial component of diffusion models, governing how noise is added during the forward process and removed during the reverse process. Optimizing the noise schedule can significantly enhance the training efficiency of diffusson models by enabling faster convergence and more effective learning of data distributions. Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020) introduced linear noise schedule that gradually decreases the variance of the noise added in the forward process, defined in Eq However, the linear schedule requires calculating complex noise terms across numerous timesteps, which can slow down the training process and highlight the need for more efficient noise schedule designs. As shown in Figure [ efficient noise schedules can be classified into two main categories: systematic noise addition and dynamic noise adjustment.
Fixed:
Linear
Cosine
Laplace
Adaptive:
Data
Loss
Aware
Aware
1.0
0.8
0.8
Loss Aware
0.6
0.6
Data Aware
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.8
0.2
0.8
Diffusion Step (t/T)
Diffusion Step (t/T)
(a) Fixed Noise Schedule
(b) Adaptive Noise Shcedule
Figure 7: Illustration of two categories of noise schedules.
One approach involves systematically adding noise during the training process at predefined intervals or according to specific levels. DDPM (Ho et al.) (020) employs a linear noise schedule, where the noise variance changes deterministically over time, serving as a foundational example, Building on this, the Improved Denoising Diffusion Probabilistic Model (IDDPM) (Nichol & Dhariwal 2021) introduces a cosine noise schedule, defined as
[/TT+s
COS
8,
(24)
cos
8_ I+s
where is the current timestep, is the total number of timesteps, and s a small positive constant for smoothing initial noise addition. The cosine noise schedule optimizes the noise distribution during training, enabling the model to learn data structures more effectively and achieve faster convergence. However, it allocates computational resources evenly across all noise intensities, which may not prioritize the most critical regions for training.
To address this inefficiency, Hang & Gu 2024 proposed the Laplace noise schedule, which enhances training efficiency by increasing the sampling frequency around critical noise regions. This approach ensures that the model focuses computational resourccs on modium noise intensitics, which are more effective for lcarning data structures and removing noise, leading to faster convergence and improved sample quality during training. The Laplace schedule balances noise addition across timesteps, resulting in more robust training process.
For text generation, traditional diffusion models often add uniform Gaussian noisc to cach word, which fails to leverage linguistic leat ures and increases computational burden during training. To address this, Chen et al. 20233) introduced a soft-man maseinn noise strategy that gradually adds noise based on word importance, mcasured by term frequency and information entropy. Using a square-root noise schodule (Li al. 2002). this method incrementally increases noise levels, stabilizing the training process and enabling the model learn linguistic structures more effectively from the initial latent variable Xo to noisy variables X1.T-
15

---

Published in Transactions on Machine Learning Research (05/2025)
In contrast, another set of methods dynamically adjusts the noise schedule based the model's state data during training- (Song et. al. (2020a) developed Denoising Diffusion Implicit Models (DDTM), which improve the noise schedule in DDPM by introducing a non non-Markovian forward process, defined as
-t-1 Va (I, v1 - (rt + v 111-a-1--0 - + ozEt.
(25)
where Og controls the noise addition over time, and or is dynamically adjusted based on the current state 24 and initial state Tp- This dynamic adjustment loverages the entire trajectory, enabling more procise noise control during training, which enhances the model's ability to learn data distributions efficiently and improves training convergence.
Inspired by DDIM, Yue et al. (2024) proposed ResShift, which introduces noise schodule that constructs shorter Markov chain by shilting residuals between high-resolution (HR) and low-resolution (LR) images instead of adding Gaussian noise. The noise schedule is defined as
TTx = v x by' == T
(26)
where
1 T 1 (T - 1), bo exp 11) log 1/1
3: with 7 as the total timesteps, / as the current timestep, controlling the growth rate of /T, and and yr as the initial and final noise levels. This non-uniform noise progression allows the model to focus training on key regions, improving convergence and learning efficiency for super-resolution tasks.
(27)
7/1
To further optimize training, Li al. (2024d) proposed Immiscible Diffusion, inspired by the physical phenomenon of immiscibility. Unlike traditional methods that diffuse cach image across the entire noise space, this approach reassigns noise to images within a mini-batch to minimize the distance between image- noise pairs. By matching each image with nearby noise, Immiscible Diffusion reduces the complexity of denoising during training, enabling the model to learn data structures more efficiently.
For lext generation, traditional methods (Gong et al. 2022) often use fixed noise schedules, requiring recalcula- tion the input sequence at each timestep, which leads to inefficient training. In contrast, Yuan et al. (2024) intro- duced dynamic noise mejustment technique that records loss values at each timestep during training and uses lin- car interpolation to map these losses to noise schodule parameters. This adaptive approach optimizes noise lev- els at each timestep and token position, improving train- ing efficiency and enabling the model to better capture linguistic features.
Fine-tuning Data
tune
fix
output
Figure Illustration of Hu al. (2021) 's repa- rameterization approach, where only parameters A and B are trained.
3.2 Efficient Fine- Tuning
Fine-tuning pre-trained diffusion models demands efficient resource use, leading distinct strategies: LoRA, Adapters, and ControlNet. LoRA stands out for its low-rank parameter updates, slashing memory needs by up to 90% and enabling rapid inference, though it struggles with nuanced spatial control. Adapters, by contrast, inject lightweight modules for task-specific tweaks, excelling in conditional guidance (c.g., sketches via Adi-eeapeer but relying on input quality, which can limit robustness. ControlNet, however, leverages additional network branches for precise spatial conditioning, offering unmatched control over structure and style, yet at the cost higher computational load mitigated variants like ControlNet-XS. These trade-offis are explored in detail in the subsections.
3.2.1 LoRA
Low-Rank Adaptation (LoRA) HHu al. 2021 trained model weights while enabling efficient task adaptation through the injection of low-rank decom- position matrices into each Transformer layer. The core mathematical foundation of this approach lies in
model adaptation method that maintains frozen pre-
16

---

Published in Transactions on Machine Learning Research (05/2025)
LoRA (Hu al.
2021
Concept Sliders (Gandikota
2023
LoRA
LCM-LoRA (Luo el al.
2023b
LoRA-- Conp-ser Yang cl al.
2024
LoRA conditioning CChoi et al.
2024b
T2I-Adapter (Mou ct al
2024
P-Adapeter YYe et al
2023
Efficient Fine-tuning
Adapter
ParaTAA (Tang et al.
CTRI-Adaptor
Lin et 2024a
SimDA (Xing et al. 2024
NoutrolNet Zhang et al.
2023d
ControlNet +
Li et al
2025
NontrolNet
Control Xs
Zavariski al.
2023
ControlNeXi Peng
2024a
Uni-ControlNet Zhao et
2024
UniControl Qin al.
2023
Figure 9: Summary of efficient fine-tuning techniques for diffusion models.
its representation of the weight update mechanism: for a pre-trained weight matrix Wo E Rdxk LoRA represents the weight update as:
W Wo + AW, where AW BA
(28)
where B E Rdxr and A E Rex are trainable low-rank matrices, and the rank < min(d, k). During forward propagation, for an input E R&, the model computes the hidden representation h E Rd as:
Wor + AW2 x Wor BAr
(29)
The complete process is illustrated in Figure g A key advantage of this design lies in its deployment effi- ciency, where the explicit computation and storage of W = Wo + BA enables standard inference procedures without introducing additional latency. Originally proposed for fine-tuning Large Language Models (LLMs), LoRA has demonstrated remarkable paramoter efficiency and memory reduction in model adaptation. While predominantly utilized LLM fine-tuning, recent. research has extended its application to diffusion mod- els, indicating its potential as a versatile adaptation technique across different deep learning architectures. LCM-LoRA (Luo al., 2023b) proposes a universal accelcration approach for diffusion models. As shown in
Figure 10 this method achieves fast sampling by adding an Acceleration vector TLOM to the Base LDM Rom- bach et al. 2022). This module adopts LoRA (Hu et al. 2021) to attach low-rank matrices to the original model without architectural modifications. For customized diffusion models that are fine-tuned for specific text-to-image generation tasks, the task-specific LoRA (7') and accelcaatin LoRA (FLCM) can be lincarly combined through (q.(30 to achieve fast inference while maintaining generation quality. More importantly, it provides plug-and-play solution that reduces sampling steps from dozens to around 4, while maintaining compatibility with any pre-trained text-to-image diffusion model.
T.M = A1T' + AATLOM
(30)
Beyond the acceleration achieved by LCM-LoRA, Concept Sliders (Gandikota et al. 2023 extends LoRA's 's application to precise control over image generation attributes. This method identifies low-rank directions in the diffusion parameter space corresponding to specific concepts through LoRA adaptation. The method freezes the original model parameters and trains LoRA adapter to learn concept editing directions. Given an input (z,,C,, C,, t), where 2, is the noisy image at timestep t. For a target concept c,, the model is guided through score function to enhance certain attributes c4 while suppressing others c.. This training objective can be formulated as: eg- (Ir, Gr + ea cr 1) + 17(oo(,, c4, c4,1) - c0(x, e),
(31)
Here, 6o represents the denoising model's prediction, and is a guidance scale. With this formulation, the method enables smooth control over concept strength through the guidance scale while maintaining concept independence in the learned directions. By leveraging LoRA's parameter-effiient nature, it achieves precise attribute manipulation with minimal computational overhead.
Besides, LoRA-Composer Yang et al. (2244) advances LoRA's 's application in diffusion models toward seam- less multi-concept integration. While previous works focus acceleration or single-concept control, this
17

---

Published in Transactions on Machine Learning Research (05/2025)
Style:
Fast:
X
Style
Fast:
Customized LDM
Customized LCM 0luM
Style (Style-LoRA)
Combination
Ppe
LoN 4, 2' AZTLCM
(LCMM-LoRA)
Base LDM
LCM BLCM
Accelerating vector
OLON
Style:
Fast:
Style: X
Fast:
Figure 10: Illustration of LCM-LORA Luo et al.
2023b
approach tackles the challenging task of combining multiple LoRA-castomized concepts within a single im- age generation process. It combines multiple LoRAs diffusson models by modifying the U-Net's 's attention blocks. Specifically, it enhances both cross-attention and self-attention layers within U-Net enable direct fusion of multiple LoRAs. Compared to traditional methods like Mix-of-Show Gu et al. (2024) that require training fusion matrix to merge multiple LoRAs, which increases computational overhead and may de- grade generation quality. It directly combines multiple lightweight LoRAs through modified attention blocks, avoiding the overhead of retraining models. While LoRA-Cone NoRA-Composer focuses on fusing multiple LoRAs for multi-concept. control, Choi et al. (2024b) explores the fundamental application of LoRA in attention layers. Both these works enhance diffusion models by modifying the attention mechanism in U-Net. The latter proposes a structured conditioning approach in U-Net blocks with three key components: (1) conventional convolutional blocks using scale-and-shift conditioning for feature normalization adjustment, (2) attention blocks enhanced by LoRA adapters that condition both QKV computation and projection layers through learnable low-rank matrices, and (3) two LoRA conditioning implementations- TimeLoRA/ ClassLoRA for discrete-time settings and UC-LoRA for continuous SNR settings, which utilize MLP- ILP--genraate weights to combine multiple LoRA bases. Them method achieves improved performance over tradit ional conditioning while only increasing the parameter count by approximately 10% through efficient low-rank adaptations in the attention layers.
3.2.2 Adapter
Adapters are lightweight modules designed enable efficient task adaptation by introducing small network layers into pre-trained models, allowing task-specific feature learning while keeping the original weights frozen. As illustrated in Figure adapter layers are placed within the transformer block, positioned between normalization and feed-forward layers. Each adapter module consists of a down-projection, nonlinearity, and up-projection, which generates lask-specific transformations without altering the core model's structure.
This design significantly reduces memory and computational requirements, making adapters well-suited for tasks requiring lightweight parameter updates, such as eext-to-mmage generation (T2I) and simulated domain adaptation (SimDA).
T21-Adapter (Mou et al. 2024) is an adapter designed enhance control in text-to-image generation models by introducing conditional features such as sketches, depth maps, and semantic segmentation maps, allowing for structural guidance in generated images. Unlike approaches that require modifying the model's core architecture, T2I-Adapter uses lightweight modules to incorporate external condition information into the generation process without altering the pre-trained model itself. This method improves the accuracy and consistency of generated images without increasing computational costs. In implementation, T2I-Adapter employs convolutional and residual blocks align conditional inputs with the spatial dimensions of interme- diate features in the UNet model, thus capturing structural information at multiple scales. This integration allows T2T-Adapter flexibly incorporate conditional features, such sketches and depth maps, providing
18

---

Published in Transactions on Machine Learning Research (05/2025)
enhanced control over text-to-image generation. Such multi-adapter strategies particularly suitable for tasks requiring high customization in image generation, enabling simultaneous input of various structural features to refine the output.
IPaAtererr Ye et al. 2023) enhances the consis- tency and visual quality text-to-image genera- tion by incorporating image prompts. Unlike T21- Adapter (Mou et al., 2024 which provides struc- tural guidance through sketches or depth maps, IP- Adapter focuses on capturing visual details from an input image, making it ideal for tasks requiring high visual consistency with a reference image. This adapter processes the input image prompt into la- tent features, allowing the generation model to cap- ture visual information from the targel image and maintain detail alignment throughout the genera- tion process. In its workflow, the image prompt is first mapped into the latent space and then pro- cessed through convolution and normalization mod- ules within the adapter, enabling the model to uti- lize these features during inference. This setup en- ables the generation model to draw rich visual infor- mation from the image prompt, making IP-Adapter particularly suitable for tasks requiring high detail consistency, such as generating images with a style similar to the input image. CTRI-Adapter (Lin et al.
Add Norm
Adapter
Layer
Adapter
Feed Forward up-project
Feed Forward
Adapter
Nonlinearity
Add Norm
Feed Forward
Multi-Head Attention
down-project
Frozen modules
Figure 11: Architecture of the Adapter module: demonstrating the integration of adapter layers within a transformer block to achieve efficient task adaptation by adding lightweight transformations, while kceping the core model weights frozen.
2024a) is designed to enhance attribute control during
generation by guiding specific attributes such as emotion or object type, enabling precise customization in generated results. Unlike T2T-Adapter (Mou et al. structural and detail consistency respectively, CdRt-Adapter tailored to provide diversity control for the generation model. For example, illustrated in Figure 12 the IP-Adapter architecture employs decoupled
2024h and IP-Adapter (Ye al., 2023), which focus on
Decoupled Cruss-Attention
Linear
Image Features
Image
Cross
Cross
Encoder
Attention
Attention
Frozen modules
Xt
Denoising U-Net
X1-1
Trainable modules
Text Features
Text Encoder
A girl with sunglasses
Figure 12: Architecture of TP-Adapter Ye al., 2023 using a decoupled cross-attention strategy, where only newly added modules are trainod, and the pro-trained text-to-image model remains frozen.
cross-atiention stralegy, where only newly added modules are trained while the pre-trained lext-to-image model remains frozen. In contrast, CTRL-Adapter can adjust the style of generated images based on specified emotions or object types, achicving controllable content generation without altering the core architecture of the model. This makes CTRL-Adapter particularly suitable for tasks requiring high customization in generation, such emotion-driven text generation stylized image synthesis.
19

---

Published in Transactions on Machine Learning Research (05/2025)
SimDA (Xing et al. 2024) is adapter suited for cross-domain generation tasks, achieving domain adapta- tion by utilizing simulated data within the adapter to enhance the model's performance on previously unseen data distributions. Unlike CTRL-Adapter (Lin et al.) 2024a), which primarily focuses on attribute control, SimDA is designed to improve the model's generalization ability, allowing it to generate high-quality content even unfamiliar data environments. SimDA is particularly useful in generation tasks that require domain transfer, such as adapting model trained on one image dataset to perform well on another dataset. This enables the model to align with new data characteristics without compromising generation quality.
zero convolution
neural network
trainable copy
block (locked)
scribble
canny
zero convolution
y
ControlNet
openpose
depth
Figure 13: Illustration of Nol.
3.2.3 ControlNet
ControlNet (Zhang et al. 2023d and its derivatives represent significant advancement in adding spa- tial conditioning cont to pre-trained text-lo-image diffusion models. The original ControlNet architec- ture (Zhang et al., 2023d as illustrated in Figure 13 presents a novel approach to integrating various spatial conditions such as scribbles, edge maps, open-pose skeletons, or depth maps into the generative process while preserving the robust features of pre-trained diffusion models. The architecture employs zero convolution layers that gradually develop parameters without disrupting the pre-trained model's stability. This design enables versatile conditioning, allowing the model to effectively leverage different types of spatial information. Through these conditioning methods, ControlNet demonstrates remarkable ability to guide generation with fine-grained control over structure, style, and composition. Building upon this foundation, several works have proposed improvements and alternatives. ControlNet++ (Li et al. 2025) addresses the challenge aligument. between generated images and conditional controls by introducing pixel-level ey- cle consistency optimization. Through a pre-trained discriminative reward model and an efficient. reward strategy involving single-step denoised images, it achieves significant improvements in control accuracy, with notable gains in metrics such as mloU (11.1%), SSIM (13.4%), and RMSE (7.6%) across various conditioning types. ControlNet-XS (Zavadski et al. 2023 reimagines the control system by enhancing the communication bandwidth between the controlling network and the generation process. This redesign not only improves im- age quality and control fidelity but also significantly reduces the parameter count, resulting in approximately twice the speed during both inference and training while maintaining competitive performance in pixel-level guidance tasks. The field has also seen efforts unify multiple control capabilities. UniControl (Qin et al. 2023) introduces task-aware HyperNet approach that enables a single model handle diverse visual condi- tions simultaneously. Similarly, Uni- ControlNet (Zhao et al. 2024 proposes a unified framework supporting both local controls and global controls through just two additional adapters, significantly reducing training costs and model size while maintaining high performance. Most recently, ControlNeXt. (Peng et al. 2024a) has pushed the boundaries of efficiency even further by introducing a streamlined archilecture that minimizes computational overhead. It replaces the traditional heavy additional branches with a more concise structure and introduces Cross Normalization (CN) as an alternative to zero convolutions. This approach achieves fast and stable training convergence while reducing learnable parameters by up to 90% compared to previous methods.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Diffiew
Ziaur & Chas
DiNGF
Zond al
222g
Jeliownr-Mor Mortinau et
9021
SDE Solver
GMS
42 2 mh.
MT Solmer2
Xne e
Xizih
Solvet Aile all 20c4h
Restart
Edficient Solver
ODR Solver
DDIM
Seg
272t
DPM-sdloz
DEES
Zlanng &
DMCCMC
Ram A 10 21122
ZZnng di
Efflclent Sampling
Parallel Sampling
DEQ Toble ct ol
2
0.22
ParaDiGMS (Sah
StresmDiifffemn pomiiia
2021
TAA
Tang al.
2024
Sampling Schedloling
Timestap Schoddulng
PuIDPM (Rony & Ping
2012
Wal mi al
2021
AYS
Selavar et,
2024
Early Exit
CAls Scluster al. 20211
ASE Bloom
2024b
Truncated Sampling
Ruidee Initisliaation
RDM Blettmaan di all 2022
DNN-Difiaion
Sheyula ct
Re-Iinages
Cluaa et al.
20222
RcDi
Zining (f
Vector Field Distillation
Tenoteing Studont Meng es al.
Tohman
2012
Progeesivv Distillation
Rollmane
2020
2023
SANA-spmi Cleen al.
2055 DMD
Knowhadge Distillation
Generator Dissillation
SUS CSD
Pode es al
2022
VSD
Wang ei al.
2014b
Dill Instruc
Lao
2026
Figure 14: Summary of efficient sampling techniques for diffusion models.
3.3 Efficient Sampling
The standard diffusion sampling process is computationally intensive because it requires soquentially exe- cuting a large number of denoising steps, with each step dependent on the output of the previous one as formulated in Fq(4]. This inherent sequential dependency makes the sampling procedure time-consuming, resulting in significantly slower generation. To address these computational challenges, researchers have developed efficient sampling methods through four principal approaches. As illustrated in Figure [4] these encompass efficient. SDE and ODE solvers that reduce the required number of function evaluations, advanced sampling scheduling strategies including parallel sampling techniques and timestep optimization methods, knowledge distillation techniques that transfer diffusion model capabilities to more efficient representations, and truncated sampling approaches that leverage early exit mechanisms and retrieval-base techniques to further accelerate the generation process while preserving output quality.
3.3.1 Efficient Solver
As we mentioned in Section. 2.1.3 although diffusion solvers avoid the need to compute the intractable partition function, they still require numerous function evaluations with fixed step sizes for SDEs in Eq [1 or generic numerical integration methods for ODEs in Eq. (2), resulting in slow sampling procedures and substantial computational costs. These limitations create substantial computational costs and lengthy gen- eration times. In the following sections, we present advances in both SDE and ODE methods that address these efficiency constraints.
SDE Solver.
Nowadays, there are many ways to efficiently implement SDE-based solvers. Zhang & Chen iinrrouus a novel generative modeling and density estimation algorithm called Diffusion Normalizing Flow (DiffFlow). Similar to the SDE of diffusion models Eq. (0). the DiffFlow model also has a forward
process:
dx = f(=;((t,, (
(32)
and a backward process:
dx = ([(x))1,)) - 1,0)] di g(()d
(33)
As result of the learnable parameter 0, the drift term f is also learnable in DiffFlow, compared to the fixed liner function as in most diffusion models. Besides, these SDEs are jointly trained by minimizing the KL divergence. This allows the model to better adapt to changes in the data distribution, thus speoding up the convergence of the backward diffusion process. Similar to DiffFlow, Zand et al. (2023) proposes a method called Diffusion with Normalizing Flow priors that also combines diffusion models with normalizing flows. The method first uses a linear SDE in the forward process to convert the data distribution into a noise distribution gradually. In the reverse process, a normalizing flow network is introduced to map the standard Gaussian distríbution to latent variables close to the data distribution through series of reversible
21

---

Published in Transactions on Machine Learning Research (05/2025)
transformations, which allows the samples to return to the data distribution more quickly, rather than relying on a large number of small incremental adjustments.
However, the fixed step sizes in existing SDE solvers Eq. 10 which usually require tremendous iterative steps, significantly affect generation efficiency. To address this, novel adaptive step-size SDE solver that dynamically adjusts the step size based on error tolerance, thherbyy reducing the number of evaluations. Specifically, the proposed method dynamically adjusts the step size by estimating the error between first-order and second-order approximations, leveraging a tolerance mechanism that incorporates both absolute and relative error thresholds. Furthermore, the use of extrapolation enhances precision without incurring additional computational overhead. This approach obviates the need for manual step-size tuning and is applicable across a range of diffusion processes, including Variance Exploding and Variance Preserving models. As a result of Gaussian assumption for reverse transition kernels becomes invalid when using limited sampling steps. The Gaussian Mixture Solver (GMS) (Guo et al., 2024) optimized SDE solver by using Gaussian mixture distribution. It addresses the limitations of the traditional process of SDE solvers in Eq. 11, which assume a Gaussian distribution for the reverse transition kernel. Specifically, GMS replaces the Gaussian assumption with more flexible Gaussian mixture mode and utilizes noise prediction network with multiple heads to estimate the higher-order moments of the reverse transition kernel. At cach sampling step, it employs the Generalized Method of Moments to optimize the parameters of the Gaussian mixture transition kernel, allowing for a more accurate approximation of the true reverse process, even with a limited number of discretization steps.
et al. (2021) proposes a
Instead, Xue et al. (2224a) unifies Bayesian Flow Networks (BFNs) with Diffusion Models (DMs) by intro- ducing time-dependent SDEs into the BFN framework. BFNs work by iteratively refining the parameters of distributions at different noise levels through Bayesian inference, rather than directly refining the samples as in traditional diffusion models. To achieve theoretical unification between BFNs and DMs, the authors introduce a time-dependent linear SDE that governs the noise addition process in BFNs. This forward pro- cess includes two timo-dependent functions: one controlling the drift of parameters and another controlling their diffusion. Based on this forward equation, they derive a corresponding reverse-time SDE for generating data from noise. This reverse process combines the drift term with a score-based correction term. This reverso-time SDE directly aligns with the denoising process in diffusion models, enabling the BFN sampling process to effectively replicate the behavior of diffusion models.
By optimizing the solving process of SDE in Eq. 10 Eq.(
Stochastic Adams Solver (SA-Solver) Xue et al. 2024b)
was presented. It is an innovative method designed to efficiently sample from Diffusion SDEs in Diffusion Probabilistic Models (DPMs) (Ho et al., 2020). By addressing the significant computational burden of traditional samplers, SA-Solver achieves this through a clever combination of variance-conttrlleed diiffusion SDE's and a stochastic Adams method (Buckwar & Winkler 2006), which is multi-step numerical technique that leverages prior evaluations to enhance efficiency. The method introduces noise control function T(t), enabling dynamic adjustment of the noise injected during sampling, which in turn strikes an optimal balance between sampling speed and the quality of the generated data. Operating within a predictor-correctt framework, SA-Solver first makes an initial estimate through the predictor step and then refines this estimate using the corrector step, ensuring greater accuracy in the final output. This strategic integration significantly reduces the number of function evaluations required. Xu et al. (2023b) introduces Restart, a novel sampling algorithm for diffusion models that strategically combines the strengths of SDE and ODE solvers. By theoretically analyzing Wasserstein bounds, the authors demonstrate that SDEs contract accumulated errors via stochasticity, while ODEs excel in low-NFE regimes due to smaller discretization errors. Restart bridges these regimes by alternating between noise injection and deterministic ODE backward steps, decoupling stochasticity from drift updates to amplify error contraction while retaining ODE efficiency.
ODE Solver. For efficiently implement the ODE-based solvers, Denoising Diffusion Implicit Models (DDIM) (Song et al. 2020a) builds upon the framework of Denoising Diffusion Probabilistic Models (DDPM) (Ho et al.) (020)), offering significant enhancements in sampling efficiency, which is one of the first models to leverage ODEs explicitly for the accelerating sampling process.
T
4a (x = qo (x7|Xo) II
go go (X-- [x1, Xo)
(34)
22

---

Published in Transactions on Machine Learning Research (05/2025)
Unlike DDPM's Markovian forward process Eq. I where each state only depends on its immediate predeces- sor, DDIM utilizes the Non-Markovian Forward Process Eq. (44..34.... These formulas allow each state not only to depend on its immediate predecessor but also on the imitial state or a series of previous states. Specifically, Eq. (35) outlines how DDIM generates X from X by predicting the denoised observation, which essentially approximales reversing the diffusion process:
X vau-1
X4
v1 040 (xx)
+ 11111-111 a:-1 oo cg (xx) Jf60
(35)
Va
During the process, DDIM employs latent space:
ODE solver to manage the continuous transformation across the
( x(t) dx(t) = =6/ Vo +1 do(()
((x(t)
(36)
Eq. 36) is key to the efficient management of the generation process, allowing for fewer steps in the generative sequence by smoothly interpolating between states using an ODE solver, thus significantly reducing the time complexity compared to traditional methods.
While DDIM's ODE formulation 12) and its implementation through Eq. 36 provide a foundation for 2022a identifies two critical issues in the ODE formulation of DDIM: first,
12)
deterministic sampling, Liu et al. the neural network and ODE are only well-defined within a narrow data manifold, while numerical hods generate samples outside this region. second, the ODE becomes unbounded as t + 0 for linear schedules. Therefore PNDM proposed to decompose the numerical solver into gradient and transfer components. It achieves second-order convergence, enabling 20x speedup while maintaining quality and reducing FID by 0.4 points at the same step count across different datasets and variance schedules.
The DPM-solver (Lu et al. 2022) and Diffusion Exponential Integrator Sampler (DEIS)
Zhang & Chen
2022) innovate by leveraging the semi-linear structure of the probability flow ODE Eq. 12 to design custom
ODE solvers that outperform traditional Rungo-Kutta (Hochbruck & Ostermann 2010 methods in terms of efficiency. Specifically, DPM-solver solves the linear part of the equation and uses neural networks to approximate the nonlinear component. Compared to PNDM, DPM-solver maintains lower FID scores at the same NFF. Further, DEIS employs Exponential Integrator (Hochbruck & Ostermann, 2010) to discretize ODEs. This method simplifies the probability flow ODE by transforming the probability ODE into simple non-stiff ODE. Both of the innovalions reduce the number of iterations needed producing high-quality samples within just 10 to 20 iterations.
To reduce the computational overhead, Zheng et al. (2023) presents an improved technique for maximum likelihood estimation of ODEs. Instead of directly working with the drift and score terms [ it introduces velocity parameterization to predict and optimize velocity changes dx, during the diffusion process directly. The method improves upon previous ODE-based approaches by incorporating second- order flow matching for more precise trajectory estimation. Additionally, it introduces negative log- signal-to-noise-ratio (log-SNR) for timing control of the diffusion process, alongside normalized velocity and importance sampling to reduce variance and optimize training. These enhancements significantly improve the model's likelihood estimation performance on image datasets without variational dequantization or data augmentation. While previous methods focus on improving reverse ODE integrators based on Eq. 12 Denoising MCMC (DMCMC) (Kim Yo, 2022) takes a different approach by integrating Markov Chain Monte Carlo (MCMC) with ODE integrators to optimize the data sampling process. In DMCMC, MCMC first generates initialization points in the product space of data and diffusion time, which are closer to noise-free state, significantly reducing the noise levels that need to be processed by the ODE integrators. This hybrid approach complements rather than improves the ODE integrators directly, enhancing overall sampling efficiency.
Besides, Lu & Song (0244) focuses on improving contimuous-time consistency modcls(CMs) (Song et al. 2023 Song Dhariwal, 2023) for efficient diffusion sampling by modifying the ODE parameterization and training objectives of continuous-time CMs. The core contribution is TrigFlow, unified framework that
23

---

Published in Transactions on Machine Learning Research (05/2025)
bridges EDM
[Karaa et al. 2022) and Flow Matching Pelluchttti
2023 Lipman et al. 2022 Liu et al,
2022b Albergo et al. 2023 Heitz et al. 2023).
While the traditional probability flow framework is governed by Eq. 12 they propose simplified param- eterization. To model these dynamics, they introduce a neural network Fe with parameters that takes normalized samples and time encodings as input. The time variable t is transformed by Croise (1) to better condition the network. This results in a concise probability flow ODE:
dx: oaFol od
Croise (t))
(37)
dt
By introducing this simplified ODE parameterization, TrigFlow enables training large-scale CMs (up to 1.5B parameters) that achieve state-of-theeart performance with just two sampling steps, significantly reducing computational costs compared to DPM-solver (Lu et al. 2022) and other traditional diffusion models.
Morcover; (Du et al. 2024) replaces SDE with ODE's deterministic trajectory, combined with fixed noise and Consistency Distillation Sampling loss, which minimizes stochastic fluctuations and computational re- dundancy. These designs enable faster convergence while maintaining high fidelity.
In conclusion, recent research has produced numerous works on faster diffusion samplers based on solving the ODE Eq. 12 Research shows that ODE samplers are highly effective when only a limited nunber of NFEs is available, while SDE samplers offer better resilience to prior mismatches (Nie et al., 2023 and exhibit superior performance with a greater Number Function Evaluations (NFEs) (Lu et al.
2022
3.3.2 Sampling Scheduling
In diffusion models, sampling schedule outlines structured approach for timing and managing the sampling steps to improve both the efficiency and quality of the model's output. It focuses on optimizing the sequence and timing of these steps, utilizing advanced techniques to process multiple steps simultancously or in an improved sequential order. Specifically, this scheduling primarily targets the optimization of the reverse process in DDPM, as described in Eq. 44, where each step requires model prediction to gradually denoise from pure noise to the target sample. This scheduling is crucial for reducing computational demands and enhancing the model's performance in generating high-quality samples.
Parallel Sampling. Parallel sampling is process that schodules sampling tasks in parallel. Traditional diffusion models require extensive series of sequential denoising steps to generate single sample, which can be quite slow. For instance, Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al. 2020) might need thousands of these steps to produce one sample. However, parallel sampling leverages the power of multi- core GPU to compute multiple sampling steps. This approach optimizes the use of computational resources and reduces the time needed for model generation. Currently, there significant. work on autoregressive models that employ parallelization to speed up the sampling process.
However, these techniques cannot be directly applied to diffusion models. This is because the computational frameworks and infer- ence efficiency in autoregressive models differ from those in diffusion models. Therefore, designing algorithms tailored to parallelize the sampling process of diffusion models is crucial. An innovative exten- sion of the Denoising Diffusion Implicit Model (DDTM) (Song et al. 2020a) using Deep Equilibrium (DEQ) models is presented (Pokle et al. 2022), where the sampling sequence is conceptualized as a mul- tivariate fixed-point system. This approach focuses finding the system's fixed point during the forward pass and utilizes implicit differentiation during the backward pass to enhance computational efficiency. By treating the sampling steps as an equilibrium system and solving for their fixed points simultancously, parallel processing on multiple GPUs is achieved by batching the workload. Notably, dependencies (Shih et al., 2024). it improves efficiency by updating each state x, based on predictions from the noise prediction network
Threod
Thread
Thread
Thread
z
x
Kip iteration
6-1
A++
Picard iteration
Figure 15: Computation graph of Pi- card iterations, which introduces skip
24

---

Published in Transactions on Machine Learning Research (05/2025)
ca, which takes into account all subsequent states X++1:7, unlike traditional diffusion processes that update states sequentially based only on the immediate next state X++1-
PirMD (Shih et al. 2024) employs Picard iterations to parallelize the sampling process in diffusion models. This method models the denoising process using ordinary differential equations (ODEs) (Song et al. 2020c), where Picard iterations approximate the solution these ODFs concurrently for multiple state updates. ParaDDGMSS operates within sliding window framework, enabling the simultaneous up- date of multiple state transitions. Each state is iteratively connected to different generations, allowing for information integration from several previous iterations Figure 15
Timestep schedule process
loss
0
Uniform Timestep
001
Chosen Timestep
Tn-2
Tn-3
Tn-4
Figure 16: Illustration of timestep schedule optimization process.
Building upon these parallel processing concepts, ParaTAA (Tang et al. 2024 also adopts an iterative approach, primarily applied in practical deployments for image generation tasks such as text-to-image trans- formations using Stable Diffussio. Specifically, ParaTAA enhances parallel sampling by solving triangular nonlinear equations through fixed-point iteration. Furthermore, the study introduces a novel variant of the Anderson Acceleration (Walker & Ni, 2011) technique, named Triangular Anderson Acceleration, designed to accelcrate computation speed and improve the stability of iterative processos. Kodaira et al. (2023) in- troduces StreamDilfusion for real-time interactive diffusion by combining batched denoising to exploit GPU parallelism and Residual Classifier Free Muidance(RCFC) to minimize redundant guidance computations, and input-output queues for asynchronous processing. It further optimizes energy efficiency via stochas- tie similarity filtering, which dynamically skips processing for near-identical frames using cosine similarity thresholds.
Timestep Schedule. In the sampling process of diffusion models, the entire process is discrete, and the model progressively restores data from noise through series of discrete timesteps. Each timestep represents small denoising step that moves the model from its current state closer to the real data. The timestep schedule refers to the strategy for selecting and arranging these timesteps. It may involve distributing them evenly or performing denser sampling during key stages to ensure the efficiency of the sampling process and the quality of the generated results. Selecting an appropriate method to choose a series of timesteps can enable the sampling process to converge quickly, which the process is shown in Figure
FastDPM [Kong& & Ping 2021 is unified framework for fast sampling in diffusion models that innovatively generalizes discrete diffusion steps to continuous ones and designs bijective mapping between these continu- ous diffusion steps and noise levels. By utilizing this mapping, FastDPM constructs an approximate diffusion and reverse process, significantly reducing the number of steps required (S & T). It allows for the flexible determination of sampling points by selecting specific steps or variances from the original diffusion process, thereby enhancing efficiency. Watson al. (2021) proposes dynamic programming algorithm optimize timestep scheduling Denoising Diffusion Probabilistic Models (DDPMs). The algorithm efficiently de- termines the optimal timestep schedule from thousands of possible steps by leveraging the decomposable
25

---

Published in Transactions on Machine Learning Research (05/2025)
property of Evidence Lower Bound (ELBO) across consecutive timesteps and treating timestep selection optimization problem. Experiments show that the optimized schedule requires only 32 timesteps to achieve comparable performance to the original model with thousands of steps, effectively balancing effi- ciency and quality. However, optimizing an exact Evidence Lower Bound (ELBO) typically not conducive
Reverse-time iterative generative denoising process
n
bn-4
n
arg min KLUB(to, (LU .nn.,,)
Figure 17: Minimizing an upper bound on the Kullback-Leibler divergence (KLUB) between the true and linearized generative SDEs to find optimal DM sampling schedules (Sabour et al. 2024
to enhancing image quality. To address this, Align Your Steps (AYS) (Sabour et al. 2024 proposes a compute--fficient paradigm for diffusion models by co-optimizing sampling schedules and solvers. Through stochastic calculus-guided Kullback-Leibler Divergence Upper Bound (KLUB) minimization, shown in 17. AYS derives adaptive schedules that reduce sampling steps by 40% while maintaining caality--chhiving equivalent FID scores with 20 steps versus 30 steps in baseline methods. The optimized schedules are versatile across modalities and solver types, enabling plug-and-play acceleration without model retraining.
3.3.3 Truncated Sampling
Truncated sampling enhances the efficiency of sample generation in diffusion models by strategically reducing redundant computations, thereby lowering computational costs. This optimization category encompasses several approaches, with Early Exit and Rotrioval-Guided Initialization representing two primary strategies that larget different phases of the diffusion process to improve computational efficiency. Specifically, early Exit focuses on terminating unnecessary computations in later stages of the diffusson process when predictions are confident. Conversely, Retrieval-Guided Initialization improves efficiency in the early stages by leveraging retrieved examples to provide better initialization, effectively bypassing parts of the iterative refinement process. These approaches allocate computat ion more effectively by focusing resources on the most critical steps of the sampling process.
Early Exit. Recent papers that focus on early exit mechanisms have gained significant attention in the Large Language Models (LLMs) domain (Schuster et al. 2021). By implementing early exit strategies, these methods bypass unnocessary computations in deeper layers when outputs can be generated earlier stages, thereby substantially redueing inference time and compulational resources.
Similarly, for computation-inteenn exit techniques present promising approach to accelerate the generation process dramatically. [Monn et al. (0244) proposes a simple yet ellective early exiting framework called Adaptive Score Estimation (ASE) specifically designed to accelerate the sampling process of diffusion models. The method based on the key obsorvation that score estimation difficulty varies across different timesteps the diffusion process, with timesteps closer to the noise distribution requiring fewer computational resources than those closer to the data distribution.
diffusion models that typically require numerous denoising steps, early
26

---

Published in Transactions on Machine Learning Research (05/2025)
The schematic in Figure 18 illustrates the time-dependent exit schedule used in ASE. As shown in the figure, the model progressively skips more building blocks as the diffusion process moves closer to the noise distribution, where the score estimation becomes easier. For example, in the DiT model (Peebles & Xic 2023 the blocks are dropped progressively as the timestep approaches 1, closer to the noise regime. Conversely, when the timestep is closer the data regime, more blocks are relained to ensure accurale score ostimation. Moreover, the U-ViT model (Bao et al.) 2023) follows a similar dropping schodule, but due the long skip connections between the encoder and decoder, only decoder blocks are skipped. This preserves critical information from the encoder while still speeding up the process.
Through carcfully designed time-varying exit schedules, the method siguificantly accelerates sampling speed while preserv- ing generation quality. It can also be scamlessly integrated with various diffu- sion model solvers to enhance overall ef- ficiency further.
2~ ((0
Block
Decoder
Black
Block
Decoder
Decoder
Block 3
Black 3
In contrast, Tang et al. (0233)) intro- duces DeeDiff, which approaches early exiting through a dynamic uncertainty- aware framework. While ASE relies on static, predefined exit schedules that are fixed during inference, DeeDiff incorpo- rates a tinesteep-aware uncertainty esti- mation module (UEM) that adaptively assesses the prediction uncertainty of each intermediate layer at every sampling slep. The UEM directly estimates uncer- tainty values from the features each layer using lightweight, colly-connected networks, which are trained to indicate how well the current layer's output matches the final layer's prediction. During inference, when layer's uncertainty falls below predefined threshold, the model exits carly that layer, bypassing deeper layers for that particular timestep. Despite their different implementation strategies, both methods demon- strate that early exiting frameworks can effectively balance computational efficiency and generation quality, offering practical solutions for deploying large-scale diffusion models.
Decoder
Decader
Decoder
T
du = [f(z,t) (t log pz dt + g()
score Function
Figure 18: Illustration of the time-dependent exit schedule, where more blocks are skipped the sampling process moves toward the noise distribution, optimizing computational efficiency Moon et al. (2024a).
Retrieval-Guided Denosing. Retrieval-Guided Initialization combines the efficiency of retrieval mech- anisms with the generative power of diffussio models, and can be applied across various generative tasks. This method guides the generation process by retrieving samples data relevant to the input. For example, illustrated in Figure 19 when the input is text prompt, the retriever selects relevant images from database, which then serve as contextual guidance for the diffusion model to generate coherent output. By providing a more informed state for the diffussin model, this approach helps the model converge more quickly and generate nigher-quality outputs.
Retrieval-Guider Denoising widely usod in text-to-image diffusion tasks, offering an efficient way to gener- ale images that align with textual descriptions. The kNN-Diffusion (Sheynin al. 2022) method integrates large-scale retrieval techniques with the generative power of diffusion models, offering efficient. approach for image generation. The key idea is to use the CLIP (Radford et al. (021) image encoder to map images into a shared embedding space and employ the ".Nearest-Neighbor similar image embeddings. Notably, this method does not require paired text-image datasets, which a sig- nificant advantage for domains where such datasets are scarce. During training, the model conditioned on image embeddings,, and kNN retrieval is used to extend the distribution of conditioning embeddings, which helps the model generalize better and bridge the gap between text and image distributions. In the sampling phase, the model receives text input, which is converted to text embedding using CLIP. The kNN retrieval mechanism is then applied to find the k most similar image embeddings based on the text embedding. These retrieved image embeddings serve as additional conditional information, guiding the generation process and ensuring that the generated image aligns with the textual description. Consequently, kNN-Diffusion im-
(kNN) retrieval to identify the k most
27

---

Published in Transactions on Machine Learning Research (05/2025)
proves both the efficiency of image generation and the computational resource consumption, making it highly efficient solution for text-to-image generation tasks. Unlike KNN-Diffusion, which primarily relies image retrieval to guide the sampling process, Re-Imagen (Chen et al. 2022b) enhances the generation process by incorporating multimodal retricval to balance computational cost and output quality, using both image and lext pairs. Re-Imagen's 's retrieval method is based on an external multimodal knowledge base. During sampling, given an input text, the model queries an external database to retrieve the most relevant image-tett pairs. These retrieved pairs are then used as additional conditional imputs for the generation process. To identify the most relevant pairs, the model employs either BM25 (Robertson et al. 2009) CLIP (Radford et al. (021) similarity measures, which evaluate the alignment between the input text and the image-text pairs in the database. Therefore, this approach provides more diverse visual information, especially for rare or unseen entities. Additionally, Re-Imagen employs a cascaded diffusiio architect (Ho jet al., 2022a), which allows to generate high-quality images more efficiently compared to kNN- LNN-Diffusion, as reduces the computational cost by progressively refining images at different resolutions.
Query: Labrador
00
Generator (Diffusion Model)
sitting on bench near river.
Retriever
Labrador
Database
River
Figure 19: Mustratiio of the retrieval-based diffusion model. The retriever selects relevant images from a database based input text. These retrieved images provide contextual guidance for the generator (diffusion model) to produce new, coherent output image.
Besides approaches that rely similarity measures like CLIP embeddings to retrieve images or texts, ReDi (Retrieval-based Diffusion) (Zhang et al. 2023ch offers a fundamentally different approach to accelerating diffusion model inference. ReDi directly retrieves precomputed trajectories from the diffusion process itself. The method constructs knowledge base of sample trajectories where each entry contains key-value pair, For instance, in the forward diffusion process (noise-adding process), an early state sample z serves as the key, while later state sample Ze serves the value. During inference, ReDi first generates the initial steps of trajectory up to d'g, uses this as query to find similar state in the knowledge base, and then skips intermediate computational steps by jumping to the retrieved 2, before continuing the generation process. By skipping large portion of intermediate steps, ReDi significantly reduces the number of function estimations (NFEs) required during sampling. Experiments with Stable Diffusion demonstrate that ReDi achieves a two times speedup with comparable quality and enables effective zero-shot domain adaptation for tasks like image stylization without requiring domain-specific knowledge bases.
3.3.4 Knowledge Distillation
Knowledge distillation (Hinton et al. 2015 is technique that compresses complex models into smaller, efficient versions with minimal performance loss. The process of knowledge distillation can be captured by minimizing the following loss function:
LKD = alce((, + BLMsE (T)))
(38)
28

---

Published in Transactions on Machine Learning Research (05/2025)
XT-1
XT
XT-1
X1 XD
Vector Field Distiliation
Generator Dietillation
Grad(e)
x))
XT
XD
Figure 20: Illustrations the knowledge distillation.
where Tr and T, are the leacher and student models, respectively, is the softmax function, LeF is the cross-entropy loss, and LMSE is the mean squared error loss, with and / balancing hyperparameters. In DMs, known for generating high-quality data, this approach is increasingly applied to improve efficiency by addressing slow sampling speeds caused by the numerous neural function evaluations in the diffuusin process. By distilling the knowledge from DMs into more efficient forms, researchers aim to accelerate sampling while preserving the generative performance of the original models. Follow Luo 20233 knowledge distillation for DMs can be categorized into vector field distillation and generator distillation.
Vector Field Distillation. Vector field distillation improves the efficiency of deterministic sampling in dif- fusion models by transforming the generative ODE into new generative vector field. This approach reduces the number of NFEs needed to produce samples of similar quality. Luhman & Luhman (2021) first proposes a strategy to distill DDIM sampler into a Gaussian model that needs only one NFE for sampling. In this ap- proach, a conditional Gaussian model serves as the student model, and the training process involves minimiz- ing the conditional KL divergence between this student model and the DDIM sampler. While this method ad- vances the application of knowledge distillation to diffusion models, still has computational inefficiencies, it necessitates generating the final outputs of DDIM other ODE samplers, which entails hundreds of NFEs for each training batch. Salimans & Ho (022 proposes progrossive distillation strategy to train student model to use half the NFEs of the teacher model by learning its two-step prediction strategy, as illustrated in Figure 21 Once the student model accurately predicts the teacher's two-step sampling strategy, it replaces the teacher model, and new student model trained to further reduce the sampling steps. This mothod reduces the NFEs significantly, achieving 250 times greater efficiency with only a 5% drop in performance. A two-stage distillation strategy pro- posed by Meng al. (2023) to address the challenge of transferring knowledge from classifier-free guided conditional dif- fusion models like DALL E-2 (Ramesh et al. 2022) and Stable Diffusion (Rom- bach et al. 2022). In the first stage, a stu- dent model is trained with classifier-free guidance to learn from the teacher dif- fusion model. The second stage employs the progressive diffusion strategy to fur- ther reduce the number of diffusion steps for the student model. This two-stage ap- proach is applied to both pixel-space and latent-space models for various tasks, in- cluding text-guided generation and image inpainting.
23/4 f(mLi))
1122 f(Paaai))
Disiilation
x f(z;80)
2144 ((aa))))
Distilation
t=
Figure 21: The progressive distillation, where the original sam- pler derived from integrating learned diffusion model's proba- bility flow ODE, efficiently condensed into new sampler that achieves the same task in fewer steps. (Salimans & Ho 2022
29

---

Published in Transactions on Machine Learning Research (05/2025)
Song al 2023) firtstly introduces the Consistency Model (CM), which leverages the self-consistency prop- erty of generative ODEs in diffusion models. Instead of directly mimicking the output of the generative ODE, their method focuses on minimizing the difference in the self-consistency function. By randomly diffuusing a real data sample and simulating a few steps the gencrative ODE to generate another noisy sample on the same ODE path, the model inputs these two noisy samples into a student model. Consequently, Chen el al. (2025) accelerates consistency models via hybrid distillation, enabling 1-4 step generation. It transforms pre-trained flow models into TrigFlow (Lu & Song) 2024) without retraining, preserving trajectory align- ment while boosting fidelity via adversarial training LADD. Wu et al. (0024] proposes Multi-scale Latent
Ancestral Sampling
Score Distillation Sampling = g
((ts)
2T
Z(-1
Updates sample in pixel space: 2-1 ddpm_ update(z;)
Updates parameters with SGD: 0,41 opt.step(0;; Vol(x;))
Figure 22: Illustration as
Poole et al. 2022 it utilizes score distillation sampling.
Point Consistency Model (MLPCM), which implements one-step generation through consistency distillation, combines multi-scale latent space and 3D attention mechanisms to reduce computational complexity. More- over, Yin al. (2024) proposes Distribution Matching Distillation (DMD), which distills multi-stop diffusion models into single-step generator by rntroiigg distribution-level matching objective that minimizes the KL divergence between real and synthetic data distributions, alongside regression loss to align large-scale structural features.
Generator Distillation. Unlike vector field distillation, which primarily focuses on distilling knowledge into student models with identical input and output dimensions, generator distillation aims to transfer the complex distributional knowledge embedded in a diffusion model into a more efficient generator. The Neural Radiance Field (NeRF) (Mildenhall et al.] 2021) a powerful technique for reconstructing 3D scenes from 2D images by learning continuous volumetric scene representation. NeRFs generate photorealistic views of scenes from novel angles, making them valuable for applications in computer vision and graphics.
However, the limited availability of data for constructing NeRFs is an issue. Therefore, exploring distillation methods to obtain NeRF's with contents related to given text prompts is promising way. (Poole et al.) 2022) first proposed Score Distillation Sampling (SDS) to distill a 2D text-to-image diffusion model into 3D NeRFs, as illustrated Figure 22 Unlike traditional NeRF
that requires images from multiple views of the target 3D objects, text-driven construction of NeRF lacks both the 3D object and the multiple views. The SDS method optimizes the NeRF by minimizing the diffusion model's loss function using NeRF-rendered images from fixed view.
Wang et al. 2024b) introduce Variational Score Distillation (VSD), which extends SDS by treating the 3D scene corresponding to a lextual prompt as a distribution rather than single point. Compared to SDS, which generates a single 3D scene and often suffers from limited diversity and fidelity, VSD is capable of generating more variod and realistic 3D scenes, even with single particle. Luo et al. (2024) propose Dift-Instruct, which can transfer knowledge from pre-trained diffusion models wide range of generative models, all without requiring additional data. The key innovation in Diff-Instruee is the introduction of Integral Kullback-Leibh Leibler divergence, which is specifically designed to handle the diffusion process and offers more robust way to compare distributions. Decatur et al. (2024) present Cascaded Score Distillation (CSD), an advancement by addressing a key limitation of standard SDS. Specifically, while traditional SDS only leverages the initial low-resolution stage of cascaded model, CSD distills scores across multiple resolutions in a cascaded manner, allowing for nuanced control over both fine details and the global structure of the supervision. By formulating a distillation loss that integrates all cascaded stages, which are trained independently, CSD enhances the overall capability of generating high-quality 3D representations.
30

---

Published in Transactions on Machine Learning Research (05/2025)
Post-Trainnig Quantisation
PTQUDM Stang et al
2023
Q-Diltsuion Lietal 2023
BREQQ
Li al. 2021
Compression
(Kim
20240
Quantization
Quantization-Awee Training
TDQ
So
2024.
QALoRA He
2023
Pruning
Dill-Pruiing
Fang
2723
JD-Prumer
Casttell al.
2024
LayerMenge
Kim
20246 LAPTOPDiI Zhang et al. 20244s
Figure 23: Summary of compression techniques for DMs.
3.4 Compression
Model compression enhances efficiency by reducing the sizes and the amount arithmetic operations of DM. As summarized in Figure 23 model compression techniques for DMs can be grouped into quantization and pruning. These two categories orthogonal to each other, and compress DMs from different perspectives.
High Precision Woight with Different Values
High Precision Activation with Different Values
Zero Weight / Activation Value
Low-Precision Weight with Different Values
Low-Precision Activation with Different Values
Calibration Data
Training Data
updated
updated
PTQ
QAT
updated
(a) Post Training Quantization
(b) Quantization Aware Training
Figure 24: Illustrations of the quantization.
3.4.1 Quantization
Quantization compresses neural networks by converting model weights and/or activations of nigh-precision data types X such as 32-bit floating point into low-precision data types X such as 8-bit integer (Dettmers ct al. 2024). Quantization techniques can be classified into post-training quantization (PTQ) and quantization-aware training (QAT).
Post-Training Quantization. PTQ involves selecting operations for quantization, collecting calibration samples, and determining quantization parameters for weights and activations. While collecting calibration samples is straightforward for CNNs and ViTs using real training data, it poses a challenge for Diffusion
Models (DMs). In DMs, the inputs are generated samples x, at various time steps (t 0, 1, ..., T), where T is large to ensure convergence to isotropic Normal distribution. To address this issue, Shang al. (2023) proposes PTQ4DM, the first DM-specific calibration set collection method, generating calibration data across all time steps with a specific distribution. However, their explorations remain confined to lower resolutions and 8-bit precision. Q-Diffsuion (Li et al. 2023) propose a time step-aware calibration data sampling to improve calibration quality and apply BRECQ (Li et al. 2021], which a commonly utilized PTQ framework, improve performance. Furthermore, compared conventional PTQ calibration methods, they identify the accumulation of quantization error across time steps as another challenge in quantixing DMs Figure 25
31

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Calibration dataset creation
(b) Model Inference
Traditional PTQ
Training Dataset
Sample from Training Data Synthesizad by Random Sampling
Input
Output
Calibration Dataset
Quanlization emrors accumulate each time step
Q-Diffusion (Ours)
Pylk
%,.1
X,.
Time Step-aware Uniform Sampling
Calibration Dataset
Naive Linear Quantization incurs severe quality loss
Figure 25: Traditional PTQ scenarios and Q-Diffusion differ in (a) the creation of calibration datasets and (b) the workflow for model inference (Li et al. 2023).
(a). Therefore, they also propose a specialized quantizer for the noise estimation network shown in Figure 25
(b). Based on Q-Difiusion, Kim et al. (2024b) find that inaccurate computation during the early stage of the reverse diffusion process has minimal impact on the quality generated images. Therefore, they introduce a method that focuses on further reducing the number of activation bits for the early reverse diffusion process while maintaining high-bit activations for the later stages. Lastly, He et al. (2024) presents PTQD, unified formulation for quantization noise and diffusion perturbed noise. Additionally, they introduce step-aware mixed precision scheme, which dynamically selects the appropriate bitwidths for synonymous steps.
Quantization-Awan Training. Different from PTQ. QAT quantizes diffusion models during the training process, allowing models to Iearn quantization-fricndy representations. Since QAT re- quires additional training after introducing quantization operators, it much more expensive and time-consuming than PTQ. So et al. (2024) proposes a novel quantization method that enhances
Random noise (Latent vector fn)
Latent vector 4+1
N gen generations
Condition
Condition Encoder
Condition embedding
Unetoriginal
Latentoriginal
For conditional tasks (example: Text-to-lmage)
Latent vector In
Unetmadt
Latent mod!
Score mod1
4; 1t time step time steps) Unetmotii Unetarigioal with x" operator modified (e.g. conv) Latent;; latent vector generated by Unet, Score; score computed between Latent original and Latent,
Unelmoak
Latent
Score mork
Figure 26: Pruming evaluates changes in the central tendency and variability to determine the significance of each operator. (Castells et al. 2024).
output quality by dynamically adjusting the quantization interval based on time step information. The proposed approach integrates with the Learned Step Size Quantization (Esser et al. 2019) framework, replacing the static quantization interval with a dynamically generated out- put from the Time-Dynamic Quantization module. This dy- namic adjustment leads to significant improvements in the quality of the quantized outputs. He et al. (2023) introduces a quantization-a aware low-rank adapter that integrates with model weights and is jointly quentized to low bit-width. This
Calbbratio
Data
Pruning
Figure 27: Illustrations the pruning.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Optimized Hardware-Software Co-design
Chen et al.
2023d
SDA
Yong al.
2023a
Choi et al. 2024a
System
DistriFusion
Li al.
2
PipeFusion
Wang ct al
2024a
Parallel Computing
SwiftDiffusion
Li ct a..
202
Diffusionl Ipe Tian
2024
NIRVANA Agurrall ct al.
2024
DeepCache Ma et al.
2024b
Caching Technique
Block Caching [Wimbauer
2024
L2C (Ma ct al.
2024n
FORA (Slvaraju al.
2024
MD-DT
Shee al.
A-DIT
Chen al.
Figure 28: Summary of system-level efficiency optimization techniques for diffusson models.
approach distills the denoising capabilitics of full-precision models into their quantized versions, utilizing only a few trainable quantization scales per layer and eliminating the need for training data.
3.4.2 Pruning
Pruming compresses DMs by removing redundant or less important model weights. Currently, most pruning methods for DMs focus on pruming structured palterns such as groups of consecutive parameters hierarchi- cal structures. For instance, Diff- feff-Prunin (Fang et al. 2023) introduces the first dedicated method designed for pruning diffusion models. Diff-Pruning leverages Taylor expansion over pruned timesteps to estimate the importance of weights. By filtering out non-contributory diffusion sleps and aggregating informative gradi- ents, Diff-Pruning enhances model efficiency while preserving essential features. LD-Pruner (Castells et al. 2024), as illustrated in Figure 26 on the other hand, proposes a pruning method specifically designed for Latent Diffusion Models (LDMs) The key innovation of LD-Pruner lies in its utilization of the latent space to guide the pruning process. The method enables precise assessment of pruning impacts by generating multiple sets of latent vectors
set for the original Unet and additional sets for each modified Unet where a single operator is altered. The importance of each operator is then quantified using a specialized formula that considers shifts in both the central tendency and variability of the latent vectors. This approach ensures that the pruning process preserves model performance while adapting to the specific characteristics of LDMs.
Kim et al. (2024a) introduces a technique known as LayerMerge, designed to jointly prune convolution layers and activation functions to achieve a desired inference speedup while minimizing performance degradation. LayerMerge addresses the challenge of selecting which layers to remove by formulating new surrogate optimization problem. Given the exponential nature of the selection space, the authors propose an efficient solution using dynamic programming. Their approach involves constructing dynamic programming (DP) lookup tables that exploit the problem's inherent structure, thereby allowing for an exact and efficient solution to the pruning problem.
Lastly, LAPTOPDIf (Zhang et al.) 2024a) introduces a layer-pruning technique aimed at automatically compressing the U-Net architecture of diffusion models. The core of this approach is an effective onc-shot pruning criterion, distinguished by its lavorable additivity property. This property ensures that the one-shot performance of the pruning is superior to other traditional layer pruning methods and manual layer removal techniques. By framing the pruning problem within the context of combinatorial optimization, LAPTOPDif simplifies the pruning process while achieving significant performance gains. The proposed method stands out for its ability to provide a robust one-shot pruning solution, offering a clear advantage compressing diffusion models efficiently.
4
System-Level Efficiency Optimization
4.1
Hardware- Soottare Co- Co-Design
The co-design of hardware and software is pivotal for achieving efficient deployment of diffusson models in real-time and resourco-constrained environments. Following algorithm-level optimizations, system-level tech- niques focus on integrating hardware-specific features, distributed computation, and caching mechanisms. These strategies aim to address the computational complexity and memory demands of large-scale diffusion
33

---

Published in Transactions on Machine Learning Research (05/2025)
models, enabling more practical applications across various platforms like GPUs, FPGAs, and mobile devices. One significant contribution is the work by Chen et al. (2023d), which explores GPU-aware optimizations for accelerating diffusion models directly on mobile devices. Implementing specialized kernels and optimized softmax operations reduces inference latency, achicving near real-time performance on mobile GPUs.
In related effort, Yang al. 2023a) propose SDA, a low-bit. stable diffusion accelerator designed specifically for edge FP- GAs. Utilizing quantization-aware training and hybrid sys- tolic array architecture illustrated in Figure 29 SDA effec- tively balances computational efficiency with flexibility, han- dling both convolutional and attention operations efficiently. Through a two-level pipelining structure, the nonlinear opera- tors are efficiently integrated with the hybridSA, enabling coor- dinated operation that enhances processing speed while reduc- ing resource usage. Finally, SDA achieves speedup of 97.3x when compared to ARM Cortex-A53 CPU. Furthermore, Choi ct al. 2024a) introduces stable diffusion processor optimized for mobile platforms through patch similarity-base sparsity, peeeeddpreision strategies and and a Dual-mode Bit-Slice Core (DBSC) architecture that supports mixed-precision computation, which particularly targeting resource-constrained devices such as mobile platforms. Together, these optimizations significantly improve throughput and energy efficiency, making Stable Diffusion more viable for
Stationary
Danolsing Process
HybridBA
Conv
Weight
Stationary
Figure 29: Illustration of the HybridSA ar-
chitecture from Yang et al. (2023a).
applications.
For GPU-accelerated mobile applications requiring real-time interactivity, Chen al. 2002d provides
kernel-level optinizations and mobile-specific operator tuning with strongest latency. When targeting emn
bedded edge devices with strict power budgets and static workloads, Yang et al. (2023a) achieves superior
performance via hardwwarealloorthm synergy. Meanwhile, Choi et al. (2024a) firstly approach dynamic
enorgy constraints in always-on sccnarios through runtime sparsity adaptation.
4.2 Parallel Computing
Data
Assembling
Data
Layer Pipellne
Model
Edge Asyne
Edge
Edge
Async
Asyne
Asyni
Async
GPUs
Patching
Batching
(a) Parallel Data Patching
(b) Model Layer Parallel Pipeline
Figure 30: Illustrations of the parallel computing for diffusson models.
Parallel computing Figure 30 plays a critical role in the efficient exccution of diffusion models, especially given the computaaiioniinlensiee nature of these algorithms. Recent advances in parallel computing strategies have enabled significant improvements in inference speed and scalability, often without compromising the quality of the generated output [Li et al. 2024a Wang et al. 2024a Li et al.) 2024b: Tian et al. 2024). This section highlights several notable contributions that tackle the challenge of parallelizing diffusion models across multiple GPUs and other distributed architectures.
Li al. (2024a) introduced DistriFusion, framework designed for distributed parallel inference tailored reghhutsolttion diffusion models such SDXL. Their approach involves partitioning the model inputs into distinct patches, which then processed independently across multiple GPUs. This method leverages
34

---

Published in Transactions on Machine Learning Research (05/2025)
the available hardware resources more effectively, achieving 6.1x speedup 8xA100 GPUs compared to single-card operation, all while maintaining output quality. To address potential issues arising from the loss of inter-patch interaction, which could compromise global consistency, DistrilFusion employs dynamic synchronization of activation displacements, striking a balance between preserving coherence and minimizing communication overhead.
Computation
Asynchromous Communication
OP
Scatter
MDD
FAAgyO
Layer
Layer L
A 6G
cd(x))
Seatter
Fj
AlllGather
Layer!
F(A
Update
A
Figure 31: Illustrations the diffusion architecture from et al. 2024a
Building on the insights gained from DistriFusion, Wang et al. (2024a] further refined the distributed infer- ence paradigm with PipeFusion. This system not only splits images into patches but also distributes the network layers across different devices, thereby roducing the associated communication costs and enabling the use of PCle-linked GPUs instead of NVLinnkconeeeled ones. PipeFusion integrates sequence parallelism, tensor parallelism, displaced patch parallelism, and displaced patch pipeline parallelism, optimizing workflow for wider range of hardware configurations. For applications involving add-on modules such as ControlNet and LoRA, Li et al. (2024b) developed SwiftDifflsion, as illustrated in Figure 31 This framework optimizes the serving workflow of these modules, allowing them to run in parallel multiple GPUs. As a result, SwiftDiffussin delivers 5x reduction in inference latency and a 2x improvement in throughput, ensuring that enhanced speed does not come at the expense output quality. Lastly, Tian et al. (2024) focused on the training phase with DiffusionPPpe, demonstrating that pipeline parallelism can produce a 1.41x training speedup, while data parallelism contributes an additional 1.28x acceleration. Although the optimization methods for DiffusionPipe were not detailed in the notes, the combination of these parallelization strategies offers promising direction to improve the efficiency of both the training and inference pipelines for diffusion models. These methodologies encompass hierarchical optimization, progressing from coarse-grained spatial partitioning to fine-grained module-specific adaptations, thereby empowering practitioners to strategically align parallelism configurations with underlying hardware constraints.
4.3 Caching Technique
In diffusion models, the computational hotspot often centers around discrete time-step diffusion, which is characterized by strong temporal locality. Consequently, building an efficient. caching system for diffusion models is nonnegligible enhance its performance. Indcod, extensive rescarch has been conducted on optimizing caching systems in Figure 32 resulting in significant advancements in this field.
Agarwal et al. 2024) proposed NIRVANA, novel system designed to enhance the efficiency of text-to-imast generation using diffusion models. Specifically, the key innovation lics in its approximate caching technique, which reduces computational costs and latency by reusing intermediale noise states from previous image generation processes. Instead of starting from scratch with every new text prompt, NIRVANA retrieves and reconditions these cachod states, allowing to skip several initial denoising steps. Additionally, the system uses custom cache management policy called Least Computationally Beneficial and Frequently Used (I.CBFU,, which optimizes the storage and reuse of cached states maximize computational efficiency. This
35

---

Published in Transactions on Machine Learning Research (05/2025)
Transformer
Feature Cache
Xt+1
Xt
X-1
MLP
Feed Forward
Feature Reuse
U-Net
U-Net
Caching System
Feature Cache
High-level
Cached Features
Multi-Head
Self Attention
Features
Feature Reuse
Caching System
Data
Figure 32: Illustrations of the caching system for diffusion models focus on the U-Net block and the Trans- former layer, critical components for effectively implementing caching techniques.
makes NIRVANA particularly suited for large-scale, production-level deployments of text-to-inage diffusion models. From another perspective, Ma et al. (202bb) introduces an innovative approach called DeepCache, designed to accolerate the image gencration process by leveraging the temporal redundancy in the denoising steps of diffusson models, without the need for additional model training, as ialustrated in Figure 33 The key insight is the observation that high-level features, such the main structure and shape of an image, exhibit minimal changes between adjacent denoising steps. These features can be cached and reused in subsequent steps, thereby avoiding redundant computations. This method takes advantage of the U-Net architecture by combining these cached high-level features with low-level features, updating only the low-level features to reduce computational load, leading to a significant acceleration in the overall process. WWimbauer et al. (2024) proposed Block Caching, a technique that identifies and caches redundant computations within the model's layers during the denoising process. By reusing these cached outputs in subsequent timesteps, the method significantly speeds up inference while maintaining image quality. To optimize this caching process, they introduce an Automatic Cache Scheduling mechanism, which dynamically determines when and where to cache based the relative changes in layer outputs over time. Additionally, the paper addresses potential misalignment issues from aggressive caching by implementing Scale-Shift Adjustment mechanism, which fine-tunee cached outputs to align with the model's expectations, thereby preventing visual artifacts.
Xt+1
X
X--1
D
Skip Branch
U1
D
D2
D2
Skip Branch
U2
D2
U2
D3
Brunch
U3
D3
U3
M
M
Chosen Path
Ft
Excluded Path
Figure 33: Illustration of the caching system from (M et al. 2024b
Recently, the application of diffusion with transformer models has yielded considerable success. Ma et al. (2024a) is concerned with the introduction of a layer caching mechanissm, designated Learning-tor taaning-to-Cache Cache (L2C), accelerate diffusion transformer models. L2C exploits the redundancy between layers within the transformer architecture, dynamically caching computations from certain layers to reduce redundant caleu-
36

---

Published in Transactions on Machine Learning Research (05/2025)
lations and lower inference costs. The implementation entails transforming the layer selection problem into differentiable optimization problem, using interpolation to determine whether to perform a full computation or utilize cached results at different timesteps during inference. In contrast to the emphasis on layer caching, Sclvaraju et al. (2024) proposed Fast-Forward Caching (FORA), a technique designed to accelcrate Diffusion Transformers (DiT) by reducing redundant compulations ions during the inference phase. The key insight behind FORA the observation that the outputs from the self-attention and MLP layers in a Transformer exhibit high similarity across consecutive time steps in the diffusion process. To leverage this, FORA implements static caching mechanism where these layer outputs cached regular intervals, which are determined by N, and reused for set number of subsequent steps, thereby avoiding recomputing similar outputs.
Shen et al. introduces MD-DiT, unified framework for efficient diffusion transformers by integrating block skipping and caching strategies, enabling dynamic depth adjustment across timesteps without additional training. optimizes computation by caching incremental changes from previous timesteps and selectively skipping non-critical blocks. Chen et al. proposes A-DiT, a groundbreaking framework for optimizing diffusion transformers through synergistie innovations: step-sensitive block caching and intelligently allocated computational resources. By exploiting the temporal consistency inherent in diffusion trajectories, the system strategically reuses intermediate residual features across sequential steps. Notably, it harmonizes both computation-skippiin and feature-caching mechanisms within a cohesive optimization architecture.
Generally, these methods demonstrate an trajectory from coarse temporal reuse to fine-grained layer adapta- tion, guiding practitioners to employ noise-state caching for prompt-variant scenarios, feature-level caching for structural consistency, and hybrid static-dynamic strategies for transformer- based models.
5 Frameworks
Comparison of Optimization Support in Sclected Diffusion Model Framoworks and Techniques. this table, 'Training' indicates whether the framework accelerates the training process of diffusion models, while 'Inference' indicates whether accelerates the generation process.
Table 2:
Framework
Training
Inference
Key Features
Flash Attention
High-efficinccy attention computation for Diffusion Transformers (DIT)
xFoomers
Memory-eeficient attention and modular ops tailored for diffusion Transformer speedups
DeepSpeed
Scalable distributed training and inference optimizations for large diffusion models
OneFlow
Compiler-opiimizeed pipeline for faster diffu- sion model training and sampling
Stable-Fast
x
Fast inference optimization for Diffusers with CUDNN fusson
Onediff
Diffusion-specific acceleration with Deep- Cache and quantization
DeepCache
Reuses cached diffusion features to speed up inference iterations
TGATE
Temporal gating to streamline cross- attention in diffusion inference
xDIT
Parallel inference engine for Diffusion Trans- formers
Frameworks in the efficient diffusion landscape refer high-performance tools and libraries designed optimize training and/or inference. DM frameworks can be in general grouped based whether they
37

---

Published in Transactions on Machine Learning Research (05/2025)
support the tasks of training and inference. Specifically, frameworks that support training aim to provide scalable, efficient, and flexible rnfrastru that improves computation efficiency, reduces memory footprint, optimizes communication efficiency, and ensures reliability of the training process. Frameworks that support inference focus on optimizing inference throughput and reducing memory footprint and latency.
A subset of frameworks, including Flash Attention (Dao et al.] 2022), xFormers (Tefaudeux et al. 2022 DeepSpeed (Rasley al.) 2020), and OneFlow (Yuan al.) 2021), distinguishes itself by offering compre- hensive support for both training and inference phases of diffusion models. These tools, rooted in broader deep learning optimization efforts, cater to the full lifecycle of model development. Flash Attention acceler- ates both training and inference by optimizing attention computation, reducing memory usage and latency through techniques like tiling and recomputation, making it particularly effective for Diffusion Transform- ers (DiT). xFormers provides -eemory-eficient Transformer optimizations with a modular design, enabling flexible and efficient computation across a range of resource-intensive tasks. DeepSpeed, originally designed for large-scale model training, extends its distributed capabilities to inference, enabling scalable deployment across multiple GPUs. Similarly, OneFlow leverages its compiler-driven architecture to streamline both training and inference workflows, appealing to researchers and practitioners seeking end-to-end optimiza- tion. Together, these frameworks provide robust foundations for advancing diffusion model rescarch and deployment, balancing flexibility with high performance.
In contrast, a growing array of tools Stable- Stat (chengzeyi 2024), Onediff (Contributors 2022 Deep- Cache (Ma et al. 2024b), TGATE (Zhang et al. 2024b Liu et al.), and xDiT (Fang et al.) 2024a Fang & Zhao 2024 Sun et al. 2024) focuses exclusively on accelerating the inference stage of diffusion mod- els, addressing the demand for rapid and resource-efficient generation. Stable-Fast optimizes the Hugging Face Diffusers ecosystem, prioritizing low-latency inference for real-time applications. Onediff builds on this trend by integrating cutting-edge techniques like caching and quantization, tailoring its acceleration to diffusion-specifie workloads. DeepCache and TGATE introduce innovative caching strategies, exploiting temporal redundancics to reduce computational overhead in inference, particularly for U-Net-basod models. Meanwhile, xDiT largets the emerging Diffusion Transformer (DiT) architecture, employing parallelization to enhance inference scalability. These inference-centril tools reflect a shift toward specialized optimizations, catering to the practical neods of deployment in constrained environments novel model paradigms.
6 Future Work
Despile the significant. progress made in elficient diffusion models, several promising research directions remain open. In particular, we identify the following key areas for future work:
Hybridizing Diffusion and Autorcgressive Models. One promissing future direction is to explore hybridizing autoregressive and diffusion models to combine the strengths of both paradigms. This integration enables the use of key-value (KV) caching, a technique from autoregressive transformers, to accelerate diffusion and support streamable generation. A representative work is Block Diffusion Arrfusion(Arrolaa Arriola al. 2025) which segments generation into blocks and applies autoregressive-styyle caching across denoising steps. This approach opens up promising opportunities to unify architectural benefits, allowing diffusson models to inherit the efficiency and online capabilities of autoregressive methods.
Without Classifier-Free Guidance (CFG). Although classifier guidance has been widely adopted to enhance generation quality, it introduces substantial computational overhead. Tang et al. (Tang et al. 2025) propose Model-guidance (MG), a novel training objective that eliminates the need for CFG by di- rectly incorporating the posterior probability of conditions rather than solely modeling data distribution. This approach not only doubles inference speed by avoiding the second network forward pass required by CFG but also significantly accolerates model training with 6.5x faster convergence and approximately 60% performance improvement. Notably, when compared to concurrent methods, MG achieves state-of-the-art performance on ImageNet 256 with an FID of 1.34 while requiring only about 12% of the computational resources of comparable approaches. The effectiveness of MG suggests a promising direction for future re- search to explore alternative guidance mechanisms or fundamentally rethink the training paradigm to better balance computational efficiency and generation quality.
38

---

Published in Transactions on Machine Learning Research (05/2025)
Efficient Attention Mechanisms for Video Diffusion. Bidirectional attention has become domi- nant component in diffusion architectures, but it introduces substantial computational overhead that scales quadratically with sequence length. This is particularly problematic for video diffusion models where the sequence length grows lincarly with the number of frames. Recent works have begun to explore efficient allention mechanisms specifically designed for diffusion transformers (DiTs). Xia et al. (Xia et al. 2025) propose AdaSpa, which leverages the hicrarchical sparsity inherent in DiTs through a blockified pattern approach and adaptive search methods. Their work demonstrates that sparse characteristics of DiTs exhibit hierarchical structures between different modalities and remain invariant across denoising steps, enabling significant computational savings. Similarly, Ding et al. (Ding et al. 2025) identify tile-style repetitive pat- terns in 3D attention maps for video data, introducing sparse 3D attention with linear complexity relative to frame count. Their approach combines efficient atlention with consistency distillation techniques to enable up to 7.8x faster generation for regl-resolution videos. These developments suggest promising directions for mitigating the computational burden of attention in diffusion models without sacrificing generation quality.
7 Conclusion
In this survey, we provide a systematic review of efficient diffusion models, an important area of research aimed at democratizing diffusion models. We start with motivating the necessity for efficient diffusion models. Guided by faxonomy, review efficient techniques for diffusion models from algorithm-level and system-level perspectives respectively. Furthermore, we review diffusion models frameworks with specific optimizations and features crucial for efficient diffusion models. We believe that efficiency will play an increasingly important role in diffusion models and diffusion models-oriented systems. We hope this survey could enable rescarchers and practitioners to quickly get started in this field and act as a catalyst to inspire new research on efficient diffusion models.
References
Shubham Agarwal, Subrata. Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjoe, and Shiv Kumar Saini. Approximate caching for efficiently serving (Texx-to-Image) diffusion models. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 1173-1189, 2024.
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying frame- work for flows and diffusions. arXiv preprint arXio:2303.00 037777777 2023.
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https://arziv http::://axxiv.or//bbbb.. wwwp/rgg//abbbaeeeeeee 2511//0533003333333
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669,26779,.. 2023.
Florentin Bieder, Julia Wolleb, Alicia Durrer, Robin Sandkuehler, and Philippe C Cattin. Memory-efficient 3d denoising diffusion models for modical image processing. In Medical Imaging with Deep Learning, 2023.
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Semi-parametria neural image synthesis. XrXiv preprint arXiv:2204.10 1 11824, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution vidco synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563-22578 2023.
Evelyn Buckwar and Renate Winkler. Multistep methods for sdes and their application to problems with small noise. SIAM journal numerical analysis, 44(2):779 803, 2006.
39

---

Published in Transactions on Machine Learning Research (05/2025)
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ama Ann Heng, and Stan Z. Li. A survey on generative diffusion models. TEEE Transactions on Knowledgge and Data Engineering, 36(7): 2814 2830, 2024. doi:
Thibault Castells, Hyoung-Kyu Song, Bo-Kyeong Kim, and Shinkook Choi. Ld-pruner: Efficient pruming of latent diffusion models using lask-agnostic insights. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 821 111-830, 2024.
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. Tn Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23206 23217, 2023.
Jiaao Chen, Aston Zhang, Mu Li, Alex Smola, and Diyi Yang. A cheaper and better diffusion language model with soft-masked noise. arXiv preprint or Xiv:2300 000..044444 04746, 2023a.
Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Enze Xie, and Song Han. Sana-sprint: One-step diffusion with contimuous-iim consistency distillation. arXiv preprint arX1v:2503.09641, 2025.
Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Applications, guided generation, statistical rates and optimization, arXiv preprint arXiv:2404.00 077771, 2024.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Essimating gradients for waveform generation. arXiv preprint arXiv:2009.0071 2020.
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. d-dit: Accelerating dillusion transformers without training via denoising property align- ment.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In 2023b.
Proceedings of the IEEE/CVF internalional conference on compuler vision, pp.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint ar.i::2208. 2022:2208.04202, 2022a.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image generator. In The Eleventh International Conference Learning Represenlations, 2022b.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. Tn Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18000 18010, 2023c.
Yu-Hui Chen, Raman Sarokin, Juhyun Lec, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. Speed all you need: On-device acceleration of large diffusion models via gpu-aware op- timizations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4651-4655, 2023d.
chengzeyi. Stable fast.
2024.
Jiwon Choi, Wooyoung Jo, Scongyon Hong, Beomscok Kwon, Wonhoon Park, and Hoi-Jun Yoo. A 28.6 mj/iter stable diffusion processor for text-to-image generation with patch similarity-based sparsity aug- mentation and text-based mixed-precision. arXiv preprint xxXx:2408.04982, 2024a.
Joo Young Choi, Jacsung R Park, Inkyu Park, Jacwoong Cho, Albert No, and Ernest Ryu. Simple drop-in Iora conditioning on attention layers will improve your diffusion model. arXiv preprint orXiv:2405.0000 03958, 2024b.
40

---

Published in Transactions on Machine Learning Research (05/2025)
OneDiff Contributors. Onediff: An out-of-the-box acceleration library for diffusion models. https://gith 2022.
ub. www.oo/iiiooooloo///oooooooooo/e
Guillaume Cousiron, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: villusion-based semantic image editing with mask guidance. arXiv preprint 1xXi:2210011499 2022.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850 1000000009,,-.0 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficien exact attention with io-awareness. Advances neural information processing systems, 35:16344-16359, 2022.
Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Jong Chul Ye, Peyman Milanfar, Alexan- dros G. Dimakis, and Mauricio Delbracio. survey on diffusion models for inverse problems, 2024. URL
https://arxiv. wwwpeii.org/abb.a.. wttgg///g//...o..////// hatps//2100.00000 00083
Dale Decatur, Itai Lang, Kfir Aberman, and Rana Hanocka. 3d paintbrush: Local stylization of 3d shapes with cascaded score distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4473 4483, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Hangliang Ding, Dacheng Li, Runlong Su, Peiyuan Zhang, Zhijie Deng, lon Stoica, and Hao Zhang. Efficient- vdit: Efficient vidco diffusion transformers with attention tilc. arXiv preprint arXiv:2502.06155 2025.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXio:2112.07065 2021.
Bi'an Du, Wei Hu, and Renjic Liao. Multi-scale latent point consistency models for 3d shape generation. arxiv preprint ar Xrxi::2412 19413, 2024.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint 1XXt:1902.001533 2019.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Advonces in Newral Information Processing Systems, 2023.
Jiarui Fang and Shangehun Zhao. A unified sequence parallelism approach for long context generative ai. 2024.
arXiv preprinl ar 2x0:2405.007
Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint 241::2411.0700 2024a.
Jiarui Fang, Jinzhe Pan, Jiannan Wang, Aoyu Li, and Xibo Sun. Pipefusion: Patch-level pipeline parallelism for diffusion transformers inference. arXiv preprint 4rXiv:2405.14490 2024b.
Yuxiang Fu, Qi Yan, Lele Wang, Ke Li, and Renjie Liao. Moflow: One-step flow matching for human
trajectory forecasting via implicit maximum likelihood estimation based distillation. arXio preprint arXiv:2503.0995: 2025.
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint ar Xiv:23111 12092, 2023.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Murphy, and Tim Salimans, Diffusion meets flow matching: Two sides of the same coin. 2024. URL https://aafffusioooooooo .iv.wggiii gitbub
.1o/
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence sequence text generation with diffusion models. Xi preprint or 2Xix:221000899 2022.
41

---

Published in Transactions on Machine Learning Research (05/2025)
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yumpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang, Yu Bao, Jian Peng, Jianzhu Ma, Qiang Liu, Liang Wang, and Quanquan Gu. Decompdiff: Diffusion models with decomposed priors for structure-based drug design. Proceedings of Machine Learning Research, 202:11827-11866, 2023.
Hanzhong Guo, Cheng Lu, Fan Bao, Tianyu Pang, Shuicheng Yan, Chao Du, and Chongxuan Li. Gaussian mixture solvers for dilfusion models. Advances in Newral Information Processing Systems, 36, 2024.
Tiankai Hang and Shuyang Gu. Improved noise schedule for diffusion training. arXiv preprint arXiv:2407.03990 2024.
Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization-aware fine-tuning of low-bit diffusion models. arXiv preprint orXiv.231000327 2023.
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yingging He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. Xiv preprint arxi:2211 1.1221, 2022.
Shashank Hegde, Sumeet Batra, KR Zentner, and Gaurav Sukhatme. Generating behaviorally diverse policies with latent diffusion models. Advances in Neural Information Processing Systems, 14:7541-75544 2023.
Eric Heitz, Laurent Belcour, and Thomas Chambon. Iterative (-(de) blending: A minimalist deterministic diffusion model. In ACM SIGGRAPH 2029 Conference Proceedings, pp. 1-8, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0m 020331 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022a.
Jonathan Ho, Tim Salimans, Alexcy Gritsenko, William Chan, Mohammad Norouzi, and David J Flect. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022b.
Marlis Hochbruck and Alexander Ostermann. Exponential integrators. Acta Numerica, 19:209 286, 2010.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaplation of large language models. arXie preprinl arXiv:2106.0906 096885 2021.
Kaixuan Huang, Yukang Yang, Kaidi Fu, Yanyi Chu, Le Cong, and Mengdi Wang. Latent diffusion models for controllable rna sequence generation. orXiv preprint ar arXio:2409.098.0 2024.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Max Jiang, Yijing Bai, Andre Cormman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kul- shrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, al. Scenediffuser: Efficient and controllable driving simulation initialization and rollout. Advances in Neural Information Processing Systems, 37: 55729-55760, 2024.
Alexia Jolicoeur-Martineann Ke Li, Rémi Piché Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast. when generating data with score-based models. arXiv preprint arxiv:215.. 14080, 2021.
42

---

Published in Transactions on Machine Learning Research (05/2025)
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565- 2022.
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Mocin Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof. Diffusion models in medical imaging: A comprchensive survey. Medical Image Analysis, 88:102846, 2023. ISSN 1361-8415. doi:
wwtp:////./...111..mmma......... http://mmmmmaa222..... 102.44 URL attpps://ww. http://wwwiiinnniiieeeeeeee htt.a///ea..oooooooo....coo/aa/a
Bcomsu Kim and Jong Chul Ye. Denoising meme for accelcrating diffusion-based generative models. arXiv preprint 22v: 14593, 2022.
Jinuk Kim, Marwa Halabi, Mingi Ji, and Hyun Oh Song. Layermerge: Neural network depth compression through layer pruning and merging. arXio preprint orXiv:2406. 12837, 2024a.
Yulhwa Kim, Dongwon Jo, Hyesung Joon, Taesu Kim, Dachyun Ahn, Hyungjun Kim, et al. Leveraging carly- stage robustness in diffusson models for efficient and high-quality image synthesis. Advances in Newral Information. Processing Systems, 36, 2024b.
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer. Streamdiffusion: A pipeline-level solution for real-time interactive generation. arXiv preprint arXiv:2312. (.t.2312.12499) 12491, 2023.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv:2106.00130 2021.
arXiv preprint
Zhileng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.080 097611 2020.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. In International Conference on Learning Representations, 2021.
Sangyun Lee, Zinan Lin, and Giulia Fanti. arXiv:2405. 20320, 2024a.
Improving the training of rectified flows. arXiv preprint
Taegyeong Lee, Soyeong Kwon, and Taehwan Kim. Grid diffusson models for text-to-video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8734-8743, 2024b.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https: :
//github.com. httech/w..amcccoooorrreerrr/ttee
2022.
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet+ Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision, pp. 147. Springer, 2025.
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjic Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 7183-7193, 2024a.
Suyi Li, Lingyun Yang, Xiaoxiao Jiang, Hanfeng Lu, Zhipong Di, Weiyi Lu, Jiawei Chen, Kan Liu, Yinghao Yu, Tao Lan, Guodong Yang, Lin Qu, Liping Zhang, and Wei Wang. Swiftdiffusion: Efficient diffusion model serving with add-on modules, 2024b. URL TTRLhttps:// wwwpa/www.o//b/////0/007..
22200
43

---

Published in Transactions on Machine Learning Research (05/2025)
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Aduances in Newral Information Processing Systems, 0444328-4343 2022.
Xinyang Li, Zhangyu Lai, Linning Xu, Jianfei Guo, Liujuan Cao, Shengchuan Zhang, Bo Dai, and Rongrong Ji. Dual3d: Efficient and consistent text-to-3d generation with dual-mode multi-view latent diffusion. arXiv preprint arX6v:2405.0987 099774 2024c.
Xinyu Li, Yijiang Läu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVE International Con- ference on Computer Vision, pp.
2023.
Yiheng Li, Heyang Jiang, Akio Kodaira, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Immiscible diffusion: Accelerating diffusion training with noise assignment. arXiv preprint arXiv:2406. 12303, 2024d.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-traiine training quantization by block reconstruction. In International Conference on Learning Represenlations, 2021.
Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arx:v2440 099677 2024a.
Jianghao Lin, Jiaqi Liu, Jiachen Zhu, Yunjia Xi, Chengkai Liu, Yangtian Zhang, Yong Yu, and Weinan Zhang. A survey on diffusion models for recommender systems. arXiv preprint vrXiv:2409.0508. 2024b.
Lequan Lin, Zhengkun Li, Ruikun Li, Xuliang Li, and Junbin Gao. Diffusion models for time-series applica- tions: a survey, Frontiers of Information Technology & Electronic Engineering, 25(1):19 41, 2024c.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, orXiv preprint arXim:2210.0211 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumb- ley. Audioldm: Text-lo-audio generation with latent diffusion models. arXiv preprinl or arXix:2301. 125003 2023a.
Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio, Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan-Manuel Perez-Rua, and Jürgen Schmidhuber. Faster diffusion through temporal attention decompo- sition. Transactions on Machine Learning Research.
Jiawei Liu, Weining Wang, Wei Liu, Qian He, and Jing Liu. Ed-t2v: An efficient training framework for diffusion-based text-to-video generation. In 2023 International Joint Conference on Neural Networks (IJCNN), pp. 1 8. IEEE, 2023b.
Luping Liu, Yi Ren, Zhijic Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint ar xxXm:2202.00772, 2022a.
Qiang Liu. Rectified flow: A marginal preserving approach optimal transport. arXiv:2209.1430 14577, 2022.
arXiv preprint
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprinl X20. 03003, 2022b.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Represen- tations, 2023c.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. Advances in Neural Information. Processing Systems, 36, 2024.
44

---

Published in Transactions on Machine Learning Research (05/2025)
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-ting time consistency models. arXiv preprint arXiv:2410. 10.1 1081, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211001 1.00955, 2022.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.085 02388, 2021.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint (xXix:2310.04877 2023a.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint 3rXiv:2911.05550 2023b.
Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. ar preprint arXiv:2304.0400 04262, 2023.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching. ar preprint arXiv:2406.8m 0173,, 2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024b.
Zhiyuan Ma, Yuzhu Zhang, Guoli Jia, Liangliang Zhao, Yichao Ma, Mingjie Ma, Gaofeng Liu, Kaiyan Zhang, Jianjun Li, and Bowen Zhou. Efficient diffussion models: A comprehensive survey from principles to practices. arXiv preprint arXiv:2410. 10..2110.1199.. 2024c.
Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusson model for stochas- tic trajectory prediction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5517-5526, 2023.
Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, and Helge Ritter. Video diffusion models: A survey. arXiv preprint 2xXiv24455001550 2024.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021.
Shentong Mo. Efficient 3d shape generation via diffusion mamba with bidirectional ssms. arXiv preprint arXiv:2406.0500 05038, 2024.
Tachong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, and Juho Lee. A simple early exiting framework for accelerated sampling in diffusion models. arXiv preprint 2000022008.0027, 2024a.
Tachong Moon, Moonscok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, and Juho Lec. A simple carly exiting framework for accelerated sampling diffusion models. In Forty-first International Conference Machine Learning, 2024b.
45

---

Published in Transactions on Machine Learning Research (05/2025)
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296 4304, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162 8171. PMLR, 2021.
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: Sde beats ode in general diffusion-based image editing, arXiv preprint 0rXiv:2311.01110 2023.
Jangho Park, Gihyun Kwon, and Jong Chul Ye. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.027 027112 2023.
William Peobles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint ar arXiv:2312.44588 2023.
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint 2rXiv:2488.060 06070, 2024a.
Mingxing Peng, Kehua Chen, Xusen Guo, Qiming Zhang, Hongliang Lu, Hui Zhong, Di Chen, Meixin Zhu, and Hai Yang. Diffusion models for intelligent transportation systems: A survey, arXiv preprint 4rXiv:2499.1500 2024b.
Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975-37990, 2022.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual gener- ation in the wild. arXiv preprint ar trXiv:2305.1114 11147, 2023.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In Scale Space and Varialional Methods in Compuler Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29 June 2, 2011, Revised Selected Papers 3, pp. 435 4464 Springer, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 2021.
PmLR,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint ar Xrv: 2.t.22104..12, 2022.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Decpspoed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505-3506, 2020.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and TrendsS in Information Retrieval, 3(4):333-389 2009.
Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Bulô, Peter Kontschieder, Angela Dai, and Matthias Nießner. L3dg: Latent 3d gaussian diffusion. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
46

---

Published in Transactions on Machine Learning Research (05/2025)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the TEEE/CVF conference on compuler vision and pattern recognition, pp. 10684 10695, 2022.
Amirmojtaba. Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules diffusion models. arXiv preprint arXiv:2404. 14507, 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated inference via confident adaptive transformers. arXiv preprint XXiu:214408800, 2021.
Pratheba Selvaraju, Tianyu Ding, Tianyi Chen, llya Zharkov, and Luming Liang. Fora: Fast-forward caching in diffusion transformer acceleration. arXiv preprint arXiv:2407.01425, 2024.
Yuzhang Shang, Zhihang Yuan, Bin Xic, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on compuler vision and pollern recognition, pp- 1972 1981, 2023.
Mingzhu Shen, Pengtao Chen, Peng Ye, Guoxnnnn Xia, Tao Chen, Christos-Savvas Bouganis, and Yiren Zhao. Md-dit: Step-aware mixturo-of-depths for efficient diffusion transformers. In Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning.
Shelly Sheymin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. knn-diffusion: Image generation via large-scale retrieval. In The Fleventh International Conference Learning Representations, 2022.
Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Junhyuk So, Jungwon Lee, Dachyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quanti- zation for diffuusin models. Advances in Neural Information Processing Systems, 36, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXio preprint arXiv:2010. 00..0000,,, 2020a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. orXiv preprint arXiv:2910.2 1419 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in neural information processing systems, 32, 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainly in Artificial Intelligence, pp. 574 584. PMLR, 2020b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.19466, 2020c.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXi::2303.01408 2023.
arXiv preprint
47

---

Published in Transactions on Machine Learning Research (05/2025)
Xibo Sun, Jiarui Fang, Aoyu Li, and Jinzhe Pan. Unveiling redundancy in diffusion transformers (dits): A systematic study. arXiv preprint arXiv:2411.138 2024.
Shengkun Tang, Yaqing Wang, Caiwen Ding, Yi Liang Yao Li, and Dongkuan Xu. Deediff: Dynamic uncertainty-wware early exiting for accelerating diffusion model generation. 2023a.
Zhicong Tang, Shuyang Gu, Chunyn Wang, Ting Zhang, Jianmin Bao, Dong Chen, and Baining Guo. Volumediffusion: Flexible text-to-3d generation with efficient volumetric encoder. ar preprinl 1.Xiv:2312.11458 2023b.
Zhicong Tang, Jianmin Bao, Dong Chen, and Baining Guo. Diffusion models without classifier free guidance. arXiv preprint 2rx:22502.. 12154, 2025.
Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, and Tsung-Hui Chang, Accelerating parallel sampling of diffusion models. In Forty-first International Conference on Machine Learning, 2024.
Ye Tian, Zhen Jia, Ziyue Luo, Yida Wang, and Chuan Wu. Diffusionpipe: Training large diffusion models with efficient pipelines, 2024. URL https://arxiv org/abs/2405. http:://arriv.orr//ab////4. 01248
Anwaar Ulhaq, Naveed Akhtar, and Ganna Pogrebna. Efficient diffusion models for vision: A survey. arXiv preprint Xrx:22210. 09292, 2022.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Clement Vignac, Igor Krawczuk, Antoine Sirandin, Bohan Wang, Volkan Covher, and Pascal Frossard.
Digress: Discrete denoising diffusion for graph generation. In The Eleventh International Conference Learning Representations, 2022.
Homer F Walker and Peng Ni. Anderson acceleration for fixed- fixed-point iterations. SIAM Journal on Numerical Analysis, 4074511755)1735 2011.
Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. Pipcfusion: Displaced patch pipeline parallelism for inference of diffusion transformer models. arXie preprint Xivi::24 14430, 2024a.
Xiang Wang, Shiwci Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Vidcolem: Video latent consistency model. arXiv preprint arxiv:2312.091 290009, 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024b.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprinl urXio:2106. 00000002 2021.
Felix Wimbaner, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyen, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you Accelerating diffusion models through block caching, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6211 12200 2024.
Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp.
2024.
Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-fre free and adaptive sparse attention for efficient long vidco generation. arXiv preprint 0rXiv:2502.21077 2025.
Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. ACM Computing Surveys, 2023.
48

---

Published in Transactions on Machine Learning Research (05/2025)
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for effi- cient video generation. In Proceedings of the IERE/CVE Conference on Computer Vision and Pottern Recognition, pp. 7827 7839, 2024.
Xingqian Xu, Zhangvang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one dilfusion model. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7754-7765, 7765, 2023a.
Yilum Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36:76806-76835, 2023b.
Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, and Chongxuan Li. Unifying bayesian flow networks and diffusion models through stochastic differential equations. Xin preprinl arXiv:2404.1576, 2024a.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adans solver for fast sampling of diffusion models. Advances in Newral Information Processing Systems, 36, 2024b.
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint arXiv:2405.0777 07510, 2024.
Geng Yang, Yanyue Xie, Zhong Jia Xue, Sung-En Chang, Yanyu Li, Peiyan Dong, Jie Lei, Weiying Xie, Yanzhi Wang, Xue Lin, et al. Sda: Low-bit stable diffusion acceleration edge fpgas. 2023a.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023b.
Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, al. Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models. arXin preprint 2rX 2024.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXio:2501.01425 2025.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for texi-Lo-image diffusion models. arXiv preprint Niw:2308.06721, 2023.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint ar Xrx:23303 12346, 2023.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 6613 6623, 2024.
Yuning You, Ruida Zhou, Jiwoong Park, Haotian Xu, Chao Tian, Zhangrang Wang, and Yang Shen. Latent 3d graph diffusion. International Conference Learning Representations (ICLR), 2024.
Sibyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusson models in projected latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18456-18466, 2023.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xic. Representation alignment for generation: Training diffusion transformers is casier than you think. arxiv preprint ar Xiv::4 11.1000.50 10.06940, 2024a.
49

---

Published in Transactions on Machine Learning Research (05/2025)
Yonghao Yu, Shunan Zhu, Huai Qin, and Haorui Li. Boostdream: Efficient refining for high-quality text-to-3d generation from multi-view diffusion. arXiv preprint arXiv:2401.16764 2024b.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. Text diffusion model with oncoder-decoder transformers for scquence-to-sqquenee generation. In Proceedings of the 2024 Confer- ence of the North Americun Chapter of the Association for Compulational Linguislies: Human Language Technologies (Volume 1: Long Papers), pp. 22-39, 2024.
Jinhui Yuan, Xinqi Li, Cheng Cheng, Juncheng Liu, Ran Guo, Shenghang Cai, Chi Yao, Fei Yang, Xiaodong Yi, Chuan Wu, et al. Oneflow: Redesign the distributed deep learning framework from scratch. ar preprinl ar Xrv: 10. 15032, 2021.
Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super- resolution by residual shifting. Advances in Neural Information Processing Systems, 36, 2024.
Mohsen Zand, Ali Etemad, and Michael Greenspan. Diffusion models with deterministic normalizing flow priors. arXiv preprint arXiv:2309.011 019.27 2023.
Denis Zavadski, Johann-Priedrich Feiden, and Carsten Rother. Controlnet-ss: Rethinking the control of text-to-image diffusion models as feedback-control systems, 2023.
Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, and Haonan Tu. Laptop-diff: Layer pruning and normalized distillation for compressing diffusion models. arXiv preprint arXiv:2404.1009, 2024a.
Hengrui Zhang, Jiani Zhang, Balasubramaniam Srinivasan, Zhengyuan Shen, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis. Mixed-type tabular data synthesis with score-based diffusion latent space. arXiv preprint arXiv:2310.0080 096556, 2023a.
Hui Zhang, Zuxuan Wu, Zhen Xing, Jie Shao, and Yu-Gang Jiang. Adadiff: Adaptive step selection for fast diffusion. Xiv preprint ar Xrx::23311. 2023b.
Kexun Zhang, Xianjun Yang, William Yang Wang. and Lei Li. Redi: efficient learning-free free diffusion inference via trajectory retrieval. In International Conference on Machine Learning, pp. 11170041788 PMLR, 2023c.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 3836 3847, 2023d.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in neural informalion processing systems, 34:16280-100 16291, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint Xrx::22204. 2022.
Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-allention makes inference cumbersome in Lexl-lo-image diffusion models. ar e-prints, pp. arXiv- 2404, 2024b.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yo Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusson models. Advances in Neural Informa- lion Processing Systems, 36, 2024.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood esti- mation for diffusion odes. In International Conference on Machine Learning, pp. 42363-42389. PMLR, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXio preprint ar arxiv:221 110118 2022.
50

---

Published in Transactions on Machine Learning Research (05/2025)
Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient. motion diffusion model for fast, high-quality motion generation.
Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. Slimflow: Training smaller one-step diffusion models with rectified flow. arxiv preprint or 1771:1147.12718, 2024.
51


Reviews:
[{"review_id": "71x7kVl3zP", "reviewer": "Reviewer_AZ4W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly structured and easy to read, with different research tidily organised into their corresponding sections.\n- A wide range of topics is covered, spanning algorithms and systems, giving the reader a clear picture of efficient diffusion *across the whole stack*.\n- There is generally a good balance between detail and brevity.\n\n\n**Weaknesses**\n1. I find that the authors dive into the details too quickly, when more general background/motivation/intuition for each section would help. For example, Sec. 3.4.1 would benefit from initially discussing the *benefits* of low-bitwidth and fixed-precision inference, such as reduced memory costs and smaller-area/lower-power hardware at the start. Similarly, the first paragraph of Sec. 2 could have a sentence saying that diffusion models generate data by iterative denoising, transforming pure noise gradually into clean data. For a final example, the opening of Sec. 3.3 could include some elaboration on how the iterative denoising process is slowed down by the need to *sequentially* perform a large number of network inferences. **The paper would benefit from simply one or two additional sentences at the start of each section/subsection to help the reader in this way.**\n2. The discussion of diffusion model fundamentals relies on the original DDPM paper which complicates understanding when later discussion of SDE and ODE samplers pops up. I would suggest using a more unified mathematical framework such as the one in https://diffusionflow.github.io/. \n3. Distillation would be better placed in efficient sampling, since its purpose is not to reduce network size, but to reduce the number of sampling steps. Also, rectified flows would better be considered together with distillation. Finally, consistency models are not very well discussed (LCM-LoRA is introduced before CMs themselves).\n4. Links/references should be provided to the frameworks discussed in Sec. 6. Additionally it is not completely clear what a \"framework\" is in this context. \n5. Miscellaneous missing references: \n    - Distillation: Yin et al. One-step Diffusion with Distribution Matching Distillation, CVPR 2024\n    - Caching: Shen et al. MD-DiT: Step-aware Mixture-of-Depths for Efficient Diffusion Transformers, AFM Workshop NeurIPS 2024; $\\Delta$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment, Chen et al. 2024\n    - Parallel Inference: Kodaira et al. StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation,  2023\n    - Efficient Sampling: Xu et al. Restart Sampling for Improving Generative Processes,  NeurIPS 2023\n6. **Edit (additional point)** There is a lack of discussion about future work; I think the paper would benefit from identifying to the reader broader areas of research potential, especially in applications beyond image generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide a survey on *efficient* diffusion models. They cover a broad range of research, arranging it into the categories of", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above. I would classify none of them as critical, as I believe the paper to be satisfactory as a survey. However, I hope the authors consider my feedback; I especially believe 1 and 2 would improve the experience for the reader.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "phyDPCiYpA", "reviewer": "Reviewer_v2b1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Figures 1 and 2 clearly illustrate the paper's organization and are easy to follow. The writing is generally clear, at least for those familiar with diffusion models.\n2. Methods from various application domains are surveyed and discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey presents a systematic review of diffusion models with a focus on efficiency. It is a timely and popular topic in the research community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I believe that flow matching works should be highlighted in Section 2, in addition to the classical DDPM formulations, as they are increasingly demonstrating real-world impact. The authors have already cited the seminal works on flow matching (Lipman et al., 2022; Liu et al., 2022b; Albergo et al., 2023) and should expand on this. Placing it in Section 3.1.4, \"Rectified Flow,\" in my opinion, unreasonably downplays its importance in the field.  \n\n2. Diffusion generative models for video are an important topic. While this paper focuses on surveying efficient methods, it is suggested that the authors review the latest developments in this field and expand the current single-paragraph Section 5.2, \"Video Generation.\"  For example, consider referencing:  \n **Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H.** *Video Diffusion Models: A Survey.* Transactions on Machine Learning Research.  \n\n3. Please use $\\mathrm{d}$ (\\mathrm{d}) to represent the differential operator, for example, in Equations (20)–(23).  \n\n4. The mathematical formulations of score-based models, SDEs, and probability flow ODEs should be introduced earlier in Section 2, \"Fundamentals,\" to ensure that the mathematical concepts are more coherent and self-contained. After all, SDEs, ODEs, score-based models, and denoising diffusion can be understood from a very similar perspective. Presenting them together will make it clearer for readers unfamiliar with the field to grasp the connections and nuances.  \n\n5. More importantly, the survey should discuss and provide insights into potential future work, given the current research landscape on diffusion model efficiency. This will help readers not only understand the existing work, which is abundant, but also grasp key trends and research focus.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pBUuVArn0i", "reviewer": "Reviewer_TFC6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. This survey comprehensively reviews literatures about efficient diffusion, which shows the recent progress of efficient diffusion models.\n2. This survey briefly introduces some background before discussing a topic which makes it easier for readers to get some basic knowledges.\n3. Tables and figures are presented to explain some concepts and show the overall structure of the survey, which makes the survey clear to read.\n\n## Weakness:\n1. This survey does not include a good summarization or organization of papers listed.\n    - This survey introduces many papers which reads like a related work in a paper without summarization or organization for the papers discussed. For instance, in section 3.2, the authors discussed LoRA, Adapter and ControlNet without a high level of discussion about their advantages and disadvantages when compared with each other. As a reader, it is hard to get a good feeling about which method I should  investigate based on my own application or issues at hand.\n\n2. I have concerns about the discussion for \"Efficient Training\".\n    - Most papers discussed in this section are about improving the quality of generation or improving the efficiency of inference, not about reducing cost of pre-training as mentioned in Section 3.1. For instance, as mentioned in the paper \"The cosine noise schedule helps the model achieve high-quality sample generation by optimizing the noise distribution throughout the training process, allowing the model to generate samples with a reduced number of sampling steps while preserving essential structural information at each stage.\" Purpose of applying cosine noise schedule is to improve quality and reduce inference time, but no evidence or discussion shows it can improve the training efficiency.\n    - The four categories discussed in this section are not at the same concept level. For instance, score matching and rectified flow are different formulation of variants. Noise scheduling and adaptive priors are generally training tricks which can be applied to different formulation of variants. To better organize this section, the authors may refer to (Ma et al. 2024c) in which they organize these papers as principles instead of discussing improve the training efficiency.\n    - From my point of view, compare with applying diffusion model in pixel space, applying diffusion model in latent space is one way to really improve the training efficiency, however, there is no discussion about this part.\n\n3. Questions about discussing application and Frameworks.\n    - While application and frameworks are important, they are not the main focus of this survey. A short discussion about them may not good enough. Unless there are some novel techniques that are particularly designed for different applications or frameworks that cannot be included in the above discussion.\n    - What is 'Training', 'Fine-Tuning' and 'inference' mean in table 2? Does it mean whether the frame work support these oprations?\n\n4. Other issues:\n    - Duplicate citation: \\\n         (a) \"Adding conditional control to text-to-image diffusion models\". Zhang et al 2023c and 2023d. \\\n         (b) \" Diffusion models in vision: A survey\", Croitoru 2023a and 2023b. \n    - Typo:\n        (a) However, this approach has been shown **ta** failure to leverage the linguistic features of the text in question (page 5).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey comprehensively reviewed a large number of papers related to efficient diffusion models. They organize these papers following the structure of Algorithm, System, Application and Framework. When discussing these papers, the authors provide necessary background  and include tables and figures to make it easy for the readers to follow. This survey may be beneficial for readers working in related field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors may want to include more comparisons of different strategies under the same aspect of efficiency (for example, in the Efficient Fine-tuning section) to make it clear for the readers to understand their advantages and disadvantages.\n\n2. The overall structure of organizing the literatures need to change. As discussed in weakness 2, \"Efficient Training\" may not be a good aspect to discuss efficient diffusion models. A good reference would be the paper (Ma et al., 2024c) which mentioned by the authors in the introduction. \n\n3. Application and framework could be more concise as they are not the focus of this survey. Moreover, in the abstract, when discussing application, the authors may want to include text and 3D.\n\n4. Please fix the minor issues mentioned in the weakness.", "broader_impact_concerns": "I don't see broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please continune to update the literature in this field on the github page mentioned in this paper.
Decision: Accept (poster)