Diffusion Sampling Correction via Approximately 10 Parameters
Authors: Guangyi Wang
Wei Peng
lijiang Li
Wenyu Chen
Yuren Cai
Song-Zhi Su
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
accelerated sampling
low-cost training
plug-and-play
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a new plug-and-play training paradigm for accelerated sampling of diffusion models, featuring minimal learnable parameters and training costs.

Abstract: While powerful for generation, Diffusion Probabilistic Models (DPMs) face slow sampling challenges, for which various distillation-based methods have been proposed. However, they typically require significant additional training costs and model parameter storage, limiting their practicality. In this work, we propose **P**CA-based **A**daptive **S**earch (PAS), which optimizes existing solvers for DPMs with minimal additional costs. Specifically, we first employ PCA to obtain a few basis vectors to span the high-dimensional sampling space, which enables us to learn just a set of coordinates to correct the sampling direction; furthermore, based on the observation that the cumulative truncation error exhibits an ``S"-shape, we design an adaptive search strategy that further enhances the sampling efficiency and reduces the number of stored parameters to approximately 10. Extensive experiments demonstrate that PAS can significantly enhance existing fast solvers in a plug-and-play manner with negligible costs. E.g., on CIFAR10, PAS optimizes DDIM's FID from 15.69 to 4.37 (NFE=10) using only **12 parameters and sub-minute training** on a single A100 GPU. Code is available at https://github.com/onefly123/PAS.

Diffusion Sampling Correction via Approximately 10 Parameters
Guangyi Wang
123*
Wei Peng
Lijiang Li
Wenyu Chen Yuren Cai 13
Songzhi Su
Abstract
While powerful for generation, Diffusion Proba- bilistic Models (DPMs) face slow sampling chal- lenges, for which various distillation-based meth- ods have been proposed. However, they typi- cally require significant additional training costs and model parameter storage, limiting their prac- ticality. In this work, we propose PCA-based Adaptive Search (PAS), which optimizes exist- ing solvers for DPMs with minimal additional costs. Specifically, we first employ PCA to obtain a few basis vectors to span the high-dimensional sampling space, which enables us to learn just set of coordinates correct the sampling di- rection; furthermore, based the observation that the cumulative truncation error exhibits an "S"-shape, we design adaptive search strategy that further enhances the sampling efficiency and reduces the number stored parameters to ap- proximately 10. Extensive experiments demon- strate that PAS can significantly enhance existing fast solvers in plug-and-play manner with neg- ligible costs. E.g., CIFAR1O, PAS optimizes DDIM's FID from 15.69 to 4.37 (NFF=10) us- ing only 12 parameters and sub-minute train- ing on a single A100 GPU, Code is available at https: / git.ub..hb.......... wwt./o////ooooo.ooooe..oao.o / PAS.
Grordd trch
Orginal lampling (rectie
de
Orheggool unil lock
13
daaa
d, 1/
Figure I. PCA-based sampling correction. We first utilize PCA to obtain few orthogonal unit vectors that span the space of the sampling trajectories, and then learn the coordinates correct the sampling directions in regions of large curvature along the ground truth trajectory.
& Xie, 2023), text-to-image generation (Rombach al., 2022; Betker et al., 2023), video generation (Dehghani et al., 2023), and speech synthesis (Song et al., 2022), garner- ing widespread attention. DPMs introduce noise into the data through a forward process and then generate the ac- tual output by iterative denoising during the reverse process. Compared to other generative models such as Generative Adversarial Networks (GANs) (Goodfellow al., 201 4) and Variational Autoencoders (VAEs) (Kingma & Welling, 2013), DPMs offer advantages in generating high-quality outputs and maintaining training stability. However, the denoising process in DPMs often requires hundreds or thou- sands of iterative steps, resulting in slow sampling speeds that severely hinder practical applications.
1. Introduction
Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021b; Karras et al., 2022) have demonstrated im- pressive generative capabilities in various fields, includ- ing image generation (Dhariwal & Nichol, 2021; Peebles
Existing sampling algorithms for accelerated DPMs be categorized into two main categories: training-free methods and training based methods. Training-fre free methods (Song et al., 2021a; Lu et al., 2022a;b; Bao et al., 2022b; Liu et al., 2022a; Karras et al., 2022; Zhao et al., 2023; Zheng et al., 2023; Zhang Chen, 2023; Wang & Vastola, 2024; Wang et al., 2025) employ analytical solvers reduce dis- cretization errors per sampling iteration, achieving quality comparable 1000-step sampling while requiring few as 20 number of function evaluations (NFE). However, when NFE is less than 10, the accumulated truncation errors in these methods can be significantly magnified, leading to non-convergent sampling, which is ineffective and remains time-consuming. Training- methods (Salimans & Ho,
"Equal contribution 'School of Informatics, Xiamen Univer- sity, China 'Xiamen Truesight Technology Co., Ltd, China 'eyy Laboratory of Multimedia Trusted Perception and Efficient Com- puting, Ministry of Education of China, Xiamen University, China "ttanford University, USA "Shandong University, China. Corre- spondence to: Songzhi Su <ssz& xmu.cdu.cn>.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diffusion Sampling Correction via Approximately 10 Parameters
2022; Liu et al., 2022b; Song et al., 2023; Luo al., 2023; Yin et al., 2024) generally enhance sampling efficiency sig- nificantly, with the potential to achieve one-step sampling that rivals the quality of the original 1000 NFE. Nonethe- less, these methods typically demand high computational resources and require additional model parameters storage. Even for the relatively simple CIFAR 10 dataset, they may require over 100 A100 GPU hours (Salimans & Ho, 2022: Song et al., 2023), posing challenges for practical applica- tions. Moreover, training-based methods often establish new paths between noise and data distributions, disrupting the interpolation capability between two disconnected modes.
2. Background
2.1. Forward and Reverse Processes
The goal of diffusion probability models (DPMs) (Sohl- Dickstein et al., 2015; Ho et al., 2020; Song Ermon, 2019; Song et al., 2021b; Karras et al., 2022) is to generate D- dimensional random variables 2i E R.D that follow the data distribution Gdata (2o). DPMs add noise to the data distribution through a forward diffusion process; given Fo. the latent variables (1 E [RP)
are defined as:
g( 20) = N ayzo, oI
(I)
To address these issues, we propose PCA-based Adaptive Search (PAS), a plug-and-play method that can correct the truncation errors of existing fast solvers with minimal training costs and small number of learnable parame- ters. Additionally, PAS retains the interpolation capabil- ity between two disconnected modes. Inspired by previ- ous observation that the sampling trajectories of DPMs lie in a www-dimensional subspace within a nigh-diimesional space (Zhou et al., 2024; Chen et al., 2024; Wang Vastola, 2023; 2024), we propose employing Principal Component Analysis (PCA) obtain few orthogonal unit basis vec- tors in the -wwdimensional subspace where the sampling trajectories lie, then learning the corresponding coefficients (i.e., coordinates) to determine the optimal sampling di- rection. This approach avoids training neural networks to directly produce tigh-dimensional outputs, significantly reducing the number of learnable parameters and training costs. Furthermore, we observe that the accumulated trun- cation errors of existing fast solvers exhibit "S"-shape. We have designed an adaptive search strategy to balance the sampling steps that require correction and the truncation error. This further enhances the sampling efficiency of our method while reducing the amount of parameters required for storage. We validate the effectiveness of PAS on various unconditional and conditional pre-trained DPMs, across five datasets with resolutions ranging from 32 to 512. Results demonstrate that our method can significantly improve the image quality with negligible costs. Our contributions are summarized as follows:
where 04 E R and or E R are scalar functions related to the time step t. Furthermore, Song et al. (2021b) introduced stochastic differential equations (SDE) to model the process:
dr. = [(1)x + g(()dw),
where f(-) E - R. g() + R. and w, E. is the standard Wiener process (Oksendal, 2013). Put together Equation (1) and Equation (2), we can get f() dt and (1) बi d of. Additionally, Song et dt al. (2021b) provided the corresponding reverse diffusion process from time step T to 0 as follows:
log
dož
dz.- [f(t)= (t) log (1x)) dt + g(t)dū,, (3)
where V log is referred to as the score function, which can be estimated through neural networks. Remark- ably, Song et al. (2021b) proposed a probability flow ordi- nary differential equation (PF-ODE) with the same marginal distribution as Equation (3) at any time as follows:
1 dx, -(())=-10 (t) log a (zx) dt.
(4)
Unlike Equation (3), this PF-ODE does not introduce noise into the sampling process, making it a deterministic sam- pling procedure. Due to its simpler form and more effi- cient sampling, is preferred in practical applications over SDE (Song al., 2021b;a; Lu et al., 2022a).
We propose new plug-and-play training paradigm with about 10 parameters for existing fast DPMs solvers as an efficient alternative to the high-cost training-based algorithms, rendering the learnable pa- rameters and training costs negligible.
2.2. Score Matching
To solve the PF-ODE in Equation (4), it is typically neces- sary first employ a neural network so(22, t) to estimate the unknown score function V, log qe (2x). The neural net- work sp is trained using the I.2 loss as follows:
We design an adaptive search strategy reduce correc- tion steps, further enhancing the sampling efficiency of our method and decreasing the stored parameters.
E Exorgaate E (ny=)(a [|se(az, t) - V log (=) (5)
Additionally, Ho et al. (2020) proposed using a noise pre- diction network (,,,,, t) to predict the noise added to x, at time step t. Other literature (Karras et al., 2022; Lu et al., 2022b) suggested using a data prediction network
Extensive experiments across various datasets validate the effectiivneess of the proposed PAS method in en- hancing the sampling efficiency of existing fast solvers.
2

---

Diffusion Sampling Correction via Approximately 10 Parameters
(r 1) to directly predict zi at different time steps l. The relationship among these three prediction networks can be expressed as follows:
1.000
Cumulative percent variance
Cumulative percent varlance
CIFAR10 FFHQ64
0.995
Imag NegeNet64
so s0 t) = coloe,t) t) 20( 1) 24
(6)
01
o
0.990
In this paper, we adopt the settings from EDM (Karras et al., 2022), specifically f(() 0, g(l) V21, derived from Equations (2) to (4), and a2 1, o = t as stated in Equation (1). Furthermore, utilizing the noise prediction network EØ, Equation (4) can be expressed as:
0.985
CIFAR10
FFHQ64
0.2
0.980
ImageNet64
19 20 22 42 50 10 120
Number principal component
Number principal components
(a) Single Sample
(b) K Samples
dx. <0( O)d
We utilize PCA to analyze the sampling trajectories, lustrating the trend of cumulative percent variance the number of principal components varies. The trajectories are obtained from IK samples using the Euler solver (Kloeden al., 1992) in the EDM (Karras et al., 2022) pre-trained model with 100 NFE. (a) The average results of each trajectory fær, (de, 3mm) J. (b) The results of K trajectories { {at, Nim,) K k== (FFHQ and ImageNet
Figure 2.
According to the simple PF-ODE form in Equation (7), using the Euler-Maruyama (Euler) solver (Kloeden et al., 1992), the sampling process from I3 to 4-11 can be repre- sented as:
T1-11 R St, + (4i-1 - 1a:1----ee (x)co(Tt,) 42);
(8)
curves nearly overlap).
where E .N.... .1] and iy= = T, ,to -0-00
performing PCA to decompose the entire sampling tra- jectory {x >i the cumulative percent variance satu- rates rapidly; by the time the number of principal compo- nents reaches 3, the cumulative percent variance approaches nearly 100%. This indicates that the entire sampling tra- jectory lies in diree-dimessioall subspace embedded in high-dimensional space, the finding initially revealed in works (Zhou et al., 2024; Chen et al., 2024; Wang & Vastola, 2023; 2024). Furthermore, according to Equa- tion (8), Tt.-, is a linear combination of It, and de,, al- lowing us to modify the existing trajectory { ... ,",,,,, } to {Iix dex, .......,dd ....,.... This modification enables our method to share buffers during the sampling process when combined with existing multi-step solvers that utilize histor- ical gradients (e.g., PNDM (Liu et al., 2022a), DEIS (Zhang & Chen, 2023), etc.) to optimize memory usage. To val- idate the reasonableness of this modification, we perform PCA on complete sampling trajectory {IT, {de, J ). with the resulting cumulative percent variance illustrated in Figure 2a, showing that three principal components suffice to span the space occupied by the entire sampling trajectory. Notably, the sampling trajectories of different samples do not lie in the same fitee-dimessional subspace. We apply PCA to decompose the set of K sampling trajectories from samples { )2 K k=11 with the resulting cumulative percent variance displayed in Figure 2b. We observe that the cumulative percent variance does not show saturation trend as the number of principal components increases. A theoretical analysis explaining why the sampling trajectories lie in low-dimensional subspaces, and why different samples occupy distinct subspaces, is provided in Section 3.4.
3. The Proposed PAS Method
3.1. PCA-based Sampling Correction
Utilizing the Euler solver (Kloeden al., 1992) with Equa- tion (8) to approximate Equation (7) introduces notable discretization errors that can become significantly amplified with a limited number of iterations. The exact solution of Equation (7) is given by:
St. 1 = Lt, + f're co t)dl.
Let sampling direction de, := co(ze, ts). Existing fast solvers reduce discretization errors through various numer- ical approximations. For example, the PNDM (Liu al., 2022a) employs linear multi-step methods, while the DPM- Solver (Lu et al., 2022a;b) utilizes Taylor expansion to cor- rect the sampling direction de, in Equation (8) to approach the exact solution ph:- co (x,, t)tt. Training- based meth- ods (Song et al., 2023; Salimans & Ho, 2022; Zhou et al., 2024) typically utilize neural networks to correct the direc- tion de,- In contrast to the aforementioned methods, we extract a few orthogonal unit basis vectors from the high- dimensional space of the sampling trajectory using PCA. By learning the coordinates corresponding these basis vectors, we correct the direction de, to the optimal direction de,, thereby minimizing the training cost.
Specifically, during the iteration process from It, to It,-1" we first extract a set of basis vectors from the space of the existing sampling trajectories { }r, where In = T,.... to 0. surprising finding is that when
Based on this, during the iterative process from It, to Tt;-17
3

---

Diffusion Sampling Correction via Approximately 10 Parameters
0.65
CIFARIO
0.05
CIFARIO
Noise
Data
FFHQ64
FFHQ64
0.04
ImageNlet64
0.04
ImaggeNet64
Grund insh Injeztary
3.05
0.155
Origital samping direction
Comretted sampling direction
0.02
0.007
Origizal
2002
0.01
Carrested statis
0.00
3000
1-1
10
Regin Lang Comvaure
Euler + PAS
(a) Truncation Errors
(b) Corrected Truncation Errors
Figure 3. The truncation errors are evaluated using the Fuler solver both with and without the proposed PAS. We utilize the EDM (Kar- ras et al., 2022) pre-trained model to sample 10k samples and com- pute the average L2 distance of 10 NFE compared the ground truth trajectory (100 NFE). (a) The -shaped truncation error is produced by the Euler solver. (b) The truncation error corrected using PAS. Notably, PAS adaptively corrects only the parts of the sampling trajectory with large curvature.
((TT)
g(x²)
((20)
Figure 4. Illustration of PCA-based Adaptive Search (PAS). We demonstrate the Fuler solver with the proposed PAS method, where sampling directions are derived from the tangent direction of the ground truth trajectory. The details the correction process for the sampling direction are presented in Figure 1.
we decompose the existing sampling trajectory, requiring only the top three basis vectors to span the space of the sam- pling trajectory. Let X~ { dey..... ,.,,,, d where X E I and D denotes the dimension of Tax- When using the top k principal components, the process is described as:
trajectory lies within the span of the basis vectors. Through Schmidt orthogonalization, we can obtain new orthogonal unit basis vectors as follows:
= [u1, U2 U3 un] = Schmidt v', V2, Va), (14)
WEVT = SVD((X, [v.} 1 = Vi::: k),
(10)
where Schmidt represents the Schmidt orthogonalization and U E Dxd consists of four orthogonal unit basis vec- tors. It is noteworthy that increasing a basis vector incurs less computational cost relative to the projection operation, and the additional single parameter ligible. After obtaining the basis vectors U that span the space of the sampling trajectory, we can initialize the learn- able coordinate parameters. Since our goal is to correct the sampling direction de,, and we have already specified the first basis vector u1 = Vi = d lla. we initialize the first coordinate as Ci = I|de, ll2. with the remaining coordinates initialized zero, as follows:
(11)
where SVD denotes the Singular Value Decomposition (SVD) and v, E Px represent orthogonal unit basis vectors. Further, since our goal to correct the current direction di,, we modify the above PCA process by directly specifying Vi = / Ila. The subsequent approach gen- erally involves computing the projection of X onto the basis vector V1, as follows:
be considered neg-
Xv1
projv, (X)
(12)
|
Then apply PCA to decompose X- projv, (X), obtaining the remaining two orthogonal unit basis vectors.
C = [c] = 11d,, =2 0, = 0, = =].
(15)
At this point, we have di. UCT. Through training, we can obtain the optimized C, thereby acquiring the corrected direction d. = UŌ" The specific PCA-based sampling correction schematic illustrated in Figure 1. In summary, we employ PCA to correct the sampling direction, requir- ing only few sets of coordinates as learnable parameters. This approach serves as an efficient alternative to high-cost training-basee algorithms, leveraging the geometric char- acteristics of the sampling trajectory in a low-dimensional space. As a result, it significantly reduces the number of learnable parameters and training costs.
To further optimize computation time, we omit the projec- tion step. After specifying v1= = ddd/Ild,, we modify X
as follows:
X' Concat (x, d.
(13)
where X' E (NN-++)) Subsequently,
decompose X' using Equation (10) to obtain V', and then extract two new basis vectors, v',, v22 = V'I.: 2). Due to the omission of the projection step, the new basis vectors may be collinear with v1. Nevertheless, we only need to add one new basis vector v/g = V' 2), sufficient to ensure that the sampling
4

---

Diffusion Sampling Correction via Approximately 10 Parameters
Algorithm 1 PCA--baed Adaptive Search (PAS)
Algorithm 2 Sampling Correction
1: Input: initial value PT, NEE N,, model 6Ø, given solver @, time steps (4))=N= ground truth trajectory
1: Input: initial value 2y, NFE N, model cø, given solver
&, time steps {4)) N= coordinate
{ [2
J N= tolerance
2: Q
butfer 2T, dey 6a (orr,
2: def PCA(Q, dr.):
3: for i + N to 1 do
3:
WEVT- = SVD(Connaat() Q, d:
4: if i in coordinate_diee deyt. then
4:
V1 = wi=de// /|, ll2: v,, V2, V/3 = V V[; (4:3)
5: C = coordinate coordinate.dict(i], = PCA(Q, dt,) de, UC
5:
u1, U2, u3, 41 Schmidt(V V1. V1, V22 v
6:
Return: [u,, U2 U3 uy
end if
7: Q
buller
IT, dep c (N)
8: - = du, t;-)
8:
for i + N to do
buffer de de, 10(ft:1)1i-1)
9:
Init C1 = |de, 12, C2 = 02 C3 = C4
03 0
10: end for
10:
C= = [c1, C2 C3. Cal, U PCA(Q,d,,)
11: Return: Eto
11:
It.- = o( UC (4i
12:
Ö + C -VVcl - - l1
13:
_t,- = o(x;. UCT t4, 11-1)
3.3. Adaptive Search
14:
Li = بلاسا ot 12. L2
||
- 2t ,12
In Sections 3.1 and 3.2, we introduced how to correct the iterative process from Øt, to It using our method. This section describes how to correct the iterative process from 27 to 20 using our approach. First, need to generate ground truth trajectory {=][ )o to correct {æt, 1o where xxxx Ttn- In this paper, we adopt a widely used polynomial time schedule (Karras et al., 2022) for both sampling and generating the ground truth trajectory, which is expressed as follows:
15:
if L2 (L1 +T) > then
16:
coordinate dict[i]= = C
17:
Tt, It.,. 11 d: UZT
end if
19: Q
buffer de, de,-, = co
20:
end for Return: coordinate dict
21:
3.2. Training and Sampling
4 4- (10// + 1/ (1' - 1//0))0 i (N.... O), (19)
i
N
To correct the update direction de, during the iterative pro- cess from Tt, to 1t, we need to learn the coordinates C in Equation (15) to apply to the sampling trajectory of all samples. First, given any first-order ODE solver o, the discretized solution of Equation (9) can be uniformly repre- sented as follows:
where iN = T.... to = c, and c is a value approach- ing zero To obtain the ground truth trajectory, we simply need to insert more sampling steps into the time schedule from Equation (19) to achieve a more accurate solution. Specifically, consider using a teacher Euler solver with N' NFE guide student Fuler solver with N(< N') NFE during training. First, we insert M values into the time schedule for the student solver, such that M is the smallest positive integer satisfying N N(M + 1) 2 N'. Next, We use Equation (19) to generate the time schedule for the teacher solver: (N(M++1) = T..... to = e. Finally, we only need to index the (1 from the teacher solver us- ing the E ....... .0. thereby obtaining the ground truth trajectory {0 li=N = ( J==N-
Dt., E(I1, de, t1,
(16)
where de UCT co(xi, t.). Given the ground truth Tt-11 we can train the coordinates C using the stochas-
tic gradient descent (SGD) algorithm (Robbins & Monro, 1951), with the L2 loss update process as follows:
Č- - -CC-aVella.... - - 11
(17)
where a denotes the learning rate, and the specific acqui- sition method for x is discussed in Section 3.3. After training C using multiple samples through Equation (17), we obtain the trained coordinates C.
1000 After obtaining the ground truth trajectory {ag Ti=N- we need to sequentially correct dey, de,. This is because once dey is corrected to dry -tx_, will be adjusted accord- ingly =zx This further modifies the next time point di- rection that requires correction: d ==( fN In general, we need to correct N directions sequentially, storing 4N learned coordinate parameters, and correcting n iterative processes during sampling. Nevertheless, to further reduce the additional computational cost during the sampling process and the number of stored coordinate pa-
During the iterative process from Iı, to 1 by utilizing the trained coordinates Č, we can correct the current update direction de, to de = UČT, thereby obtaining more accurate āt; as follows:
Je,, = p(ax,, de, de,, (1)
(18)
5

---

Diffusion Sampling Correction via Approximately 10 Parameters
rameters, we propose an adaptive search strategy. Specifi- cally, the cumulative truncation error of the existing solvers exhibits an "5" "S""shaped trend, as shown in Figure 3a, indi- cating that it initially grows slowly, then increases rapidly, and ultimately returns to a slow growth rate. Thus, we can infer that the sampling trajectory first appears linear, then transitions to a curve, and ultimately becomes linear again under the attraction of a certain mode. Consequently, only the parts of the sampling trajectory with large curvature require correction; the linear sections do not. We employ PCA to obtain the basis of the space containing the sampling trajectory, also aiming to compensate for the missing direc- tions in other bases due to discretization of Equation (9) in cases of large curvature. The specific implementation of the adaptive search determined by the loss of the optimized state. When using L2 loss, we obtain:
tic theoretical analysis explain why these key observations hold. Inspired by Wang et al. (2023; 2024), and acknowl- edging that directly analyzing neural networks is inherently intractable, Wang et al. (2023; 2024) derived an analytical form of Gaussian score structures to approximate diffusion trajectories, and provided both theoretical and empirical support. For the EDM (Karras et al., 2022) trajectories, an approximate analytical form is given:
Tt OT 32 = -
(21)
OT
k=1
where μr and uy represent the dataset-dependent mean and basis vectors that define the data manifold, and 2 denotes the off-manifold component. or and (ht,, A are functions dependent on time l. Given 2T, both or and (T) are constants. From Equation (21), it is evident that 24 is a linear combination of the vectors x7. μ4, and uk. For a fixed dataset, the rate of change of the coefficients for the vectors TTT- 14, and 4y depends only on the timestep t. As time progresses, all samples evolve along their respective off- manifold directions toward the data manifold with the same scaling. This theory heuristically explains why all samples exhibit strongly consistent geometric characteristics. Fur- thermore, Wang et al. (2023) theoretically derived that the diffusion trajectories resemble 2D rotations on the plane spanned by the initial noise øT and the final sample xo. Since FT varies across different samples, the corresponding planes also differ, providing theoretical underpinning for why the trajectories of different samples lie in distinct sub- spaces. At the same time, this also theoretically supports the observation that the sampling trajectories of the samples lie in a low-dimensional (2D) subspace.
L = || 2- 20,,,11 112. L2 = || 2 12 (20)
where o is the corrected state. We introduce a tolerance 7 to determine whether C2 - (Ci + 7) is greater than zero. If it is greater than zero, correction is required for that step: otherwise, the step is considered to lie within the linear part of the sampling trajectory, and no correction is neces- sary. The tolerance r is set to positive value, e.g. 10-4. The truncation error after correction using our algorithm is depicted in Figure 3b, clearly showing a significant reduc- tion in truncation error in the large curvature regions. Now that we have thoroughly presented the proposed PCA-based Adaptive Search (PAS) algorithm, detailing the complete training and sampling processes in Algorithms I and 2. The specific schematic is illustrated in Figure 4.
3.4. Theoretical Insights and Discussion
We attribute the efficiency of the proposed PAS to several key observations about the geometric structure of sampling trajectories: First, the sampling trajectories of the sam- ples lie in low-dimensional subspace (Figures 2a and 6c), which enables PAS to correct the sampling direction effec- tively by learning only a few sets of dow-dimensional coor- dinates corresponding the basis vectors of the sampling subspace. Second, although the sampling trajectories of different samples lie in distinct subspaces (Figure 2b), their geometric shapes exhibit strong consistency (Figures 3a and 6d). The correction coordinates learned by PAS at each time step t; implicitly capture the curvature information of sample trajectories. In other words, for a given dataset, PAS leverages the strongly consistent geometric characteristies learned from small number of samples and generalizes them to all samples within the dataset, thereby achieving efficient sampling correction and acceleration.
3.5. Comparing with Training-based Methods
As discussed in Section 1, while training- based methods can achieve one-step sampling (Salimans & Ho, 2022; Liu et al., 2022b; Song et al., 2023; Yin et al., 2024), they often incur substantial training costs (e.g., exceeding 100 A100 GPU hours simple CIFARIO), 10). Moreover, distillation- based methods tend to disrupt the original ODE trajectories, resulting in the loss of interpolation capability between two disconnected modes, thereby limiting their effectiveness in downstream tasks that rely on such interpolation. Although some low-cost training methods (Kim et al., 2023a; Hsiao et al., 2024; Bao et al., 2022a; Kim et al., 2023b; Na et al., 2024: Zhang et al., 2024; Zhou et al., 2024) have been proposed, as diseussed in Appendix A, these methods still do not address the essential issue and require training new, relatively small neural network.
We have already empirically validated the aforementioned key observations through experiments and prior studies in Sections and 4 and Appendix A. Next, we provide heuris-
In contrast to these methods, PAS introduces a new training paradigm that corrects dimeeniimmiio vectors by learning low-dimensional coordinates, achieving minimal learnable
6

---

Diffusion Sampling Correction via Approximately 10 Parameters
v1.4 (Rombach et al., 2022) belongs to the conditional latent- space pre-trained model.
Table 1. On the CIFARIO, the list of numbers denotes all the time points requiring correction from adaptive search for the DDIM and iPNDM solvers, with each element ranging from NFE (N) to I.
We provide comparative results from previ- ously state-of-the-an fast solvers, including DDIM (Song et al., 2021a), DPM-Solver-2 (Lu et al., 2022a), DPM- Solver++ (Lu et al., 2022b), DEIS-IAB3 (Zhang & Chen, 2023). UniPC (Zhao al., 2023), DPM-Solver-v3 (Zheng et al., 2023), and improved PNDM (iPNDM) (Liu et al.. 2022a; Zhang & Chen, 2023).
Solvers.
NFE
Method
5
8
10
DDIM + PAS
3,1 2
4,2.1
5,3,2
6,4.2
iPNDM + PAS
3,1
4,2
parameters and training costs, For instance, using a single NVIDIA A100 GPU, training on CIFARIO takes only 0-2 minutes, and merely 10~20 minutes on datasets with a maximum resolution of 256. Notably, the computational cost for PCA is negligible compared that of NFE. For example, in Stable Diffusion v1.4, one NEE takes approx- imately 30.2 seconds for 128 samples, whereas one PCA computation takes only 0.06 seconds. Additionally, based on adaptive search, PAS only requires correcting I~3 time points on the CIFARIO, as shown in Table 1 (results for additional datasets, see Table 6 in Appendix C.I). This means that PAS only requires 4- 12 parameters during the sampling correction process. This is not in the same order of magnitude as the aforementioned training- based methods. Furthermore, PAS preserves the original ODE trajectories, thereby retaining the interpolation capability of DPMs
Evaluation. We evaluate the sample quality using the widely adopted Fréchet Inception Distance (FID.) (Heusel et al., 2017) metric. For Stable Diffusion, we sample 10k samples from the MS-COCO (Lin et al., 2014) validation set to compute the FID, while for other datasets, we uniformly sample 50k samples.
Training, In Section 4.3 and Appendix C.2, we present ab- lation experiments related to the hyperparameters involved in training, and Appendix B provides detailed training con- figurations for different datasets that we used. Below, we outline some recommended settings for training hyperpa- rameters: utilizing Heun's 2nd solver (Karras et al., 2022) from EDM to generate 5k ground truth trajectories with 100 NFE, employing the L1 loss function, setting the learning rate to 10 and using a tolerance T of 10 10-4.
4. Experiments
4.2. Main Results
To validate the effectiveness of PAS a plug-and-play and low-cost training method, we conducted extensive experi- ments on both conditional (Karras et al., 2022; Rombach et al., 2022) and unconditional (Karras et al., 2022; Song et al., 2023) pre-trained models.
In this section, we present the experimental results of PAS across various datasets and pre-trained models with NFE E {5, 6, 8, 10]. In Table 2, we report the experimental results of PAS correcting DDIM and iPNDM solvers, cover- ing the CIFARIO, FFHQ, ImageNet, and LSUN Bedroom datasets. Notably, for the LSUN Bedroom, the order of iPNDM is set 2: for the other datasets, the order is set to 3, which yields better average performance (for more results regarding the order of iPNDM, see Appendix C.3). Experi- mental results demonstrate that PAS consistently improves the sampling quality of both DDIM and iPNDM solvers, regardless of image resolution (large or small) or model type (conditional or unconditional prained. Particularly, PAS combined with iPNDM surpasses the previously state- of-the-art solvers. Furthermore, we compared PAS with previous works that enhance sampling efficiency through trajectory regularity on the CIFAR10 and FFHQ datasets, including AMED (Zhou et al., 2024) and GITS (Chen et al., 2024). The results demonstrate that PAS provides a more significant improvement. To further explore the potential of PAS, we employ the analytical approach proposed by Wang et al. (2023) "warm up" the basis vectors from the Gaus- sian score structure, and then perform sampling correction starting from the teleported solution. The results combining teleportation (TP) with skip = 10.0 and PAS are presented in Table 2, demonstrating the enhanced performance and
4.1. Settings
In this paper, we adopt the design from the EDM frame- work (Karras et al., 2022) uniformly, as shown in Equa- tion (7). Regarding the time schedule, to ensure a fair comparison, we consistently utilize the widely used poly- nomial schedule with p 7 in Table 2, described in Equation (19).
Datasets and pre-trained models. We employ PAS across a wide range of images with various resolutions (from 32 to 512). These include CIFAR10 32x32 (Krizhevsky et al., 2009), FFHQ 64x64 (Karras et al., 2019), ImageNet 64 64 (Deng et al., 2009), LSUN Bedroom 256x255 (Yu et al., 2015), and images generated by Stable Diffusion v1.4 (Rombach et al., 2022) with 512 resolution. Among these, the CIFAR1O. FFHQ, and LSUN Bedroom datasets are derived from the pixel-space unconditional pre-trained models (Karras et al., 2022; Song et al., 2023); the Ima- geNet comes from the pixel-space conditional pre- trained model (Karras et al., 2022); and the Stable Diffusion
7

---

Diffusion Sampling Correction via Approximately 10 Parameters
Table Sample quality measured by Fréchet Inception Distance (FIDI) on CIFARIO, FFHQ. ImageNet, and LSUN Bedroom datasets. "\" indicates missing data due to the inherent character- isties of the algorithm.
Table 3. Sample quality measured by FID, on Stable Diffusion v1.4 with a guidance scale of 7.5. Te borrow the results reported in Zheng et al. (2023) directly.
NHE
Method
Method
NET:
5
6
8
10
8
DDIM (Song, et al., 202
23.42
20.08
17.72
16.56
CIFARIO 32 (Krizhevsky et al., 2009)
DPM-Solver++ (Lu et al., 20226)
18.87
17.44
16.40
15.93
UniPC (Zhao et al., 2023)
18.77
17.32
16,20
16.15
DDIM (Sang et al.,, 2021a)
49.68
35.63
22.32
15.69
D0 (Zheng et al., 2023)
18.83
16.41
15.41
15.32
DDIM AMED (7hou at., 2024)
25.15
17.03
11.33
DDIM PAS (Ours)
17.70
13.93
14.74
14.23
DDIM + GITS (Chen et al., 2024)
28.05
21.04
13.30
10.37
DDIM + TP (Wang& & Vastola, 2024)
24.50
18.41
12.04
8.78
DDIM PAS (Ours)
17.13
12:11
7.07
4.37
DDIM + TP + PAS (Ours)
9.15
5.16
3.65
3.16
PAS when combined with DDIM surpasses the sampling re- sults of previous state-of-the-an methods, further validating the effectiveness of the proposed PAS.
DPM-Solver-2 et al. 2022n)
60.00
10.30
5.01
DPM-Solvers+(3M) (Lu et al., 20226)
31.65
17.89
8,30
5,16
DEIS-ABB (Zhang& & Chen, 2023)
17.65
11.84
6.82
5.64
UniPC(3M) (Zhao et 2023)
31.44
17.74
8.42
5.31
iPNDM (Zhang & Chen, 2023)
16:55
9.74
5.23
3.69
In Figure 5 and Appendix C.4, we present the visualization results for Stable Diffusion, as well as for the CIFARIO. FFHQ, ImageNet, and Bedroom datasets. The results in- dicate that the samples generated by PAS exhibit higher quality and richer detail. Through the experiments above, we have demonstrated that PAS serves as plug-and-play, low-cost training method that can effectively enhance the performance of existing fast solvers (including DDIM and iPNDM), thereby validating the effectiveness of PAS.
iPNDM TP (Wang & Vastola, 2024)
7.25
4.89
3.08
2.49
1PNDM (Ours)
13.61
7.47
3.87
2.84
iPNDM TP PAS (Ours)
5,16
3.76
2.77
2.40
FFRQ 64 64 (Kamas et al., 2019)
DDIM (Song al., 2021a)
43.92 )
35.21
24.38
18.37
DDIM + AMED (7hou et al., 2024)
32.46
20.72
15.52
DDIM + GITS (Chen et al., 2024)
29.80
23.67
16.60
13.06
DDIM TP (Wang& & Vastola, 2024)
26.61
21.34
15.11
11.69
DDIM + PAS (Ours)
29.07
17.63
8.16
5.61
DDIM TP PAS (Ours)
10.37
7.37
4.54
4.50
DPM- (Lu al. 2022a)
83.17
22.84
9.46
DPM-Solver++(3M) (Lu ct al., 20226)
23.50
14.93
9.58
6.96
DEISAAB3 (Zhang& & Chen, 2023)
19:47
11.61
8.64
7.07
UmPCC(M) (Zhao al., 2023)
22.82
14.30
10.07
7.39
4.3. Ablation Study
iPNDM (Zhang & Chen, 2023)
17.26
11.31
6.82
4.95
iPNDM TP (Wang & Vastola, 2024)
10:26
7.28
4.83
3.94
In this section, we conduct ablation experiments on sev- eral key modules used during the training process, as illus- trated in Figure 6. Additionally, ablation experiments on the learning rate, the solver for generating trajectories, and the tolerance 7 are presented Appendix C.2. Notably, the selection of these modules in the overall PAS is not critical factor and only has a slight impact on performance.
iPNDM + PAS (Ours)
15.89
10.29
5.85
4.28
iPNDM TP PAS (Ours)
8.20
5.49
3.99
3.44
ImageNNtt 64 64 (Deng et al., 2009)
DDIM (Song et al., 2021a)
43.81
34.03
22.59
16.72
DDIM PAS (Ours)
31.37
26.21
12.33
9.13
2PM-Solver-2 (Lu al., 2022n)
44.83
12.42
6.84
DPM-Solver++(3M) (Lu et al., 20226)
27.72
17.18
8.88
6.44
DEIS-AB3 (Zhang& & Chen, 2023)
21.06
14.16
8.43
6.36
UmPPC/3M (Zhao et 2023)
27.14
17.08
9.19
6.89
iPNDM (Zhang Chen, 2023)
19.75
13.48
7.75
5.64
Adaptive search. In Figure 6a, we present the experimental results comparing the PAS with and without the adaptive search strategy (-AS). The findings reveal that the sampling quality of the PAS(-AS) is even inferior to DDIM. This degradation may be attributed the linear segments in the sampling trajectory, where the errors from DDIM are negli- gible. However, the PCA-based sampling correction does not reduce error and instead introduces biases in other basis vectors. This further validates the necessity and effective- ness of the proposed overall PAS.
iPNDM + PAS (Ours)
23.33
12.89
7.27
5.32
LSUN Bedroom 256 256 (Yu et al., 2015) (picel-space)
DDIM (Song al., 2021a)
34.34
25.25
15.71
11.42
DDIM + PAS (Ours)
28.22
13.31
7.26
6.25
DPM Salver (Lu al., 2022a)
80.59
23.26
9.61
DPM Salvere- (I.i et al., 2022h)
17.39
11.97
9.86
7.11
DEIS-ABB3 (Zhang & Chen, 2023)
16.31
11.75
7.00
5.18
UnPCC3M) (Zhao et al., 2023)
17.29
12.73
11.91
7.79
iPNDM (Zhang & Chen, 2023)
18.15
12.90
7.98
6.17
iPNDM + PAS (Ours)
15.48
10.24
6.67
5.14
strong plug-and-play potential of PAS. For Stable Diffusion, we report the experimental results of PAS correcting DDIM. Additionally, we introduce previous state-of-the-an meth- ods, including Sole--oler++, UniPC, and IPMM-Sllve--3,, for comparison. These methods utilize their optimal con- figurations on Stable Diffusion, including the 2M version and logSNR schedule (Lu et al., 2022a), etc. As shown in Table 3, PAS significantly improves the sampling quality of DDIM on Stable Diffusion. Notably, the performance of
Loss function. We evaluated the L1, L2. and previously established effective loss functions: LPIPS (Zhang et al., 2018) 8) and Pseudo- Huber (Song & Dhariwal, 2024). Here, the hyperparameter for the Pseudo- as recommended by Song et al. (2024). The results are presented in Figure 6b. Surprisingly, LPIPS exhibits the lowest average performance, possibly because it was trained on clean images rather than noisy ones. Overall, the L, loss function demonstrates superior average performance, which
set to 0.03,
8

---

Diffusion Sampling Correction via Approximately 10 Parameters
Text Prompt: village, shrouded by fierce blizzard night.
CIFARIO-32
FFHO-64
ImageNet-64
Bedroom-266 -266
+ SHSS
DDIM NFE-6, FID-20.08
DDIM + PAS (Ours) NFE-6, FID-15.93
DDIM NFE-100, FID-15.34
NFE=
NP
NFR=
NER=5
Figure 5. Visualization results using DDIM with and without the proposed PAS. Left: Sampling results Stable Diffusion v1.4 with guidance scale of 7.5. Right: Sampling results on the CIFAR1O, FFHQ 64 x64, ImageNet x 64, and LSUN Bedroom 256 256 datascts.
150
DOIM
DDIM
DDIM PAS, bosk::2
DDIM
80
PAS, Io
60
PAS,
trajectoces
PAS (-AS)
FAS, banis-3
trajecturian
100
PAS. LPIPS
PAS, bush-m4
PAS,, trajectoriss 5k
RD
603
FID
40
FID
40
PAS. trajectoriss- 20k
trajecturion-
20
20
01
NFEE
NFE
NFE
NFE
(a) Adaptive Search
(b) Loss Function
(c) Number of Basis Vectors
(d) Number of Trajectories
Figure 6. Ablation study CIFARIO, utilizing PAS correct DDIM, exploring the impact of adaptive search, loss function, the number of orthogonal unit basis vectors, and the number ground truth trajectories on FID (recommended setting: the red solid line).
may be attributed to its larger scale.
5. Conclusion
Number of basis vectors. We demonstrate the ablation results in Figure 6c, by varying the number of basis vectors used. Experimental results indicate that PAS can signifi- cantly improve the sampling quality of DDIM using only the top 2 basis vectors, while employing the top tors yields slightly better performance. Notably, the experi- mental results in Figure 6c exhibit the same trend as those in Figure 2a, further validating that the sampling trajectory of DPMs lies in oiw-dimesiional subspace. Four basis vectors suffice to span the sampling trajectory space, en- abling PAS to achieve minimal training costs and learnable parameters compared to other training-based algorithms.
In this paper, we introduced novel training paradigm, PAS, for accelerating DPMs with minimal training costs and learn- able parameters. Our key strategy is obtain few basis vectors via PCA. and then learn their low-dimensional coor- dinates to correct the high- nigh-dimensional sampling direction vectors. Moreover, based on the observation that the trunca- tion error of existing fast solvers exhibits an "S"'shape, we design an adaptive search strategy to balance the correction steps, further enhancing sampling efficiency and reducing the number of stored parameters approximately 10. Ex- tensive experiments on both unconditional and conditional pre-trained DPMs demonstrate that PAS can significantly improve the sampling quality of existing fast solvers, such as DDIM and iPNDM, in a plug-and-play manner.
vec-
Number of trajectories. In Figure 6d, vary the number of ground truth trajectories from 500 to 20k. We find that learning coordinates from few as 500 trajectories can significantly enhance the sampling quality of DDIM. This demonstrates that the sampling trajectories of all samples exhibit strong consistent geometric characteristics, specif- ically the "S"-shaped truncation error. This also explains why coordinates can effectively adapt to samples within single dataset. However, increasing the number of trajectories generally results in more generalized learned coordinates, with 5k trajectories being the optimal balance,
In future work, we plan to further investigate the regular- ity of diffusion trajectories, focusing on leveraging their geometric characteristics to enable rapid sampling at minimal cost. For instance, given the low dimen- sionality and strong consistency in the geometric shapes of sampling trajectories across different samples, it is promis- ing to learn the integrated form of analytical trajectory equa- tions from small number of samples, thereby enabling one-step sampling with minimal cost.
one-step

---

Diffusion Sampling Correction via Approximately 10 Parameters
Impact Statement
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Diffusion models can generate images, audio, and video content that is indistinguishable from human-created me- dia. This work proposes an accelerated sampling algo- rithm based the geometrie characteristics of the diffusion model's sampling trajectory. However, this method could potentially be misused, accelerating the generation of ma- licious content, such as fake news or manipulated images, leading to detrimental societal impacts. We are fully aware of this potential risk and plan to focus on the discrimina- tion of generated content in future research to mitigate its possible negative effects on society.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 18556,0-6851,, 2020.
Hsiao, Y.-T., Khodadadeh, S., Duarte, K., Lin, W.-A., Qu, H., Kwon, M., and Kalarot, R. Plug-and-play diffusion distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13743- 13752, 2024.
References
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 4401-4410, 2019.
Bao, F., Li, C., Sun, J., Zhu, J., and Zhang, B. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. In International Conference on Machine Learning, pp. 1555-1584. PMLR, 2022a.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion- generative models. Advances in neural information processing systems, 35: 26565-26577 2022.
Bao, F., Li, C.,, Zhu, J., and Zhang, B. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffu- sion probabilistic models. In International Conference on Learning Representations, 2022b.
Kim, B.-K., Song, H.-K., Castells, T., and Choi, S. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. arXiv preprint ar3iv:2305.. 15798, 2023a.
Betker, J., Goh, G., Jing, L.,, Brooks, T., Wang. J., Li, L.., Ouyang, L., Zhuang. J., Lee, J., Guo, Y., et al. Improving image generation with better captions. Computer Sci- ence. https://cdn. openai. com/papers/dallle-3 pdf. 2(3): 8, 2023.
Kim, D., Kim, Y., Kwon, S. J., Kang, W., and Moon, I.-C. Refining generative process with discriminator guidance in score-based diffusion models. In International Con- ference on Machine Learning, pp. 16567-16598. PMLR, 2023b.
Chen, D., Zhou, Z., Wang, C., Shen, C., and Lyu. S. On the trajectory regularity ode-based diffusion sampling. In Forty-first International Conference on Machine Learn- ing, 2024.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Dehghani, M., Mustafa, B., Djolonga, J., Heek, J., Minderer, M., Caron, M., Steiner, A., Puigcerver, J., Geirhos, R., Alabdulmohsin, I., al. Patch pacck: Navit, vision transformer for any aspect ratio and resolution. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, pp.
Kloeden, P. E., Platen, E., Kloeden, P. E., and Platen, E.
Stochastic differential equations. Springer, 1992.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. Technical Report, 2009.
2023.
Li, I.,, Li, H., Zheng, X., Wu, J., Xiao, X., Wang, R., Zheng, M., Pan, X., Chao, F., and Ji, R. Autodiffusion: Training- free optimization of time steps and architectures for au- tomated diffusion model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7105-7114, 2023.
Deng, J., Dong, W., Socher, R., L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Farle, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
10

---

Diffusion Sampling Correction via Approximately 10 Parameters
Liu, L., Ren, Y., Lin, Z.,, and Zhao, Z. Pseudo numerical methods for diffusion models manifolds. In Interna- tional Conference on Learning Representations, 2022a.
Shih, A., Belkhale, S., Ermon, S., Sadigh, D., and Anari, N. Parallel sampling of diffusion models. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 4263 020334426,440 2023.
Liu, X., Gong, C., et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Repre- sentations, 2022b.
Si, C., Huang, Z., Jiang, Y., and Liu, Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVV Con- ference Computer Vision and Pattern Recognition, pp. 4733-4743, 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C.. and Zhu, J. Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 5775-5787, 2022a.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.0100 1.01055, 2022b.
Song, K., Leng, Y., Tan, X., Zou, Y., Qin, T., and Li, D. Transcormer:
Transformer for sentence scoring with slid- ing language modeling. Advances in Neural Information Processing Systems, 15:11160-11174, 2022.
Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Song, Y. and Dhariwal, P. Improved techniques for train- ing consistency models. In The Twelfth International Conference on Learning Representations, 2024.
Ma, X., Fang, G., and Wang, X. Deepcache: Acceler- ating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
Na, B., Kim, Y., Park, M., Shin, D., Kang, W., and Moon, I.c. Diffusion rejection sampling. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Oksendal, B. Stochastic differential equations: an intro- duction with applications. Springer Science Business Media, 2013.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In International Conference on Machine Learning, pp. 3221 1-2222, PMLR, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 4195-4205, 2023.
Wang, B. and Vastola, J. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. Transactions on Machine Learning Research, 2024.
Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, pp. 400- 407, 1951.
Wang. B. and Vastola, J. J. Diffusson models generate images like painters: an analytical theory of outline first, details later. arXiv preprint arXiv:2303.0244
2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and paltern recognition, pp. 10684-10695, 2022.
Wang, G., Cai, Y., Peng, W., Su, s.-.., et al. Pfdiff: Training- free acceleration of diffusion models combining past and future scores. In The Thirteenth International Conference Learning Representations, 2025.
Salimans, T. and Ho, J. Progressive distillation for fast sam- pling of diffusion models. In International Conference on Learning Representations, 2022.
Wimbauer, F. Wu, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., Cache me if you can: Accelerating diffusion models

---

Diffusion Sampling Correction via Approximately 10 Parameters
through block caching. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 62116112222 0211-6220, 2024.
Xia, M., Shen, Y., Lei, C., Zhou, Y., Zhao, D., Yi, R., Wang, W., and Liu, Y.-J. Towards more accurate diffusion model acceleration with timestep tuner. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5736-5745, 2024.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024.
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint
2015.
Zhang, J., Liu, D., Park, E., Zhang, S., and Xu, C. Resid- ual learning in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7289-7299, 2024.
Zhang, Q. and Chen, Fast sampling of diffusion models with exponential integrator. Proceedings of Machine Learning Research, 2023.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effecctiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. Unipe: unified predictor-corrector framework for fast sampling of diffusion models. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Systems, pp. 49842-49869, 2023.
Zheng, K., Lu, C., Chen, J., and Zhu, J. Dpm-solver-v3: improved diffusion ode solver with empirical model statis- tics. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 55502- 55542, 2023.
Zhou, Z., Chen, D.,. Wang, C., and Chen, C. Fast ode- based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7777-7786, 2024.
12

---

Diffusion Sampling Correction via Approximately 10 Parameters
A. Related Works
Table 4. Training settings for learning rate (I.R), loss function (Loss), number of ground truth trajectories (Trajectory). and tol- erance T (Tolerance) when applying PAS to correct DDIM (Song et al., 2021a) and iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) solvers across various datasets and pre-trained models.
Trajectory-base methods. Recently, several works (Zhou et al., 2024; Chen et al., 2024; Wang & Vastola, 2023; 2024) have designed efficient sampling methods by analyzing the geometric properties of the sampling trajectories of diffu- sion models. For instance, Zhou et al. (2024) observed that the sampling trajectories of diffusion models lie in low- dimensional subspaces, thereby utilizing the Mean Value Theorem to reduce the output dimensions of neural net- works and optimize training costs. Chen et al. (2024) found that the shape of the sampling trajectories exhibits strong consistency, and based on the shape of the trajectories, they employed dynamic programming algorithms to design more efficient sampling schedules. The proposed PAS method builds upon these in-depth studies of the geometric proper- ties of sampling trajectories, particularly the characteristic that the trajectories of diffusion models lie within a low- dimensional subspace embedded in nigh-dimensional space. In contrast the aforementioned methods, PAS innovatively proposes using PCA to obtain few basis vectors of the sampling space, and further corrects dige-dimensional vec- tors by learning their low-dimensional coordinates, thereby significantly reducing additional computational costs. Fur- thermore, we observe that the truncation errors of existing fast solvers exhibit an "S" "S""shape, leading to the proposal of an adaptive search strategy to further reduce overhead.
Method (+ PAS)
I.
Loss
Trajectory
Tolerance
CIFARIO 32 (Krizhevsky et al., 2009)
DDIM (Song al., 2021a)
10-2
L1
10k
10-2 10-1
iPNDM (Zhang & Chen, 2023)
L
5k
FFHQ 64x 64 (Karras et al., 2019)
DDIM (Song et al., 202
2
L1
10k
10 2
iPNDM (Zhang Chen, 2023)
5k
10-4
ImageNet 64 x 64 (Deng et al., 200M)
DDIM (Song al., 2021a)
10
L1
10k
10-2
iPNDM (Zhang & Chen, 2023)
10
L2
5k
10
LSU Bedroom 256 x 256 (Yu et al., 2015)
DDIM (Song et al., 2021a)
10-2 2
10-2 2
iPNDM (Zhang Chen, 2023)
10 2
5k
10 4
Stable Diffusion 512x512 (Rombach et al., 2022)
DDIM (Song et al., 2021a)
10
L1
5k 10-2
scores to efficiently utilize information and thus reduce discretization error. Ma et al. (2024) and Wimbauer et al. (2024) reduced the computational load of neural net- works by caching their low-level features. Shih et al. (2023) suggested utilizing more computational resources and im- plementing parallelized sampling processes to shorten sam- pling times. Orthogonal to these studies, the proposed PAS method introduces a new orthogonal axis for accelcrated sampling in DPMs, which can be further integrated with these approaches to enhance the sampling efficiency of ex- isting fast solvers
Low-cost training. Previous studies (Salimans & Ho, 2022; Liu et al., 2022b; Song et al., 2023) have shown that directly learning the mapping between noise and data distributions necessitates high training costs for minimal-step sampling. Recently, several low-cost training methods have been pro- posed. Kim al. (2023a) and Hsiao al. (2024) explored how to reduce the number of parameters in student models to achieve efficient distillation. Bao et al. (2022a), Kim et al. (2023b), and Na et al. (2024) corrected errors arising during the sampling process by training smaller neural net- works. Zhang et al. (2024) suggested training the neural network only for the last step of sampling to eliminate ac- cumulated residuals, thereby reducing training costs. How- ever, these methods typically still require training a new, relatively small neural network. Unlike these approaches, the proposed PAS method requires learning only a few sets of coordinates, which results in minimized learnable param- elers and training costs.
B. Training Details and Discussion
In this section, we provide training details regarding the PAS correction for different solvers (including DDIM (Song et al., 2021a) and iPNDM (Liu al., 2022a; Zhang & Chen, 2023)) across various datasets. Unless mentioned in the ablation experiments or special notes, all experimental set- tings related to training are based on what is described in this section. First, we outline some common experimen- tal settings: we use Heun's 's 2nd solver from EDM (Karras et al., 2022) with 100 NFE to generate ground truth trajecto- ries. We uniformly apply four orthogonal unit basis vectors (where u =de+: / /I|de443 l12) correct the sampling di- rections. Notably, for the PCA process in Equation (10), we utilize the torch. pcca_pclll lpc_alowrreaapppppppppppppppppaa owwank function to obtain the basis vectors, as it offers a faster computation speed compared to torch. svd. Second, other hyperparameters such as learning rate, loss function, the number of ground truth trajectories, and tolerance 7 are specified in Table 4.
Plug-and-play acceleration. Numerous studies (Ma et al., 2024; Wimbauer et al., 2024; Li al., 2023; Chen et al., 2024; Shih al., 2023; Si al., 2024; Xia et al., 2024; Wang al., 2025) have explored ways to accelerate existing fast solvers, such as DDIM (Song et al., 2021a), DPM- Solver (Lu et al., 2022a;b). PNDM (Liu et al., 2022a), and DEIS (Zhang& & Chen, 2023). Specifically, Wang et al. (2025) proposed the combination of past and future
13

---

Diffusion Sampling Correction via Approximately 10 Parameters
Table Sample quality measured by Fréchet Inception Distance (FIDI) on CIFARIO 32x32 (Krizhevsky et al., 2009), FFHQ 64x64 (Kar- ras et al., 2019) datasets, varying the number of function evaluations (NFE) from 4 to 10. "\\ indicates missing data due to the inherent characteristics of the algorithm.
NFE
Method
10
CIFARIO 32 x2 (Krizhevsky al, 2009)
(Song 2021a)
DDIM PAS (Ours)
41.14
17.13
12.11
11:77
7.07
5.56
4.37
Heun's 's 2nd (Karms et al., 2022)
319.87
99.74
38.06
15.93
DPM-Solver22 (Lu al., 2022a)
145.98
60.00
10.30
5.01
DPM-Salver-+(3M) (Lu el 2022b)
50.39
31.65
17.89
11.30
8.30
6.45
5.16
IABS (7hang & Chen, 2023)
47.13
17.65
11.84
10.89
6.82
6.21
5.64
UnIPC(33M (Zhao et al., 2023)
49.79
31.44
17.74
11.24
8.42
6.69
5.31
iPNDM (Zhang Chen, 2023)
9.74
6.92
5,23
4.33
3.69
iPNDM PAS (Ours)
27.59
13.61
7.47
5.59
3.87
3.17
2.84
FFTQ 64 (Karms al., 2019)
DDIM (Song 2021a)
24.38
21.01
DDIM PAS (Ours)
39.09
29.07
17.63
12:47
8.16
8.26
5.61
2nd (Karms et 2022)
344.87
142.39
57.21
29.
DPM-Solver-2 (Lu al., 2022n)
238.57
83.17
22.84
9.46
DPM-Solver--(MM) (Ta 2022h)
39.50
23.50
14.93
11.04
9.58
8.36
6.96
DEIS-tBB3 (Zhang Chen, 2023)
35.34
11.61
11.70
8.64
7.72
7.07
UnIPC(3M) (Zhao et 2023)
2023)
38.60
22.82
14.30
10.90
10.07
9.00
7.39
iPNDM (Zhang Chen, 2023)
17.26
11.31
8.56
6.82
5.71
4.95
iPNDM PAS (Ours)
41.89
15.89
10.29
7.59
5.85
4.88
4.28
Regarding the aforementioned hyperparameter settings, we conducted extensive ablation experiments in Section 4.3 and Appendix C.2 to elucidate the rationale behind these choices. Furthermore, note that the impact of hyperparameter set- tings the correction of DDIM is not critical factor, DDIM exhibits substantial truncation error; regardless of how hyperparameters are configured, PAS significantly en- hances the sampling quality of DDIM. In contrast, since the iPNDM solver's sampling quality is already relatively high, certain hyperparameters need to be adjusted when using PAS to achieve better FID scores. Nevertheless, due to the extremely low training cost of PAS (requiring only 0~2 minutes on single NVIDIA A100 GPU for the CIFAR10 10 and 10~20 minutes for larger datasets at a resolution of 256), we can easily conduct hyperparameter searches. Coin- cidentally, the final training loss can serve as reference for assessing the effectiveness of hyperparameter choices.
even the FID score does not improve during the correction of the iPNDM solver, the L, and L2 metrics show improve- ment, as demonstrated in Table 11. This further corroborates the effectiveness of the proposed PAS plug-and song-and-playy -layppaay correction algorithm.
C. Additional Experiment Results
In this section, we present additional experimental results on NFE, corrected time points, the order of iPNDM, ablation studies, and visualization studies. Except for the ablation experiments and specific clarifications, the experimental setup and training details are consistent with Section 4 and Appendix B.
C.1. Additional Results on NFE and Corrected Time Points
Regarding hyperparameter search recommendations for solvers with significant truncation errors (e.g., DDIM), we suggest using learning rate of 10-2, the L1 loss func- tion, 5k ground truth trajectories, and a tolerance 7 of 10-2 which generally applies to all datasets. Conversely, for solvers with smaller truncation errors (e.g., iPNDM (Liu al., 2022a; Zhang & Chen, 2023)), we recommend con- ducting learning rate search from 10-1 to 10 for different datasets, while fixing the L2 loss function, using 5k ground truth trajectories, setting the tolerance to 10-4
In this section, we first extend the FID results on the CIFARIO 32x32 (Krizhevsky et al., 2009) and FFHQ 64x64 (Karras et al., 2019) datasets for more values of NFE E [4, 6, 7, 10). The results of PAS correcting DDIM (Song et al., 2021a) and iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) with the order of 3 shown in Ta- ble 5 (for more results regarding the order of iPNDM, Appendix C.3). The results indicate that PAS can signifi- cantly improve the sampling quality of DDIM and iPNDM across different NFE.
It is important to emphasize that the mainstream evaluation metric for the quality of generated samples FID; however, there no corresponding FID loss function. Therefore, when training coordinates using the L1 or La loss functions,
Additionally, in Table 6, we present the time points cor- rected by PAS for the DDIM and iPNDM solvers across various datasets, which correspond to Tables 2 and 3. From
14

---

Diffusion Sampling Correction via Approximately 10 Parameters
Table 6. The list of numbers denotes all the time points requiring correction from adaptive search for the DDIM (Song al., 2021a) and iPNDM (Liu et al., 2022a: Zhang & Chen, 2023) solvers, with each element i ranging from NFE (N) to 1.
Table 7. Ablation study regarding adaptive search conducted on the CIFARIO 32 (Krizhevsky et al., 2009) and FFHQ 64 x4 (Kar- ras et al., 2019) datasets, employing PAS alongside PAS without adaptive search (PAS (-AS)) to correct DDIM (Song et al., 2021a). We report the Fréchet Inception Distance (FID.) score, varying the number of function evaluations (NFE).
NFE
Method
10
NFE
CIFARIO 32 X2 (Krizhevsky et al., 2009)
Method
10
DDIM PAS
3,1
4,2,1
5,1.2
6,4.2
iPNDM PAS
3,1
4,2
CIFARIO 32 x 32 (Krizhevsky et al., 2009)
FFHQ 64 (Karras al., 2019)
DDIM (Sung et al., 2021a)
49.68
35.63
22.32
15.69
DDIM PAS (-AS)
132.12
139.45 48
143.54
120.32
DDIM PAS
3,2,1
4,3,1
5,4,2,1
7,5,3.2
DDIM PAS
17.13
12.11
7.07
4.37
iNDDM PAS
3.1
4,2
FFHQ 64 (Karmus al., 2019)
ImageNet 64 (Deng al., 2009)
DDIM + PAS
3,2,1
4,3,1
5,4.21
7,5,2,11
DDIM (Song et 2021a)
DDIM + PAS (-AS)
43.92
78.
98.88 98.84
35.21
24.38
98.67
93.62
18.37
iPNDM- PAS
DDIM PAS
29.07
17.63
8.16
5.61
LSUN Bedroom 256 x 256 (Yu et al., 2015)
DDIM + PAS
4,3,1
5,4.2
6,5.3,2
8,7.5.4
iNDMM PAS
4,2
5,3.2.1
6,4.3.1
8,6.4.3
30
DUIM
IPNOM
Stable Diffusion 512 (Rombach et al., 2022)
60
PAS, 25
RAS, I--10
RAS, In
PAS. N--5
DDIM + PAS
3,2,1
2,1
2,1
RAS,
20
PAS,
201
PAS. 1-
PAS, I-
RAS, I-100
15
PAS. I-100
PAS, In-10
PAS, I-10
Table 6, it can be observed that the DDIM, which has large truncation error, requires correction of more sampling steps, while the iPNDM, which exhibits a smaller truncation error, requires relatively fewer correction steps, which aligns with our intuition. Overall, PAS only needs to correct 1~55 time points, which corresponds to requiring only 4~200 learnable parameters, to significantly enhance the sampling quality of the baseline solvers. This validates the effectivvness of PAS as acceleration algorithm with minimal training costs.
PAS, In 10
10
PAS. 1-110-1
5
6
8
10
5
8
0
10
NFE
NFE
(a) DDIM PAS
(b) iPNDM + PAS
Figure 7. Ablation study regarding learning rate conducted on the CIFARIO 32 (Krizhevsky al., 2009), utilizing PAS to correct DDIM (Song et al., 2021a) and iPNDM (Liu et al., 2022a: Zhang & Chen, 2023) solvers. We report the Fréchet Inception Distance (FIDL) score, varying the number of function evaluations (NFE).
C.2. Additional Ablation Study Results
In this section, we first supplement additional ablation periments concerning the adaptive search. Subsequently, we provide further ablation experiments on learning rate, tolerance t, and solvers for trajectory generation.
adaptive search strategy. We analyzed the ground truth tra- jectory transitioning from a straight line to curve, and ultimately back to straight line. The adaptive search is de- signed to correct the errors in the large curvature regions of the sampling trajectory. In the straight linear segments, the errors introduced by the existing fast solvers (e.g., DDIM) are negligible, and the application of PCA-based sampling correction does not provide any further error adjustment. Instead, it introduces biases on other basis vectors, leading to a decline in sampling quality. Therefore, it is essential to combine the adaptive search strategy with PCA-based sampling correction, specifically to correct truncation er- rors in regions of high curvature the sampling trajectory. Combining Figure 6a and Table 7, further validates the ne- cessity and effectiveness of the proposed overall algorithm, PCA-based Adaptive Search (PAS).
More results on adaptive search. Regarding the adap- tive search strategy, we first supplement the specific FID values corresponding Figure 6a the CIFAR10 10 32x32 (Krizhevsky et al., 2009) dataset, as shown in Ta- ble 7. Additionally, in Table 7, we also provide experimental results on the FFHQ 64x64 (Karras et al., 2019) dataset, us- ing PAS alongside PAS without the adaptive search strategy (PAS(-AS)) to correct DDIM (Song et al., 2021 Con- sistent with the results in Figure 6a. when the adaptive search strategy is not based sampling correction is applied at each time step the sampling quality is inferior to that of the baseline DDIM. Furthermore, the absence of the adaptive search strategy results in increased computational time, as PCA-based sam- pling correction is required at each step. In Section 3.3, we described the motivation and process of the proposed
leeeyydd-secfficaall when PCA-
Learning rate. In Figure 7, we present the ablation results of the PAS correcting DDIM (Song et al., 2021a) and iP. NDM (Liu et al., 2022a: Zhang & Chen, 2023) with the order of on the CIFARIO 32x32 (Krizhevsky et al., 2009)
15

---

Diffusion Sampling Correction via Approximately 10 Parameters
Table 8. Ablation study regarding tolerance conducted on the CIFARIO 32 (Krizhevsky et al., 2009), utilizing PAS to correct DDIM (Song et al., 2021a) and iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) solvers. We report the Fréchet Inception Distance (FIDL) score, varying the number of function evaluations (NFE).
Table 10. Sample quality measured by Fréchet Inception Distance (FIDL) on the LSUN Bedroom x56 (Yu et al., 2015) dataset and Stable Diffusion (Rombach et al.. 2022), varying the order iPNDM (Liu et al., 2022a; Zhang & Chen, 2023).
NFF
Method
Order
NFT
Method
8
10
LSUN Bedroom 256 256 (Yu et al., 20155)
CIFARIO 32 (Krizhevsky et al., 2009)
iNNDM (Zhung & Chen, 2023)
1
34.34
25.25
15.71
11.42
DDIM (Song et al., 2021a)
35.63
22.32
15.69
iPNDM (Zhang Chen, 2023)
12.90
7.98
6.17
DDIM PAS
10
49.68
35.63
22.32
15.69
iPNDM (Zhang & Chen, 2023)
3
16:57
10.83
6.18
4.92
iPNDM (Zhang & Chen, 2023)
4
26.65
20.72
11.77
5.56
DDIM + PAS
10
17.13
12.11
7.07
4.37
DDIM+ + PAS
10-3
17.13
12.11
7.07
4.37
INNDM + PAS iPNDM PAS
2
15.48
10.24
6.67
5.14
DDIM + PAS
10
17.13
12.11
7.07
4.37
5.92
INDDM (Zhang & Chen, 2023)
16.55
9.74
5.23
3.69
Stable Diffusion 512 (Rombach et al., 2022)
iPNDM + PAS
10
13.61
9.74
5,23
3.69
IPNDM (Zhang & Chen, 2023) iNDDM (Zhang Chen, 2023)
3
22.31
17.21
13.60
13.71
iPNDM + PAS
10 3
13.61
7.47
3.87
2.91
28.12
25.69
20.68
16.45
iPNDM PAS
10-4
7.47
3,87
2,84
DDIM (Song et al., 2021a)
23.42
20.08
17.72
16.56
DDIM + PAS
17.70
15.93
14.74
14.25
Table 9. Ablation study regarding solvers for generating ground truth trajectories, including Heun's 2nd (Heun) (Karras al., 2022), DDIM (Song et al., 2021a), and DPM-Solver-2 (DPM) (Lu al., 2022a), conducted on the CIFARIO 32x32 (Krizhevsky et al., 2009) and FFHQ 64x64 (Karras et al., 2019). We report the Fréchet Inception Distance (FIDL) score, employing PAS to correct DDIM (Song et al., 2021a), varying the NFE.
point at which correction should commence, while subse- quent time points have their tolerance fixed at 10-4, We adjusted the tolerance T for the initial correction point from 10-1 to 10 and the experimental results on the CIFARIO 32x32 (Krizhevsky et al., 2009) dataset are presented in Ta- ble 8. The experimental findings indicate that PAS is not sen- sitive to the configuration of the hyperparameter tolerance r. PAS consistently demonstrates a significant improvement in the sampling quality of DDIM (Song et al., 2021a) and iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) solvers with tolerance ranging from 10 2 to 10-4. Lastly, we recommend setting the tolerance T to 10-2 for solvers with substantial truncation error (e.g.. DDIM), while for solvers with relatively smaller truncation error (e.g., iPNDM), the tolerance 7 should be set to 10-4
NFE
Method
Solver
5
CIFARIO 32 x2 (Krizhevsky et al., 2009)
DDIM (Song et al., 2021a)
\.
49.68
35.63
22.32
15.69
DDIM PAS
17.13
12.11
7.07
4.37
DDIM+ PAS
DDIM
17.10
12.44
6.97
4.87
DDIM PAS
DPM
17.12
12.10
7.10
4.40
FFHQ 64 (Karras et al. 2019)
DDIM (Song et al., 2021a)
43.92
35.21
24.38
18.37
DDIM + PAS
Heun
29.07
17.63
8.16
5.61
DDIM+ PAS
DDIM
30.49
15.26
8.65
6.37
DDIM PAS
DPM 29.11
17.58
8.12
5.64
Solvers for trajectory generation We investigated the im- pact of solver selection for generating ground truth tra- jectories on the performance of PAS using the CIFARIO 32x (Krizhevsky et al., 2009) and FFHQ 64 X64 (Karras et al., 2019) datasets. We employed Heun's 's 2nd (Karras et al., 2022), DDIM (Song et al., 2021a), and DPM-Solver- (Lu et al., 2022a) solvers with 100 NFE to generate 10k trajectories for training the PAS to correct the DDIM solver, as shown in Table 9. Our findings indicate that the choice of solver for generating ground truth trajectories has negli- gible impact on the performance of PAS. This suggests that regardless of the solver used, a sufficient number of NFE (e.g.. 100 NFE) enables the solving process to approximate the ground truth trajectories closely. Therefore, we simply fixed Heun's 2nd solver for all experiments.
dataset. We varied the learning rate from 10-4 to 10. The results demonstrate that, regardless of the learning rate set- ting, PAS significantly improves the sampling quality of both DDIM and iPNDM on the CIFARIO. 10, and further ploration exhibits slightly better sampling performance.
Tolerance 7. In Section 3.3, we designed an adaptive search strategy to enhance the overall performance of PCA-bo -based sampling correction. This adaptive search strategy relies on the condition L2- - (C, + -) > 0. Therefore, we further investigate the impact of the tolerance on the adaptive search strategy. is important to emphasize that the adap- tive search strategy is aimed at correcting the sections of the sampling trajectory with large curvature, while the tol- erance - serves as the criterion for determining whether the current sampling state has reached a region of large curvature in the trajectory. Consequently, the tolerance T initially treated as a hyperparameter to indicate the time
C.3. Additional Results on the Order of iPNDM
In this section, we discuss why we mainly chose to apply PAS correcting iPNDM with the order of 3 in Table 2. First,
16

---

Diffusion Sampling Correction via Approximately 10 Parameters
Table 11. Sample quality measured by Fréchet Inception Distance (FIDL), L2 (,SE) and L metrics on CIFARIO 32x32 (Krizhevsky et al.. 2009) dataset, varying the order of iPNDM (Liu et al., 2022a; Zhang & Chen, 2023). The L2 (MSE) and Ln metrics were evaluated against Heun's 2nd solver (Karras et al., 2022) with 100 NFE, using 50k samples.
NFE
Method
Order
Metrics
5
6
7
8
9
10
CIFARIO 32 x. (Krizhevsky 2009), Fréchet Inception Distance (FID.) J)) metric
iPNDM (Zhang Chen, 2023)
FID
49.68
35.63
27.93
22.32
18.43
iPNDM + PAS (Ours)
FID.
41.14
17.13
12.11
11.77
7.07
5.56
4.17
iPNDM (Zhang & Chen, 2023)
FID
39.02
25.24
16.19
11.85
9.08
7.39
IPNDM + PAS (Ours)
FID
33.54
16.59
9.77
5.87
4.51
3.55
3.01
iPNDM (Zhang & Chen, 2023)
3
FIDI
29.49
16.55
9.74
6.92
5.23
4.33
3.69
iPNDM PAS (Ours)
FIDL
27.59
13.61
7.47
5.59
3.87
3.17
2.84
iPNDM (Zhang & Chen, 2023)
FID
24.82
13.58
7.05
5.08
3.69
3.17
2.77
iPNDM PAS (Ours)
FID4
25.79
7.95
CIFARIO 32 (Krizhevsky et al., 2009), L2 (MSE) and L, metrics
iPNDM (Zhang & Chen, 2023)
4
L2 (MSE)
0.027
0.016
0.009
0.006
0.004
0.003
0.002
iPNDM PAS (Ours)
La (MSE)
0.014
0.009
0.006
0.004
0.003
IPNDM (Zhang & Chen, 2023)
0.126
0.089
0.063
0.047
0.037
0.030
0.025
iPNDM + PAS (Ours)
L1
0.100
0.078
0.059
0.047
0.037
0.031
0.026
we evaluated the sampling quality of iPNDM by adjust- ing its order on the CIFARIO 32 (Krizhevsky et al., 2009), LSUN Bedroom 256 (Yu et al., 2015) datasets, and Stable Diffusion (Rombach et al., 2022), as shown in Tables 10 and 11. We found that increasing the order of iPNDM does not always enhance sampling quality. Particu- larly on high-resolution datasets, optimal sampling quality is often not achieved at iPNDM with the order of 4; in com- parison, iPNDM with order 3 demonstrates better average performance. Therefore, we mainly selected PAS to correct the iPNDM with the order of in Table 2.
Text Prompts:
Majestic eagle soaring above a mountainlandscape. (Top) Warm living room with a crackling fireplace and soft lighting. (Bottom)
DDIM
Furthermore, on low-resolution datasets, such CIFARIO 32x 32, iPNDM with the order of 4 displays the best per- formance. Consequently, we employed PAS to correct its truncation error; however, we observed no improvement in its FID score. Nevertheless, the iPNDM corrected by PAS
DDIM + PAS
with the order of 4 performs better in terms of L1 and L2 metrics, as indicated Table 11. This phenomenon may be attributed to the fact that PAS uses LA La loss functions during training, and when the solver's 's sampling quality is al- ready relatively satisfactory, there is not always consistency between the L,. L2 metrics and the FID score.
DDIM
C.4. Additional Visualize Study Results
DDIM PAS PAS
We present additional visual sampling results using Stable Diffusion v1.4 (Rombach et al., 2022), as shown in Fig- ure 8. Furthermore, more visual results on the CIFARIO 10 32x 32 (Krizhevsky et al., 2009), FFHQ 64 64 (Karras al., 2019), ImageNet 64x64 (Deng et al., 2009), and LSUN Bedroom 256x256 (Yu al., 2015) datasets with NFE of 6 and 10 are displayed in Figures 9 16. These visual results demonstrate that the samples generated by PAS exhibit higher quality and richer details compared to the corresponding baselines.
NFE=6 6
NFE-8
NFE=10
Figure 8. Random samples by DDIM (Song al., 2021a) with and without the proposed PAS on Stable Diffusion v1.4 (Rombach et al., 2022) with a guidance scale of 7.5.
17

---

Diffusion Sampling Correction via Approximately 10 Parameters
(a) DDIM, NFE = 6. FID= = 35.63
(b) DDIM, NFE= = 10. FID 15.69
(c) DDIM + PAS, NFE 6, FID= 12.11
(d) DDIM PAS, NFE 10, FID 4.37
Figure? 9. Random samples by DDIM (Song et al.. 2021a) with and without the proposed PAS on CIFARIO 32x32.
(a) iPNDM, NFE- 6. FID 9.74
(b) iPNDM. NFE = 10, FID = 3.69
(c) iPNDM PAS, NEE = 6, FID 7.47
(d) iPNDM + PAS, NFE = 10, FID 2.84
Figure 10. Random samples by iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) with and without the proposed PAS on CIFARIO 32
18

---

Diffusion Sampling Correction via Approximately 10 Parameters
(a) DDIM, NFE- = 6, FID = 35.21
(b) DDIM, NFE= 10, FID 18.37
(c) DDIM PAS, NFE 6, FID 17.63
(d) DDIM PAS, NFE 10, FID 5.61
Figure 11. Random samples by DDIM (Song al., 2021a) with and without the proposed PAS on FFHQ 64x64.
(a) iPNDM, NFE = 6. FID 11.31
(b) iPNDM. NFE = 10, FID = 4.95
(c) iPNDM + PAS, NEE 6, FID 10.29
(d) iPNDM + PAS, NFE 10, FID 4.28
Figure 12, Random samples by iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) with and without the proposed PAS on FFHQ 64x64.
19

---

Diffusion Sampling Correction via Approximately 10 Parameters
(a) DDIM, NFE- = 6, FID 34.03
(b) DDIM, NFE = 10, FID 16.72
(c) DDIM + PAS, NFE 6, FID 26.21
(d) DDIM PAS, NFE 10, FID 9.13
Figure 13. Random samples by DDIM (Song et al., 2021a) with and without the proposed PAS on ImageNet x4.
(a) iPNDM, NFE- = 6, FID 13.48
(b) iPNDM. NFE = 10, FID = 5.64
(c) iPNDM + PAS, NEE 6, FID 12.89
(d) iPNDM + PAS, NFE 10, FID 5.32
Figure 14, Random samples by iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) with and without the proposed PAS on ImageNet 64x64,
20

---

Diffusion Sampling Correction via Approximately 10 Parameters
(a) DDIM, NFE = 6, FID 25,25
(b) DDIM, NFE 10, FID 11.42
DDIM + PAS, NFE 6, FID 13.31
DDIM PAS, NFE 10, FID 6.23
Figure 15. Random samples by DDIM (Song al., 2021a) with and without the proposed PAS LSUN Bedroom 256x256.
(a) iPNDM, NFE FID 12.90
(b) iPNDM. NFE- = 10, FID 6.17
(c) iPNDM + PAS, NFE = 6, FID = 10.24
(d) iPNDM + PAS, NFE= 10, FID = 5.14
Figure Random samples by iPNDM (Liu et al., 2022a; Zhang & Chen, 2023) with and without proposed PAS LSUN Bedroom 256x 256.
21


Reviews:
[{"review_id": "5tdpKTxq7b", "reviewer": "Reviewer_pbLK", "summary": "The authors leveraged the previous finding that the diffusion sampling trajectories are low dimensional, and that part of it is more curvy. Then they developed a method to learn the PCA basis of current ongoing trajectory during sampling and then learn coefficient to recombine the PC vectors to correct for the truncation errors in the small NFE solvers. They can train the PC correction coefficients efficiently on a bunch of “ground truth” trajectory with higher NFEs, and obtain corrections for sampler with fewer NFEs, effectively distilled out a dataset specific fast sampler based on the PC correction coefficients. \n\nThe method is evaluated on CIFAR10 32x32 to Stable diffusion 512, and showed effectiveness through the scales.\n\n## Update after rebuttal\nThe authors showed a close comparison with the Gaussian approximation based teleportation / acceleration method in [WV24], and an updated benchmark. The updated theoretical link between the low dimensional trajectory to the Gaussian theory also made it stronger. \nThe reviewer is happy to maintain the score as such.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's detailed review and insightful suggestions!\n\n**Abbreviation: CaE (Claims And Evidence), MaEC (Methods And Evaluation Criteria), ERND (Essential References Not Discussed)**\n\n>***CaE1: The later trajectory may not be linear but instead takes smaller steps.***\n\n**A**: Your interpretation is reasonable. However, [KMTS22] and [WV23] only analyze the trajectory in 2D, which may lose some information. [R1] visualizes trajectory in 3D ([R1] Fig. 4) and offers a \"boomerang\" explanation (linear-nonlinear-linear) in Sec. 3, showing that the later part is indeed linear. We have added a detailed analysis in the revised version.\n\n[R1] Chen D et al. On the trajectory regularity of ode-based diffusion sampling. ICML2024.\n\n>***CaE2: The degradation without adaptive search may be due to the obtained PCs not being good.***\n\n**A**: Notably, this paper has conducted a similar analysis in lines 377-384(Right): \"The errors in the linear part are negligible; the correction does not further reduce errors and instead introduces noise\", highlighting the **necessity and effectiveness of the adaptive search**.\n\n>***Expand theory (MaEC1, Theoretical Claims, and ERND_M1): The study misses citing [WV23] and [WV24], which could theoretically motivate Fig. 2b and the correction of the more curved parts(Fig. 1).***\n\n**A**: We have carefully revisited [WV23] and [WV24], which approximate diffusion trajectories by Gaussian score structure and provide both theoretical and empirical insights. These works are indeed inspiring and insightful!\n\nSpecifically, [WV23] offers a theoretical analysis in Sec. 4.2, suggesting that diffusion trajectories resemble 2D rotations on the plane spanned by the initial noise ($x_T$) and the final sample ($x_0$). This provides a theoretical underpinning for our Fig. 2b, where trajectories corresponding to different initial noises lie in different subspaces.\n\n**Correction of the more curved parts**: [WV24] derives an analytical form of the EDM trajectory in Eq. 15 (Gaussian structure):\n\n$x_t=\\mu+\\frac{\\sigma_t}{\\sigma_T} x_T^\\perp +\\sum_{k=1}^r \\psi (t,\\lambda_k)c_k(T)u_k$,\n\nwhich shows that $x_t$ is a linear combination of $x_T^\\perp$ and $\\mu,u_k$. PAS is designed to learn statistical patterns of the dataset that are independent of the initial noise $x_T$, i.e., the coefficients of $\\mu,u_k$. As $\\sigma_t \\to 0$, the influence of $x_T^\\perp$ on $x_t$ diminishes. So, in the low-noise regime (i.e., more curved parts), PAS batter captures the PCs of the data ($\\mu,u_k$), leading to more accurate correction. This offers a theoretical explanation for why PAS tends to work better in the more curved regions (Fig. 1). We have added relevant discussions in Sec. 3 and the Related Work of the revised manuscript. The alignment between PAS and the conclusions of [WV23], [WV24] is indeed intriguing!\n\n>***MaEC2: Discuss the time taken to obtain the sampling trajectory for training (similar to [WV24]).***\n\nA: We did not discuss the cost because other training-based methods also require sampling full or partial trajectory for training. In the revised version, we added a discussion on the time taken: 3.26m for CIFAR10 and 1.79h for Bedroom256 with 50 NFE and a 5k trajectory on an A100 GPU.\n\n>***ERND_M2 and ERND_R1-3: Discuss the connections and differences between PAS and [WV23], [WV24], and include them as a benchmark.***\n\n**A**: The proposed PAS corrects sampling in the low-noise regime, while [WV24] accelerates sampling in the high-noise regime. So, combining [WV24] with PAS is indeed interesting. Theoretically, we can analytically warm up the PCs from the Gaussian structure and then **correct the sampling starting from the teleported solution** $x_{t'}$, further improving the FID in [WV24] Fig. 15. However, [WV24] is limited by the computational cost of estimating means and covariances on large pixel datasets. Furthermore, another key challenge is how to effectively combine the Gaussian (analytical but imprecise under low noise) and neural (precise but costly) structure to obtain more exact PCs with minimal cost. This will be a focus of our future work.\n\n>***ERND_R4: The PAS is hard to apply to very high-res images.***\n\n**A:** No, the PAS can **be easily applied to high-res images**. This is because we use `torch.pca_lowrank` to decompose the trajectory matrix as $\\mathbf{X}^{\\prime}\\in \\mathbb{R}^{(N-i+2)\\times D}$ (Eq. 13), where $i\\leq N,N\\leq 10$, so its dimension is low and the rank $r\\ll D$. For Stable Diffusion(SD), $D$=64x64x4 from the latent space. In the table below, we compare the time for 1PCA and 1NFE, and the PCA time is negligible.\n\npre 128 samples time↓ (s)\n| |SD|Bedrooom256|\n|-|-|-|\n|1NFE|30.20|10.05|\n|1PCA|0.06|0.2|\n\n>***Other Suggestions: Table1 caption needs clarification.***\n\n**A**: Thanks for pointing out the issues; we have revised them. Briefly, the list of numbers denotes all the time points requiring correction from adaptive search, with each element $i\\in [N,1]$.", "claims_and_evidence": "Most claims are clear and supported. \n\nSome interpretation of why the method works could be made clearer. \n\n- **Minor point** “*Thus, we can infer that the sampling trajectory first appears linear, then transitions to a curve, and ultimately becomes linear again under the attraction of a certain mode.*” \nIn Fig. 3a the authors interpreted the S shape truncation error as the error at middle range is due to higher curvature. This is possible but my interpretation is the earlier trajectory is quite linear, and later trajectory may not be linear but took smaller steps, so the L2 error do not increase that much (See [KMTS22] Fig.3 [WV23] Fig.1). If the author do want to consolidate the curvature intuition, you could directly perform an empirical analysis of how much the trajectory rotate at each part.\n- The failure of ablation study without Adaptive search is worth noticing, from the FID, it seems it failed spectacularly. \nI’m curious about the author’s interpretation of it (e.g. curvature). My interpretation is that the PCA basis you got from some part of the trajectory (e.g. see Fig. 18 in [WV23]) may be not good, or too contaminated with noise, so not useful to correct the trajectory. Thus those PC may not generally help correct for truncation error. \nI think a deeper analysis of that part may illustrate why this method may or may not work.\n\n[KMTS22] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.\n\n[WV23] Wang, B., & Vastola, J. J. (2023). Diffusion models generate images like painters: an analytical theory of outline first, details later.", "methods_and_evaluation_criteria": "The experimental set up and evaluations are quite standard and performed well!  \n\n- There are some very recent results the authors didn’t include in the benchmark, which could have some shared mechanism with the paper. [WV24] (detailed in missing reference part.)\n- One aspects the authors didn’t mention is that even though training is super efficient, obtaining the sampling trajectory for training is likely not efficient. In your case obtaining 5k to 10k sampling trajectory with high NFE could take some time. The authors could mention the time of obtaining such trajectories in additional to training (around L311-318).  \nThis is actually a bit similar to [WV24], where their acceleration is parameter / training free, but to sample the data to compute PCA takes time, if they don’t have access to training set.\n\n[WV24] Wang, B., & Vastola, J. J. (2024). The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. arXiv preprint arXiv:2412.09726.", "theoretical_claims": "There is not much theoretical claims in this paper. The intuition of correcting the more curvy part of trajectory is interesting, the authors could provide a slightly more formal treatment of the \n\nThough more connection to some theoretical framework can better frame or motivate the method. [WV24]", "experimental_designs_or_analyses": "Standard design and analysis.", "relation_to_broader_scientific_literature": "See below.", "essential_references_not_discussed": "**Methods**\n\n- Regarding Figure 2 results of PCA of sampling trajectory, one reference the authors missed is [WV23], where they also systematically studied the PC structure of sampling trajectory and provided a theoretical account explaining the low dimensional (mostly 2d) nature of the sampling trajectory. Basically they showed the low dim subspace is the one spanned by the initial noise and the final image sample, which explains your Fig.2B, since different trajectory starts from different initial noise, they won’t have a shared subspace / low dimensional structure.\n    \n    Authors of [WV23] also found a better way to understand these trajectories is to look at the PCs of the projected outcome x0_hat, their PC should be closer to those of the image manifold. Seems Fig 2. in [WV23] and Fig.1 in your paper has interesting connections. \n    \n    The authors should mention these results in line 156-160 “*This indicates that the entire sampling trajectory lies in a three-dimensional subspace embedded …* ”\n    \n- The theory and observations of early diffusion trajectory is well approximated by Gaussian score trajectory leads to some strong predictions about what the PC’s you are getting, they should somehow be aligned with the PC of target data manifold [WV23]. So the authors could discuss those as further theoretical justifications of the method.\n\n**Results**\n\n- In the main benchmark table 2, it seems the results the authors get from PAS-acceleration are similar or sometimes slightly worse than the results from a recent paper [WV24] Fig.15, which also achieves accelerated sampling via analytical teleportation, based on the PCA of data manifold without any retraining. They also tested on EDM pretrained on CIFAR and FFHQ, AFHQ, so the NFE and FID numbers are totally comparable to those in your table. The authors could consider adding them to the benchmark.\n    - I have some suspicion that the mechanism of acceleration may be shared, i.e. due to the Gaussian score structure in diffusion. Due to the Gaussian structure, the trajectory from any initial noise state can be analytically predicted based on mean and PC of data, so they can teleport the solution. I guess the mechanism of your PAS acceleration is likely to be related to that. The authors might benefit from discussing the connections.\n    - On the other hand, one difference is, their acceleration mainly happen at earlier part of trajectory (e.g. high noise regime) and from Table 1, seems PAS works by correcting more at the low noise regime, suggesting different correcting mechanism.\n    - Mentioning this is not to reduce the contribution or novelty of the current result. I think their methods were hard to be applied to Stable Diffusion or very high res images, due to they need to compute PCA of the dataset. Trajectory based PCA and training seems to circumvent a bit.\n\n[WV23] Wang, B., & Vastola, J. J. (2023). Diffusion models generate images like painters: an analytical theory of outline first, details later. \n\n[WV24] Wang, B., & Vastola, J. J. (2024). The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications. arXiv preprint arXiv:2412.09726.", "other_strengths_and_weaknesses": "- The method is simple to implement and parameter and compute efficient, also conceptually beautiful.\n- The illustrations and tables were well made and authors put efforts into explaining their ideas.", "other_comments_or_suggestions": "- Table 1 caption is not very clear, authors could more clearly annotate what are the list of numbers is showed in the table.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3HXeXARCK1", "reviewer": "Reviewer_8neu", "summary": "The paper proposes a novel PCA-based Adaptive Search (PAS) method to accelerate diffusion model sampling with minimal additional computational and parameter costs. The key idea of PAS rests on the observation that the sampling trajectory of a parameterized reverse ODE of diffusion model almost lies in a 3D subspace in a high-dimensional space. Thus, PAS involves leveraging Principal Component Analysis (PCA) to identify a small set of orthogonal basis vectors and learns low-dimensional coordinates corresponding to these vectors to correct sampling errors. The main results demonstrate substantial improvements in sampling efficiency and quality with negligible cost. Specifically, PAS reduces the FID score of DDIM from 15.69 to 4.37 on CIFAR10 with 10 sampling steps, using merely around 12 parameters and minimal training time. PAS shows similar enhancements across multiple datasets (e.g., CIFAR10, FFHQ, ImageNet, LSUN Bedroom) and various pre-trained diffusion models, consistently outperforming the state-of-the-art method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's efforts and valuable review. Below are our responses to all questions. We kindly hope you could consider increasing the score if you are satisfied.\n\n>**Experimental Designs Or Analyses: Direct performance comparisons with training-based methods are lacking, highlighting PAS's advantage.** \n\n**A**: We appreciate your suggestion! We did not include direct performance comparisons because high-cost training-based methods and PAS have their own application scenarios. Training-based methods can achieve one-step sampling but are expensive, making them difficult to apply to large models like Stable Diffusion. In contrast, PAS focuses on high-quality sampling under 10 NFE with negligible cost, making it a **more attractive and practical** solution. Furthermore, we added a comparison of parameter size and training time: Training-based one-step sampling methods require **at least 35.7M** parameters (DDPM, CIFAR10 model), while PAS uses **only ~10**. The table below summarizes training time↓ (A100 hours) on CIFAR10, using the estimation method and certain results from [R1].\n\n**Note**: Training a CIFAR10 model from scratch takes ~200 A100 hours [R1].\n|PD[R2]|Guided PD[R3]|CD[R4]|CTM[R5]|PAS(Ours)|\n|-|-|-|-|-|\n|~195|~146|~1156|~83|<0.04|\n\n>***W1: Theoretical explanation of why the sampling trajectories of all samples exhibit strongly consistent geometric characteristics.***\n\n**A**: Inspired by [WV23] and [WV24], we have added more theoretical analysis to the paper (**Reviewer pbLK, Expand theory**). Briefly, since analyzing neural networks directly is inherently intractable, [WV23] and [WV24] approximate diffusion trajectories using Gaussian score structures, providing both theoretical and empirical insights. \n\nSpecifically, [WV24] derives an analytical form of the EDM trajectory in Eq. 15(Gaussian structure):\n\n$x_t=\\mu+\\frac{\\sigma_t}{\\sigma_T} x_T^\\perp +\\sum_{k=1}^r \\psi (t,\\lambda_k)c_k(T)u_k$,\n\nwhich shows that $x_t$ is a linear combination of $x_T^\\perp$ and $\\mu, u_k$. Here, $\\mu, u_k$ are the dataset-dependent mean and basis vectors that define the data manifold, while $x_T^\\perp$ is an off-manifold component. Furthermore, given $x_T$, $c_k(T)$ is a constant, and the rate of change of the coefficients for the vectors $x_T^\\perp, \\mu$, and $u_k$ depends only on the dataset and the timestep $t$. For a fixed dataset, as $\\sigma_t$ decreases during the sampling process, each sample is constrained by the same temporal decay scale, converging to the data manifold. This theory heuristically explains why all samples exhibit strongly consistent geometric characteristics.\n\n>***W2 and Q2: Discuss the limitations of the reliance on PCA and the robustness of PAS to stochastic samplers.***\n\n**A**: The effectiveness of PAS relies on the accuracy of the basis derived from PCA, which may be influenced by noise from stochastic samplers. To further explore this, we added experiments using PAS on the stochastic sampler of EDM [R6] without the 2-order correction ([R6] Algorithm 2) on CIFAR10, with the following results:\n\nFID↓\n|Method\\NFE|5|8|10|20|\n|-|-|-|-|-|\n|Stochasticity|55.71|27.57|21.48|10.95|\n|+PAS|41.44|23.81|19.04|10.48|\n\nThe table clearly shows that PAS is **equally effective** for the stochastic sampler, further highlighting the robustness of PAS. Nonetheless, in general, SDE has a higher quality upper bound, with 1000 NFE outperforming ODE. But for sampling with fewer steps, ODE performs better. Therefore, PAS with ODE would result in faster sampling speeds.\n\n>***Q1: A comparison of sampling quality and computational trade-offs with training-based methods like PD[R2].***\n\n**A**: We have added some comparative results on the training cost and sample quality between PD and PAS, as shown in the following tables:\n\nFID↓(A100 hours↓)\n|Method\\NFE|1|2|4|8|\n|-|-|-|-|-|\n|DDIM+PD[R2]|9.12(~195)|4.51(~171)|3.00(~159)|2.57(~146)|\n\n|Method\\NFE|5|6|8|10|\n|-|-|-|-|-|\n|DDIM+PAS|17.13(~0.01)|12.11(~0.02)|7.07(~0.03)|4.37(~0.04)|\n|iPNDM+PAS|13.61(~0.01)|7.47(~0.02)|3.87(~0.03)|2.84(~0.04)|\n\nAs shown in the tables above, PD requires ~146 A100 hours of training time for 8 NFE to achieve 2.57 FID, whereas PAS with iPNDM requires **only 0.04 A100 hours with 8 parameters** to achieve 2.84 FID at 10 NFE. Therefore, PD and PAS each have their own application scenarios. PAS achieves impressive performance with negligible cost, making it a **more attractive and practical** solution.\n\n[R1] Zhou Z et al. Simple and fast distillation of diffusion models. NeurIPS 2024.\n\n[R2] Salimans T et al. Progressive distillation for fast sampling of diffusion models. ICLR 2022.\n\n[R3] Meng C et al. On distillation of guided diffusion models. CVPR 2023.\n\n[R4] Song Y et al. Consistency models. ICML 2023.\n\n[R5] Kim D et al. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv:2310.02279.\n\n[R6] Tero Karras et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.", "claims_and_evidence": "Yes, the paper's main claims are convincingly supported through empirical evaluations and comparative analyses. The paper proposes utilizing PCA to identify a small number of basis vectors spanning the high-dimensional trajectory space of sampling paths. They substantiate this claim with empirical evidence showing that sampling trajectories reside primarily within a low-dimensional subspace. Clear experimental results confirm that correction along these PCA-derived directions is effective in reducing truncation errors.", "methods_and_evaluation_criteria": "Yes, the paper proposes PAS, a lightweight, plug-and-play method to correct discretization errors during diffusion sampling. PAS leverages the inherent geometric property that sampling trajectories lie in a low-dimensional subspace, utilizing PCA to represent and correct sampling directions with minimal parameters efficiently.\nThe evaluation utilizes standard datasets, such as CIFAR10, FFHQ, ImageNet, LSUN Bedroom, and Stable Diffusion v1.4. The evaluation metric FID objectively measures the quality of generated images. Comparisons against existing state-of-the-art methods (e.g., DPM-Solver++, iPNDM) provide strong evidence for PAS’s effectiveness in terms of improved sampling quality and efficiency at significantly lower parameter and computational costs.", "theoretical_claims": "This paper do not provide theoretical claims.", "experimental_designs_or_analyses": "Yes, the paper conducts extensive experiments to validate the effectiveness of the proposed method. The authors emphasize minimal computational costs by reporting clearly the training time (e.g., less than 2 minutes on CIFAR10) and the number of parameters (approximately 10), however, direct quantitative performance comparisons against existing training-based methods is lacking. Including such comparisons could better highlight PAS's advantage over a more costly training-based method.", "relation_to_broader_scientific_literature": "The key contributions of this paper respond to training costs limitations in prior few step generation works, notably the Progressive Distillation method(Salimans & Ho, 2022). While Progressive Distillation substantially improves sampling speed by training distilled models that drastically reduce sampling steps, it faces critical limitations including significant additional training costs and storage demands for the distilled model. This paper addresses these practical bottlenecks by leveraging PCA to identify a compact set of basis vectors spanning the sampling directions, thus requiring minimal retraining and alleviating memory constraints.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a highly original approach by leveraging PCA to significantly reduce the dimensionality of sampling corrections in diffusion probabilistic models, thus effectively addressing computational and storage bottlenecks prevalent in existing methods.\n2. The explanations throughout the paper are clear and logically structured.\n\nWeaknesses:\n\n1. While the paper extensively demonstrates empirical effectiveness, it could further clarify theoretical insights behind the PCA-based trajectory correction-particularly why sampling trajectories of all samples exhibit strong consistent geometric characteristics.\n2. Discussion of potential limitations regarding the reliance on PCA and robustness to stochastic samplers, remains limited and could be expanded upon.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. The paper highlights the efficiency advantages of PAS over existing training-based methods. Could you provide a more detailed comparison—especially in terms of sampling quality and computational trade-offs—against widely-used training-based methods such as Salimans & Ho (2022)? \n2. The PAS demonstrates promising results for deterministic solvers (e.g., DDIM). Could PAS be similarly effective if adapted to stochastic sampling methods? Have you conducted any preliminary experiments or analyses in this direction? Clarification on applicability or limitations would be valuable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzQklajFhf", "reviewer": "Reviewer_rBDg", "summary": "This paper proposes PCA-based Adaptive Search (PAS) to optimize the sampling process of diffusion probabilistic models (DPMs). The key of the method is leveraging Principal Component Analysis to identify a low-dimensional subspace for sampling correction. The method also includes an adaptive search strategy to reduce correction steps. PAS operates in a plug-and-play fashion, enhancing existing fast solvers without significant additional training costs. Experimental results show that PAS significantly improves sampling quality on diverse datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's recognition of our work and meticulous review. Please find below our responses to all the questions. We would greatly appreciate it if you could consider increasing the score if you are satisfied with our response.\n\n**Abbreviation: CaE (Claims And Evidence), EDoA (Experimental Designs Or Analyses)**\n\n>***CaE and EDoA2: The study lacks a comparison of training efficiency between PAS and other methods.***\n\n**A**: We provide a comparison of PAS with the previously well-performing AMED and GITS methods, while also introducing the high-cost PD method as a contrast. The comparative experiments, based on sample quality and training cost on CIFAR10, are shown in the table below:\n\nFID↓(A100 hours↓)\n\n| Method\\NFE  | 1          | 2          | 4          | 8          |\n| ----------- | ---------- | ---------- | ---------- | ---------- |\n| DDIM+PD[R1] | 9.12(~195) | 4.51(~171) | 3.00(~159) | 2.57(~146) |\n\n| Method\\NFE      | 5            | 6            | 8            | 10           |\n| --------------- | ------------ | ------------ | ------------ | ------------ |\n| DDIM+AMED[R2]   | \\            | 25.15(~0.08) | 17.03(~0.10) | 11.33(~0.11) |\n| DDIM+GITS[R3]   | 28.05(<0.01) | 21.04(<0.01) | 13.30(<0.01) | 10.37(~0.01) |\n| DDIM+PAS (Ours) | 17.13(~0.01) | 12.11(~0.02) | 7.07(~0.03)  | 4.37(~0.04)  |\n\nAs shown in the table above, PAS demonstrates higher training efficiency compared to AMED, and better sample quality than GITS with negligible additional cost. In contrast, PD, which yields higher sample quality, incurs a significantly larger training cost. Notably, **PAS is theoretically orthogonal to PD, AMED and GITS**, and can further improve their sampling efficiency.\n\n\n>***Theoretical Claims: Lacking theoretical justification for sampling trajectories in a low-dimensional subspace.***\n\n**A**: Inspired by [WV23] and [WV24], we have added more theoretical analysis to the paper (see **Reviewer pbLK, Expand theory**). In brief, since analyzing neural networks directly is inherently intractable, [WV23] and [WV24] approximate diffusion trajectories using Gaussian score structures, providing both theoretical and empirical insights. \n\nFurthermore, [WV23] offers a theoretical analysis in Sec. 4.2, suggesting that diffusion trajectories **resemble 2D rotations on the plane spanned by the initial noise ($x_T$) and the final sample ($x_0$)**. This offers a theoretical justification for why sampling trajectories lie in a low-dimensional subspace. It also explains why trajectories corresponding to different initial noises lie in different subspaces (see our Fig. 2b).\n\n\n>***EDoA1: Add a comparison with other trajectory-based methods (e.g., AMED, GITS) combined with other SOTA solvers (e.g., iPNDM).***\n\n**A**: We did not include additional comparisons between PAS and other trajectory-based methods (e.g., AMED, GITS) primarily because the integration of PAS with DDIM already demonstrates significant improvements in sampling efficiency compared to AMED and GITS, thereby validating the effectiveness of PAS. Furthermore, PAS is theoretically orthogonal to methods like AMED and GITS, and can potentially be combined with them to further enhance sampling efficiency. Therefore, we believe that exploring how to better combine PAS with trajectory-based methods (e.g., AMED, GITS) is a more meaningful direction, which will be considered in our future work.\n\n\n>***Questions For Authors: Have you experimented with alternative dimensionality reduction techniques in place of PCA?***\n\n**A**: We have experimented with both standard PCA (`torch.svd`) and low-rank or sparse matrix PCA (`torch.pca_lowrank`). Notably, PCA_lowrank (used in our paper) computes approximately 1.58× faster than standard PCA, with some loss in accuracy. We provide a comparative experiment on the CIFAR10, as shown in the following table:\n\nPAS+DDIM, FID↓\n| Method\\NFE  | 5     | 6     | 8    | 10   |\n| ----------- | ----- | ----- | ---- | ---- |\n| PCA         | 16.38 | 13.08 | 7.01 | 4.39 |\n| PCA_lowrank | 17.13 | 12.11 | 7.07 | 4.37 |\n\nFrom the table, we can observe that PCA_lowrank and standard PCA yield **comparable results**. Upon further analysis, we believe this is because only around 4 principal components are sufficient to span the complete sampling trajectory space (which consists of 1000 vectors). Since extracting 4 principal components from trajectory space is relatively straightforward, the choice of dimensionality reduction technique is not a critical factor in the effectiveness of the proposed PAS method.\n\n\n[R1] Salimans T et al. Progressive distillation for fast sampling of diffusion models. ICLR 2022.\n\n[R2] Zhou, Zhenyu et al. Fast ode-based sampling for diffusion models in around 5 steps. CVPR 2024.\n\n[R3] Defang Chen et al. On the Trajectory Regularity of ODE-based Diffusion Sampling. ICML 2024.", "claims_and_evidence": "The authors claim that PAS can correct the truncation errors and improve the sample quality. This claim is well-supported by the improvements of FID on multiple datasets.\n\nHowever, despite the authors claim that PAS can enhance sampling efficiency of existing fast solvers (requiring only a small number of learnable parameters and negligible training time), there is a lack of intuitive comparison with other methods regarding efficiency.", "methods_and_evaluation_criteria": "The method is well-founded, using PCA on the sampling trajectory to extract low-dimensional basis vectors for sampling correction.\n\nThe method is evaluated primarily using FID on standard image generation benchmarks. Comparisons with previously state-of-the-art fast solvers, as well as detailed ablation studies, ensure a comprehensive evaluation.", "theoretical_claims": "The derivations of PCA-based correction and adaptive search appear correct and well-grounded in numerical methods.\n\nThe claim that sampling trajectories lie in a low-dimensional subspace is experimentally validated but could benefit from more theoretical justification.", "experimental_designs_or_analyses": "The experimental design is comprehensive, covering 5 datasets, multiple solvers, and different NFE values (5–10 steps).\n\nComparisons with other trajectory-based methods (e.g., AMED, GITS) have only been done on DDIM, and it would be useful to compare and analyze combined with other SOTA solvers (e.g., iPNDM). \n\nThere is a lack of comparison and analysis about training efficiency between PAS and other methods.", "relation_to_broader_scientific_literature": "The paper is well-positioned in the broader context of: fast DPM solvers (e.g., DDIM, DPM-Solver-2, DPM-Solver++, iPNDM) and trajectory-based acceleration methods (e.g., AMED, GITS).\n\nIt builds on recent findings that sampling trajectories of DPMs lie in a low-dimensional subspace, extending this idea with a novel PCA-based correction approach.", "essential_references_not_discussed": "The paper cites all major relevant works, including prior fast solvers, trajectory-based corrections, and low-cost training strategies.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "Have you experimented with alternative dimensionality reduction techniques in place of PCA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a training paradigm called PAS (PCA-based Adaptive Search), for accelerating DPMs (Diffusion Probabilistic Models) with minimal training costs and learnable parameters. In particular, the authors propose to obtain a few basis vectors via PCA, and then learn their low-dimensional coordinates to correct the high-dimensional sampling direction vectors. Numerical experiments are provided to illustrate the method.

Reviewers generally agree that this is an original approach and appreciate the provided numerical experiments, which clearly demonstrate the effectiveness of the proposed method. All reviewers point out the lack of theoretical analysis in the current manuscript. In their response, the authors pointed out connection with the Gaussian  approximation based teleportation / acceleration method in Wang and Vastola (2024), which 
reviewers pbLK and 8neu found to be satisfactory.
 
While the paper is lacking theory, I believe the proposed method would be of interest and agree with the acceptance recommendation by all reviewers.

Note: The authors claim that the code will be available. This should be released upon acceptance of the paper.
Decision: Accept (poster)