Title: PFDiff: Training-free Acceleration of Diffusion Models through the Gradient Guidance of Past and Future
Venue: NeurIPS 2024
Accepted: no
Keywords: diffusion models

accelerated sampling

training-free sampler

orthogonal sampling method
Abstract: Diffusion Probabilistic Models (DPMs) have shown remarkable potential in image generation, but their sampling efficiency is hindered by the need for numerous denoising steps. Most existing solutions accelerate the sampling process by proposing fast ODE solvers. However, the inevitable discretization errors of the ODE solvers are significantly magnified when the number of function evaluations (NFE) is fewer. In this work, we propose PFDiff, a novel training-free and orthogonal timestep-skipping strategy, which enables existing fast ODE solvers to operate with fewer NFE. Based on two key observations: a significant similarity in the model's outputs at time step size that is not excessively large during the denoising process of existing ODE solvers, and a high resemblance between the denoising process and SGD. PFDiff, by employing gradient replacement from past time steps and foresight updates inspired by Nesterov momentum, rapidly updates intermediate states, thereby reducing unnecessary NFE while correcting for discretization errors inherent in first-order ODE solvers. Experimental results demonstrate that PFDiff exhibits flexible applicability across various pre-trained DPMs, particularly excelling in conditional DPMs and surpassing previous state-of-the-art training-free methods. For instance, using DDIM as a baseline, we achieved 16.46 FID (4 NFE) compared to 138.81 FID with DDIM on ImageNet 64x64 with classifier guidance, and 13.06 FID (10 NFE) on Stable Diffusion with 7.5 guidance scale.

Abstract

Diffusion Probabilistic Models (DPMs) have shown remarkable potential in image generation, but their sampling efficiency is hindered by the need for numerous denoising steps. Most existing solutions accelerate the sampling process by proposing fast ODE solvers. However, the inevitable discretization errors of the ODE solvers are significantly magnified when the number of function evaluations (NFE) is fewer. In this work, we propose PFDiff , a novel training-free and orthogonal timestep-skipping strategy, which enables existing fast ODE solvers to operate with fewer NFE. Based on two key observations: a significant similarity in the model's outputs at time step size that is not excessively large during the denoising process of existing ODE solvers, and a high resemblance between the denoising process and SGD. PFDiff, by employing gradient replacement from past time steps and foresight updates inspired by Nesterov momentum, rapidly updates intermediate states, thereby reducing unnecessary NFE while correcting for discretization errors inherent in first-order ODE solvers. Experimental results demonstrate that PFDiff exhibits flexible applicability across various pre-trained DPMs, particularly excelling in conditional DPMs and surpassing previous state-of-the-art training-free methods. For instance, using DDIM as a baseline, we achieved 16.46 FID (4 NFE) compared to 138.81 FID with DDIM on ImageNet 64x64 with classifier guidance, and 13.06 FID (10 NFE) on Stable Diffusion with 7.5 guidance scale.

1 Introduction

In recent years, Diffusion Probabilistic Models (DPMs) [1-4] have demonstrated exceptional mod21 eling capabilities across various domains including image generation [5-7], video generation [8], 22 text-to-image generation [9, 10], speech synthesis [11], and text-to-3D generation [12, 13]. They have 23 become a key driving force advancing deep generative models. DPMs initiate with a forward process 24 that introduces noise onto images, followed by utilizing a neural network to learn a backward process 25 that incrementally removes noise, thereby generating images [2, 4]. Compared to other generative 26 methods such as Generative Adversarial Networks (GANs) [14] and Variational Autoencoders (VAEs) 27 [15], DPMs not only possess a simpler optimization target but also are capable of producing higher 28 quality samples [5]. However, the generation of high-quality samples via DPMs requires hundreds or 29 thousands of denoising steps, significantly lowering their sampling efficiency and becoming a major 30 barrier to their widespread application. 31

Existing techniques for rapid sampling in DPMs primarily fall into two categories. First, training32

based methods [16-19], which can significantly compress sampling steps, even achieving single-step 33

sampling [19]. However, this compression often comes with a considerable additional training cost, 34

and these methods are challenging to apply to large pre-trained models. Second, training-free samplers 35

[20-30], which typically employ implicit or analytical solutions to Stochastic Differential Equations 36

Text Prompts: Winter night with snow -covered rooftops and soft yellow lights. (Left) A Corgi running towards me in Times Square. (Right)

Stable-Diffusion

Ours +

Stable-Diffusion

NFE = 6, FID = 20.33

NFE = 10, FID = 16.78

NFE = 6, FID = 15.47

NFE = 10, FID = 13.06

NFE = 250, FID = 15.86

NFE = 1000, FID = 15.82

(a) Results from Stable-Diffusion [9] on MS-COCO2014 [31] (Classifier-Free Guidance,

s

= 7

.

5

)

Guided-Diffusion

Ours +

Guided-Diffusion

NFE=4

NFE=8

NFE=10

NFE=100

NFE=4

NFE=8

NFE=10

NFE=250

(SDE)/Ordinary Differential Equations (ODE) for lower-error sampling processes. For instance, Lu 37 et al. [21, 22], by analyzing the semi-linear structure of the ODE solvers for DPMs, have sought to 38 analytically derive optimally the solutions for DPMs' ODE solvers. These training-free sampling 39 strategies can often be used in a plug-and-play fashion, compatible with existing pre-trained DPMs. 40 However, when the NFE is below 10, the discretization error of these training-free methods will be 41 significantly amplified, leading to convergence issues [21, 22], which can still be time-consuming. 42

To further enhance the sampling speed of DPMs, we have analyzed the potential for improvement 43 in existing training-free accelerated methods. Initially, we observed a notably high similarity in the 44 model's outputs for the existing ODE solvers of DPMs when time step size ∆ t is not extremely large, 45 as illustrated in Fig. 2a. This observation led us to utilize the gradients that have been computed 46 from past time steps to approximate current gradients, thereby reducing unnecessary estimation of 47 noise network. Furthermore, due to the similarities between the sampling process of DPMs and 48 Stochastic Gradient Descent (SGD) [33] as noted in Remark 1, we incorporated a foresight update 49 mechanism using Nesterov momentum [34], known for accelerating SGD training. Specifically, we 50 ingeniously employ prior observation to predict future gradients, then utilize the future gradients as a 51 ' springboard ' to facilitate larger update step size ∆ t , as shown in Fig. 2b. 52

Motivated by these insights, we propose PFDiff , a timestep-skipping sampling algorithm that rapidly 53 updates the current intermediate state through the gradient guidance of past and future. Notably, 54 PFDiff is training-free and orthogonal to existing DPMs sampling algorithms, providing a new or55 thogonal axis for DPMs sampling. Unlike previous orthogonal sampling algorithms that compromise 56

0

200

400

600

800

999

Δt

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

1

∑

t = 0

‖εθ

(xt

, t ) Δ ε

θ

(xt + Δt

, t + Δt)‖

2

DDPM(η =1.0)

DDPM(η =0.5)

DDPM(η =0.2)

DDIM

DPM-Solver-1

DPM-Solver++-1

DPM-Solver-2

DPM-Solver-3

(a) Gradient Changes in SDE/ODE Solvers

ODE trajectories

Buffer

B ffer

PFDiff-1

First-ord

Forecasting sample

Gradients saving

Sampling trajectory

'Springboard'

'Springboard'

(b) Comparison of Sampling Trajectories

sampling quality for speed [28], we prove that PFDiff corrects for errors in the sampling trajectories 57 of first-order ODE solvers. This improves sampling quality while reducing unnecessary NFE in 58 existing ODE solvers, as illustrated in Fig. 2b. To validate the orthogonality and effectiveness of 59 PFDiff, extensive experiments were conducted on both unconditional [2, 4, 20] and conditional [5, 9] 60 pre-trained DPMs, with the visualization experiment of conditional DPMs depicted in Fig. 1. The 61 results indicate that PFDiff significantly enhances the sampling performance of existing ODE solvers. 62 Particularly in conditional DPMs, PFDiff, using only DDIM as the baseline, surpasses the previous 63 state-of-the-art training-free sampling algorithms. 64

65

66

67

68

69

70

71

72

73

74

2 Background

2.1 Diffusion SDEs

Diffusion Probabilistic Models (DPMs) [1-4] aim to generate D -dimensional random variables x 0 ∈ R D that follow a data distribution q ( x 0 ) . Taking Denoising Diffusion Probabilistic Models (DDPM) [2] as an example, these models introduce noise to the data distribution through a forward process defined over discrete time steps, gradually transforming it into a standard Gaussian distribution x T ∼ N ( 0 , I ) . The forward process's latent variables { x t } t ∈ [0 ,T ] are defined as follows:

where α t is a scalar function related to the time step t , with α 2 t + σ 2 t = 1 . In the model's reverse process, DDPM utilizes a neural network model p θ ( x t -1 | x t ) to approximate the transition probability q ( x t -1 | x t , x 0 ) ,

where σ 2 θ ( t ) is defined as a scalar function related to the time step t . By sampling from a standard 75 Gaussian distribution and utilizing the trained neural network, samples following the data distribution 76 p θ ( x 0 ) = ∏ T t =1 p θ ( x t -1 | x t ) can be generated. 77

Furthermore, Song et al. [4] introduced SDE to model DPMs over continuous time steps, where the 78 forward process is defined as: 79

.

where w t represents a standard Wiener process, and f and g are scalar functions of the time step t 80 It's noteworthy that the forward process in Eq. (1) is a discrete form of Eq. (3), where f ( t ) = dlog α t d t 81 and g 2 ( t ) = d σ 2 t d t -2 dlog α t d t σ 2 t . Song et al. [4] further demonstrated that there exists an equivalent 82 reverse process from time step T to 0 for the forward process in Eq. (3): 83

84

85

86

87

88

89

90

91

92

where ¯ w denotes a standard Wiener process. In this reverse process, the only unknown is the score function ∇ x log q t ( x t ) , which can be approximated through neural networks.

2.2 Diffusion ODEs

In DPMs based on SDE, the discretization of the sampling process often requires a significant number of time steps to converge, such as the T = 1000 time steps used in DDPM [2]. This requirement primarily stems from the randomness introduced at each time step by the SDE. To achieve a more efficient sampling process, Song et al. [4] utilized the Fokker-Planck equation [35] to derive a probability flow ODE related to the SDE, which possesses the same marginal distribution at any given time t as the SDE. Specifically, the reverse process ODE derived from Eq. (3) can be expressed as:

Unlike SDE, ODE avoids the introduction of randomness, thereby allowing convergence to the data 93 distribution in fewer time steps. Song et al. [4] employed a high-order RK45 ODE solver [36], 94 achieving sample quality comparable to SDE at 1000 NFE with only 60 NFE. Furthermore, research 95 such as DDIM [20] and DPM-Solver [21] explored discrete ODE forms capable of converging in 96 fewer NFE. For DDIM, it breaks the Markov chain constraint on the basis of DDPM, deriving a new 97 sampling formula expressed as follows: 98

99

100

101

where ¯ σ t = η √ (1 -α t -1 ) / (1 -α t ) √ 1 -α t /α t -1 , and α t corresponds to α 2 t in Eq. (1). When η = 1 , Eq. (6) becomes a form of DDPM [2]; when η = 0 , it degenerates into an ODE, the form adopted by DDIM [20], which can obtain high-quality samples in fewer time steps.

102

103

104

105

106

107

108

109

110

111

112

113

114

115

Remark 1. In this paper, we regard the gradient d¯ x t , the noise network output ϵ θ ( x t , t ) , and the score function ∇ x log q t ( x t ) as expressing equivalent concepts. This is because Song et al. [4] demonstrated that ϵ θ ( x t , t ) = -σ t ∇ x log q t ( x t ) . Moreover, we have discovered that any first-order solver of DPMs can be parameterized as x t -1 = ¯ x t -γ t d¯ x t + ξϵ t . Taking DDIM [20] as an example, where ¯ x t = √ α t -1 α t x t , γ t = √ α t -1 α t -α t -1 - √ 1 -α t -1 , d¯ x t = ϵ θ ( x t , t ) , and ξ = 0 . This indicates the similarity between SGD and the sampling process of DPMs, a discovery also implicitly suggested in the research of Xue et al. [30] and Wang et al. [37].

3 Method

3.1 Solving for reverse process diffusion ODEs

By substituting ϵ θ ( x t , t ) = -σ t ∇ x log q t ( x t ) [4], Eq. (5) can be rewritten as:

Given an initial value x T , we define the time steps { t i } T i =0 to progressively decrease from t 0 = T to t T = 0 . Let ˜ x t 0 = x T be the initial value. Using T steps of iteration, we compute the sequence { ˜ x t i } T i =0 to obtain the solution of this ODE. By integrating both sides of Eq. (7), we can obtain the exact solution of this sampling ODE.

For any p -order ODE solver, Eq. (8) can be discretely represented as: 116

where ˆ t 0 = t i -1 , ˆ t p = t i , and ∆ˆ t = ˆ t n +1 -ˆ t n denote the time step size. The function h rep117 resents the different solution methodologies applied by various p -order ODE solvers to the func118 tion s . For the Euler-Maruyama solver [38], h is the identity mapping of s . Further, we define 119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

ϕ ( Q, ˜ x t i -1 , t i -1 , t i ) := ˜ x t i -1 + ∑ p -1 n =0 h ( ϵ θ (˜ x ˆ t n , ˆ t n ) , ˜ x ˆ t n , ˆ t n ) · ∆ˆ t . Here, ϕ is any p -order ODE solver, and buffer Q = ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i -1 , t i ) , where ˆ t 0 = t i -1 and ˆ t p = t i .

When using the ODE solver defined in Eq. (9) for sampling, the choice of T = 1000 leads to significant inefficiencies in DPMs. The study on DDIM [20] first revealed that by constructing a new forward sub-state sequence of length M +1 ( M ≤ T ), { ˜ x t i } M i =0 , from a subsequence of time steps [0 , . . . , T ] and reversing this sub-state sequence, it is possible to converge to the data distribution in fewer time steps. However, as illustrated in Fig. 2a, for ODE solvers, as the time step ∆ t = t i -t i -1 increases, the gradient direction changes slowly initially, but undergoes abrupt changes as ∆ t → T . This phenomenon indicates that under minimal NFE (i.e., maximal time step size ∆ t ) conditions, the discretization error in Eq. (9) is significantly amplified. Consequently, existing ODE solvers, when sampling under minimal NFE, must sacrifice sampling quality to gain speed, making it an extremely challenging task to reduce NFE to below 10 [21, 22]. Given this, we aim to develop an efficient timestep-skipping sampling algorithm, which reduces NFE while correcting discretization errors, thereby ensuring that sampling quality is not compromised, and may even be improved.

3.2 Sampling guided by past gradients

For any p -order timestep-skipping sampling algorithm for DPMs, the sampling process can be reformulated according to Eq. (9) as follows:

where buffer Q = ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i -1 , t i ) and [1 , . . . , M ] is an increasing subsequence of [1 , . . . , T ] . As illustrated in Fig. 2a, when the time step size ∆ t (i.e., t i -t i -1 ) is not excessively large, the MSE of the noise network, defined as 1 T -∆ t ∑ T -∆ t -1 t =0 ∥ ϵ θ ( x t , t ) -ϵ θ ( x t +∆ t , t +∆ t ) ∥ 2 , is remarkably similar. This phenomenon is especially pronounced in ODE-based sampling algorithms, such as DDIM [20] and DPM-Solver [21]. This observation suggests that there are many unnecessary time steps in ODE-based sampling methods during the complete sampling process (e.g., when T = 1000 ), which is one of the reasons these methods can generate samples in fewer steps. Based on this, we propose replacing the noise network of the current timestep with the output from a previous timestep to reduce unnecessary NFE without compromising the quality of the final generated samples. Initially, we store the output of the previous timestep's noise network in a buffer as follows:

Then, in the current timestep, we directly use the noise network output saved in the buffer from the previous timestep to replace the current timestep's noise network output, thereby updating the intermediate states to the next timestep, as detailed below:

By using this approach, we can effectively accelerate the sampling process, reduce unnecessary NFE, and ensure the quality of the samples is not affected. The convergence proof is in Appendix B.1.

3.3 Sampling guided by future gradients

As stated in Remark 1, considering the similarities between the sampling process of DPMs and SGD [33], we introduce a foresight update mechanism of Nesterov momentum, utilizing future gradient information as a ' springboard ' to assist the current intermediate state in achieving more efficient leapfrog updates. Specifically, for the intermediate state ˜ x t i +1 predicted using past gradients as discussed in Sec. 3.2, we first estimate the future gradient and update the current buffer as follows:

Subsequently, leveraging the concept of foresight updates, we predict a further future intermediate state ˜ x t i +2 using the current intermediate state ˜ x t i along with the future gradient information corresponding to ˜ x t i +1 , as shown below:

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

Furthermore, Zhou et al. [39] performed a Principal Component Analysis (PCA) on the sampling trajectories generated by ODE solvers for DPMs and discovered they almost lie in a two-dimensional plane embedded within a high-dimensional space. This implies that the Mean Value Theorem approximately holds during the sampling process using ODE solvers. Specifically, updating the current intermediate state ˜ x t i at an optimal time point s with the corresponding gradient information, ground truth ϵ θ (˜ x t s , t s ) , results in the smallest update error, where s is between time points i and i +2 . Further, we can reason that for any first-order ODE solver, under the same time step, the use of future gradient information ϵ θ (˜ x t i +1 , t i +1 ) from Eq. (13) to update the current intermediate state ˜ x t i results in a smaller sampling error compared to using the gradient information at the current time point ϵ θ (˜ x t i , t i ) . A detailed proof is provided in Appendix B.2. However, for higher-order ODE solvers, the solving process implicitly utilizes future gradients as mentioned in Sec. 3.5, and the additional explicit introduction of future gradients increases sampling error. Therefore, when using higher-order ODE solvers as a baseline, the sampling process is accelerated by only using past gradients. It is only necessary to modify Eq. (14) to ˜ x t i +2 ≈ ϕ ( Q, ˜ x t i +1 , t i +1 , t i +2 ) while keeping Q constant. Ablation experiments can be found in Sec. 4.3.

3.4 PFDiff: sampling guided by past and future gradients

̸

Combining Sec. 3.2 and Sec. 3.3, the intermediate state ˜ x t i +1 obtained through Eq. (12) is used to update the buffer Q in Eq. (13). In this way, we achieve our proposed efficient timestep-skipping algorithm, which we name PFDiff, as shown in Algorithm 1. For higher-order ODE solvers ( p > 1 ), PFDiff only utilizes past gradient information, while for first-order ODE solvers ( p = 1 ), it uses both past and future gradient information to predict further future intermediate states. Notably, during the iteration from intermediate state ˜ x t i to ˜ x t i +2 , we only perform a single batch computation (NFE = p ) of the noise network in Eq. (13). Furthermore, we propose that in a single iteration process, ˜ x t i +2 in Eq. (14) can be modified to ˜ x t i +( k +1) , achieving a k -step skip to sample more distant future intermediate states. Additionally, when k = 1 , the buffer Q , which acts as an intermediate 'springboard' from Eq. (13), has various computational origins. This can be accomplished by modifying ˜ x t i +1 in Eq. (12) to ˜ x t i + l . We collectively refer to this multi-step skipping and different 'springboard' selection strategy as PFDiffk _ l ( l ≤ k ). Further algorithmic details can be found in Appendix C. Finally, through the comparison of sampling trajectories between PFDiff-1 and a first-order ODE sampler, as shown in Fig. 2b, PFDiff-1 showcases its capability to correct the sampling trajectory of the first-order ODE sampler while reducing the NFE.

Proposition 3.1. For any given DPM first-order ODE solver ϕ , the PFDiffk _ l algorithm can describe the sampling process within an iteration cycle through the following formula:

Algorithm 1 PFDiff-1

Require: initial value x , NFE N , model ϵ , any p -order solver ϕ

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

where the value of ϵ θ (˜ x t i -( k -l +1) , t i -( k -l +1) ) can be directly obtained from the buffer Q , without the need for additional computations. The iterative process defined by Eq. (15) ensures that the sampling outcomes converge to the data distribution consistent with the solver ϕ , while effectively correcting errors in the sampling process (Proof in Appendix B).

It is noteworthy that, although the PFDiff is conceptually orthogonal to the SDE/ODE solvers of DPMs, even when the time size ∆ t is relatively small, the MSE of the noise network in the SDE solver exhibits significant differences, as shown in Fig. 2a. Consequently, PFDiff shows marked improvements on the ODE solver, and our experiments are almost exclusively based on ODE solvers, with exploratory experiments on SDE solvers referred to Sec. 4.1.

3.5 Connection with other samplers

Relationship with p -order solver [21, 22, 27]. According to Eq. (10), a single iteration of the p -order solver can be represented as:

A single iteration of the p -order solver uses p NFE to predict the next intermediate state. The intermediate step gradients obtained during this process can be considered as an approximation of future gradients. This approximation is implicitly contained within the sampling guided by future gradients that we propose. Furthermore, as shown in Eq. (15), a single iteration update of PFDiff based on a first-order solver can be seen as using a 2-order solver with only one NFE.

4 Experiments

In this section, we validate the effectiveness of PFDiff as an orthogonal and training-free sampler through a series of extensive experiments. This sampler can be integrated with any order of ODE solvers, thereby significantly enhancing the sampling efficiency of various types of pre-trained DPMs. To systematically showcase the performance of PFDiff, we categorize the pre-trained DPMs into two main types: conditional and unconditional. Unconditional DPMs are further subdivided into discrete and continuous, while conditional DPMs are subdivided into classifier guidance and classifier-free guidance. In choosing ODE solvers, we utilized the widely recognized first-order DDIM [20], Analytic-DDIM [23], and the higher-order DPM-Solver [21] as baselines. For each experiment, we use the Fréchet Inception Distance (FID ↓ ) [40] as the primary evaluation metric, and provide the experimental results of the Inception Score (IS ↑ ) [41] in the Appendix D.7 for reference. Lastly, apart from the ablation studies on parameters k and l discussed in Sec. 4.3, we showcase the optimal results of PFDiffk _ l (where k = 1 , 2 , 3 and l ≤ k ) across six configurations as a performance demonstration of PFDiff. As described in Appendix C, this does not increase the computational burden in practical applications. All experiments were conducted on an NVIDIA RTX 3090 GPU.

4.1 Unconditional sampling

For unconditional DPMs, we selected discrete DDPM [2] and DDIM [20], as well as pre-trained models from continuous ScoreSDE [4], to assess the effectiveness of PFDiff. For these pre-trained models, all experiments sampled 50k instances to compute evaluation metrics.

For unconditional discrete DPMs, we first select first-order ODE solvers DDIM [20] and AnalyticDDIM [23] as baselines, while implementing SDE-based DDPM [2] and Analytic-DDPM [23] methods for comparison, where η = 1 . 0 is from ¯ σ t in Eq. (6). We conduct experiments on the CIFAR10 [42] and CelebA 64x64 [43] datasets using the quadratic time steps employed by DDIM. By varying the NFE from 6 to 20, the evaluation metric FID ↓ is shown in Figs. 3a and 3b. Additionally, experiments with uniform time steps are conducted on the CelebA 64x64, LSUN-bedroom 256x256 [44], and LSUN-church 256x256 [44] datasets, with more results available in Appendix D.2. Our experimental results demonstrate that PFDiff, based on pre-trained models of discrete unconditional DPMs, significantly improves the sampling efficiency of DDIM and Analytic-DDIM samplers across multiple datasets. For instance, on the CIFAR10 dataset, PFDiff combined with DDIM achieves a FID of 4.10 with only 15 NFE, comparable to DDIM's performance of 4.04 FID with 1000 NFE. This is something other time-step skipping algorithms [23, 28] that sacrifice sampling quality for speed

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

6

8

10

12

15

20

NFE

10

20

30

40

50

60

70

FID

DDPM(η =1.0)

Analytic-DDPM(η = 1.0)

DDIM

Analytic-DDIM

DDIM+Ours

Analytic-DDIM+Ours

(a) CIFAR10 (Discrete)

6

8

10

12

15

20

NFE

5

10

15

20

25

30

35

40

45

FID

DDPM(η =1.0)

Analytic-DDPM(η = 1.0)

DDIM

Analytic-DDIM

DDIM+Ours

Analytic-DDIM+Ours

(b) CelebA 64x64 (Discrete)

6

8

9

10

12

15

16

20

21

NFE

5

10

20

50

300

FID

DPM-Solver-1

DPM-Solver-2

DPM-Solver-3

DPM-Solver-1+Ours

DPM-Solver-2+Ours

DPM-Solver-3+Ours

4

6

8

10

15

20

NFE

0

10

20

30

40

50

60

FID

DDIM

DPM-Solver-2

DPM-Solver-3

DPM-Solver++(2M)

∗ AutoDiffusion

DDIM+Ours

(a) ImageNet 64x64

(Guided-Diffusion)

(Classifier Guidance,

s

= 1

.

0

)

5

6

8

10

15

20

NFE

20

30

40

50

60

70

FID

DDIM

DPM-Solver-2

DPM-Solver-3

DPM-Solver++(2M)

DDIM+Ours

(b) MS-COCO2014

(Stable-Diffusion)

(Classifier-Free Guidance,

s

= 1

.

5

)

5

6

8

10

15

20

NFE

14

16

18

20

22

24

FID

DDIM

DPM-Solver-2

† DPM-Solver++(2M)

† DPM-Solver-v3(2M)

† UniPC

DDIM+Ours

(c) MS-COCO2014

(Stable-Diffusion)

(Classifier-Free Guidance,

s

= 7

.

5

)

cannot achieve. Furthermore, in Appendix D.2, by varying η from 1.0 to 0.0 in Eq. (6) to control the scale of noise introduced by SDE, we observe that as η decreases (reducing noise introduction), the performance of PFDiff gradually improves. This once again validates our assumption proposed in Sec. 3.2, based on Fig. 2a, that there is a significant similarity in the model's outputs at the time step size that is not excessively large for the existing ODE solvers.

For unconditional continuous DPMs, we choose the DPM-Solver-1, -2 and -3 [21] as the baseline to verify the effectiveness of PFDiff as an orthogonal timestep-skipping algorithm on the first and higher-order ODE solvers. We conducted experiments on the CIFAR10 [42] using quadratic time steps, varying the NFE. The experimental results using FID ↓ as the evaluation metric are shown in Fig. 3c. More experimental details can be found in Appendix D.3. We observe that PFDiff consistently improves the sampling performance over the baseline with fewer NFE settings, particularly in cases where higher-order ODE solvers fail to converge with a small NFE (below 10) [21].

4.2 Conditional sampling

For conditional DPMs, we selected the pre-trained models of the widely recognized classifier guidance paradigm, ADM-G [5], and the classifier-free guidance paradigm, Stable-Diffusion [9], to validate the effectiveness of PFDiff. We employed uniform time steps setting and the DDIM [20] ODE solver as a baseline across all datasets. Evaluation metrics were computed by sampling 50k samples on the ImageNet 64x64 [32] dataset for ADM-G and 10k samples on other datasets, including ImageNet 256x256 [32] in ADM-G and MS-COCO2014 [31] in Stable-Diffusion.

For conditional DPMs employing the classifier guidance paradigm, we conducted experiments on the 261 ImageNet 64x64 dataset [32] with a guidance scale ( s ) set to 1.0. For comparison, we implemented 262 DPM-Solver-2 and -3 [21], and DPM-Solver++(2M) [22], which exhibit the best performance on 263 conditional DPMs. Additionally, we introduced the AutoDiffusion method [26] using DDIM as a 264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

baseline for comparison, noting that this method incurs additional search costs. We compared FID ↓ scores by varying the NFE as depicted in Fig. 4a, with corresponding visual comparisons shown in Fig. 1b. We observed that PFDiff reduced the FID from 138.81 with 4 NFE in DDIM to 16.46, achieving an 88.14% improvement in quality. The visual results in Fig. 1b further demonstrate that, at the same NFE setting, PFDiff achieves higher-quality sampling. Furthermore, we evaluated PFDiff's sampling performance based on DDIM on the large-scale ImageNet 256x256 dataset [32]. Detailed results are provided in Appendix D.4.

For conditional, classifier-free guidance paradigms of DPMs, we employed the sd-v1-4 checkpoint and computed the FID ↓ scores on the validation set of MS-COCO2014 [31]. We conducted experiments with a guidance scale ( s ) set to 7.5 and 1.5. For comparison, we implemented DPM-Solver-2 and -3 [21], and DPM-Solver++(2M) [22] methods. At s = 7 . 5 , we introduced the state-of-the-art method reported in DPM-Solver-v3 [27] for comparison, along with DPM-Solver++(2M) [22], UniPC [29], and DPM-Solver-v3(2M) [27]. The FID ↓ metrics by varying the NFE are presented in Figs. 4b and 4c, with additional visual results illustrated in Fig. 1a. We observed that PFDiff, solely based on DDIM, achieved state-of-the-art results during the sampling process of Stable-Diffusion, thus demonstrating the efficacy of PFDiff. Further experimental details can be found in Appendix D.5.

4.3 Ablation study

We conducted ablation experiments on the six different algorithm configurations of PFDiff mentioned in Appendix C, with k = 1 , 2 , 3 ( l ≤ k ). Specifically, we evaluated the FID ↓ scores on the unconditional and conditional pre-trained DPMs [2, 4, 5, 9] by varying the NFE. Detailed experimental setups and results can be found in Appendix D.6.1. The experimental results indicate that for various pre-trained DPMs, the choice of parameters k and l is not critical, as most combinations of k and l within PFDiff can enhance the sampling efficiency over the baseline. Moreover, with k = 1 fixed, PFDiff-1 can significantly improve the baseline's sampling quality within the range of 8 ∼ 20 NFE. For even better sampling quality, one can sample a small subset of examples (e.g., 5k) to compute evaluation metrics or directly conduct visual analysis, easily identifying the most effective k and l combinations.

To validate the PFDiff algorithm as mentioned in Sec. 3.3, which necessitates the joint guidance of past and future gradients for first-order ODE solvers, and only past gradients for higher-order ODE solvers, offering a more effective means of accelerating baseline sampling. This study employs the first-order ODE solver DDIM [20] as the baseline, isolating the effects of both past and future gradients, and uses the higher-order ODE solver DPM-Solver [21] as the baseline, removing the influence of future gradients for ablation experiments. Specific experimental configurations and results are shown in Appendix D.6.2. The results indicate that, as described by the PFDiff algorithm in Sec. 3.3, it is possible to further enhance the sampling efficiency of ODE solvers of any order.

5 Conclusion

In this paper, based on the recognition that the ODE solvers of DPMs exhibit significant similarity in model outputs when the time step size is not excessively large, and with the aid of a foresight update mechanism, we propose PFDiff, a novel method that leverages the gradient guidance from both past and future to rapidly update the current intermediate state. This approach effectively reduces the unnecessary number of function evaluations (NFE) in the ODE solvers and significantly corrects the errors of first-order ODE solvers during the sampling process. Extensive experiments demonstrate the orthogonality and efficacy of PFDiff on both unconditional and conditional pre-trained DPMs, especially in conditional pre-trained DPMs where PFDiff outperforms previous state-of-the-art training-free sampling methods.

Limitations and broader impact Although PFDiff can effectively accelerate the sampling speed of existing ODE solvers, it still lags behind the sampling speed of training-based acceleration methods and one-step generation paradigms such as GANs. Moreover, there is no universal setting for the optimal combination of parameters k and l in PFDiff; adjustments are required according to different pre-trained DPMs and NFE. It is noteworthy that PFDiff may be utilized to accelerate the generation of malicious content, thereby having a detrimental impact on society.

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

References

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, et al. Patch n'pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems , 36, 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8, 2023.

Kaitao Song, Yichong Leng, Xu Tan, Yicheng Zou, Tao Qin, and Dongsheng Li. Transcormer: Transformer for sentence scoring with sliding language modeling. Advances in Neural Information Processing Systems , 35:11160-11174, 2022.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:2206.02262 , 2022.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint 364 arXiv:2303.01469 , 2023. 365

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 366 arXiv:2010.02502 , 2020. 367

368

369

370

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503 , 2022.

Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309 , 2022.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7105-7114, 2023.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. arXiv preprint arXiv:2310.13268 , 2023.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. arXiv preprint arXiv:2312.00858 , 2023.

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

Yurii Nesterov. A method of solving a convex programming problem with convergence rate o (1/k** 2). Doklady Akademii Nauk SSSR , 269(3):543, 1983.

Bernt Øksendal and Bernt Øksendal. Stochastic differential equations . Springer, 2003. 404

John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational 405 and applied mathematics , 6(1):19-26, 1980. 406

Kai Wang, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural 407 network diffusion. arXiv preprint arXiv:2402.13144 , 2024. 408

Peter E Kloeden, Eckhard Platen, Peter E Kloeden, and Eckhard Platen. Stochastic differential equations . 409 Springer, 1992. 410

Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in 411 around 5 steps. arXiv preprint arXiv:2312.00094 , 2023. 412

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans 413

trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information 414

processing systems , 30, 2017. 415

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved 416 techniques for training gans. Advances in neural information processing systems , 29, 2016. 417

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 418

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 419 Proceedings of the IEEE international conference on computer vision , pages 3730-3738, 2015. 420

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con421 struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 422 arXiv:1506.03365 , 2015. 423

Elliott Ward Cheney, EW Cheney, and W Cheney. Analysis for applied mathematics , volume 1. Springer, 424 2001. 425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

A Related work

While the solvers for Diffusion Probabilistic Models (DPMs) are categorized into two types, SDE and ODE, most current accelerated sampling techniques are based on ODE solvers due to the observation that the stochastic noise introduced by SDE solvers hampers rapid convergence. ODE-based solvers are further divided into training-based methods [16-19] and training-free samplers [20-30]. Trainingbased methods can notably reduce the number of sampling steps required for DPMs. An example of such a method is the knowledge distillation algorithm proposed by Song et al. [19], which achieves one-step sampling for DPMs. This sampling speed is comparable to that of GANs [14] and VAEs [15]. However, these methods often entail significant additional costs for distillation training. This requirement poses a challenge when applying them to large pre-trained DPM models. Therefore, our work primarily focuses on training-free, ODE-based accelerated sampling strategies.

Training-free accelerated sampling techniques based on ODE can generally be applied in a plugand-play manner, adapting to existing pre-trained DPMs. These methods can be categorized based on the order of the ODE solver-that is, the NFE required per sampling iteration-into first-order [20, 23-25] and higher-order [21, 22, 27, 29, 36]. Typically, higher-order ODE solvers tend to sample at a faster rate, but may fail to converge when the NFE is low (below 10), sometimes performing even worse than first-order ODE solvers. In addition, there are orthogonal techniques for accelerated sampling. For instance, Li et al. [26] build upon existing ODE solvers and use search algorithms to find optimal sampling sub-sequences and model structures to further speed up the sampling process; Ma et al. [28] observe that the low-level features of noise networks at adjacent time steps exhibit similarities, and they use caching techniques to substitute some of the network's low-level features, thereby further reducing the number of required time steps.

The algorithm we propose belongs to the class of training-free and orthogonal accelerated sampling techniques, capable of further accelerating the sampling process on the basis of existing first-order and higher-order ODE solvers. Compared to the aforementioned orthogonal sampling techniques, even though the skipping strategy proposed by Ma et al. [28] effectively accelerates the sampling process, it may do so at the cost of reduced sampling quality, making it challenging to reduce the NFE below 50. Although Li et al. [26] can identify more optimal subsampling sequences and model structures, this implies higher search costs. In contrast, our proposed orthogonal sampling algorithm is more efficient in skipping time steps. First, our skipping strategy does not require extensive search costs. Second, we can correct the sampling errors of first-order ODE solvers while reducing the number of sampling steps required by existing ODE solvers, achieving more efficient accelerated sampling.

B Proof of convergence and error correction for PFDiff

In this section, we prove the convergence of PFDiff and elaborate on how it theoretically corrects first-order ODE solver errors. To delve deeper into PFDiff, we propose the following assumptions:

Assumption B.1. When the time step size ∆ t = t i -t i -( k -l +1) is not excessively large, the output estimates of the noise network based on the p -order ODE solver at different time steps are approximately the same, that is, ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i , t i + l ) ≈ ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i -( k -l +1) , t i ) .

Assumption B.2. For the integral from time step t i to t i +( k +1) , ∫ t i +( k +1) t i s ( ϵ θ ( x t , t ) , x t , t )d t , there exist intermediate time steps t ˜ s , t s ∈ ( t i , t i +( k +1) ) such that ∫ t i +( k +1) t i s ( ϵ θ ( x t , t ) , x t , t )d t = s ( ϵ θ ( x t ˜ s , t ˜ s ) , x t ˜ s , t ˜ s ) · ( t i +( k +1) -t i ) = h ( ϵ θ ( x t s , t s ) , x t s , t s ) · ( t i +( k +1) -t i ) holds, where the definition of the function h remains consistent with Sec. 3.1.

The first assumption is based on the observation in Fig. 2a that when ∆ t is not excessively large, the MSE of the noise network remains almost unchanged across different time steps. The second assumption is based on the Mean Value Theorem for Integrals, which states that if f ( x ) is a continuous real-valued function on a closed interval [ a, b ] , then there exists at least one point c ∈ [ a, b ] such that

b

∫ a f ( x )d x = f ( c )( b -a ) holds. It is important to note that the Mean Value Theorem for Integrals originally applies to real-valued functions and does not directly apply to vector-valued functions [45]. However, the study by Zhou et al. [39] using Principal Component Analysis (PCA) on the trajectories of the ODE solvers for DPMs demonstrates that these trajectories are primarily distributed

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

on a two-dimensional plane, which allows the Mean Value Theorem for Integrals to approximately hold under Assumption B.2.

B.1 Proof of convergence for sampling guided by past gradients

Starting from Eq. (8), we consider an iteration process of a p -order ODE solver from ˜ x t i to ˜ x t i + l , where l is the ' springboard ' choice determined by PFDiffk _ l . This iterative process can be expressed as:

Discretizing Eq. (B.1) yields:

where ˆ t 0 = t i and ˆ t p = t i + l . Consistent with Sec. 3.1, the function h represents the different solution methodologies applied by various p -order ODE solvers to the function s . To accelerate sampler convergence and reduce unnecessary evaluations of the NFE, we adopt Assumption B.1, namely guiding the sampling of the current intermediate state by utilizing past gradient information. Specifically, we approximate that ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i , t i + l ) ≈ ( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i -( k -l +1) , t i ) , where k represents the number of steps skipped in one iteration by PFDiffk _ l . Eq. (B.2) can be rewritten as:

where ϕ is any p -order ODE solver. Eq. (B.3) demonstrates that under Assumption B.1, for any p -order ODE solver ϕ , PFDiffk _ l utilizes past gradient information as a substitute for current gradient information to update the current intermediate state. This method not only reduces the NFE but also approximates the solution of ˜ x t i + l , ensuring convergence to the data distribution corresponding to the solver ϕ . It is important to note that the sampling process described in Eq. (B.3) relies solely on past gradient information and does not estimate the output of the noise network based on the current intermediate state.

In particular, within Proposition 3.1 for any first-order ( p = 1 ) ODE solver ϕ , according to Eq. (B.3), we can approximate ˜ x t i + l ≈ ϕ ( ϵ θ (˜ x t i -( k -l +1) , t i -( k -l +1) ) , ˜ x t i , t i , t i + l ) . Thus, Eq. (15) can be rewritten as:

For any first-order ODE solver ϕ , Eq. (B.3) and (B.4) utilize the gradient information from both past and future to constitute a complete sampling iteration process for PFDiffk _ l . Eq. (B.4) indicates that under the Assumption B.1 and upon the convergence of Eq. (B.4), PFDiffk _ l is guaranteed to converge to the data distribution corresponding to the sampler ϕ for any first-order ODE solver.

B.2 Error correction and proof of convergence of Proposition 3.1

Based on Eq. (8), we consider an iteration process of a first-order ( p = 1 ) ODE solver from ˜ x t i to ˜ x t i +( k +1) , which can be expressed as:

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

where the second line of Eq. (B.5) is obtained by discretizing the first line with an existing first-order ODE solver ( p = 1 ), and the definition of ϕ and h are consistent with Appendix B.1. It is well-known that the discretization method used in Eq. (B.5) restricts the sampling step size ∆ t = t i +( k +1) -t i of the first-order ODE solver. A too-large step size will cause the first-order ODE solver to not converge. This indicates that although Assumption B.2 points out that the sampling trajectory of the first-order ODE solver lies on a two-dimensional plane, this trajectory is not a straight line (if it were a straight line, a larger sampling step size could be used). Therefore, using ϵ θ (˜ x t i , t i ) for discretized sampling in Eq. (B.5) introduces a significant sampling error, as shown by the first-order ODE sampling trajectory in Fig. 2b. To reduce the first-order ODE solver sampling error, we have revised Eq. (B.5) based on Assumption B.2, as follows:

̸

where k and l are determined by the selected PFDiffk _ l and the second and third lines are obtained based on Assumption B.2. Combining Eq. (B.6) and Eq. (B.3) leads to the complete Eq. (15), thereby completing the convergence proof of Proposition 3.1. Moreover, t s falls within the interval [ t i , t i +( k +1) ] , and since the sampling trajectory of the first-order ODE solver is not a straight line, generally t s = t i and t s = t i +( k +1) . The interval [ t i , t i +( k +1) ] is amended to ( t i , t i +( k +1) ) . By adopting the foresight update mechanism of the Nesterov momentum [34], and guiding the current intermediate state sampling with future gradient information, we replace ϵ θ (˜ x t s , t s ) with ϵ θ (˜ x t i + l , t i + l ) . According to the definition of PFDiffk _ l , t i + l also lies within the interval ( t i , t i +( k +1) ) , and for different combinations of k and l , this means searching and approximating the ground truth t s within the interval ( t i , t i +( k +1) ) . Among the six different versions of PFDiffk _ l defined in Appendix C, we believe that the optimal t s has been approximated. Compared to the direct discretization of ϵ θ ( x t , t ) in Eq. (B.5), we corrected the sampling error of the first-order ODE solver and proved its convergence by guiding sampling based on the future gradient information ϵ θ (˜ x t i + l , t i + l ) under Assumption B.2, as shown in the sampling trajectory of PFDiff-1 in Fig. 2b. Together with this section and Appendix B.1, this completes the error correction and convergence proof of Proposition 3.1.

̸

C Algorithms of PFDiffs

As described in Sec. 3.4, during a single iteration, we can leverage the foresight update mechanism to skip to a more distant future. Specifically, we modify Eq. (14) to ˜ x t i +( k +1) ≈ ϕ ( Q, ˜ x t i , t i , t i +( k +1) ) to achieve a k -step skip. We refer to this method as PFDiffk . Additionally, when k = 1 , the computation of the buffer Q , originating from Eq. (13), presents different selection choices. We modify Eq. (12) to ˜ x t i + l ≈ ϕ ( Q, ˜ x t i , t i , t i + l ) , l ≤ k to denote different ' springboard ' choices with the parameter l . This strategy of multi-step skips and varying 'springboard' choices is collectively termed as PFDiffk _ l ( l ≤ k ). Consequently, based on modifications to parameters k and l in Eq. (12) and Eq. (14), Eq. (13) is updated to Q buffer ← - - -( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i + l , t i +( k +1) ) , and Eq. (11) is updated to Q buffer ← - - -( { ϵ θ (˜ x ˆ t n , ˆ t n ) } p -1 n =0 , t i -( k -l +1) , t i ) . When k = 1 , since l ≤ k , then l = 1 , and PFDiffk _ l is the same as PFDiff-1, as shown in Algorithm 1 in Sec. 3.4. When k = 2 , l can be either 1 or 2, forming Algorithms PFDiff-2_1 and PFDiff-2_2, as shown in Algorithm 2. Furthermore, when k = 3 , this forms three different versions of PFDiff-3, as shown in Algorithm 3. In this study, we utilize the optimal results from the six configurations of PFDiffk _ l ( k = 1 , 2 , 3 ( l ≤ k )) to demonstrate the performance of PFDiff. As described in Appendix B.2, this is essentially an approximation of the ground truth t s . Through these six different algorithm configurations, we approximately search for the optimal t s . It is important to note that despite using six different algorithm configurations, this does not increase the computational burden in practical applications. This is because, by visual analysis of a small number of generated images or computing specific evaluation metrics, one can effectively select the algorithm configuration with the best performance. Moreover, even without any selection, directly using the PFDiff-1 configuration can achieve significant

̸

performance improvements on top of existing ODE solvers, as shown in the ablation study results in 553 Sec. 4.3. 554

Algorithm 2 PFDiff-2

Algorithm 3 PFDiff-3

D Additional experiment results 555

In this section, we provide further supplements to the experiments on both unconditional and 556 conditional pre-trained Diffusion Probabilistic Models (DPMs) as mentioned in Sec. 4. Through 557 these additional supplementary experiments, we more fully validate the effectiveness of PFDiff as an 558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

orthogonal and training-free sampler. Unless otherwise stated, the selection of pre-trained DPMs, choice of baselines, algorithm configurations, GPU utilization, and other related aspects in this section are consistent with those described in Sec. 4.

D.1 License

In this section, we list the used datasets, codes, and their licenses in Table 1.

D.2 Additional results for unconditional discrete-time sampling

In this section, we report on experiments with unconditional, discrete DPMs on the CIFAR10 [42] and CelebA 64x64 [43] datasets using quadratic time steps. The FID ↓ scores for the PFDiff algorithm are reported for changes in the number of function evaluations (NFE) from 4 to 20. Additionally, we present FID scores on the CelebA 64x64 [43], LSUN-bedroom 256x256 [44], and LSUN-church 256x256 [44] datasets, utilizing uniform time steps. The experimental results are summarized in Table 2. Results indicate that using DDIM [20] as the baseline, our method (PFDiff) nearly achieved significant performance improvements across all datasets and NFE settings. Notably, PFDiff facilitates rapid convergence of pre-trained DPMs to the data distribution with NFE settings below 10 , validating its effectiveness on discrete pre-trained DPMs and the first-order ODE solver DDIM. It is important to note that on the CIFAR10 and CelebA 64x64 datasets, we have included the FID scores of Analytic-DDIM [23], which serves as another baseline. Analytic-DDIM modifies the variance in DDIM and introduces some random noise. With NFE lower than 10 , the presence of minimal random noise amplifies the error introduced by the gradient information approximation in PFDiff, reducing its error correction capability compared to the Analytic-DDIM sampler. Thus, in fewer-step sampling (NFE < 10), using DDIM as the baseline is more effective than using Analytic-DDIM, which requires recalculating the optimal variance for different pre-trained DPMs, thereby introducing additional computational overhead. In other experiments with pre-trained DPMs, we validate the efficacy of the PFDiff algorithm by combining it with the overall superior performance of the DDIM solver.

Furthermore, to validate the motivation proposed in Sec. 3.2 based on Fig. 2a-that at not excessively 583 large time step size ∆ t , an ODE-based solver shows considerable similarity in the noise network 584 outputs-we compare it with the SDE-based solver DDPM [2]. Even at smaller ∆ t , the mean 585 squared error (MSE) of the noise outputs from DDPM remains high, suggesting that the effectiveness 586 of PFDiff may be limited when based on SDE solvers. Further, we adjusted the η parameter in 587 Eq. (6) (which controls the amount of noise introduced in DDPM) from 1.0 to 0.0 (at η = 0 . 0 , 588 the SDE-based DDPM degenerates into the ODE-based DDIM [20]). As shown in Fig. 2a, as η 589 decreases, the MSE of the noise network outputs gradually decreases at the same time step size ∆ t , 590 indicating that reducing noise introduction can enhance the effectiveness of PFDiff. To verify this 591 motivation, we utilized quadratic time steps on CIFAR10 and CelebA 64x64 datasets and controlled 592

593

594

595

596

597

598

599

600

601

602

603

604

the amount of noise introduced by adjusting η , to demonstrate that PFDiff can leverage the temporal redundancy present in ODE solvers to boost its performance. The experimental results, as shown in Table 3, illustrate that with the reduction of η from 1.0 (SDE) to 0.0 (ODE), PFDiff's sampling performance significantly improves at fewer time steps (NFE ≤ 20). The experiment results regarding FID variations with NFE as presented in Table 3, align with the trends of MSE of noise network outputs with changes in time step size ∆ t as depicted in Fig. 2a. This reaffirms the motivation we proposed in Sec. 3.2.

D.3 Additional results for unconditional continuous-time sampling

In this section, we supplement the specific FID ↓ scores for the unconditional, continuous pretrained DPMs models with first-order and higher-order ODE solvers, DPM-Solver-1, -2 and -3, [21] as baselines, as shown in Table 4. For all experiments in this section, we conducted tests on the CIFAR10 dataset [42], using the checkpoint checkpoint_8.pth under the

vp/cifar10_ddpmpp_deep_continuous configuration provided by ScoreSDE [4]. For the hyper605 parameter method of DPM-Solver [21], we adopted singlestep_fixed ; to maintain consistency 606 with the discrete-time model in Appendix D.2, the parameter skip was set to time_quadratic (i.e., 607 quadratic time steps). Unless otherwise specified, we used the parameter settings recommended by 608 DPM-Solver. The results in Table 4 show that by using the PFDiff method described in Sec. 3.4 609 and taking DPM-Solver as the baseline, we were able to further enhance sampling performance on 610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

the basis of first-order and higher-order ODE solvers. Particularly, in the 6 ∼ 12 NFE range, PFDiff significantly improved the convergence issues of higher-order ODE solvers under fewer NFEs. For instance, at 9 NFE, PFDiff reduced the FID of DPM-Solver-3 from 233.56 to 5.67, improving the sampling quality by 97.57%. These results validate the effectiveness of using PFDiff with first-order or higher-order ODE solvers as the baseline.

D.4 Additional results for classifier guidance

In this section, we provide the specific FID scores for pre-trained DPMs in the conditional, classifier guidance paradigm on the ImageNet 64x64 [32] and ImageNet 256x256 datasets [32], as shown in Table 5. We now describe the experimental setup in detail. For the pre-trained models, we used the ADM-G [5] provided 64x64_diffusion.pt and 64x64_classifier.pt for the ImageNet 64x64 dataset, and 256x256_diffusion.pt and 256x256_classifier.pt for the ImageNet 256x256 dataset. All experiments were conducted with uniform time steps and used DDIM as the baseline [20]. Weimplemented the second-order and third-order methods from DPM-Solver [21] for comparison and explored the method hyperparameter provided by DPM-Solver for both singlestep (corresponding to 'Single' in Table 5) and multistep (corresponding to 'Multi' in Table 5). Additionally, we implemented the best-performing method from DPM-Solver++ [22], multi-step DPM-Solver++(2M), as a comparative measure. Furthermore, we also introduced the superior-performing AutoDiffusion [26] method as a comparison. ∗ We directly borrowed the results reported in the original paper, emphasizing that although AutoDiffusion does not require additional training, it incurs additional search costs. '\' represents missing data in the original paper. The specific experimental results of the configurations mentioned are shown in Table 5. The results demonstrate that PFDiff, using DDIM as the baseline on the ImageNet 64x64 dataset, significantly enhances the sampling efficiency of DDIM and surpasses previous optimal training-free sampling methods. Particularly, in cases where NFE ≤ 10, PFDiff improved the sampling quality of DDIM by 41.88% ∼ 88.14%. Moreover, on the large ImageNet 256x256 dataset, PFDiff demonstrates a consistent performance improvement over the DDIM baseline, similar to the improvements observed on the ImageNet 64x64 dataset.

D.5 Additional results for classifier-free guidance

In this section, we supplemented the specific FID ↓ scores for the Stable-Diffusion [9] (conditional, classifier-free guidance paradigm) setting with a guidance scale ( s ) of 7.5 and 1.5. Specifically, for

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

the pre-trained model, we conducted experiments using the sd-v1-4.ckpt checkpoint provided by Stable-Diffusion. All experiments used the MS-COCO2014 [31] validation set to calculate FID ↓ scores, with uniform time steps. PFDiff employs the DDIM [20] method as the baseline. Initially, under the recommended s = 7 . 5 configuration by Stable-Diffusion, we implemented DPM-Solver-2 and -3 as comparative methods, and conducted searches for the method hyperparameters provided by DPM-Solver as singlestep (corresponding to 'Single' in Table 6) and multistep (corresponding to 'Multi' in Table 6). Additionally, we introduced previous state-of-the-art training-free methods, including DPM-Solver++(2M) [22], UniPC [29], and DPM-Solver-v3(2M) [27] for comparison. The experimental results are shown in Table 6. † We borrow the results reported in DPM-Solver-v3 [27] directly. The results indicate that on Stable-Diffusion, PFDiff, using only DDIM as a baseline, surpasses the previous state-of-the-art training-free sampling methods in terms of sampling quality in fewer steps (NFE<20). Particularly, at NFE=10, PFDiff achieved a 13.06 FID, nearly converging to the data distribution, which is a 14.25% improvement over the previous state-of-the-art method DPM-Solver-v3 at 20 NFE, which had a 15.23 FID. Furthermore, to further validate the effectiveness of PFDiff on Stable-Diffusion, we conducted experiments using the s = 1 . 5 setting with the same experimental configuration as s = 7 . 5 . For the comparative methods, we only experimented with the multi-step versions of DPM-Solver-2 and -3 and DPM-Solver++(2M), which had faster convergence at fewer NFE under the s = 7 . 5 setting. As for UniPC and DPM-Solver-v3(2M), since DPM-Solverv3 did not provide corresponding experimental results at s = 1 . 5 , we did not list their comparative results. The experimental results show that PFDiff, using DDIM as the baseline under the s = 1 . 5 setting, demonstrated consistent performance improvements as seen in the s = 7 . 5 setting, as shown in Table 6.

D.6 Additional ablation study results

D.6.1 Additional results for PFDiff hyperparameters study

In this section, we extensively investigate the impact of the hyperparameters k and l on the perfor664 mance of the PFDiff algorithm, supplemented by a series of ablation experiments regarding their 665 configurations and outcomes. Specifically, we first conducted experiments on the CIFAR10 dataset 666

667

668

669

670

671

672

673

674

675

676

677

678

679

[42] using quadratic time steps, based on pre-trained unconditional discrete DDPM [2] and continuous ScoreSDE [4] DPMs. For the conditional DPMs, we used uniform time steps in classifier guidance ADM-G [5] pre-trained DPMs, setting the guidance scale ( s ) to 1.0 for experiments on the ImageNet 64x64 dataset [32]; for the classifier-free guidance Stable-Diffusion [9] pre-trained DPMs, we set the guidance scale ( s ) to 7.5. All experiments were conducted using the DDIM [20] algorithm as a baseline, and PFDiffk _ l configurations ( k = 1 , 2 , 3 ( l ≤ k )) were tested in six different algorithm configurations. The change in NFE and the corresponding FID ↓ scores are shown in Table 7. The experimental results show that under various combinations of k and l , PFDiff is able to enhance the sampling performance of the DDIM baseline in most cases across different types of pre-trained DPMs. Particularly when k = 1 is fixed, PFDiff-1 significantly improves the sampling performance of the DDIM baseline within the range of 8 ∼ 20 NFE. For practical applications requiring higher sampling quality at fewer NFE, optimal combinations of k and l can be identified by fixing NFE and sampling a small number of samples for visual analysis or computing specific metrics, without significantly

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

increasing the computational burden. However, as discussed in Sec. 5, although the experimental results presented in Table 7 demonstrate the excellent performance of the combinations of k and l under various pre-trained DPMs and NFE settings, no universally optimal configuration exists. This finding somewhat limits the generality of the proposed PFDiff algorithm and sets objectives for our future research.

D.6.2 Ablation study of gradient guidance

To further investigate the impact of gradient guidance from the past or future on the rapid updating of current intermediate states, this section supplements experimental results and analysis using first-order and higher-order ODE solvers as baselines. Specifically, as described in Sec. 3.3, PFDiff uses a first-order ODE solver as a baseline, where future gradient guidance corrects sampling errors, with detailed proofs provided in Appendix B.2. Hence, using the first-order ODE solver DDIM [20] as a baseline, we removed past and future gradients separately and employed quadratic time steps. This was based on the pre-trained model from DDPM [2] on the CIFAR10 [42] dataset, evaluating the FID ↓ metric by changing the number of function evaluations (NFE). For higher-order ODE solvers, the solving process implicitly utilizes future gradients as mentioned in Sec. 3.5, and the additional explicit introduction of future gradients increases sampling error. Therefore, when using higher-order ODE solvers as a baseline, PFDiff accelerates the sampling process using only past gradients. Specifically, for higher-order ODE solvers, we selected DPM-Solver-2 and -3 [21] as the baseline, also employing quadratic time steps, and based on the ScoreSDE [4] pre-trained model on CIFAR10 [42]. Only the future gradients were removed to validate the effectiveness of the PFDiff algorithm by changing the NFE and evaluating the FID ↓ metric. As shown in Table 8, the experimental results indicate that using the first-order ODE solver DDIM as a baseline, employing only past gradients (similar to DeepCache [28]), or only future gradients, only slightly improves the baseline's sampling performance; however, combining both significantly enhances the baseline's sampling performance. Meanwhile, using higher-order ODE solvers DPM-Solver-2 and -3 as the baseline, because the algorithm inherently contains future gradients, continuing to explicitly introduce future gradients increases the overall error. Therefore, using only past gradients (PFDiff) significantly improves the baseline's sampling efficiency, especially under fewer steps (NFE<10), where PFDiff markedly ameliorates the non-convergence issues of the higher-order ODE solvers.

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

D.7 Inception score experimental results

To evaluate the effectiveness of the PFDiff algorithm and the widely used Fréchet Inception Distance (FID ↓ ) metric [40] in the sampling process of Diffusion Probabilistic Models (DPMs), we have also incorporated the Inception Score (IS ↑ ) metric [41] for both unconditional and conditional pre-trained DPMs. Specifically, for the unconditional discrete-time pre-trained DPMs DDPM [2], we maintained the experimental configurations described in Table 2 of Appendix D.2, and added IS scores for the CIFAR10 dataset [42]. For the unconditional continuous-time pre-trained DPMs ScoreSDE[4], the experimental configurations are consistent with Table 4 in Appendix D.3, and IS scores for the CIFAR10 dataset were also added. For the conditional classifier guidance paradigm of pre-trained DPMs ADM-G [5], the experimental setup aligned with Table 5 in Appendix D.4, including IS scores for the ImageNet 64x64 and ImageNet 256x256 datasets [32]. Considering that the computation of IS scores relies on features extracted using InceptionV3 pre-trained on the ImageNet dataset, calculating IS scores for non-ImageNet datasets was not feasible, hence no IS scores were provided for the classifier-free guidance paradigm of Stable-Diffusion [9]. The experimental results are presented in Table 9. A comparison between the FID ↓ metrics in Tables 2, 4, and 5 and the IS ↑ metrics in Table 9 shows that both IS and FID metrics exhibit similar trends under the same experimental settings, i.e., as the number of function evaluations (NFE) changes, lower FID scores correspond to higher IS scores. Further, Figs. 1a and 1b, along with the visualization experiments in Appendix D.8, demonstrate that lower FID scores and higher IS scores correlate with higher image quality and richer

728

729

730

details generated by the PFDiff sampling algorithm. These results further confirm the effectiveness of the PFDiff algorithm and the FID metric in evaluating the performance of sampling algorithms.

D.8 Additional visualize study results

To demonstrate the effectiveness of PFDiff, we present the visual sampling results on the CIFAR10 731 [42], CelebA 64x64 [43], LSUN-bedroom 256x256 [44], LSUN-church 256x256 [44], ImageNet 732 64x64 [32], ImageNet 256x256 [32], and MS-COCO2014 [31] datasets in Figs. 5-10. These results 733 illustrate that PFDiff, using different orders of ODE solvers as a baseline, is capable of generating 734 samples of higher quality and richer detail on both unconditional and conditional pre-trained Diffusion 735 Probabilistic Models (DPMs). 736

NFE = 4

NFE = 10

FID = 65.70

FID = 13.66

NFE = 4

NFE = 10

FID = 37.76

FID = 10.86

NFE = 8

FID = 18.45

NFE = 8

FID = 14.10

(a) DDIM [20]

FID = 106.86

FID = 10.09

FID = 69.75

FID = 9.37

FID = 14.21

FID = 11.84

(b) Analytic-DDIM [23]

FID = 22.38

FID = 4.57

FID = 13.29

FID = 4.71

FID = 5.64

FID = 5.06

(c) DDIM+PFDiff (Ours)

NFE = 5

FID = 71.02

NFE = 10

NFE = 10

NFE = 5

FID = 19.26

FID = 73.43

FID = 22.04

(a) DDIM [20]

FID = 25.32

FID = 9.28

FID = 26.69

FID = 13.27

(b) DDIM+PFDiff (Ours)

NFE = 6

NFE = 12

(a) DPM-Solver-1 [21]

(b) DPM-Solver-1+PFDiff (Ours)

(c) DPM-Solver-2 [21]

(d) DPM-Solver-2+PFDiff (Ours)

(e) DPM-Solver-3 [21]

(f) DPM-Solver-3+PFDiff (Ours)

NFE = 4

(a) DDIM [20]

(b) DPM-Solver-2 [21]

(d) DDIM+PFDiff (Ours)

(b) DDIM+PFDiff (Ours)

NFE = 8

Text Prompts (listed from left to right):

A large bird is standing in the water by some rocks.

A candy covered cup cake sitting on top of a white plate.

People at a wine tasting with a table of wine bottles and glasses of red wine.

A bathtub sits on a tiled floor near a sink that has ornate mirrors over it while greenery grows on the other side of the tub.

A kitchen and dining area in a house with an open floor plan that looks out over the landscape from a large set of windows.

DDIM

(FID = 23.92)

DPM-Solver++

(FID = 18.87)

DDIM+PFDiff

(FID = 18.31)

NFE = 5

DDIM

(FID = 16.78)

DPM-Solver++

(FID = 15.93)

DDIM+PFDiff

(FID = 13.06)

NFE = 10

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See abstract and section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See section 4 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code in the supplemental materials, which will be made available on the GitHub platform.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our method is training-free, but we also report the hyperparameters used when evaluating our proposed method. Details can be found in section 4 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We generate 50k images for 32x32, 64x64 datasets and 10k for 256x256 to evaluate the FID metric. According to previous works [20, 21, 26, 27], when evaluating with the generated samples mentioned above, the standard deviation of the FID evaluations is rather small (mainly less than 0.01). These small standard deviations do not change the conclusions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In section 4, we mentioned that all experiments were conducted on an NVIDIA RTX 3090 GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work is conducted with the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method is a training-free accelerated sampling approach. It relies on existing pre-trained DPMs and does not involve the release of data or models, thus there is no need for safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Appendix D.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve human subjects nor crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

1047

1048

1049

1050

1051

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
To accelerate the sampling speed in diffusion models, this paper proposes a training-free denoising method, dubbed PFDiff. 
Concretely, PFDiff employs the gradient from past time steps to update intermediate states, aiming to reduce unnecessary NFEs while correcting for discretization errors.
In this manner, PFDiff enables to improve classic samplers without any training computation.
Importantly, experimental results demonstrate the effectiveness of the proposed PFDiff.

STRENGTHS:
1. Reducing discretization errors in diffusion models in a training-free manner is attractive and practical.

2. The motivation of using previous gradients to guide the current sampling direction is intuitively plausible, and the proposed method is technically sound.

3. The presentation is excellent and the figures all are readable.

WEAKNESSES:
1. In my humble opinion, the theoretical analysis part is naive. Can you provide more explanation about why previous gradients is helpful to guide current sampling direction? Since different noise levels correspond to different gradients, is there any harm in denoising images with the proposed method?

2. Many works investigate using previous gradients to improve sampling speed, so the contribution is limited.

QUESTIONS:
1. Is there any memory overhead for saving the gradients?

2. Can you present more experiments on ImageNet 256? Including conditional and unconditional.

3. How about the comparison with DEIS sampler?

4. Can the proposed method connect with distillation models?

5. If more metrics are tested, such as recall and precision, it would be better.

LIMITATIONS:
Please see in Weaknesses and Questions. If all of my concerns are addressed, I will improve my score.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PFDiff, a fast training-free sampler for diffusion models. PFDiff updates the current state with both the past score network evaluation and the future score network evaluation. It can achieve good sample quality with less than 10 NFE. The authors showcase the effectiveness of PFDiff on various pre-trained diffusion models.\n\nSTRENGTHS:\n1. With proper tuning, the proposed PFDiff can outperform existing ODE solvers in the low-NFE regime on various datasets. \n2. The authors provide comprehensive technical details about the proposed algorithm.\n\nWEAKNESSES:\n1. Flawed justification for future gradient: The authors' claim that using future gradient information is better than using current gradient information is based on the mean value theorem (lines 164-167 and Appendix B.2). However, this theorem only guarantees the existence of an optimal point within an interval, not its specific location. Therefore, the mean value theorem itself doesn't justify the preference for future gradients. \n2. Missing justification for the approximation: While the authors claim that their approximation is better, there is no theoretical justification for it. The proof in Appendix B.2 assumes that the optimal point is already known, which is not informative. The manuscript will benefit from a further approximation error analysis. \n3. Expensive and case-specific tuning: the proposed method essentially defines a set of candidate points and searches for the optimal point by tuning parameters $k$ and $l$. This tuning process can be computationally expensive and needs to be done for each specific case, limiting its practicality.\n\nQUESTIONS:\nLine 163-164: the mean value theorem always holds for any continuous differentiable function. What does it even mean for the mean value theorem to hold \"approximately\"? Also, it's unclear how the observations mentioned in lines 161-163 indicate the mean value theorem.\n\nLIMITATIONS:\nThe algorithm performance depends heavily on parameters $k$ and $l$ as shown in Table 7.  Optimal values for $k$ and $l$ vary based on the pre-trained model and the number of function evaluations. This necessitates extensive parameter tuning when applying the proposed method in practice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper proposes PFDiff, a training-free approach for accelerating diffusion models. Motivated by the high similarity of the diffusion network outputs at adjacent timesteps on the sampling trajectory, PFDiff utilizes past and future information for sampling with time-skipping, 
and decreases the number of function evaluations (NFEs) significantly. Experiments on various settings show significant acceleration, especially in the low NFE regime.

STRENGTHS:
- The method is training-free and can be plug into existing solvers.
- The motivation and overall method seem reasonable.
- The improvements is significant especially in the low NFE regime. State-of-the-art diffusion solvers like UniPC and DPM-Solver-v3 are compared.
- The finding that first-order solver (DDIM), along with PFDiff, can outperform high-order solvers, is intriguing.

WEAKNESSES:
- The highly concise writing and complex notations might be a bit confusing. Additional illustrations for certain local algorithm procedures can be helpful for understanding the overall idea.
- There are fundamental mistakes in the writing. Eqn. (8) (9) are represented as Euler discretizations of the original PF-ODE. However, both DDIM and the series of DPM-Solvers rely on exponential integrators to transform the PF-ODE into other forms, so that the linear term $x_t$ is cancelled. Though this does not mean the method is wrong, such simplified writing can be misleading. The authors are obligated to correct this, or I will be forced to reject this paper.
- It will be more convincing to include experiments on EDM, the SOTA diffusion model on CIFAR-10 and ImageNet 64x64.

QUESTIONS:
- Are there any insights why PFDiff is more effective on first-order solvers (DDIM+PFDiff even outperforms high-order solvers)?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper proposes a new training-free acceleration method for the inference of diffusion probabilistic models. The key components of the presented time-skipping strategy are the use of past and future gradients to eliminate redundant neural function evaluations (NFE). The proposed method is shown effective compared to other training-free acceleration methods, leading to solid performance improvements especially for ODE solvers with less than 10 NFEs.

STRENGTHS:
*    The method is training-free and can complement existing fast ODE solvers
*    The paper is well structured and puts the presented method in the proper context with respect to existing methods
*    The experimental results cover conditional and unconditional settings, showing performance improvements across the board.

WEAKNESSES:
*    The performance gap compared to training-based methods is still apparent, especially considering the latest distillation techniques resulting in one-step models.
*    The mathematical notations are a bit hard to follow up. I would advise the authors to add a schematic clarifying for a given setting of hyperparameters, which timepoints are being evaluated and which are being skipped.
*    The optimal setting of hyperparameters *k,l* is model/dataset dependent and it is not clear apriori how to set these. Therefore, this requires empirical experimentation which makes it time-consuming to get optimal performance when using the method out-of-the-box.

QUESTIONS:
*    How does your samples fare in terms of diversity compared to the original model with enough NFE? In other words, does the faster sampling somehow come at the cost of reduced diversity?

LIMITATIONS:
The authors were upfront about the limitations of their method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a training-free time step-skipping method that can be used with existing ODE solvers for reduced NFE. The method was motivated by two observations: 1) a significant similarity in the model's outputs at time step size during the denoising process and 2) a high resemblance between the denoising process and SGD. The proposed method employed gradient replacement from past time steps and rapidly updated intermediate states inspired by Nesterov momentum. The proposed method yielded promising results.\n\nSTRENGTHS:\n- Experimental results look promising with multiple diffusion models on diverse datasets.\n- Accelerating diffusion models for sampling is an important issue and this work tried to address it.\n\nWEAKNESSES:\n- There have been a lot of prior works on accelerating diffusion models for sampling. While this manuscript cited many, it still missed important prior works - some of them look quite similar to the proposed method. Thus, the novelty of the proposed method is unclear in the current form of this manuscript. For example, using Nesterov acceleration for fast diffusion models is not really new (e.g., R Li et al., Hessian-Free High-Resolution Nesterov Acceleration For Sampling, ICML 2022). Eq (15) of this work can be seen as a special case of the following prior works such as [R1], DeepCache [28] (using past), [R3] (using three moments or future) or [R2] (using all). Some recent work like [R4] even used partial caching instead of using the whole results. A more theoretically grounded work on using Nesterov momentum for sampling can be found in [R5]. \n[R1] M Xia et al., Towards More Accurate Diffusion Model Acceleration with A Timestep Tuner, CVPR 2024.\n[R2] A Pokle et al., Deep Equilibrium Approaches to Diffusion Models, NeurIPS 2022.\n[R3] H Guo et al., Gaussian Mixture Solvers for Diffusion Models, NeurIPS 2023.\n[R4] F Wimbauer et al., Cache Me if You Can: Accelerating Diffusion Models through Block Caching, CVPR 2023.\n[R5] R Li et al., Hessian-Free High-Resolution Nesterov Acceleration For Sampling, ICML 2022.\n- A number of acceleration works for diffusion models also investigated the feasibility of the parallel computation. Will the proposed method be parallelized for computation? \n- It is unclear if the proposed method was compared with other methods in terms of computation. Will 1 NFE of the proposed method take the same computation time as 1 NFE of other methods since the proposed method contains multiple evaluations of the neural network as in Eq. (15).\n- The notation and explanation are quite confusing, so it is not easy to understand the whole idea as well as the algorithm itself.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}