Self-Refining Diffusion Samplers: Enabling Parallelization via Parareal Iterations
Authors: Nikil Roashan Selvam
Amil Merchant
Stefano Ermon
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion
sampling
parallel
Primary area: diffusion_based_models
TL;DR: We introduce a novel technique to improve sampling speed of diffusion models by exploiting parallel-in-time integration.

Abstract: In diffusion models, samples are generated through an iterative refinement process, requiring hundreds of sequential model evaluations. Several recent methods have introduced approximations (fewer discretization steps or distillation) to trade off speed at the cost of sample quality. In contrast, we introduce Self-Refining Diffusion Samplers (SRDS) that retain sample quality and can improve latency at the cost of additional parallel compute. We take inspiration from the Parareal algorithm, a popular numerical method for parallel-in-time integration of differential equations. In SRDS, a quick but rough estimate of a sample is first created and then iteratively refined in parallel through Parareal iterations. SRDS is not only guaranteed to accurately solve the ODE and converge to the serial solution but also benefits from parallelization across the diffusion trajectory, enabling batched inference and pipelining. As we demonstrate for pre-trained diffusion models, the early convergence of this refinement procedure drastically reduces the number of steps required to produce a sample, speeding up generation for instance by up to 1.7x on a 25-step StableDiffusion-v2 benchmark and up to 4.3x on longer trajectories.

Self-Refining Diffusion Samplers: Enabling Parallelization via Parareal Iterations
Nikil Roashan Selvam
Amil Merchant Department of Computer Science Stanford University
Stefano Ermon
{nrs, amil, ermon @rco stanford.
Abstract
In diffusion models, samples are generated through an iterative refinement process, requiring hundreds of sequential model evaluations. Several recent methods have introduced approximations (fewer discretization steps or distillation) to trade off speed the cost of sample quality. contrast, introduce Self-Refining Diffu- sion Samplers (SRDS) that retain sample quality and can improve latency at the cost of additional parallel compute. We take inspiration from the Parareal algorithm, a popular numerical method for parallel-in-time integration of differential equations. In SRDS, quick but rough estimate of sample is first created and then iteratively refined in parallel through Parareal iterations. SRDS is not only guaranteed to accurately solve the ODE and converge to the serial solution but also benefits from parallelization across the diffusion trajectory. enabling batched inference and pipelining. As we demonstrate for pre-trained diffusion models, the early convergence of this refinement procedure drastically reduces the number of steps required to produce a sample, speeding up generation for instance by up to 1.7x on 25-step StableDiiffsion-Ve -2 benchmark and up to 4.3x on longer trajectories.
Introduction
Deep generative models based diffusion processes have showcased the capability to produce high-fidelity samples in a wide-range of applications [2 11 39 27]. From their origins in image and audio generation BI 33 10], diffusion models have enabled robotic applications well as scientific discovery via drug design []. Despite this promise, sampling from diffusion models can still be prohibitively slow. Early Denoising Diffusion Probabilistic Models (10) required thousand sequential model evaluations (steps), and state-of-the-art models such as StableDiffusion (7 can still require up to 100s of iterations for high-quality generations. This large number of sampling steps leads to high-latencies associated with diffusion models, limiting applications such as real-time image music editing and trajectory planning in robotics [2].
As sampling involves solving an ordinary differential equation (ODE), a prominent body of research including works such Denoising Diffusion Implicit Models (DDIM, 32). Diffusion Exponential Integrator Sampler (DEIS, [42)). and DPM-Solver [19] has tried to reduce the number of model evaluations required by introducing various approximations. For example, progressive distillation 29 requires re- training models to approximate the solution to the ODE at larger timesteps. However, such approaches trade-off speed at the cost of sample quality.
In this work, we instead take an orthogonal approach: we focus on additional parallel compute and show how this can be used reduce latencies while still providing accurate solutions to the original ODE, thereby preserving sample quality. Recently Shih et al. (30) leveraged a parallel-in-Ue integration method to introduce the first perall-aarallllizable algorithm for diffusion model sampling. The presented ParaDiGMs algorithm builds on Picard iterations 26 to perform denoising steps
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure I: A visualization of the iterative refinement provided by the SRDS algorithm on sample from StableDiffrusion with the prompt beautiful castle, matte painting." The initial coarse solve (left) via limited steps provides rough estimate of the sample, which iteratively refined through iterations of our algorithm. Early convergence is observed as the 3rd output nearly matches, a key feature that enables efficient generation.
across the trajectory in parallel, leading to state-of-the-art sampling speeds on popular benchmarks 30]. Despite the promising results, ParaDiGMs can be highly memory bound due to the of sliding window methods and also has a sequential convergence criteria requiring communication- expensive cumulative sums across devices coordinate parallel-same suralle--sampling. The method also has limited controllability over the accuracy of the final solution, assuming convergence per step rather than in the final generation. a completely different approach to accelerating parallel sampling: special techniques in solving triangular nonlinear systems through fixed point iteration. Instead, this paper, we turn to multiple- shooting methods from the parallel-in- -immteel ODE integration literature [14, B 6 and aim to improve parallelization of diffusion sampling by utilizing multiple resolutions across the time domain [2].
also worth noting that in concurrent work, Tang al. [37 take
Specifically, we present Self- Reffnnngi Diffusion Samplers (SRDS) that start with a quick but rough solve of the diffusion trajectory, achieved by limiting the number of total steps taken (for instance, using a few-step DDIM solver). The trajectory can then be simulated to higher fidelity via a highly- parallel algorithm that updates the final generation iteratively until convergence. At high level, each refinement step of SRDS partitions the current guess of the trajectory into blocks, and simulates each of these blocks at the desired (high) resolution. The running guess for the overall trajectory is then updated via convergence. This iterative refinement allows us to efficiently interpolate in parallel between coarse solution corresponding
coorectt mechanism based on the Parareal algorithm to accelerate
low-fidelity sample and an accurate solution corresponding to high-fidelity sample. The key benefits of SRDS are three-fold:
By design, SRDS computes an accurate solution to the reverse process (as defined by the practitioner's choice of diffusion solver), thereby maintaining high quality of samples. Importantly, as it is purely a sampling algorithm, it does so without incurring any retraining cost.
Approximation-Fiee Free:
Extensive Control and Compatibility: By serving an efficient interpolation method between the coarse and fine-ge mee-grained solvers, SRDS provides the practitioner with flexible control of the tradeoff between sample quality and speed. For instance, (corresponding to, say. few-step DDIM). Then, desired, one can add a budget-appropriate number of parallel SRDS iterations (instead of sequential) refine the obtained sample. Furthermore, SRDS is compatible with most existing off-the-shelf solvers (such as Fuler, Heun, DPM etc), thereby providing direct benefits virtually any diffusion workflow.
could first start with rough solve
Low Latency: Most importantly, we find that the number of iterations required by SRDS for convergence is quite low, leading to drastic improvements in latency for sampling. We present results using the Self-Refining Diffusion Samplers on a wide range of benchmarks, starting with paxel-based image diffusion models and further exploring latent-methods where SRDS leads to 1.7x improvement in the sampling speed on the 25-step StableDiffusson-v2 benchmark and up to 4.3x on longer trajectories. Through enabling faster sampling, SRDS aims unlock capabilities for real-time interaction with diffusion models.
'Code for paper be found at :tttps:://itte /ttp//ggtttttttt/// htt.p//....oonniiillsssssa/////s

---

2 Background
Diffusion models are a general class of generative models that rely on a noising procedure that converts the data distribution into noise via series of latent variables updates. For the purposes of this work, we will consider the continuous-ting
generalization presented by Song [35] and Denoising Diffusion Implicit Models [00] that formulate sampling as solving the initial value problem characterized by probability flow ordinary differential equation (ODE):
dæ
f(x 20( (2 (xx,t
dt; a(t= 20 - ~O,,
(1)
hg
where so t) is time-conditional prediction of the score function V z log p: (z)) from the diffusion
model. To be consistent with prior work on parallelized diffusion sampling Shih et al. [30], we use reversed time index (from traditional notation) where 2g refers to pure Gaussian noise, and ær refers to the denoised image after I denoising steps.
2.1 Solving the Differential Equation
Given the dynamics governing the differential equation, our goal to provide accurate solutions to:
[ ho t)dt
FT = zo +
(2)
1=
in order to produce sample from the diffusion model. Common approaches discretize the time interval [0, 7] into N pieces (to=0, 12, ty. =7) and solve a sequence of initial value problems yield an approximation (Io, I1, IN IN= the trajectory.
Formally, a solver a function F(xstort: starts tend) that propagates x from = tstrt with initial value starrt to f lend- Solving the differential equation corresponds to approximating the solution 2T to the given initial value problem by sequence of N solves:
x++1 F (a,, 2,, t++1) Vi E [0, N - 1]; given initial value 2o
The choice of solver JF dictates the sampling speed and accuracy of the solution. In practice, solvers which are accurate are often slow (due to high number evaluations of hg), whereas solvers that are fast tend to have reduced accuracy. Initial works on diffusion models used the classical Euler method as choice F, and it can be expressed as:
Fi+1 Feater (Fss 4, li+1) w. (1 (Ir5 (5)* * (li+1 - (2)
However, DDIM 32 quickly became a popular choice of F for its improved efficiency. Other recent works have tried to rely on approximations or leverage various ideas from the numerical methods literature to design solvers F that require fewer denoising steps. For instance, Diffusion Exponential Integrator Sampler (DEIS, 42), and flow ODE to design special solvers where the linear part of the ODE is solved analytically and the non-linear part is solved by incorporating ideas from exponential integrators in the numerical methods literature. Karras et al. [33 leverage the Heun's 's second order method and demonstrate favorable tradeoff between number of model evaluations and quality of generated samples for a small number of denoising steps. In this work, SRDS presents an orthogonal improvement these methods via parallelization, and by default we will assume all our solvers to be DDIM.
19 exploit the special structure of the probability
3
Self-Refining Diffusion Samplers
Attempts to reduce the number of steps in diffusion samplers can provide speedups in sample generation 29 23 but unfortunately often lead to lower-sample quality. While low-frequency components (in the Fourier sense) of the images may be well-established, the generations miss the high-frequency details that leads to good generations [40]. To fix sample quality while maintaining the latency benefits of reducing the number of steps, we turn to numerical methods introduced the parallel-in -nnlell-i--tiee integration literature where dynamics with different components having different rates
3

---

1xx9)
xx
axx))
c0P9)
x(0
X6
10
01
03
Figure 2: First iteration of the parareal algorithm to solve example ODE. The black represents the desired solution from the fine solver. The magenta dots indicate the running solution after one iteration of predictor-corrector updates. Figure inspiration from Pentland al. (25)
of convergence has been extensively studied. Specifically, multiple-shooting and multigrid methods have seen success in wide range of domains from convection-diffusia equations to eigenvalue problems (2 22 24 4 by creating rough but efficient solve of the prescribed differential equation that can then be iteratively updated via a highly parallelizable simulation. One such algorithm Parareal [T8 serves as the backbone for our Self-Refining Diffusion Samplers that we describe below.
3.1 Parareal Algorithm
Parareal makes use of two solvers: solver (called the 'fine solver') provides accurate solutions but slow to evaluate, and g (called the 'coarse solver') provides rough solutions but is much quicker.
Parareal targets general purpose initial value problems of forms similar to Equation T. Consider partition (4o, f..... iN 7) of the time axis (fa, T) into N intervals of equal width. Using the same solver notation as above, the goal is to approximate the solution nN to the initial value problem that would be produced using sequence of fine solves:
Fi+1 = (Ia;, li, 4--1). E [0, 1
The key insight of parareal is that we can first use the coarse solver G to quickly produce a rough trajectory, and this rough solution can be iteratively refined using parallel calls to the fine solver F.
Formally, the parareal algorithm begins with a rough estimate of the trajectory, initialzied via a series of coarse solves from G.
2f = g (x., 4, (441) Vi N-
(5)
where the notation x denotes the initial estimate the trajectory from the coarse solver (orange curve in Figure
Parareal then proceeds in iterations until convergence, where each iteration corresponds to a refine- ment of the trajectory. At each iteration, we solve the differential equation in each of the N time intervals at higher resolution using the fine solver F., where the initial value for each interval is given by the estimate of the trajectory from the previous iteration. Crucially, these fine solves (blue Figure 2 can be performed in parallel. Lastly, at the end of each iteration, we perform another coarse sequential solve through the trajectory (magenta in Figure 2 and incorporate the results of the fine solves into running solution for the trajectory using a predictor-correettrr method, where coarse solver "predictions" are 'corrected' via the updates from the parallel fine solves. Formally,
++1 p+1 = F (af to +++1) + (g (o[+),t,,t4t)
G (a t4, t(+1)) i O,..., - (6)
where the notation o. denotes the running estimate of the trajectory Parareal iteration number p.
3.2 Self-Refining Diffusion Samplers
Now, we turn our attention back to drawing a sample from our diffusion model, which as discussed corresponds to estimating solution to the initial value problem defined in Equation

---

Algorithm SRDS: Self-Refining Diffusion Sampler
Require: Diffusion model ps with denoising steps N, tolerance т, and corresponding DDIM solver 1( lstarts fend; sleps)
Ensure: A sample from Po
a (0)
// Sample initial condition for Initial Value Problem from prior
for i to VN do
3: prev: + (21-1,41-1,4,.)
x + prev,
// Initialize æ with a coarse solve
5: p 6-1
// SRDS refinement iteration number
6: while S VN do 1 VN in parallel do Y: - h t
for
8:
// Perform fine solves in parallel // Perform a coarse sweep
9:
for
i - 1 to VN do
10:
cur; <- h [ 41,14,11
11:
x - yi + cur; pret:
// Take predictor step
12:
prev; t cur;
13:
|
x p-1
<T then
// Check for convergence
vN
14:
break
15:
p p+1
16:
return x"
We leverage the Parareal algorithm order to parallelize sampling. Our idea is compute a solution on the N N-discretization of the interval by instead considering coarser v -N-discretizationf of the interval [0, 27. Let AT T/VN. 4++1 4, + AT', partitioning [0, I'] into vN intervals. We pick v step DDIM solver as our fine solver F. In other words, F(x,, 4, (4+4) ) the result of V VN-step DDIM solve propagating æ from = f with initial value æ; to = f4+1. We pick 1-step DDIM solver as our coarse solver G. That is, G(v, 4, (441) denotes the result of the corresponding 1-step DDIM solve propagating æ from = t, with initial value I; to t++1 ("step" refers to denoising step involving an he evaluation).
The SRDS algorithm proceeds as follows. We start with the coarse-solve to generate a rough estimate of the trajectory and final sample, achieved by taking /N DDIM steps in total. Then, each of the vN coarse predictions in the trajectory simulated higher resolution with further /N DDIM-iterations in parullel, each with an effective time step corresponding to the original N-step discretization of the diffusion model. Iterative updates to the running solution then proceed in a manner equivalent to Parareal updates until convergence, as measured by the change in the outputted generation. Our SRDS algorithm is summarized Algorithm|
3.3 Convergence Guarantee
The ideal result for diffusion sampling is to get the solution arising from N sequential denoising score steps. SRDS however only starts with rough solve of the diffusion trajectory taking v sequential denoising steps. Nevertheless, we can show that each iteration of SRDS (line 6 of Algorithm refines the generated sample and leads us closer to the ideal solution.
Proposition 1. The sample output by SRDS converges to the output of the N-step sequential solver at most V refinement iterations.
A key property of our algorithm is that after iterations (refinements to the diffusion trajectory) of SRDS, the first steps of the running trajectory exactly matches the trajectory generated by the sequential solver for the corresponding intervals. Consequently, the algorithm is guaranteed to converge in at most VN iterations. We defer the formal proof to Appendix A It is also worth
II is not required for N to be perfect square. We can extend the described techniques in straightforward manner to perform a [/]ddiscretiizttion instead, with the last interval in the partition having a smaller size. The special choice of /N as discretization level further explained in Appendix B
Thii refers solver that takes v/ DDIM steps (or model evaluations) to solve the initial value problem.
5

---

30
211
22
2.7
x
...
2N
(a) Sequential Sampling
x
x RR
.2
NN
potl W
p
P+1
(h) SRDS
Figure 3: Computation graph for diffusion sampling. For SRDS, the red arrows correspond to fine solves, and each block [0, VN1, [VN, 2V NV, and so on can be perform the fine solves independently in parallel. The blue arrows correspond 1-step coarse solves.
noting that this worst case guarantee similar in spirit to Proposition in 30 and its generalization (Theorem 3.6 in (37).
worth noting, however, that in practice we observe that the number of refinement iterations required till convergence - which is defined as difference in consecutive sample generations not exceeding a chosen threshold - is much less than the worst case bound of VN. This early conver- gence critical speedups from SRDS. The exact choice of threshold t in line 13 of Algorithm [ is a hyperparameter that is empirically chosen so as to avoid measurable degradation in sample quality.
3.4 Batched Inference and Pipelining
SRDS benefits from two key features to reduce latencies: batched inference and pipelining.
First, the fine solves that are used in order to refine the trajectories DDIM-steps, which means that they be performed in batched manner even for single sample generation. This parallelization allows for a single sample generation to incur the benefits batched inference, introducing higher device utilization or device parallelism.
Secondly, we observe that the dependency graph for SRDS enables pipelined parallelism. As outlined in Figure 3 we find that F Fx= t2, ti+1) and (x"" t4, 1++1) both only depend on x!". The tasks for computing F (x), .t. t++1) and (x, t,, t++1) can be spawned mechanism to finish updating the SRDS solution for iteration i. This leads to efficiently pipelined version of the algorithm, further speeding up the sampling process by a factor of two. See Figure 4 for an illustration of this pipelined algorithm with N 16. Pipelining furthers the benefits of batched inference the coarse solver is simply DDIM-step with a larger time-step, so it can be batched with fine solves when applicable.
x is computed, without
waiting for the entire predictor-correct
3.5 Sampling Latency
Proposition 2. [Worst-Case Behavior] Ignoring GPU overhead, the worst wall-clock time of generating sample through SRDS is no worse than that of generating through sequential sampling.
Referring to the pipelined implementation of SRDS, it is easy to see that the fine solve F (ar), li, 4++ 1) starts immediately after F (x:-1 1,-1, t.). Subsequently, from Proposition i then follows that in the worst case, the final sample of SRDS x= VN is computed time VN. VN N as desired. A formal argument can be found in Appendix A It is worth noting, however, that this property of SRDS comes at the cost of much higher parallel compute compared to sequential sampling.
NN

---

SRDS
10
12
16
18
21
28
28
30
32
01
02
04
F2
F4
Pipelined SRDS
F
62
F3
G4
Iteation
Color
F2
63
F4
3
Figure 4: Illustration of the pipelined version of the SRDS algorithm N 16 denoising steps, which results a direct 2x speedup compared to the vanilla version.
3.6 Memory and Communication Overhead
Proposition [Memory] SRDS requires memory corresponding to O(VN) model evaluations.
Once again referring to the pipelined implementation SRDS, easy to see that at any given time there most one model evaluation corresponding a coarse solve, and up vN parallel model evaluations corresponding to the fine solves. It is worth contrasting this with the quadratically higher (NN) memory requirement of the full ParaDiGMs algorithm (30.. necessitating the use of sliding window tricks to reverse the process in a piece-wise fashion.
It is finally worth noting that there is minimal Inter-GPU communication in SRDS. In particular, most one sample is passed between adjacent GPUs in each SRDS iteration. Once again, it is worth contrasting this with ParaDiGMs algorithm, which by its use of parallel prefix sum operations to sync the solutions at each Picard iteration incurs greater GPU communication overhead. See Appendix for more discussion.
Experiments on Diffusion Image Generation
To showcase the capabilities of the prescribed SRDS algorithm, we apply the diffusion sampler to pretrained diffusson models and present the difference in sample time and quality to ensure that applied convergence criteria do not reduce generation metries. We start with pixel-based diffusion before expanding experiments applied to latent methods such as StableDiffusion-22 Across the range of tasks, we show consistent speedups while maintaining quality of sample generation.
In this section, we perform an extensive comparison with ParaDiGMa 33 as our baseline. Nonethe- less, provide high level empirical comparison concurrent work ParaTAA 37) in Appendix where we demonstrate the superiority of SRDS.
4.1 Pixel Diffusion Image Generation
We start with pixel-space diffusion models. In particular, we test our SRDS algorithm and demonstrate capabilities in performing diffusion directly on the pixel space of 128x 128 LSUN Church and Bedroom [41] 64x64 Imagenet [5], and 32x32 CIFAR [16] using pretrained diffusion models (29]. which all use N 1024 length diffusion trajectories.
We measure the convergence via 4 norm in pixel space with values [0, 255]. We conservatively set r = 0.1, meaning that convergence occurs when on average each pixel in the generation differs by only 0.1 after a refinement step (see Appendix for an ablation on choice of r). Through our experiments, we quantitatively showcase how the SRDS algorithm can provide signficant speedups in generation without degrading model quality (as measured by FID score 9 on 5000 samples). seen in Table I SRDS remarkably converges in 4-6 iterations across all datasets; this corresponds to roughly 200 effective serial steps (counting all model evaluations simultaneously performed in parallel as evaluation), which is only 15 20% of the serial steps required by sequential solve
7

---

Table 1: Evaluating FID score (lower better) of SRDS on various datasets using 5000 samples generated using DDIM solver. Effective serial evals refers to the number of serial model evaluations taken by the pipelined SRDS algorithm (counting all model evaluations simultaneously performed in parallel as one evaluation). Total evals refers to the total number of model evaluations.
Sequential
SRDS
Serial Evals
FID Score
SRDS Iters
Eff. Serial
Total Evals
FID
Dataset
Evals
Score
LSUN Church
1024
12.8
5.7
209
5603
12.8
LSUN Bedroom
1024
10.0
5.8
212
5692
10.0
Imagenet
1024
9.0
4.6
175
4612
9.0
CIFAR
1024
7.6
3.7
147
3771
7.6
Table 2: CLIP scores of SRDS on StableDiffusion-22 over 1000 samples from the COCO2017 captions dataset, with classifier guidance w = 7.5, evaluated on ViT-g- 14. Time is measured on A100 GPUs without pipeline parallelism, showcasing speedups with early convergence of the SRDS sample.
Sequential
Vamilla SRDS
Stable
Scrial Evals
CLIP
Time per Sample
Max Iter
Eff. Serial Evals
Total Evals
CLIP Score
Time per Sample
Speedup
Diffusion-v2
Score
DDIM
100
31.9
4.6
19
119
31.9
2.0
2.3x
DDIM
25
31.7
1.2
9
34
31.4
0.8
1.5x
DDIM
31.7
1.2
17
74
31.9
1.7
0.7x
(N = 1024). We clarify that effective serial evaluations is referred to Parallel Iters in Steps in 337.
30 and
While we are pretty conservative above in measuring convergence through distance in pixel space, we can also simply limit the number of SRDS iterations 2 and achieve further speedups without measurable degradation in sample quality. See Appendix F for more details.
It is again worth noting that this improved latency from parallelization comes the cost of greater number of total model evaluations compared to regular sequential solver. However, this tradeoff enables the diffusion models for many other use cases such as real-time image or music editing and trajectory planning in robotics. Moreover, we often empirically observe that SRDS provides reasonable predictions within single Parareal iteration; here, the total number of model evaluations is only slightly larger than the serial approach (increasing from n to n + 21)). Lastly, also worth noting that many users are often agnostic to inference time GPU compute costs as they are orders of magnitude lower than training compute costs anyway.
4.2 Latent Diffusion Image Generation
where evaluations of the CLIP score over 1000 random samples show how SRDS maintains sample quality while improving the number of parallel iterations required per sample, with summary metrics presented in Table 2 As the SRDS algorithm has small GPU overhead, achieve measured wallclock time improvements with a Diffusers compatible implementation [38] It is worth nothing that while we focus on DDIM here (as in the rest of the writing), we show speedups by readily incorporating other solvers into SRDS in Appendix
Finally, we turn to latent diffusion models, in particular StableDiftusion-22 27),
For the test bed of latent diffusion models, explore the convergence properties of our SRDS algorithm, with the average CLIP score plotted against the number iterations in Figure 5 For shorter sequences of length 25 (left), the corresponding SRDS sampler converges after approximately 3 iterations. However, for longer sequences of length 100 (right) the sampler has converged after a single SRDS iteration, showcasing the capabilities of our algorithm improves with longer trajectories.
Next, we demonstrate the additional speedup that pipeline parallelism can bring to SRDS. We plement slightly suboptimal version of pipelined SRDS for StableDiffusion and already observe

---

32.0
32.0
CLiP Score
CLIP
31.0
31.0
Iterations
Figure 5: Convergence of the SRDS algorithm for a trajectory length 25 (left) and 100 (right) showcase how early termination of the algorithm can yield equivalent sample quality. In particular, longer trajectories with increased parallelism appear to converge faster.
Table 3: Evaluation additional speedup offered by pipelined version of SRDS.
Serial Model Evals
SRDS
Pipelined SRDS
Eff. Serial
Time Per Sample
Elf. Serial
Time Per Evals
Method
Evals
Sample
DDIM
961
93
12.30
10.31
DDIM
42
3.30
2.85
DDIM
25
15
0.82
0.69
Table 4: Comparison of wallclock speedups offered Pipelined SRDS and ParaDiGMS with various thresholds, with respect to Serial image generation. These StableDiffusion experiments are performed on identical machines (4 40GB A100 GPUs) for a fair comparison.
Serial
Pipelined SRDS
ParaDi GMS Threshold
Model Evals
Time Per
Time Per
Threshold le-3
Threshold
Method
Sample
Sample
le-2
le-1
DDIM
961
44.88
10.31 (4.3x)
275.29
20.48
14.30
DDIM
196
9.17
2.85 (3.2x)
29.45
5.08
3.42
DDIM
1.18
0.69 (1.7x)
1.98
0.77
significant speedups as seen in Table 3 with some more engineering effor we can further push towards extracting the full potential pipelining. However, as this already beats the baselines, this sufficiently demonstrates the benefits of SRD
Furthermore, for our main baseline ParaDiGMs, we also perform more extensive evaluation to evaluate both methods on equal hardware to more clearly demonstrate the benefits of SRDS. In Table we demonstrate that SRDS consistently beats ParaDiGMS on wallclock speedups. Though the authors of (30 uses convergence threshold of 1e - 3, we show that SRDS can provide impressive speedups even when compared to significantly relaxed ParaDiGMS thresholds of
Finally, we finally provide few sample generations on standard text prompts from Draw Bench 28 in Figure 6 and additional examples of convergence similar Figure I in the Appendix G
Related Work
Recent literature diffusion models has focused heavily reducing the cost of sampling. Tech- niques such as higher order methods 33 and exponential integrators 42 have been proposed as
"The suboptimality of the implementation arises from the use of single device to coordinate the pipeline allelism and device transfers (arising artifact from muttirrcsppor. more complete implementation would instead use ring-like like communication between devices rather than wait on the coordinator.
Wee note that the number of denoising steps in the experiments is chosen to be perfect squares merely for convenience. SRDS is general and applies to any number of denoising steps.

---

Figure 6: Sample generation from StableDiffusion-22 with the SRDS algorithm with text prompts based on examples from DrawBench. We plot the early converged SRDS figure (top) and the result of the serial trajectory (bottom); the two rows are essentially indistinguishable, highlighting the approximation-free nature of SRDS.
strategies for reducing the model evaluations required in order to build high-quality samples without any additional training. When additional training is possible, other works have proposed distillation further speed up sample generation. For the purposes of this paper, we view these approaches as orthogonal, as the resultant models could be simulated with SRDS for potential benefits from combining methods.
[99] quantization (17], and consistency (34] as alternate objectives
As discussed throughout the paper, this work most cloesly related to the ParaDiGMS sampler method developed by Shih al. (30] for parallel sampling of diffusion models. The works take a similar approach by building off popular parallel-in-time integration methods order to achieve lower latencies in simulation. In particular, ParaDiGMS builds Picard iterations converge trajectories; we, however, build on Parareal method that performs multiresolution along the time axis for faster sampling. Parareal has been well-explored [18 25 20 7 though with limited theoretical guarantees only spanming certain cases such as the heat equation and Navier-Stokes equation [8, 36]; our work is the first apply this algorithm diffusion models.
Conclusion
Limitations: Similar to previous iterations parallel-in-time integration methods, SRDS make use of additional compute that can be used in parallel in exchange for faster latencies of sampling. That is say, the total number of model evaluations in comparison to standard diffusion modelling increases in exchange for lower latencies. The additional compute may be reasonable in applications such as small-batch sampling where the additional cost can be hidden through better device utilization (e.g. sampling of single image or trajectory in robotics). Alternatively, the responsiveness of real-time image editing may make parallel sampling an appealing option for cost-insensitive users.
Future Directions: This work opens up a of interesting open questions for future exploration. Firstly, while fast convergence of parareal-style algorithms has only been proven for very limited settings, will be extremely interesting to derive convergence guarantees specifically for the diffusion process. This has the potential to further our understanding of the nature of the ODE/SDE that governs the reverse process. Another natural direction is explore the effects employing higher levels of discretization and other multigrid methods such as F-cycles and W W-cycles. As alluded to in Section [3.2] one could not only further study the optimal choice of second level of discretization, but also consider novel schedules that involve partitioning the diffusion trajectory into intervals of varying sizes. Lastly, it worth highlighting that by serving a highly modular and interoperable framework, SRDS unlocks vast landscape of interesting coarse/fine solver combinations, For instance, one can use DDIM solver to perform the parallel refinement steps, while using progressively distilled model 29 or consistency trajectory model [15] as the coarse solver in SRDS.
10

---

Acknowledgments and Disclosure of Funding
We thank the reviewers for their thoughtful feedback towards improving this paper. We also thank Aryaman Arora, Harshit Joshi, Ken Liu, Rajeshwari Jadhav, Rohit Nema, and Yanzhe Zhang for their helpful discussions and support during various stages of the project. This project was funded in part by ARO (W911NF-21-1-0125), ONR (N00014-33-1-2159) and the CZ Biohub.
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Hans Georg Bock and Plitt. A multiple shooting algorithm for direct solution of optimal control problems. IFAC Proceedings Volumes, 17:1603-1608, 1984.
[3] Bulirsch. The multiple shooting method for the numerical solution of nonlinear boundary- value problems and of optimal control problems. Report of the Carl-Cranz-Gesellschaff eV, Oberpfafsenhofen, Germany, 1971.
[4] Hongtao Chen, Hehu Xie, and Fei Xu. A full multigrid method for eigenvalue problems. J. Comput. Phys., 322:747-759, 2014.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[6] Peter Deuflhard. Recent advances in multiple shooting techniques. 1980.
[7] Paul F Fischer, Frédéric Hecht, and Yvon Maday. A parareal in time semi-impline approximation of the navier-stokes equations. In Domain decomposition methods in science and engineering, pages 433 440. Springer, 2005.
[8] Martin J. Gander and Stefan Vandewalle. Analysis of the parareal time-parallel time- integration method. SIAM Journal Scientific Computing, 29(2):556-77,, 2007. doi: 10.1 137/050666666
[9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:684006851, 2020.
[11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Flect, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. 2022.
[12] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0998 2022.
[13] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 26565-26577, 2022. 2022.
[14] M. Kiehl. Parallel multiple shooting for the solution of initial value problems. Parallel Com- puting. 20(3):275-20) 1994. ISSN 0167-8191. doi: http:///oo.org//0..0000//1111199 80013-X.
[15] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279 2023.

---

[16] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[17] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535-17545, 2023.
[18] J.-L. Lions, Yvon Maday, and Gabriel Turinici. A "parareal" in time discretization PDE's. Comples Rendus de I' l'Académie des Sciences Series Mathematics, 332:661-668, 2001. doi:
10.1016/57764444420000019999999
[19] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787 2022.
[20] Yvon Maday and Olga Mula. An adaptive parareal algorithm. Journal of computational and applied mathematics, 377:1 177:112915, 2020.
[21] Yvon Maday and Gabriel Turinici. A parareal in time procedure for the control of partial differential equations. Comptes Rendus Mathematique, 335(4):387-392, 2002.
[22] Oliver A. McBryan, Johannes Linden, A. Schuller, Karl Solchenbach, and K. Stuben. Multigrid methods on parahel computers survey on recent developments. 2014.
[23] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[24] Galina V. Muratova and Evgeniya M. Andreeva. Multigrid method for solving convection- diffusion problems with dominant convection. Journal of Computational and Applied Mathe- matics, 226(1):77-83, 2009, ISSN 0377-0427. Special Issue: The First International Conference on Numerical Algebra and Scientific Computing (NASC06).
[25] Kamran Pentland, Massimiliano Tamborrino, Debasmita Samaddar, and Lynton C Appel. Stochastic parareal: An application of probabilistic methods to iime-parallllization. SIAM Journal on Scientific Computing, 45(3):882-SI 2022.
[26] Émile Picard. Memoire sur la theorie des equations aux derivees partielles et la methode des approximations successives. Journal de Mathématiques pures et appliquées, 6:145-210) 1890.
[27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[28] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-in rext-to-image diffusion models with deep language understanding. Advances in 2022.
neural information processing systems,
[29] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models.
arXiv preprint arXiv:2202.00512, 2022.
[30] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[31] Jascha Dicksiickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using noncquilibrium thermodynamics, In International conference on machine learning, pages 2256-2266 2265.. PMLR, 2015.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
[33] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, Advances in neural information processing systems, 32, 2019.
12

---

[34] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning. 2023.
[35] Yang et al. Song. Score-based generative modeling through stochastic differential equations. arXiv preprini urXiv:2011.13456 2020.
[36] Johannes Steiner, Daniel Ruprecht, Robert Speck, and Rolf Krause. Convergence of parareal for the navier-stokes equations depending on the reynolds number. In Numerical Mathematics and Advanced Applications Applications-ENNMMATHH 2013: Proceedings ENUMATH 2013, the 10th European Conference on Numerical Mathematics and Advanced Applications, Lausanne, August 2013,
pages 195-202. Springer, 2015.
[37] Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, and Hsu Chang. Accelerating parallel sampling of diffusion models, 2024. URL btttss///aaaxx org/ abaaabs/22000 2402 09970
[38] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://gil diffusers 2022.
con/huggingfaae.
[39] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, al. De novo design of protein structure and function with rfdiffusion. Nature, 62097976):108-11100 2023.
[40] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22552-22562, 2023.
[41] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506v:/506.033360 2015.
[42] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arX1 2022.
13

---

A Proofs
Proposition [Convergence Guarantee] The sample from SRDS converges the output of the slow sequential solver in most VN refinement iterations.
Proof. We show, by induction, that o. converges in i iterations of SRDS for all E [0, 1]. Further, क. = .4-_,,,-1.4 for all 2 i. implying that the final sample indeed corresponds to the desired sample from F. The base case of i follows trivially from the initialization (initial condition). To prove the second base case of 1, notice that anj Do for all implying that g क
p-1 to. t1) is constant for all 1. Consequently,
a- F ab 10.11 + (g (g (2 (1 (0 9 (1,6))),)1)) vp21 21
= F (2o, to:
vp 2
as desired.
Assume by the induction hypothesis that for some fixed i. i. = F (i-,t) vp 2 Then, vp 2 i, we have that
a aptl F (a to t(+1) + (g (o tt,t441) g (xf, (1,
= F (a) to, t441)
+ (9 (F (F (x to tr,t1+1) - (2 t..t+1))
F ta
as desired.
Proposition (Worst-Case Sampling Latency] Ignoring GPU overhead, the worst case wall- clock time of generating single sample through SRDS is no worse than that of generating a single sample through sequential sampling.
Proof. Consider the unit of time to be the time taken for one denoising step (or one model evaluation). Referring to the pipelined implementation SRDS, it easy to see via a straightforward inductive
argument that the -t-step fine solve F x WN:li li, li+1) ends time
VN
P
VN
p.
From
Proposition 1, it then follows that in the worst case, the final sample of SRDS
is computed VN
N
time V /N VN N desired.
Proposition 3. evaluations.
[Memory]
SRDS requires memory corresponding O(vN) denoising model
Proof. In the pipelined implementation of SRDS, easy see that at any given timestep there is at most one model evaluation corresponding to coarse solve. Further, the number of parallel model evaluations corresponding to the fine solves upper bounded by the coarse discretization (or the number of "blocks"), which is VN. Thus, the memory used by SRDS corresponds to at most /N + = O(VN) model evaluations.
14

---

B Choice of Coarse Resolution
The choice of resolution for the coarse solve is not arbitrary. For practical implementations, since we use the same denoiser (say, DDIM) for both the coarse and fine solves, we choose VN as an optimal choice in the runtime sense (assuming constant number of iterations till convergened At a high level, this choice stems from the fact that we want to balance out the time the it takes to run all the fine solves parallel and the time takes perform one set of sequential predictor-correst steps through the trajectory.
Proposition [Optimal Coarse Resolution] The speed of an SRDS iteration is maximized for B
Proof. Let is denote the number of SRDS iterations until convergence, let denote the cost one denoising step or model evaluation, and let < B < N denote the "block-size": that is, the second scale of discretization. For the 1-step coarse solve, each SRDS iteration incurs a runtime cost of 1 For the B-step fine solves, as each of the [N fine solves are independently executed in F T. The baseline runtime for sequentially
parallel, each SRDS iteration incurs a runtime cost of B 1.
sampling from the diffusion model is N T. Thus, the runtime speedup (ignoring parallelization
overhead) is N-r
x[ [+++)++ T+ For a fixed value k, it easy to see that this quantity is
concave in and is maximized by choosing B Rs VN.
is worth noting, however, that we use solvers of different latencies for the coarse and fine steps, modifed analysis is required to incorporate differences in denoising step times for the two solvers. Consequently, vN might no longer be the optimal choice of coarse resolution.
C Incorporation of other Solvers
It is worth emphasizing again that SRDS provides orthogonal improvement when compared to the other lines of research on accelerating diffusion model sampling. In particular, while the main experiments (and writing) were focused on DDIM, SRDS is compatible with the other solvers and they can be readily incorporated into SRDS to speed up diffusion sampling. For example, below we show that SRDS is directly compatible with other solvers such DDPM (often requiring more steps than DDIM) and DPMSolver (often requiring fewer steps than DDIM) and can efficiently accelerate sampling in both cases. We demonstrate this StableDiffusion in Table 5 We also highlight that the Diffuser-come compatible implementation requires only minor modification to the arguments of the solver, suggesting that SRDS will also be easy to extend out-of-the-bbo to other methods that the community develops.
Table 5: Evaluation of SRDS with various off-the-shelf solvers.
Sequential Model Evals
SRDS
Speedup
Time Per Sample
Eff Scrial Evals
Time Per Sample
Model
DDPM
961
44.68
93
12.30
3.63x
DDPM
196
9.03
42
3.26
2.76x
DPM Solver
196
10.30
42
3.49
2.95x
DPM Solver
1.31
0.88
1.48x
DDIM
196
9.17
42
3.30
2,77x
DDIM
25
1.18
15
0.82
1.43x
'It is possible that a different choice might actually be optimal by enabling better flow of information down the computation graph and thereby resulting in lower number of iterations till convergence However, in our experiments (as also noted in Section 4). we observe that SRDS often converges for small k E [2,4]. validating empirical choice.
15

---

D Memory Utilization
For I' step denoising process, ParaDiGMS needs to perform model evaluations in parallel with subsequent computations needing information about all previous evaluations, while SRDS only requires vT parallel evaluations (which fits comfortably in GPU memory) and requires much lesser communication between GPUs. While the prohibitively large memory requirement can be combated with sliding window method, the significantly larger communication overhead remains because every step of Paradigms, an AllReduce over all devices must performed order calculate updates to the sliding window. (For instance, even when ParaDiGMS reduces Eff. Serial Steps by 20x, the obtained speedup only 3.4x). This is in contrast to the independent fine-solves in parareal that only need to transfer information for the coarse solve.
demonstrate how the minimal memory and communication overhead of SRDS shines through as we are able to achieve better device utilization as we increase the number of available GPUs. The following experiment was performed on 40GB A100s and used generous 1e-2 threshold for ParaDiGMS.
Below in Table
Table 6: Evaluating the device utilization of SRDS in comparison to ParaDiGMS
SRDS
ParaDiGMS
Eff Scrial Evals
Eff Scrial
Devices
Serial Model Evals
Time Per Sample
Time Per Sample
Evals
25
1.62
2.71
DDIM
25
15
1.08
16
2.01
DDIM
4
25
15
0.82
16
1.51
E
Comparison to ParaTAA
We demonstrate the superiority of SRDS baselines ParaDiGMS (30 and ParaTAA 37. Here,
demonstrate the high-level superiority of SRDS solely by using the results published by the authors in
30 (Table 5) and [37] (Table
In the table below, we show that SRDS offers better wall-clock speedups (over sequential) in sample generation time for StableDiffussin when compared to 300 and (37] We clarify that the reported speedup for each method is with respect to sequential solve on the same machine that the corresponding parallel method
evaluated. Our results particularly impressive given that the authors of (30) used 8x 80GB A A000s for the evaluation and the authors of [37] used 8x 80GB A800 for the same, while we (SRDS) only used 4x 40GB A100 for the evaluation due computational constraints. (For interpretation purposes, recall that a sequential solve is not compute/memory bound and doesn't 't benefit significantly from additional GPU compute, whereas the parallel methods certainly do!) We would also like to highlight the superiority of SRDS over the baselines in the regime of small number denoising steps (25) being particularly impactful.
Table 7: Speedup wallclock time for single sample generation offered by SRDS compared to ParaDiGMS and ParaTAA
Denoising Steps
ParaDiGKS
ParaTAA
Pipelined SRDS
DDIM 100
2.5x
1.92x
2.73x
DDIM 25
1.0x
1.17x
1.72x
16

---

Additional Convergence Experiments
In this section, we further evaluate the convergence properties of SRDS, First, in Table we analyze how the sample quality varies as we vary the tolerance threshold т.
Table 8: Evaluating the effect of SRDS tolerance parameter sample quality for pretrained diffusion model on 128x128 L.SN Church. The metric used Kernel Inception Distance (KID, lower is better) over 1000 random samples generated using DDIM solvers.
SRDS
Eff. Serial
Model Evals
KID
Method r
Iterations
Iters
Score
Sequential N/A
N/A
1024
1024
0.0146
SRDS 0.1
5.7
209
5603
0.0146
SRDS 0.5
4.3
165
4334
0.0147
SRDS 1.0
3.7
147
3771
0.0146
Next, we analyze how the FID score of generated samples varies as function of the number of SRDS iterations. As shown in Figure we observe rapid convergence the FID score to the value obtained by sequential sampling (12.8) within few SRDS iterations.
17
16
15
score
14
13
12
11
10
9
1
2
3
5
6
SRDS Iterations
Figure 7: Convergence of the SRDS algorithm LSUN Church.
17

---

G
Additional Samples from SRDS
Various samples from Drawbench prompts are provided in Figure
A black colored dog.
35mm macro shot kitten licking a baby duck, studio lighting.
A blue cup and a green cell phone.
Figure 8: Convergence of the SRDS algorithm on various samples from Benc..
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our novel algorithm and its related claims made in the abstract and introduction are discussed thoroughly in Section 3 with supporting empirical results in Section 4. The relation to existing work in the literature is presented in Sections and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are discussed Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: Complete proofs of all theoretical claims are presented in Appendix Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, all the experimental settings thoroughly described in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: There is a public link to our repository containing complete code needed to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: relevant experimental details are thoroughly described in Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Each evaluation of sample quality (using FIID score) requires thousands of samples from the model; unfortunately, however, sampling from diffusion models is computationally very expensive. Due to computational constraints, perform multiple rounds of our experiments to provide error bars for the FID scores.
were unable to
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resource details are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: The research presented conforms to the NeurIPS Code of Ethics including aspects such as disclosing essential elements for reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents foundational research: novel fast sampling algorithm for diffusion models in an attempt unlock capabilities for real-time interaction with diffusion models. While there could be potential positive and negative societal impacts of this work stemming from applications that use this (say, real-time generation of deepfakes), we believe it is not direct consequence of this work.
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper presents sampling algorithm and hence poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits and attributes the creators of all the models and datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Y80vEVCVcQ", "reviewer": "Reviewer_dCML", "summary": "The paper presents a new approach to speed up (improve latency)  the generation of samples from diffusion models. The approach is orthogonal to many other approaches present in the literature for the same task. It leverages Parareal algorithm for the task by getting a quick coarse approximate of the sample and then refining it iteratively in parallel, thus reducing the latency while maintaining sample quality. The authors present results for pre-trained pixel space diffusion models  and Stable Diffusion and find upto 2x speed ups for the latter.", "questions": "- The authors claim that the worst case sampling latency is no worse than generating a sample through sequential sampling. Yet one of their results (last on in Table 2) has speed up of 0.7x. Is this all due to GPU overhead?\n\n- Is is possible to present empirical results with pipeline parallelism? It seems like the authors expect 2x gains with it, but I assume it will require some computational overhead and based on the previous question, the degradation can be non-trivial.\n\n- While I understand the claim regarding this approach being orthogonal to other works reducing, showing some experiments around the same will make for a stronger paper.", "limitations": "There are a few limitations discussed towards the end of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Improving the latency of diffusion models is an active and important area of research. The paper presents a new strategy that relies on leveraging parallel computation of modern hardware. This approach can in theory be combined with other strategies to lead to further improvement, and does not require re-training. The presented approach seems to have guarantees on the quality of the solution and some control on trading off the speed and quality. There is also thought given to efficient batching and pipelining tasks for further potential improvements, which is good. Overall, the paper is also well written and easy to follow.", "weaknesses": "The main weakness of the paper is lack of strong and diverse results. The authors only test one  diffusion model task. Furthermore the best speed up is only 2.3x, while another experiment results in 0.7x speed up which is concerning.\nThe authors argue on theoretical grounds that this approach can be combined with other approaches to reduce latency of diffusion models. This is fair, but they do not show any empirical results around it. \nFinally, the approach relies completely on leveraging parallel refinement and the actual number of model evaluations are much larger (often by a factor of 3x) which can limit the applicability in some cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, and appreciate that they find the paper well written and easy to follow! We respond to the remaining concerns below:\n\n> The authors argue on theoretical grounds that this approach can be combined with other approaches to reduce latency of diffusion models. This is fair, but they do not show any empirical results around it.\n\nIn order to empirically demonstrate that the SRDS method can be used in conjunction with other approaches to reduce the latency of diffusion model sampling, we have extended beyond the base DDIM solver primarily used in the paper. We tested SRDS both in conjunction with DDPM (which takes relatively more steps than DDIM) and DPMSolver (which can reduce the number of steps beyond DDIM). For both methods, SRDS maintains a speedup over the corresponding baseline serial implementation (~3.6x for 1000-step DDPM and ~2.9x for 200-step DPMSolver ). In this way, we show that SRDS for example could be combined with the improvement from DDPM → DDIM or DDIM → DPMSolver in order to orthogonally improve latencies. Please refer to the combined response to all reviewers for more details on the same. \n\n> The approach relies completely on leveraging parallel refinement and the actual number of model evaluations are much larger (often by a factor of 3x) which can limit the applicability in some cases. \n\nYes, indeed there is a tradeoff: as we note in the introduction section, SRDS provides speedups in sampling at the cost of potentially additional parallel compute. While this might limit applicability in some cases (such as compute-bound, batched inference workloads), we would like to emphasize that this tradeoff enables the diffusion models for many other use cases such as real-time image or music editing and trajectory planning in robotics. In fact, there is an increasing trend of diffusion models run locally where small enough models or strong enough hardware can support the batching described in this paper. In such scenarios, strict latency requirements may justify the tradeoff of additional compute vs wall-clock times. Moreover, in a number of cases, we have found that SRDS provides reasonable predictions within a single Parareal iteration; here, the total number of model evaluations is only slightly larger than the serial approach (increasing from $n$ to $ n + 2\\sqrt{n}$). Lastly, it is also worth noting that many users are often agnostic to inference time GPU compute costs as they are orders of magnitude lower than training compute costs anyway.\n\n\n> Yet one of their results (last on in Table 2) has speed up of 0.7x. Is this all due to GPU overhead?\n\nNo, while there is certainly some loss due to GPU overhead, the <1.0x speedup is primarily due to the lack of pipeline parallelism (meaning that coarse and fine solves are performed serially rather than in parallel). Note that the worst case sampling latency being no worse than sequential sampling only holds for the pipelined version. Without pipelining, in the worst case, the speed up is 0.5x, and the 0.7x speaks to the early termination/convergence of SRDS. Please refer to Figure 4 for an illustration that clarifies this.\n\n> Is is possible to present empirical results with pipeline parallelism? It seems like the authors expect 2x gains with it, but I assume it will require some computational overhead and based on the previous question, the degradation can be non-trivial.\n\nPlease refer to the combined response to all reviewers for a broad discussion. While achieving close to 2x bonus gains with pipeline parallelism requires considerable engineering effort, we implemented a somewhat suboptimal version of pipelined SRDS, and this already showcases significant speedups compared to both non-pipelined SRDS (~20% speedup) and more importantly baselines such as ParaDiGMS (up to 38% faster than ParaDiGMS for DDIM on identical hardware even when using very generous convergence thresholds for ParaDiGMS; even faster when using stricter thresholds). The less than optimal speedups stems from GPU and scheduling overhead.\n\n> While I understand the claim regarding this approach being orthogonal to other works reducing, showing some experiments around the same will make for a stronger paper.\n\nWe hope the discussion/experiments in the common response on readily incorporating other solvers such as DDPM and DPMSolver into SRDS helps make this a stronger paper and provides context to how SRDS can be used in conjunction with other improvements in diffusion modeling!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2QpkvU5aoC", "reviewer": "Reviewer_MmLD", "summary": "This work proposes SRDS, a sampler for diffusion models that applies the Parareal algorithm, to reduce the overall sampling latency by introducing extra but parallelizable network evaluations compared to the fully sequential fashion. With higher device utilization or device parallelism through batched inference and pipelining, SRDS successfully reduces the overall sampling latency of existing diffusion models without sacrificing the sample quality.", "questions": "- In the current SRDS pipeline, the intermediate results of the fine-grained solver are only used once. Do you have any thoughts on utilizing them in subsequent steps for better sample quality or faster convergence?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written.\n- The proposed SRDS sampling pipeline is flexible and extendable, providing a neat baseline and rich directions for future technical improvements. \n- The experiment successfully demonstrates SRDS's practical value of reducing the sampling latency compared to full sequential sampling.\n- The ablation studies provide a nice practical guidance for SRDS.", "weaknesses": "- As the authors addressed in the limitation part, SRDS relies on parallelizing extra computations in exchange for a reduction in total latency, which may not be applicable to some scenario that performs full batch sequential sampling. However, I believe SRDS could still work in a fairly broad range of cases. \n- The important ParaDiGMs baseline is not compared in the experiment part.\n\n### Minor\n- The meaning of the abbreviation \"IVP\" in line `1:` of Algorithm.1  is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and are glad to hear the reviewer’s agreement of the broad applicability of SRDS! We respond to the concerns below:\n\n> ParaDiGMs baseline\n\nPlease refer to the combined response to all reviewers, where we clearly demonstrate the superiority of SRDS over ParaDiGMS. For instance, SRDS is up to 38% faster than ParaDiGMS for DDIM on identical hardware even when using very generous convergence thresholds for ParaDiGMS. We hope that addition of these results, as well as the comparisons between different samplers, helps provide more context to the impact of SRDS in diffusion model sampling.\n\n> In the current SRDS pipeline, the intermediate results of the fine-grained solver are only used once. Do you have any thoughts on utilizing them in subsequent steps for better sample quality or faster convergence?\n \nWe appreciate the question about only using the intermediate results of the fine-grained solver once, as it highlights that there is a rich body of work in parallel-in-time integration methods that may also be applicable to parallelizing the sampling of diffusion models. In particular, the intermediate re-use idea appears similar to PITA [1]; however, the inversion required in the calculation of the projection matrix makes it more complicated and less amenable to the high-dimensional outputs common to diffusion models. There may be room for such methods in latent diffusion; though have not yet explored such an approach due to the complexity. \n\nThere may be more room for simpler ideas in diffusion sampling, as the approximation to the score function may be inaccurate and simple averaging in re-use may provide a better approximation of the true score. These have not been explored as the main goal of the paper was to ensure that parallelized sampling was finding the solution to the ODE defined by the network. This would certainly be an interesting future direction to explore theoretically and empirically as an extension to SRDS! \n\n> The meaning of the abbreviation \"IVP\" in line 1: of Algorithm.1 is unclear.\n\nThanks for pointing this out. We will update the manuscript to clarify that it refers to ‘Initial Value Problem’.\n\n\n---\n\n\nReferences:\n\n- [1] C. Farhat and M. Chandesris, “Time-decomposed parallel time-integrators: theory and feasibility studies for fluid, structure, and fluid-structure applications,” International Journal for Numerical Methods in Engineering, vol. 58, no. 9, 2003, doi: 10.1002/nme.860. [Online].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tZXEua3ICA", "reviewer": "Reviewer_XqLH", "summary": "Inspired by the parallel-in-time ODE integration literature, especially Parareal method, this paper introduces SRDS as a fast sampler for diffusion models enabling efficient parallelization. Experimental results demonstrate that the proposed SRDS reduces the number of steps required to synthesize samples.", "questions": "1. Could the pipelined SRDS be easily deployed in practice?\n2. Why the SRDS Iters shown in Table 1 are not integers?\n3. Why the first and second rows of Figure 6 are almost the same?\n4. Minor points:\n\n(1) In line 10 of Algorithm 1, $x_{i-1}^p$ is undefined for p == 1.\n\n(2) It is redundant to use Figure 2, Figure 3b and Algorithm 1 to describe the same thing in the main text. They together occupy a whole page. Adding more experiments to support the effectiveness of the proposed method is more valuable.", "limitations": "Please refer to the weakness and question sections above. This manuscript needs an overhaul to make it meet the bar of a top conference.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "It is interesting to use parallel sampling algorithms to accelerate the sampling of diffusion models, which will benefit practical applications such as real-time image editing.", "weaknesses": "My concerns are mostly about the limited experimental results:\n\n1. This paper lacks of clear discussions about the differences between the proposed method and the previous work “23NeurIPS-Parallel sampling of diffusion models.”, and sufficient experimental evidence to demonstrate the effectiveness of the proposed method. Specifically, there is no quantitative comparison with the most related method ParaDiGMs. It seems that the existing method ParaDiGMs can provide better speedup according to their results.\n2. This paper lacks of a sufficient comparison to the existing fast samplers (such as Heun, DEIS, DPM-Solver). In practice, we can already synthesize samples with those advanced samplers with only around 10 steps. While this paper only presents experiments on several hundreds or one thousand steps at Table 1, which makes the method proposed in this paper unappealing.   \n3. It is claimed in Lines 38-40 and 196-198 that SRDS outperforms ParaDiGMs in terms of memory requirement. However, there is also no quantitative support from experimental results.\n4. It is claimed in Line 62-64 that SRDS is compatible with off-the-shelf solvers. However, the speedup is only shown on DDIM. The speedup offered by SRDS is only explicitly shown in Table 2, making it hard to fully assess the effectiveness of the proposed method.\n5. There is no quantitative comparison between SRDS and pipelined SRDS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, and appreciate that they recognize the practical applicability of our work! We respond to the concerns below:\n\n> ParaDiGMs baseline\n\nPlease refer to the combined response to all reviewers, where we clearly demonstrate the superiority of SRDS over ParaDiGMS. For instance, SRDS is up to 38% faster than ParaDiGMS for DDIM on identical hardware even when using very generous convergence thresholds for ParaDiGMS.\n\n> Comparison to existing solvers\n\nWe'd like to re-emphasize that SRDS provides an orthogonal improvement compared to the other lines of research on accelerating diffusion model sampling. In particular, while the main experiments (and writing) were focused on DDIM, SRDS is compatible with other solvers and they can be readily incorporated into SRDS to speed up diffusion sampling. We have empirically also evaluated that the SRDS method can be used with other solvers, such as DDPM (often requiring more steps than DDIM) and DPMSolver (often requiring fewer steps than DDIM). For both methods, SRDS maintains a speedup over the corresponding baseline serial implementation (~3.6x for 1000-step DDPM and ~1.5x for 25-step DPMSolver ).  Please see the combined response for a greater discussion of this extension. \n\n> SRDS outperforms ParaDiGMs in terms of memory requirement\n\nApologies for the confusion. Yes, indeed ParaDiGMs has a higher memory requirement. Specifically, for a T step denoising process, ParaDiGMS needs to perform $T$ model evaluations in parallel with subsequent computations needing information about all previous evaluations, while SRDS only requires $\\sqrt{T}$ parallel evaluations (which fits comfortably in GPU memory) and requires much lesser communication between GPUs. While the prohibitively large memory requirement can be combated with a sliding window method, the significantly larger communication overhead remains because at every step of Paradigms, an AllReduce over all devices must be performed in order to calculate updates to the sliding window. (For instance, even when ParaDiGMS reduces Eff. Serial Steps by 20x, the obtained speedup is only 3.4x). This is in contrast to the independent fine-solves in parareal that only need to transfer information for the coarse solve. \n\nBelow, we demonstrate how the minimal memory and communication overhead of SRDS shines through as we are able to achieve better device utilization as we increase the number of available GPUs. The following experiment was performed on 40GB A100s and used a generous 1e-2 threshold for ParaDiGMS.\n\n|  |  | Serial | SRDS | SRDS | ParaDiGMS | ParaDiGMS |\n|---|---|:---:|:---:|:---:|:---:|:---:|\n|  | Devices | Model Evals | Eff Serial Evals | Time Per Sample | Eff. Serial Evals | Time Per Sample |\n| DDIM | 1 | 25 | 15 | **1.62** | 16 | 2.71 |\n| DDIM | 2 | 25 | 15 | **1.08** | 16 | 2.01 |\n| DDIM | 4 | 25 | 15 | **0.82** | 16 | 1.51 |\n\nIn the table above, we can observe how SRDS has a significantly better time per sample than ParaDiGMS despite having a similar number of effective serial evaluations.\n\n> Pipelined SRDS\n\nWe compare SRDS and pipelined SRDS in the overall response to all reviewers. We hope this provides more context and strengthens the empirical evaluation!\n\n\n> Could the pipelined SRDS be easily deployed in practice?\n\nYes, the pipelined SRDS can be easily deployed in practice, though it does take minor adjustments to common diffusion modeling code in order to be efficiently implemented. For example, the StableDiffusion pipeline makes use of a scheduler that ahead-of-time sets the number of inference steps (as a class variable). Adjusting this parameter to be modified at runtime is required for pipelining. Additionally, a framework is required to coordinate computation and data transfers; our implementations make use of torch.multiprocessing and the queue organization to coordinate the launches of the different nodes.\n\nWhile extracting close to optimal bonus speedup (2x) from pipelining might require considerable engineering effort, we implemented a suboptimal version of pipelined SRDS and already achieve substantial speedups compared to non-pipelined SRDS and more importantly baselines such as ParaDiGMS. Please refer to the combined response for quantitative results on the same.\n\n> Why the SRDS Iters shown in Table 1 are not integers?\n\nIn Table 1, for pixel-based diffusion, we used a very tight/conservative threshold on the convergence of the output in order to define early convergence. We then measure the average number of SRDS iterations required until convergence across 5000 samples, which ends up not necessarily being an integer. This contrasts from Table 2 (Stable Diffusion) which caps the number of iterations with no measurable degradation in sample quality. We will make this more clear in the revised manuscript text, as well as add an equivalent version of Table 1 that includes a capped number of iterations in order to obtain a fairer comparison to baselines (e.g. ParaDiGMS) and demonstrates much greater speedups without loss of sample quality. In fact, Figure 7 in Appendix C already provides support for the fact that even for pixel-based diffusion models, we can similarly limit the number of SRDS iterations to 1 or 2 without degradation in sample quality, while achieving much greater speedups than reported in Table 1.\n\n> Why the first and second rows of Figure 6 are almost the same?\n\nThe first and second rows of Figure 6 seek to highlight the early convergence of the SRDS algorithm to the output of sequential sampling. For a visualization of the ‘iterative refinement’ through rounds of parareal iterations, we direct the reviewer to Figure 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrTgkHaG52", "reviewer": "Reviewer_pK7d", "summary": "This paper proposes Self-Refining Diffusion Samplers (SRDS), which draws inspiration from the Parareal algorithm, aims to solve the reverse process accurately without retraining models, balance the tradeoff between sample quality and sampling speed. lower latency for requiring fewer sampling steps to reach convergence. The experiment section shows that the proposed algorithm speeds up the convergence without degrading sample quality in image generation tasks.", "questions": "1. For the experiment section, are the numbers intentionally matched in ``FID Score`` in Table 1 and ``CLIP Score`` in Table 2? If that's true, it would be also interesting to see how the performance changes when you have different SRDS Iters and their corresponding Total Evals.", "limitations": "This paper discussed the limitations and pointed out the potential positive or negative social impacts will not be the direct consequence of this work in the checklist guidelines.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is motivated by Parareal algorithm to speed up sampling process without lowering sample quality which seems to work well with image generation.\n2. This paper provides theoretical analysis for sampling convergence and latency investigation.\n3. This paper observes the pipelined SRDS, which further speeds up the sampling process.", "weaknesses": "1. In related work, the authors mentioned that ParaDiGMS proposed by Shih et al [1] is another parallel-in-time integration method, and this paper also works on image generation. In this case, I believe their results could be baselines that this paper should compare against.\n\n[1] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, and are happy to hear that the experimental results are convincing! We respond to the concerns below:\n\n> ParaDiGMs baseline\n\nPlease refer to the combined response to all reviewers, where we clearly demonstrate the superiority of SRDS over ParaDiGMS. For instance, SRDS is up to 38% faster than ParaDiGMS for DDIM on identical hardware even when using very generous convergence thresholds for ParaDiGMS.\n\n\n> Are the numbers intentionally matched in FID Score in Table 1 and CLIP Score in Table 2\n\nThanks for the question! Yes, the FID Score in Table 1 and CLIP Score in Table 2 are presented to show a tolerance/SRDS iterations so as to have no measurable degradation in sample quality. In the original version of the manuscript, we have included an ablation study for the number of SRDS iterations and the corresponding CLIP score in Figure 5; showcasing the capability of the model to achieve early convergence. For pixel space diffusion, we provide an ablation of tolerance threshold vs sample quality in Appendix C. We note that we use KID score instead of FID score due to computational budget constraints (KID is an unbiased estimator and requires much fewer samples than FID).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmlB177c1w", "reviewer": "Reviewer_KhBd", "summary": "This paper proposes a new method for parallelizing the sampling process of diffusion models, offering a novel way to trade compute for speed. In particular, it relies on the parareal iteration in this setting, where the main idea is to split the solving timesteps into multiple groups and then update them in parallel. To ensure that the output from these timestep groups matches the exact solution, parareal performs predictor-corrector steps after each group is updated. Experiments demonstrate the efficacy of applying parareal iteration in speeding up the sampling process.", "questions": "1. In my understanding, the technique in Parareal have a different spirit in parallelization with [1][2]. For example, consider that we wish to do n-step sampling with $\\sqrt n$ batchsize. Given a initial sequence $[x_1,...,x_n]$, parareal first group them into $g_1=[x_1,...,x_{\\sqrt n}],...,g_{\\sqrt n}=[x_{n-\\sqrt n},x_n]$. The parallelism is that to update $g_1$, ..., $g_n$ in parallel, while the update of each $g_i$ consists of $\\sqrt n$ sequential solving. On the other hand, [1][2] consider update $[x_{i},...,x_{i+\\sqrt n}]$ in parallel until convergence. It is not clear to me which one is better from the theoretical aspects, as they have the same worst-case convergence bound. I suspect that it should depends on the form of ODE and also the batchsize? I wonder if the authors could provide some thought and discussion on this point?\n\n2. Do you think parareal can be applied to the SDE sampler as [1] and [2] did?\n\n\n[1] Shih, Andy, et al. \"Parallel sampling of diffusion models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Tang, Zhiwei, et al. \"Accelerating parallel sampling of diffusion models.\" Forty-first International Conference on Machine Learning. 2024.", "limitations": "See my comments on weaknesses and questions above. This work lacks a correct and thorough discussion comparing it to existing works.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper demonstrates a very interesting direction by introducing the parareal iteration into diffusion models.\n2. The experiments indeed demonstrate the efficacy of this approach.", "weaknesses": "1. I think the author misses a rigorous discussion comparing the parareal algorithm with ParaDiGMS [1] and also completely ignores a more recent reference [2]. It is clear that all these works, including parareal, focus on parallelizing the sampling process via certain forms of fixed-point iteration, although the computing graphs of these algorithms differ. The only discussion in the current manuscript is that ParaDiGMS is highly memory-intensive and must use a sliding window. While this statement is certainly true, I believe it is not a disadvantage of ParaDiGMS and does not imply that ParaDiGMS is inferior to parareal. I do not see why using a sliding window should be considered a disadvantage.\n\n2. As far as I can see, the Eff. Serial Evals in Table 1 and Table 2 should correspond to the Parallel Iters in [1] and Steps in [2], am I correct? Because Eff. Serial Evals essentially represent the number of batch evaluations on the models. If this is the case, based on the current numbers in Table 1, it seems quite inferior to the results in [1] and [2]. Of course, I understand that the batch size may be different in this work, but I still wish the author would provide a detailed discussion on this point.\n\n3. Proposition 1 in the manuscript is somewhat trivial and exists in many existing works. Note that each refinement consists of $\\sqrt n$ sequential evaluation, so the parareal will also require at most $\\sqrt n \\times \\sqrt n=n$ sequential evaluation on the models to converge. I think the author should mention that this result is fundamentally equivalent to Proposition 1 in [1], and a more general result is found in Theorem 3.6 in [2]. The key point is that when performing fixed-point iteration for an $n$-order triangular system, it will certainly converge within $n$ steps, and this bound can not be further improved without extra assumption.\n\n\n\n[1] Shih, Andy, et al. \"Parallel sampling of diffusion models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Tang, Zhiwei, et al. \"Accelerating parallel sampling of diffusion models.\" Forty-first International Conference on Machine Learning. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, and are excited to hear that the reviewer finds the novel ideas presented to be very interesting, and that the experiments indeed demonstrate the efficacy of this approach! We respond to the concerns below:\n\n> Baselines\n\nPlease refer to the combined response to all reviewers. We thank the reviewer for bringing [2] to our attention; we were not aware of this very recent work (ICML 2024) at the time of writing this paper. However, their method uses completely different techniques (namely, formulating and solving a triangular system in a specialized way) to solve the same problem. We have discussed this work in the common response, and will make sure to include a detailed discussion in the final manuscript.  \n\n> As far as I can see, the Eff. Serial Evals in Table 1 and Table 2 should correspond to the Parallel Iters in [1] and Steps in [2] ... based on the current numbers in Table 1, it seems quite inferior to the results in [1] and [2]. Of course, I understand that the batch size may be different in this work, but I still wish the author would provide a detailed discussion on this point.\n\nThank you for highlighting the difference in notation between the papers. You are correct that the Eff. Serial Evaluations correspond to the Parallel Iters in [1] and Steps in [2], where all refer to the number of batched evaluations. We will be sure to clarify this notation in updates to the paper.\n\nThe reasons for the seemingly high number of Eff. Serial Iterations is twofold:\n\n- *Stricter convergence criteria for SRDS*: We note that there is a considerable difference in the how the convergence criteria is calculated for ParaDiGMS and SRDS. ParaDiGMS using a per step tolerance proportional to the scheduler’s noise magnitude and aggressively (and irrecoverably) slides the sliding window if an early step has “converged”; this gives very loose guarantees of the final generated image. On the other hand, SRDS only directly considers differences in the final output representation to decide convergence. This strict/conservative threshold for our pixel diffusion results in Table 1 causes larger number of effective serial iteration. One can comfortably relax this threshold and still maintain good sample quality (as seen in the ablation study in Appendix C). Alternatively, as we show for StableDiffusion, one can instead limit the number of iterations and drastically reduce the Eff. Serial Evaluations (comparable to ParaDiGMS) with no measurable degradation in sample quality! In fact, even for pixel-based diffusion models, we can similarly limit the number of SRDS iterations to 1 or 2 without degradation in sample quality (Figure 7 in Appendix C).\n\n- *Batch size*: As the reviewer hinted correctly, ParaDiGMS has a much higher needed batch size. Specifically, for a T step denoising process, ParaDiGMS needs to perform T model evaluations in parallel, while SRDS only requires $\\sqrt{T}$ evaluations, which fits comfortably in GPU memory. To combat this using sliding windows, ParaDiGMS is able to arbitrarily increase the compute and create larger sliding windows (in exchange for reducing the number of serial iterations). It is worth noting that the comparison performed for the number of serial evaluations in the ParaDiGMS paper is on a system of 8 A100 80Gb GPUs whereas our experiments are performed on 4 A100 40Gb GPUs. To shed light on this difference in platforms, we also try to provide a more fair comparison in our experiments in the combined reviewer response, and we will add this clarification to the revised manuscript.\n\n\n> fixed-point iteration \n\nOur rationale to include Proposition 1 in the manuscript, in a similar manner to Proposition 1 in [1] is to highlight the worst case behavior of the SRDS algorithm and provide the guarantee on convergence, despite the triviality. We will be sure to include the cross-references to the appropriate equations in [1] and [2], especially in the larger context of fixed-point iteration for n-order triangular systems.\n\n> different spirit in parallelization with [1][2] ... which one is better from the theoretical aspects\n\n This is a great question! Indeed, SRDS has a different spirit of parallelization compared to [1] and [2]. We like to view it as a multigrid/multiresolution method where we solve for the trajectory at a low resolution and use parallel high-resolution solves to correct the trajectory. This is in contrast to [1] and [2] which have the flavor of more traditional fixed point iteration/updates. From a theoretical point of view, it is actually unclear which is fundamentally better. Even if SRDS might seem slightly superior empirically today, all three (SRDS [1] and [2]) are still the first works in this area of parallel sampling, and it is entirely possible that one of them possesses superior theoretical properties/guarantees that can be exploited. As we briefly discussed in the future work section of the paper, convergence guarantees/properties for parareal have been analyzed only in a handful of special cases such as heat equations, and it would be a very interesting future direction to see if the diffusion ODE admits better convergence guarantees for parareal (in line with what is observed empirically). \n\n> SDE sampler\n\nYes, as we mention in the paper, SRDS can indeed be readily applied to the SDE sampler. As seen in [1], we can just use the trick of pre-sampling the noise upfront, and the rest of the algorithm remains the same. Notably, it is even possible to ensure that the noise at the coarse and fine-time scales is aligned; however, this is not necessary due to the use of the predictor corrector updates in the parareal algorithm.\n\n--- \nReferences:\n- [1] Shih, Andy, et al. \"Parallel sampling of diffusion models.\" Advances in Neural Information Processing Systems 36 (2023).\n- [2] Tang, Zhiwei, et al. \"Accelerating parallel sampling of diffusion models.\" Forty-first International Conference on Machine Learning. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful reviews! We are thrilled to hear that the reviewers find that our paper is “well written and easy to follow”, “demonstrates a very interesting direction”, provides “a neat baseline and rich directions for future technical improvements”, and “will benefit practical applications such as real-time image editing”.

Here, we answer common questions:
- Empirical comparison to ParaDiGMS baseline: Up to 38% faster than ParaDiGMS for DDIM on identical hardware even when using very generous convergence thresholds for ParaDiGMS; even faster when using stricter thresholds!
- Quantitative evaluation of pipelined SRDS: Up to 20% faster than non-pipelined SRDS even with a suboptimal implementation!
- Incorporation of other samplers: SRDS with DDPM and DPMSolver yield similar speedups of up to ~3.6x over sequential!

Other questions are answered in individual responses to reviewers. Further, we’ll include these results/clarifications in our revised manuscript.

# Comparison to Baselines
We demonstrate the superiority of SRDS to baselines ParaDiGMS [1] and ParaTAA [2]. We especially thank reviewer KhBd for bringing [2] to our attention. [2] takes a different approach to accelerating parallel sampling (solving triangular nonlinear systems using special techniques) and we'll include a thorough discussion of it in the final manuscript. Unfortunately, as this is a new paper (ICML 2024), we were unaware of it while writing our paper, and consequently we're unable to provide a comprehensive empirical comparison with [2] with identical hardware at the moment. Nonetheless, for now, we demonstrate the high-level superiority of SRDS solely by using the results published by the authors in [1] (Table 5) and [2] (Table 1).

In the table below, we show that SRDS offers better wall-clock speedups (over sequential) in sample generation time for StableDiffusion when compared to [1] and [2]. We clarify that the reported speedup for each method is w.r.t sequential solve on the same machine that the corresponding parallel method was evaluated. Our results below are particularly impressive given that the authors of [1] used 8x 80GB A100s for the evaluation and the authors of [2] used 8x 80GB A800 for the same, while we (SRDS) only used 4x 40GB A100 for the evaluation due to computational constraints. (When interpreting, recall that a sequential solve isn't compute/memory bound and doesn’t benefit significantly from more GPU compute, whereas the parallel methods certainly do!) We'd also like to highlight the superiority of SRDS over the baselines in the regime of small number of denoising steps (25) as particularly impactful.
||Denoising Steps|ParaDiGMS|ParaTAA|Pipelined SRDS|
|---|:---:|:---:|:---:|:---:|
|DDIM|100|2.5x|1.92x|**2.73x**|
|DDIM|25|1.0x|1.17x|**1.72x**|

For the main baseline ParaDiGMS that we consider in our paper, we also performed more extensive evaluation to evaluate both methods on equal hardware to more clearly demonstrate the benefits of SRDS. Below, we demonstrate that SRDS consistently beats ParaDiGMS on wallclock speedups. Though the ParaDiGMS paper uses a convergence threshold of 1e-3, we show that SRDS can provide impressive speedups even when compared to significantly relaxed ParaDiGMS thresholds of 1e-1. The following StableDiffusion experiments are performed on identical machines (4 40GB A100 GPUs) for a fair comparison.
||Serial|Serial|Pipelined SRDS|ParaDiGMS|ParaDiGMS|
|---|:---:|:---:|:---:|:---:|:---:|
||Model Evals|Time Per Sample|Time Per Sample|Threshold|Time Per Sample|
|DDIM|961|44.88|**10.31 (4.3x)**|1e-3|275.29|
|||||1e-2|20.48|
|||||1e-1|14.30|
|DDIM|196|9.17|**2.85 (3.2x)**|1e-3|29.45|
|||||1e-2|5.08|
|||||1e-1|3.42|
|DDIM|25|1.18|**0.69 (1.7x)**|1e-3|1.98|
|||||1e-2|1.51|
|||||1e-1|0.77|

# Pipelined SRDS
We demonstrate that pipelining can indeed further speed up SRDS. We implemented a slightly suboptimal version of pipelined SRDS for StableDiffusion below and already observe significant speedups; with some more engineering effort, we can further push towards extracting the full potential of pipelining. However, we believe that because this already beats the baselines, this sufficiently demonstrates the benefits of SRDS.

Our implementation is suboptimal due to using a single device for coordinating pipeline parallelism and device transfers (artifact from torch.multiprocessing). A better approach would use ring-like communication between devices instead of relying on a coordinator.
||Serial|SRDS|SRDS|Pipelined SRDS|Pipelined SRDS|
|---|:---:|:---:|:---:|:---:|:---:|
||Model Evals|Eff Serial Evals|Time Per Sample|Eff Serial Evals|Time Per Sample|
|DDIM|961|93|12.30|63|**10.31**|
|DDIM|196|42|3.30|27|**2.85**|
|DDIM|25|15|0.82|9|**0.69**|

# Incorporation of Other Solvers
As stated in our paper, we'd like to emphasize that SRDS provides an orthogonal improvement compared to the other lines of research on accelerating diffusion model sampling. In particular, while the main experiments (and writing) were focused on DDIM, SRDS is compatible with other solvers and they can be readily incorporated into SRDS to speed up diffusion sampling. For example, SRDS is directly compatible with other solvers such as DDPM (often requiring more steps than DDIM) and DPMSolver (often requiring fewer steps than DDIM) and can efficiently accelerate sampling in both cases, as shown with StableDiffusion below.
||Sequential|Sequential|SRDS|SRDS|SRDS|
|---|:---:|:---:|:---:|:---:|:---:|
||Model Evals|Time Per Sample|Eff Serial Evals|Time Per Sample|Speedup|
|DDPM|961|44.68|93|12.30|**3.63x**|
|DDPM|196|9.03|42|3.26|**2.76x**|
|DPMSolver|196|10.30|42|3.49|**2.95x**|
|DPMSolver|25|1.31|15|0.88|**1.48x**|
|DDIM|196|9.17|42|3.30|**2.77x**|
|DDIM|25|1.18|15|0.82|**1.43x**|

We also highlight that our Diffusers-compatible implementation needs only minor changes to solver arguments, indicating that SRDS can likely be easily extended to future community-developed methods.

Meta-review: This paper addresses the computational complexity of Diffusion Models, and seek to overpower the slow sequential evaluations which typically drive  diffusion model generation.  Specifically they propose Self-Refining Diffusion Samplers (SRDS) to improve the speed at a cost of parallel compute while preserving the sample quality.  This is in contrast to prior attempts at the problem toying with discretization. Convergence results on an ODE have been shown and comparative experiments were carried out. 
 While all 4  reviews out of 5 rated the paper as a "weak accept", and one as reject,  several critical comments were made, although most were successfully rebutted by the authors to the reviewers' satisfaction.  One significant one pertained to the proposed Parallel computational  non-novelty  with  respect to some specific recent references cited refs [1]& [2] by reviewer kHBd.
The authors have responded to the reviewer's satisfaction that in contrast to the multigrid/multiresolution flavor, the prior cited refs.  used classical  fixed point iteration/updates. 
The "Reject" review also raised several questions and some due to confusion and others were in the end rebutted to the reviewer's agreement who nevertheless kept their reject intact. Judging by the comments, I believe that the reviewer is a little confused, 
I think sufficient experimental evidence was also provided as attested by the other 4 reviewers.

.
Decision: Accept (poster)