Self-Refining Diffusion Samplers: Enabling Parallelization via Parareal Iterations

Authors: nikil  roashan
selvam amil merchant
stefano  ermon
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:41:19

Abstract: In diffusion models, samples are generated through an iterative refinement process, requiring hundreds of sequential model evaluations. Several recent methods have introduced approximations (fewer discretization steps or distillation) to trade off speed at the cost of sample quality. In contrast, we introduce Self-Refining Diffusion Samplers (SRDS) that retain sample quality and can improve latency at the cost of additional parallel compute. We take inspiration from the Parareal algorithm, a popular numerical method for parallel-in-time integration of differential equations. In SRDS, a quick but rough estimate of a sample is first created and then iteratively refined in parallel through Parareal iterations. SRDS is not only guaranteed to accurately solve the ODE and converge to the serial solution but also benefits from parallelization across the diffusion trajectory, enabling batched inference and pipelining. As we demonstrate for pre-trained diffusion models, the early convergence of this refinement procedure drastically reduces the number of steps required to produce a sample, speeding up generation for instance by up to 1.7x on a 25-step StableDiffusion-v2 benchmark and up to 4.3x on longer trajectories.

Main paper: 
 introduction: 
 Deep generative models based on diffusion processes have showcased the capability to produce high-fidelity samples in a wide-range of applications [28, 11, 39, 27] . From their origins in image and audio generation [31, 33, 10] , diffusion models have enabled robotic applications as well as scientific discovery via drug design [1] . Despite this promise, sampling from diffusion models can still be prohibitively slow. Early Denoising Diffusion Probabilistic Models [10] required a thousand sequential model evaluations (steps), and state-of-the-art models such as StableDiffusion [27] can still require up to 100s of iterations for high-quality generations. This large number of sampling steps leads to high-latencies associated with diffusion models, limiting applications such as real-time image or music editing and trajectory planning in robotics [13, 12] .As sampling involves solving an ordinary differential equation (ODE), a prominent body of research -including works such as Denoising Diffusion Implicit Models (DDIM, [32] ), Diffusion Exponential Integrator Sampler (DEIS, [42] ), and DPM-Solver [19] -has tried to reduce the number of model evaluations required by introducing various approximations. For example, progressive distillation [29] requires re-training models to approximate the solution to the ODE at larger timesteps. However, such approaches trade-off speed at the cost of sample quality.In this work, we instead take an orthogonal approach: we focus on additional parallel compute and show how this can be used to reduce latencies while still providing accurate solutions to the original ODE, thereby preserving sample quality. Recently Shih et al. [30] leveraged a parallel-in-time integration method to introduce the first highly-parallelizable algorithm for diffusion model sampling. The presented ParaDiGMs algorithm builds on Picard iterations [26] to perform denoising steps Figure 1 : A visualization of the iterative refinement provided by the SRDS algorithm on a sample from StableDiffusion with the prompt 'a beautiful castle, matte painting.' The initial coarse solve (left) via limited steps provides a rough estimate of the sample, which iteratively refined through iterations of our algorithm. Early convergence is observed as the 3rd output nearly matches, a key feature that enables efficient generation. across the trajectory in parallel, leading to state-of-the-art sampling speeds on popular benchmarks [30] . Despite the promising results, ParaDiGMs can be highly memory bound due to the use of sliding window methods and also has a sequential convergence criteria requiring communicationexpensive cumulative sums across devices to coordinate parallel-sampling. The method also has limited controllability over the accuracy of the final solution, assuming convergence per step rather than in the final generation. It is also worth noting that in concurrent work, Tang et al. [37] take a completely different approach to accelerating parallel sampling: special techniques in solving triangular nonlinear systems through fixed point iteration. Instead, in this paper, we turn to multipleshooting methods from the parallel-in-time ODE integration literature [14, 3, 6] and aim to improve parallelization of diffusion sampling by utilizing multiple resolutions across the time domain [21] .Specifically, we present Self-Refining Diffusion Samplers (SRDS) that start with a quick but rough solve of the diffusion trajectory, achieved by limiting the number of total steps taken (for instance, using a few-step DDIM solver). The trajectory can then be simulated to higher fidelity via a highlyparallel algorithm that updates the final generation iteratively until convergence. At a high level, each refinement step of SRDS partitions the current guess of the trajectory into blocks, and simulates each of these blocks at the desired (high) resolution. The running guess for the overall trajectory is then updated via a predictor-corrector mechanism based on the Parareal algorithm to accelerate convergence. This iterative refinement allows us to efficiently interpolate in parallel between a coarse solution corresponding to a low-fidelity sample and an accurate solution corresponding to a high-fidelity sample. The key benefits of SRDS are three-fold: Approximation-Free: By design, SRDS computes an accurate solution to the reverse process (as defined by the practitioner's choice of diffusion solver), thereby maintaining high quality of samples. Importantly, as it is purely a sampling algorithm, it does so without incurring any retraining cost.Extensive Control and Compatibility: By serving as an efficient interpolation method between the coarse and fine-grained solvers, SRDS provides the practitioner with flexible control of the tradeoff between sample quality and speed. For instance, one could first start with a rough solve (corresponding to, say, few-step DDIM). Then, if desired, one can add a budget-appropriate number of parallel SRDS iterations (instead of sequential) to refine the obtained sample. Furthermore, SRDS is compatible with most existing off-the-shelf solvers (such as Euler, Heun, DPM etc), thereby providing direct benefits to virtually any diffusion workflow.Low Latency: Most importantly, we find that the number of iterations required by SRDS for convergence is quite low, leading to drastic improvements in latency for sampling. We present results using the Self-Refining Diffusion Samplers on a wide range of benchmarks, starting with pixel-based image diffusion models and further exploring latent-methods where SRDS leads to a 1.7x improvement in the sampling speed on the 25-step StableDiffusion-v2 benchmark and up to 4.3x on longer trajectories. 1 Through enabling faster sampling, SRDS aims to unlock capabilities for real-time interaction with diffusion models. 
background: 
 Diffusion models are a general class of generative models that rely on a noising procedure that converts the data distribution into noise via a series of latent variables updates. For the purposes of this work, we will consider the continuous-time generalization presented by Song [35] and Denoising Diffusion Implicit Models [10] that formulate sampling as solving the initial value problem characterized by the probability flow ordinary differential equation (ODE):EQUATIONwhere s θ (x, t) is a time-conditional prediction of the score function ∇ x log p t (x) from the diffusion model. To be consistent with prior work on parallelized diffusion sampling Shih et al. [30] , we use a reversed time index (from traditional notation) where x 0 refers to pure Gaussian noise, and x T refers to the denoised image after T denoising steps.Given the dynamics governing the differential equation, our goal is to provide accurate solutions to:EQUATIONin order to produce a sample from the diffusion model. Common approaches discretize the time interval [0, T ] into N pieces (t 0 =0, t 1 , t 2 , ..., t N =T ) and solve a sequence of initial value problems to yield an approximation (x 0 , x 1 , ..., x N =x T ) to the trajectory.Formally, a solver is a function F(x start , t start , t end ) that propagates x from t = t start with initial value x start to t = t end . Solving the differential equation corresponds to approximating the solution x T to the given initial value problem by a sequence of N solves:EQUATIONThe choice of solver F dictates the sampling speed and accuracy of the solution. In practice, solvers which are accurate are often slow (due to high number of evaluations of h θ ), whereas solvers that are fast tend to have reduced accuracy. Initial works on diffusion models used the classical Euler method as choice of F, and it can be expressed as:EQUATIONHowever, DDIM [32] quickly became a popular choice of F for its improved efficiency. Other recent works have tried to rely on approximations or leverage various ideas from the numerical methods literature to design solvers F that require fewer denoising steps. For instance, Diffusion Exponential Integrator Sampler (DEIS, [42] ), and DPM-Solver [19] exploit the special structure of the probability flow ODE to design special solvers where the linear part of the ODE is solved analytically and the non-linear part is solved by incorporating ideas from exponential integrators in the numerical methods literature. Karras et al. [13] leverage the Heun's second order method and demonstrate a favorable tradeoff between number of model evaluations and quality of generated samples for a small number of denoising steps. In this work, SRDS presents an orthogonal improvement to these methods via parallelization, and by default we will assume all our solvers to be DDIM.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of the work are discussed in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
self-refining diffusion samplers: 
 Attempts to reduce the number of steps in diffusion samplers can provide speedups in sample generation [29, 23] , but unfortunately often lead to lower-sample quality. While low-frequency components (in the Fourier sense) of the images may be well-established, the generations miss the high-frequency details that leads to good generations [40] . To fix sample quality while maintaining the latency benefits of reducing the number of steps, we turn to numerical methods introduced in the parallel-in-time integration literature where dynamics with different components having different rates of convergence has been extensively studied. Specifically, multiple-shooting and multigrid methods have seen success in a wide range of domains from convection-diffusion equations to eigenvalue problems [2, 22, 24, 4] by creating a rough but efficient solve of the prescribed differential equation that can then be iteratively updated via a highly parallelizable simulation. One such algorithm -Parareal [18] -serves as the backbone for our Self-Refining Diffusion Samplers that we describe below.Parareal makes use of two solvers: solver F (called the 'fine solver') provides accurate solutions but is slow to evaluate, and G (called the 'coarse solver') provides rough solutions but is much quicker.Parareal targets general purpose initial value problems of forms similar to Equation 1. Consider a partition (t 0 , t 1 , ..., t N = T ) of the time axis [t 0 , T ] into N intervals of equal width. Using the same solver notation as above, the goal is to approximate the solution x N to the initial value problem that would be produced using a sequence of fine solves:x i+1 = F (x i , t i , t i+1 ) , ∀i ∈ [0, N -1]The key insight of parareal is that we can first use the coarse solver G to quickly produce a rough trajectory, and this rough solution can be iteratively refined using parallel calls to the fine solver F.Formally, the parareal algorithm begins with a rough estimate of the trajectory, initialzied via a series of coarse solves from G.EQUATIONwhere the notation x 0 i denotes the initial estimate of the trajectory from the coarse solver (orange curve in Figure 2 ). Parareal then proceeds in iterations until convergence, where each iteration corresponds to a refinement of the trajectory. At each iteration, we solve the differential equation in each of the N time intervals at a higher resolution using the fine solver F, where the initial value for each interval is given by the estimate of the trajectory from the previous iteration. Crucially, these fine solves (blue in Figure 2 ) can be performed in parallel. Lastly, at the end of each iteration, we perform another coarse sequential solve through the trajectory (magenta in Figure 2 ) and incorporate the results of the fine solves into the running solution for the trajectory using a predictor-corrector method, where the coarse solver 'predictions' are 'corrected' via the updates from the parallel fine solves. Formally,x p+1 i+1 = F (x p i , t i , t i+1 ) + G x p+1 i , t i , t i+1 -G (x p i , t i , t i+1) , i = 0, . . . , N -1 (6) where the notation x p i denotes the running estimate of the trajectory at Parareal iteration number p.Now, we turn our attention back to drawing a sample from our diffusion model, which as discussed corresponds to estimating a solution to the initial value problem as defined in Equation 1. We leverage the Parareal algorithm in order to parallelize sampling. Our idea is to compute a solution on the N -discretization of the interval by instead considering a coarser√ N -discretization 2 of the interval [0, T ]. Let ∆T = T / √ N , t i+1 = t i + ∆T , partitioning [0, T ] into √ N intervals.We pick √ N -step DDIM solver3 as our fine solver F. In other words, F(x i , t i , t i+1 ) is the result of a √ N -step DDIM solve propagating x from t = t i with initial value x i to t = t i+1 . We pick 1-step DDIM solver as our coarse solver G. That is, G(x i , t i , t i+1 ) denotes the result of the corresponding 1-step DDIM solve propagating x from t = t i with initial value x i to t = t i+1 ("step" refers to denoising step involving an h θ evaluation).The SRDS algorithm proceeds as follows. We start with the coarse-solve to generate a rough estimate of the trajectory and final sample, achieved by takingN DDIM steps in total. Then, each of the √ N coarse predictions in the trajectory is simulated at higher resolution with furtherN DDIM-iterations in parallel, each with an effective time step corresponding to the original N -step discretization of the diffusion model. Iterative updates to the running solution then proceed in a manner equivalent to Parareal updates until convergence, as measured by the change in the outputted generation. Our SRDS algorithm is summarized in Algorithm 1.The ideal result for diffusion sampling is to get the solution arising from N sequential denoising score steps. SRDS however only starts with a rough solve of the diffusion trajectory takingN sequential denoising steps. Nevertheless, we can show that each iteration of SRDS (line 6 of Algorithm 1) refines the generated sample and leads us closer to the ideal solution. Proposition 1. The sample output by SRDS converges to the output of the N -step sequential solver in at most √ N refinement iterations.A key property of our algorithm is that after i iterations (refinements to the diffusion trajectory) of SRDS, the first i steps of the running trajectory exactly matches the trajectory generated by the sequential solver for the corresponding intervals. Consequently, the algorithm is guaranteed to converge in at most √ N iterations. We defer the formal proof to Appendix A. It is also worth x0 x1 x2 • • • x √ N x √ N +1 x √ N +2 • • • x 2 √ N • • • xN (a) Sequential Sampling x p 0 x p 1 x p 2 • • • x p √ N x p √ N +1 x p √ N +2 • • • x p 2 √ N • • • x p N x p+1 0 x p+1 1 x p+1 2 • • • x p+1 √ N x p+1 √ N +1 x p+1 √ N +2 • • • x p+1 2 √ N • • • x p+1 N (b) SRDS√ N ], [ √ N , 2 √ N ],and so on -can be perform the fine solves independently in parallel. The blue arrows correspond to 1-step coarse solves.noting that this worst case guarantee is similar in spirit to Proposition 1 in [30] and its generalization (Theorem 3.6 in [37] ).It is worth noting, however, that in practice we observe that the number of refinement iterations required till convergence -which is defined as difference in consecutive sample generations not exceeding a chosen threshold -is much less than the worst case bound ofN . This early convergence is critical to speedups from SRDS. The exact choice of threshold τ in line 13 of Algorithm 1, is a hyperparameter that is empirically chosen so as to avoid measurable degradation in sample quality.SRDS benefits from two key features to reduce latencies: batched inference and pipelining.First, the fine solves that are used in order to refine the trajectories implementation-equivalent DDIM-steps, which means that they can be performed in a batched manner even for a single sample generation. This parallelization allows for a single sample generation to incur the benefits of batched inference, introducing higher device utilization or device parallelism.Secondly, we observe that the dependency graph for SRDS enables pipelined parallelism. As outlined in Figure 3 , we find that F (x p i , t i , t i+1 ) and G (x p i , t i , t i+1 ) both only depend on x p i . The tasks for computing F x i j , t i , t i+1 and G (x p i , t i , t i+1 ) can be spawned as soon as x i j is computed, without waiting for the entire predictor-corrector mechanism to finish updating the SRDS solution for iteration i. This leads to an efficiently pipelined version of the algorithm, further speeding up the sampling process by a factor of two. See Figure 4 for an illustration of this pipelined algorithm with N = 16. Pipelining furthers the benefits of batched inference as the coarse solver is simply a DDIM-step with a larger time-step, so it can be batched with fine solves when applicable.Ignoring GPU overhead, the worst case wall-clock time of generating a sample through SRDS is no worse than that of generating through sequential sampling.Referring to the pipelined implementation of SRDS, it is easy to see that the fine solve F x i i , t i , t i+1 starts immediately after F x i-1 i , t i-1 , t i . Subsequently, from Proposition 1, it then follows that in the worst case, the final sample of SRDS x√ N √ N is computed at time √ N • √ N = N as desired.A formal argument can be found in Appendix A. It is worth noting, however, that this property of SRDS comes at the cost of much higher parallel compute compared to sequential sampling. Once again referring to the pipelined implementation of SRDS, it is easy to see that at any given time there is at most one model evaluation corresponding to a coarse solve, and up to √ N parallel model evaluations corresponding to the fine solves. It is worth contrasting this with the quadratically higher O(N ) memory requirement of the full ParaDiGMs algorithm in [30] , necessitating the use of sliding window tricks to reverse the process in a piece-wise fashion.It is finally worth noting that there is minimal inter-GPU communication in SRDS. In particular, at most one sample is passed between adjacent GPUs in each SRDS iteration. Once again, it is worth contrasting this with ParaDiGMs algorithm, which -by its use of parallel prefix sum operations to sync the solutions at each Picard iteration -incurs greater GPU communication overhead. See Appendix D for more discussion.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 
experiments on diffusion image generation: 
 To showcase the capabilities of the prescribed SRDS algorithm, we apply the diffusion sampler to pretrained diffusion models and present the difference in sample time and quality to ensure that applied convergence criteria do not reduce generation metrics. We start with pixel-based diffusion before expanding experiments applied to latent methods such as StableDiffusion-v2. Across the range of tasks, we show consistent speedups while maintaining quality of sample generation.In this section, we perform an extensive comparison with ParaDiGMs [30] as our baseline. Nonetheless, we provide a high level empirical comparison to our concurrent work ParaTAA [37] in Appendix E, where we demonstrate the superiority of SRDS.We start with pixel-space diffusion models. In particular, we test our SRDS algorithm and demonstrate capabilities in performing diffusion directly on the pixel space of 128x128 LSUN Church and Bedroom [41] , 64x64 Imagenet [5] , and 32x32 CIFAR [16] using pretrained diffusion models [29] , which all use N = 1024 length diffusion trajectories.We measure the convergence via l 1 norm in pixel space with values [0, 255]. We conservatively set τ = 0.1, meaning that convergence occurs when on average each pixel in the generation differs by only 0.1 after a refinement step (see Appendix F for an ablation on choice of τ ). Through our experiments, we quantitatively showcase how the SRDS algorithm can provide signficant speedups in generation without degrading model quality (as measured by FID score [9] on 5000 samples). As seen in Table 1 , SRDS remarkably converges in 4-6 iterations across all datasets; this corresponds to roughly 150 -200 effective serial steps (counting all model evaluations simultaneously performed in parallel as one evaluation), which is only 15 -20% of the serial steps required by a sequential solve Table 1 : Evaluating FID score (lower is better) of SRDS on various datasets using 5000 samples generated using a DDIM solver. Effective serial evals refers to the number of serial model evaluations taken by the pipelined SRDS algorithm (counting all model evaluations simultaneously performed in parallel as one evaluation) . Total evals refers to the total number of model evaluations. (N = 1024). We clarify that effective serial evaluations is referred to as Parallel Iters in [30] andSteps in [37] .While we are pretty conservative above in measuring convergence through distance in pixel space, we can also simply limit the number of SRDS iterations to 1 -2 and achieve further speedups without measurable degradation in sample quality. See Appendix F for more details.It is once again worth noting that this improved latency from parallelization comes at the cost of greater number of total model evaluations compared to a regular sequential solver. However, this tradeoff enables the diffusion models for many other use cases such as real-time image or music editing and trajectory planning in robotics. Moreover, we often empirically observe that SRDS provides reasonable predictions within a single Parareal iteration; here, the total number of model evaluations is only slightly larger than the serial approach (increasing from n to n + 2 √ n). Lastly, it is also worth noting that many users are often agnostic to inference time GPU compute costs as they are orders of magnitude lower than training compute costs anyway.Finally, we turn to latent diffusion models, in particular StableDiffusion-v2 [27] , where evaluations of the CLIP score over 1000 random samples show how SRDS maintains sample quality while improving the number of parallel iterations required per sample, with summary metrics presented in Table 2 . As the SRDS algorithm has small GPU overhead, we achieve measured wallclock time improvements with a Diffusers compatible implementation [38] . It is worth nothing that while we focus on DDIM here (as in the rest of the writing), we show speedups by readily incorporating other solvers into SRDS in Appendix C.For the test bed of latent diffusion models, we explore the convergence properties of our SRDS algorithm, with the average CLIP score plotted against the number of iterations in Figure 5 . For shorter sequences of length 25 (left), the corresponding SRDS sampler converges after approximately 3 iterations. However, for longer sequences of length 100 (right) the sampler has converged after a single SRDS iteration, showcasing the capabilities of our algorithm improves with longer trajectories.Next, we demonstrate the additional speedup that pipeline parallelism can bring to SRDS. We implement a slightly suboptimal version of pipelined SRDS for StableDiffusion and already observe significant speedups as seen in Table 3 ; with some more engineering effort 4 , we can further push towards extracting the full potential of pipelining. However, as this already beats the baselines, this sufficiently demonstrates the benefits of SRDS5 .Furthermore, for our main baseline ParaDiGMs, we also perform more extensive evaluation to evaluate both methods on equal hardware to more clearly demonstrate the benefits of SRDS. In Table 4 , we demonstrate that SRDS consistently beats ParaDiGMS on wallclock speedups. Though the authors of [30] uses a convergence threshold of 1e -3, we show that SRDS can provide impressive speedups even when compared to significantly relaxed ParaDiGMS thresholds of 1e -1.Finally, we finally provide a few sample generations on standard text prompts from DrawBench [28] in Figure 6 and additional examples of convergence similar to Figure 1 in the Appendix G. 
related work: 
 Recent literature on diffusion models has focused heavily on reducing the cost of sampling. Techniques such as higher order methods [13] and exponential integrators [42] have been proposed as strategies for reducing the model evaluations required in order to build high-quality samples without any additional training. When additional training is possible, other works have proposed distillation [29] , quantization [17] , and consistency [34] as alternate objectives to further speed up sample generation. For the purposes of this paper, we view these approaches as orthogonal, as the resultant models could be simulated with SRDS for potential benefits from combining methods.As discussed throughout the paper, this work is most cloesly related to the ParaDiGMS sampler method developed by Shih et al. [30] for parallel sampling of diffusion models. The two works take a similar approach by building off popular parallel-in-time integration methods in order to achieve lower latencies in simulation. In particular, ParaDiGMS builds on Picard iterations to converge on trajectories; we, however, build on Parareal method that performs multiresolution along the time axis for faster sampling. Parareal has been well-explored [18, 25, 20, 7] though with limited theoretical guarantees only spanning certain cases such as the heat equation and Navier-Stokes equation [8, 36] ; our work is the first to apply this algorithm to diffusion models. 
conclusion: 
 Limitations: Similar to previous iterations of parallel-in-time integration methods, SRDS make use of additional compute that can be used in parallel in exchange for faster latencies of sampling. That is to say, the total number of model evaluations in comparison to standard diffusion modelling increases in exchange for lower latencies. The additional compute may be reasonable in applications such as small-batch sampling where the additional cost can be hidden through better device utilization (e.g. sampling of a single image or trajectory in robotics). Alternatively, the responsiveness of real-time image editing may make parallel sampling an appealing option for cost-insensitive users.Future Directions: This work opens up a ton of interesting open questions for future exploration. Firstly, while fast convergence of parareal-style algorithms has only been proven for very limited settings, it will be extremely interesting to derive convergence guarantees specifically for the diffusion process. This has the potential to further our understanding of the nature of the ODE/SDE that governs the reverse process. Another natural direction is to explore the effects of employing higher levels of discretization and other multigrid methods such as F -cycles and W -cycles. As alluded to in Section 3.2, one could not only further study the optimal choice of second level of discretization, but also consider novel schedules that involve partitioning the diffusion trajectory into intervals of varying sizes. Lastly, it is worth highlighting that by serving a highly modular and interoperable framework, SRDS unlocks a vast landscape of interesting coarse/fine solver combinations. For instance, one can use a DDIM solver to perform the parallel refinement steps, while using a progressively distilled model [29] or consistency trajectory model [15] as the coarse solver in SRDS.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All relevant experimental details are thoroughly described in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Each evaluation of sample quality (using FID score) requires thousands of samples from the model; unfortunately, however, sampling from diffusion models is computationally very expensive. Due to computational constraints, we were unable to perform multiple rounds of our experiments to provide error bars for the FID scores.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper presents foundational research: a novel fast sampling algorithm for diffusion models in an attempt unlock capabilities for real-time interaction with diffusion models. While there could be potential positive and negative societal impacts of this work stemming from applications that use this (say, real-time generation of deepfakes), we believe it is not a direct consequence of this work.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper presents a sampling algorithm and hence poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly credits and attributes the creators of all the models and datasets used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The sample from SRDS converges to the output of the slow sequential solver in at most √ N refinement iterations.Proof. We show, by induction, that x p i converges in i iterations of SRDS for all i ∈ [0, N -1]. Further, x p i = F x p-1 i-1 , t i-1 , t i for all p ≥ i, implying that the final sample indeed corresponds to the desired sample from F. The base case of i = 0 follows trivially from the initialization (initial condition). To prove the second base case of i = 1, notice that x p 0 = x 0 for all p, implying that G x p-1 0 , t 0 , t 1 is constant for all p ≥ 1. Consequently,x p 1 = F x p-1 0 , t 0 , t 1 + G (x p 0 , t 0 , t 1 ) -G x p-1 0 , t 0 , t 1 , ∀p ≥ 1 = F (x 0 , t 0 , t 1 ) , ∀p ≥ 1 as desired.Assume by the induction hypothesis that for some fixed i,x p i = F x i-1 i-1 , t i-1 , t i , ∀p ≥ i.Then, ∀p ≥ i, we have thatx p+1 i+1 = F (x p i , t i , t i+1 ) + G x p+1 i , t i , t i+1 -G (x p i , t i , t i+1 ) = F x i i , t i , t i+1 + G F x i-1 i-1 , t i-1 , t i , t i , t i+1 -G F x i-1 i-1 , t i-1 , t i , t i , t i+1 = F x i i , t i , t i+1 as desired.Proposition 2. [Worst-Case Sampling Latency] Ignoring GPU overhead, the worst case wallclock time of generating a single sample through SRDS is no worse than that of generating a single sample through sequential sampling.Proof. Consider the unit of time to be the time taken for one denoising step (or one model evaluation). Referring to the pipelined implementation of SRDS, it is easy to see via a straightforward inductive argument that the √ N -step fine solve F x p √ N , t i , t i+1 ends at timeN √ N p + √ N -p. FromProposition 1, it then follows that in the worst case, the final sample of SRDS x√ N √ N is computed at time N √ N √ N + √ N - √ N = N as desired.Proposition 3.[Memory] SRDS requires memory corresponding to O( √ N ) denoising model evaluations.Proof. In the pipelined implementation of SRDS, it is easy to see that at any given timestep there is at most one model evaluation corresponding to a coarse solve. Further, the number of parallel model evaluations corresponding to the fine solves is upper bounded by the coarse discretization (or the number of "blocks"), which is √ N . Thus, the memory used by SRDS corresponds to at most√ N + 1 = O( √ N ) model evaluations.The choice of resolution for the coarse solve is not arbitrary. For practical implementations, since we use the same denoiser (say, DDIM) for both the coarse and fine solves, we choose √ N as an optimal choice in the runtime sense (assuming constant number of iterations till convergence 6 ). At a high level, this choice stems from the fact that we want to balance out the time the it takes to run all the fine solves in parallel and the time it takes perform one set of sequential predictor-corrector steps through the trajectory. Proof. Let k denote the number of SRDS iterations until convergence, let τ denote the cost of one denoising step or model evaluation, and let 1 < B < N denote the "block-size": that is, the second scale of discretization. For the 1-step coarse solve, each SRDS iteration incurs a runtime cost of1 • ⌈ N B ⌉ • τ .For the B-step fine solves, as each of the ⌈ N B ⌉ fine solves are independently executed in parallel, each SRDS iteration incurs a runtime cost of B • 1 • τ . The baseline runtime for sequentially sampling from the diffusion model is N • τ . Thus, the runtime speedup (ignoring parallelization overhead) isN •τ k(⌈ N B ⌉•τ +B•τ ) = N k(⌈ N B ⌉+B). For a fixed value of k, it is easy to see that this quantity is concave in B and is maximized by choosing B ≈ √ N .It is worth noting, however, that if we use solvers of different latencies for the coarse and fine steps, a modifed analysis is required to incorporate differences in denoising step times for the two solvers. Consequently,N might no longer be the optimal choice of coarse resolution.It is worth emphasizing again that SRDS provides an orthogonal improvement when compared to the other lines of research on accelerating diffusion model sampling. In particular, while the main experiments (and writing) were focused on DDIM, SRDS is compatible with the other solvers and they can be readily incorporated into SRDS to speed up diffusion sampling. For example, below we show that SRDS is directly compatible with other solvers such as DDPM (often requiring more steps than DDIM) and DPMSolver (often requiring fewer steps than DDIM) and can efficiently accelerate sampling in both cases. We demonstrate this on StableDiffusion in Table 5 . We also highlight that the Diffuser-compatible implementation requires only minor modification to the arguments of the solver, suggesting that SRDS will also be easy to extend out-of-the-box to other methods that the community develops. For a T step denoising process, ParaDiGMS needs to perform T model evaluations in parallel with subsequent computations needing information about all previous evaluations, while SRDS only requires √ T parallel evaluations (which fits comfortably in GPU memory) and requires much lesser communication between GPUs. While the prohibitively large memory requirement can be combated with a sliding window method, the significantly larger communication overhead remains because at every step of Paradigms, an AllReduce over all devices must be performed in order to calculate updates to the sliding window. (For instance, even when ParaDiGMS reduces Eff. Serial Steps by 20x, the obtained speedup is only 3.4x). This is in contrast to the independent fine-solves in parareal that only need to transfer information for the coarse solve.Below in Table 6 , we demonstrate how the minimal memory and communication overhead of SRDS shines through as we are able to achieve better device utilization as we increase the number of available GPUs. The following experiment was performed on 40GB A100s and used a generous 1e-2 threshold for ParaDiGMS. We demonstrate the superiority of SRDS to baselines ParaDiGMS [30] and ParaTAA [37] . Here, we demonstrate the high-level superiority of SRDS solely by using the results published by the authors in [30] (Table 5 ) and [37] (Table 1 ).In the table 7 below, we show that SRDS offers better wall-clock speedups (over sequential) in sample generation time for StableDiffusion when compared to [30] and [37] . We clarify that the reported speedup for each method is with respect to sequential solve on the same machine that the corresponding parallel method was evaluated. Our results are particularly impressive given that the authors of [30] used 8x 80GB A100s for the evaluation and the authors of [37] used 8x 80GB A800 for the same, while we (SRDS) only used 4x 40GB A100 for the evaluation due to computational constraints. (For interpretation purposes, recall that a sequential solve is not compute/memory bound and doesn't benefit significantly from additional GPU compute, whereas the parallel methods certainly do!) We would also like to highlight the superiority of SRDS over the baselines in the regime of small number of denoising steps (25) as being particularly impactful. In this section, we further evaluate the convergence properties of SRDS. First, in Table 8 , we analyze how the sample quality varies as we vary the tolerance threshold τ . Next, we analyze how the FID score of generated samples varies as a function of the number of SRDS iterations. As shown in Figure 7 , we observe rapid convergence of the FID score to the value obtained by sequential sampling (12.8) within a few SRDS iterations. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is a public link to our repository containing complete code needed to reproduce the main experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Code for our paper can be found at https://github.com/nikilrselvam/srds.It is not required for N to be a perfect square. We can extend the described techniques in a straightforward manner to perform a ⌈ √ N ⌉-discretization instead, with the last interval in the partition having a smaller size. The special choice of√N as the discretization level is further explained in Appendix B.This refers to a solver that takes √ N DDIM steps (or model evaluations) to solve the initial value problem.The suboptimality of the implementation arises from the use of a single device to coordinate the pipeline parallelism and device transfers (arising as an artifact from torch.multiprocessing). A more complete implementation would instead use ring-like communication between devices rather than wait on the coordinator.5 We note that the number of denoising steps in the experiments is chosen to be perfect squares merely for convenience. SRDS is general and applies to any number of denoising steps.It is possible that a different choice of B might actually be optimal by enabling better flow of information down the computation graph and thereby resulting in lower number of iterations till convergence k. However, in our experiments (as also noted in Section 4), we observe that SRDS often converges for small k ∈ [2, 4], validating our empirical choice. 


Decision: This paper addresses the computational complexity of Diffusion Models, and seek to overpower the slow sequential evaluations which typically drive  diffusion model generation.  Specifically they propose Self-Refining Diffusion Samplers (SRDS) to improve the speed at a cost of parallel compute while preserving the sample quality.  This is in contrast to prior attempts at the problem toying with discretization. Convergence results on an ODE have been shown and comparative experiments were carried out. 
 While all 4  reviews out of 5 rated the paper as a "weak accept", and one as reject,  several critical comments were made, although most were successfully rebutted by the authors to the reviewers' satisfaction.  One significant one pertained to the proposed Parallel computational  non-novelty  with  respect to some specific recent references cited refs [1]& [2] by reviewer kHBd.
The authors have responded to the reviewer's satisfaction that in contrast to the multigrid/multiresolution flavor, the prior cited refs.  used classical  fixed point iteration/updates. 
The "Reject" review also raised several questions and some due to confusion and others were in the end rebutted to the reviewer's agreement who nevertheless kept their reject intact. Judging by the comments, I believe that the reviewer is a little confused, 
I think sufficient experimental evidence was also provided as attested by the other 4 reviewers.

.

Reviews: review: paper_summary: paper_summary: The paper presents a new approach to speed up (improve latency)  the generation of samples from diffusion models. The approach is orthogonal to many other approaches present in the literature for the same task. It leverages Parareal algorithm for the task by getting a quick coarse approximate of the sample and then refining it iteratively in parallel, thus reducing the latency while maintaining sample quality. The authors present results for pre-trained pixel space diffusion models  and Stable Diffusion and find upto 2x speed ups for the latter.
strength_weakness: strength_weakness: The main weakness of the paper is lack of strong and diverse results. The authors only test one  diffusion model task. Furthermore the best speed up is only 2.3x, while another experiment results in 0.7x speed up which is concerning.
The authors argue on theoretical grounds that this approach can be combined with other approaches to reduce latency of diffusion models. This is fair, but they do not show any empirical results around it. 
Finally, the approach relies completely on leveraging parallel refinement and the actual number of model evaluations are much larger (often by a factor of 3x) which can limit the applicability in some cases.
questions: questions: - The authors claim that the worst case sampling latency is no worse than generating a sample through sequential sampling. Yet one of their results (last on in Table 2) has speed up of 0.7x. Is this all due to GPU overhead?

- Is is possible to present empirical results with pipeline parallelism? It seems like the authors expect 2x gains with it, but I assume it will require some computational overhead and based on the previous question, the degradation can be non-trivial.

- While I understand the claim regarding this approach being orthogonal to other works reducing, showing some experiments around the same will make for a stronger paper.
limitations: limitations: There are a few limitations discussed towards the end of the paper.
paper_summary: paper_summary: This work proposes SRDS, a sampler for diffusion models that applies the Parareal algorithm, to reduce the overall sampling latency by introducing extra but parallelizable network evaluations compared to the fully sequential fashion. With higher device utilization or device parallelism through batched inference and pipelining, SRDS successfully reduces the overall sampling latency of existing diffusion models without sacrificing the sample quality.
strength_weakness: strength_weakness: - As the authors addressed in the limitation part, SRDS relies on parallelizing extra computations in exchange for a reduction in total latency, which may not be applicable to some scenario that performs full batch sequential sampling. However, I believe SRDS could still work in a fairly broad range of cases. 
- The important ParaDiGMs baseline is not compared in the experiment part.

### Minor
- The meaning of the abbreviation "IVP" in line `1:` of Algorithm.1  is unclear.
questions: questions: - In the current SRDS pipeline, the intermediate results of the fine-grained solver are only used once. Do you have any thoughts on utilizing them in subsequent steps for better sample quality or faster convergence?
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: Inspired by the parallel-in-time ODE integration literature, especially Parareal method, this paper introduces SRDS as a fast sampler for diffusion models enabling efficient parallelization. Experimental results demonstrate that the proposed SRDS reduces the number of steps required to synthesize samples.
strength_weakness: strength_weakness: My concerns are mostly about the limited experimental results:

1. This paper lacks of clear discussions about the differences between the proposed method and the previous work “23NeurIPS-Parallel sampling of diffusion models.”, and sufficient experimental evidence to demonstrate the effectiveness of the proposed method. Specifically, there is no quantitative comparison with the most related method ParaDiGMs. It seems that the existing method ParaDiGMs can provide better speedup according to their results.
2. This paper lacks of a sufficient comparison to the existing fast samplers (such as Heun, DEIS, DPM-Solver). In practice, we can already synthesize samples with those advanced samplers with only around 10 steps. While this paper only presents experiments on several hundreds or one thousand steps at Table 1, which makes the method proposed in this paper unappealing.   
3. It is claimed in Lines 38-40 and 196-198 that SRDS outperforms ParaDiGMs in terms of memory requirement. However, there is also no quantitative support from experimental results.
4. It is claimed in Line 62-64 that SRDS is compatible with off-the-shelf solvers. However, the speedup is only shown on DDIM. The speedup offered by SRDS is only explicitly shown in Table 2, making it hard to fully assess the effectiveness of the proposed method.
5. There is no quantitative comparison between SRDS and pipelined SRDS.
questions: questions: 1. Could the pipelined SRDS be easily deployed in practice?
2. Why the SRDS Iters shown in Table 1 are not integers?
3. Why the first and second rows of Figure 6 are almost the same?
4. Minor points:

(1) In line 10 of Algorithm 1, $x_{i-1}^p$ is undefined for p == 1.

(2) It is redundant to use Figure 2, Figure 3b and Algorithm 1 to describe the same thing in the main text. They together occupy a whole page. Adding more experiments to support the effectiveness of the proposed method is more valuable.
limitations: limitations: Please refer to the weakness and question sections above. This manuscript needs an overhaul to make it meet the bar of a top conference.
paper_summary: paper_summary: This paper proposes Self-Refining Diffusion Samplers (SRDS), which draws inspiration from the Parareal algorithm, aims to solve the reverse process accurately without retraining models, balance the tradeoff between sample quality and sampling speed. lower latency for requiring fewer sampling steps to reach convergence. The experiment section shows that the proposed algorithm speeds up the convergence without degrading sample quality in image generation tasks.
strength_weakness: strength_weakness: 1. In related work, the authors mentioned that ParaDiGMS proposed by Shih et al [1] is another parallel-in-time integration method, and this paper also works on image generation. In this case, I believe their results could be baselines that this paper should compare against.

[1] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
questions: questions: 1. For the experiment section, are the numbers intentionally matched in ``FID Score`` in Table 1 and ``CLIP Score`` in Table 2? If that's true, it would be also interesting to see how the performance changes when you have different SRDS Iters and their corresponding Total Evals.
limitations: limitations: This paper discussed the limitations and pointed out the potential positive or negative social impacts will not be the direct consequence of this work in the checklist guidelines.
paper_summary: paper_summary: This paper proposes a new method for parallelizing the sampling process of diffusion models, offering a novel way to trade compute for speed. In particular, it relies on the parareal iteration in this setting, where the main idea is to split the solving timesteps into multiple groups and then update them in parallel. To ensure that the output from these timestep groups matches the exact solution, parareal performs predictor-corrector steps after each group is updated. Experiments demonstrate the efficacy of applying parareal iteration in speeding up the sampling process.
strength_weakness: strength_weakness: 1. I think the author misses a rigorous discussion comparing the parareal algorithm with ParaDiGMS [1] and also completely ignores a more recent reference [2]. It is clear that all these works, including parareal, focus on parallelizing the sampling process via certain forms of fixed-point iteration, although the computing graphs of these algorithms differ. The only discussion in the current manuscript is that ParaDiGMS is highly memory-intensive and must use a sliding window. While this statement is certainly true, I believe it is not a disadvantage of ParaDiGMS and does not imply that ParaDiGMS is inferior to parareal. I do not see why using a sliding window should be considered a disadvantage.

2. As far as I can see, the Eff. Serial Evals in Table 1 and Table 2 should correspond to the Parallel Iters in [1] and Steps in [2], am I correct? Because Eff. Serial Evals essentially represent the number of batch evaluations on the models. If this is the case, based on the current numbers in Table 1, it seems quite inferior to the results in [1] and [2]. Of course, I understand that the batch size may be different in this work, but I still wish the author would provide a detailed discussion on this point.

3. Proposition 1 in the manuscript is somewhat trivial and exists in many existing works. Note that each refinement consists of $\sqrt n$ sequential evaluation, so the parareal will also require at most $\sqrt n \times \sqrt n=n$ sequential evaluation on the models to converge. I think the author should mention that this result is fundamentally equivalent to Proposition 1 in [1], and a more general result is found in Theorem 3.6 in [2]. The key point is that when performing fixed-point iteration for an $n$-order triangular system, it will certainly converge within $n$ steps, and this bound can not be further improved without extra assumption.



[1] Shih, Andy, et al. "Parallel sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[2] Tang, Zhiwei, et al. "Accelerating parallel sampling of diffusion models." Forty-first International Conference on Machine Learning. 2024.
questions: questions: 1. In my understanding, the technique in Parareal have a different spirit in parallelization with [1][2]. For example, consider that we wish to do n-step sampling with $\sqrt n$ batchsize. Given a initial sequence $[x_1,...,x_n]$, parareal first group them into $g_1=[x_1,...,x_{\sqrt n}],...,g_{\sqrt n}=[x_{n-\sqrt n},x_n]$. The parallelism is that to update $g_1$, ..., $g_n$ in parallel, while the update of each $g_i$ consists of $\sqrt n$ sequential solving. On the other hand, [1][2] consider update $[x_{i},...,x_{i+\sqrt n}]$ in parallel until convergence. It is not clear to me which one is better from the theoretical aspects, as they have the same worst-case convergence bound. I suspect that it should depends on the form of ODE and also the batchsize? I wonder if the authors could provide some thought and discussion on this point?

2. Do you think parareal can be applied to the SDE sampler as [1] and [2] did?


[1] Shih, Andy, et al. "Parallel sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[2] Tang, Zhiwei, et al. "Accelerating parallel sampling of diffusion models." Forty-first International Conference on Machine Learning. 2024.
limitations: limitations: See my comments on weaknesses and questions above. This work lacks a correct and thorough discussion comparing it to existing works.

Comments: title: Response to rebuttal
Thanks for your rebuttal
Thanks for the reviews!
comment: Thank you for your response and for raising the score! We are glad that your concerns have been addressed.
Thank you for your response and for raising your score! We are glad to hear that your concerns have been addressed.
Thank you for your response! We are pleased to hear that your concerns have been addressed.
Thank you for your response! We are glad to hear that your concerns have been addressed.
I would like to thank the authors for the rebuttal, especially the additional results. I will keep my current recommendation unchanged.
I have read through the responses and decide to increase the score from 5 to 6.
I thank the authors for their responses and new experiments. I am satisfied with their rebuttal and will maintain my score.
After careful consideration, I decided to raise my score to 6.
We sincerely hope that the additional experimental results/clarifications in the rebuttal clearly demonstrate/validate the usefulness of our proposed method (SRDS), thereby sufficiently answering all the reviewers’ concerns and significantly improving the strength of the paper. We once again thank the reviewers for their time and feedback, and we are happy to answer any further questions that may remain in the discussion period!

---

Author Rebuttal References:
- [1] Shih, Andy, et al. "Parallel sampling of diffusion models." Advances in Neural Information Processing Systems 36 (2023).
- [2] Tang, Zhiwei, et al. "Accelerating parallel sampling of diffusion models." Forty-first International Conference on Machine Learning. 2024.