SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 1, we present results on DDPM, DDIM, DPMSolver, and their parallel variants (ParaDDPM, ParaDDIM, ParaDPMSolver) when combined with ParaDiGMS. We plot the model evaluations (number of calls to the diffusion model  $p_{\theta}$ ), the task reward, and the sampling speed reported in time per episode. Although parallelization increases the total number of necessary model evaluations, the sampling speed is more closely tied to the number of parallel iterations, which is much lower. We see that ParaDDPM gives a speedup of  $3.7\mathrm{x}$ , ParaDDIM gives a speedup of  $1.6\mathrm{x}$ , and ParaDPMSolver gives a speedup of  $1.8\mathrm{x}$ , without a decrease in task reward. Table 2 presents similar findings on the PushT task, where we see speedups on all three methods with up to  $3.9\mathrm{x}$  speedup on ParaDDPM.
Table 1: Robosuite Square with ParaDiGMS using a tolerance of  $\tau = 0.1$  and a batch window size of 20 on a single A40 GPU. Reward is computed using an average of 200 evaluation episodes, with sampling speed measured as time to generate  $400 / 8 = 50$  samples.
Robosuite Square	Model Evals	Sequential Reward	Time per Episode	Model Evals	ParaDiGMS Parallel Iters	Reward	Time per Episode	Speedup
DDPM	100	0.85 ± 0.03	37.0s	392	25	0.85 ± 0.03	10.0s	3.7x
DDIM	15	0.83 ± 0.03	5.72s	47	7	0.85 ± 0.03	3.58s	1.6x
DPMSolver	15	0.85 ± 0.03	5.80s	41	6	0.83 ± 0.03	3.28s	1.8x
Table 2: PushT task with ParaDiGMS using a tolerance of  $\tau = 0.1$  and a batch window size of 20 on a single A40 GPU. Reward is computed using an average of 200 evaluation episodes, with sampling speed measured as time to generate  $\lceil 300 / 8\rceil = 38$  samples.
PushT	Model Evals	Sequential Reward	Time per Episode	Model Evals	ParaDiGMS Parallel Iters	Reward	Time per Episode	Speedup
DDPM	100	0.81 ± 0.03	32.3s	386	24	0.83 ± 0.03	8.33s	3.9x
DDIM	15	0.78 ± 0.03	4.22s	46	7	0.77 ± 0.03	2.54s	1.7x
DPMSolver	15	0.79 ± 0.03	4.22s	40	6	0.79 ± 0.03	2.15s	2.0x
Table 3: FrankaKitchen with ParaDiGMS using a tolerance of  $\tau = 0.1$  and a batch window size of 20 on a single A40 GPU. Reward is computed using an average of 200 evaluation episodes, with sampling speed measured as time to generate  $1200 / 8 = 150$  samples.
Franka Kitchen	Model Evals	Sequential Reward	Time per Episode	Model Evals	ParaDiGMS Parallel Iters	Reward	Time per Episode	Speedup
DDPM	100	0.85 ± 0.03	112s	390	25	0.84 ± 0.03	33.3s	3.4x
DDIM	15	0.80 ± 0.03	16.9s	47	7	0.80 ± 0.03	9.45s	1.8x
DPMSolver	15	0.79 ± 0.03	17.4s	41	6	0.80 ± 0.03	8.89s	2.0x
The final robotics task we study is FrankaKitchen, a harder task with predicted action sequences of dimension 112 and an episode length of 1200. In Table 3 we notice some decline in performance when sampling with a reduced number of steps using DDIM and DPMSolver. On the other hand, ParaDDPM is able to maintain a high task reward. Similar to before, ParaDiGMS consistently achieves a speedup across all 3 sampling methods, giving a speedup of  $3.4\mathrm{x}$  with ParaDDPM,  $1.8\mathrm{x}$  with ParaDDIM, and  $2.0\mathrm{x}$  with ParaDPMSolver. These improvements translate to a significant decrease in the time it takes to roll out an episode in the Kitchen task from 112s to 33.3s.
4.2 Diffusion image generation
Next, we apply parallel sampling to diffusion-based image generation models, both for latent-space and pixel-space models. For latent-space models, we test out StableDiffusion-v2 $^3$  [26, 24], which generates 768x768 images using a diffusion model on a 4x96x96 latent space. For pixel-space models, we study pretrained models on LSUN Church $^4$ /Bedroom $^5$  from Huggingface [9, 35], which run a diffusion model directly over the 3x256x256 pixel space.
4.2.1 Latent-space diffusion models
Even with the larger image models, there is no issue fitting a batch window size of 20 on a single GPU for parallelization. However, the larger model requires more compute bandwidth, so the parallel efficiency quickly plateaus as the batch window size increases, as the single GPU becomes bottlenecked by floating-point operations per second (FLOPS). Therefore, for image models, we leverage multiple GPUs to increase FLOPS and improve the wall-clock sampling speed.
In Fig. 4 we examine the net speedup of ParaDDPM relative to DDPM on StableDiffusion-v2 using 1000-step diffusion on A100 GPUs. The net speedup is determined by the interplay between algorithm inefficiency and hardware efficiency. Algorithm inefficiency refers to the relative number of model evaluations of ParaDDPM compared to DDPM, which arises from the parallel algorithm taking multiple iterations until convergence. We see in Fig. 4e that as the batch window size grows, ParaDDPM can require 2-3x more model evaluations. On the other hand, hardware efficiency refers to the relative empirical speedup of performing a batch of model evaluations. For example, in Fig. 4e we see that evaluating a batch window size of 80 on 4 GPUs (20 per GPU) is roughly 5x faster than performing 80 model evaluations sequentially. In Fig. 4f, we divide the hardware efficiency by the
<sup>3</sup>https://huggingface.co/stabilityai/stable-diffusion-2
4https://huggingface.co/google/ddpm-ema-church-256
<sup>5</sup>https://huggingface.co/google/ddpm-ema-bedroom-256
(a) "a beautiful castle, matte painting"
(b) "a batter swings at a pitch during a baseball game"
(c) "several sail boats in the water at night"
(d) "a grey suitcase sits in front of a couch"
Efficiency of ParaDDPM on StableDiffusion-v2, A100
(e) Hardware efficiency overtakes algorithm inefficiency as number of GPUs increase.
(f) Over 3x net wall-clock speedup for 1000-step ParaD-DPM
Figure 4: StableDiffusion-v2 generating text-conditioned  $768 \times 768$  images by running ParaDDPM over a  $4 \times 96 \times 96$  latent space for 1000 steps, on A100 GPUs. In Fig. 4e algorithm inefficiency in gray denotes the relative number of model evaluations required as the parallel batch window size increases. The colored lines denote the hardware efficiency provided by the multi-GPUs. As the batch window size increases, the hardware efficiency overtakes the algorithm inefficiency. In Fig. 4f we normalize the algorithm inefficiency to 1, to show the net wall-clock speedup of parallel sampling.
algorithm inefficiency to obtain the net relative speedup of ParaDDPM over DDPM. We observe over 3x speedup by using a batch window size of 80 spread across 8 A100s. Finally, in Table 4 we verify that ParaDiGMS increases sampling speed for ParaDDPM, ParaDDIM, and ParaDPMSolver without degradation in sample quality as measured by CLIP score [8] on ViT-g-14 [23, 11].
One important consideration is that the algorithm inefficiency is agnostic to the choice of GPU. Therefore, as the parallel efficiency of GPUs in the future improve for large batch window sizes, we will see an even larger gap between hardware efficiency and algorithm inefficiency. With enough hardware efficiency, the wall-clock time of sampling will be limited only by the number of parallel iterations, leading to much larger net speedup. For example, observe that in Table 4 the number of parallel iterations of ParaDDPM is  $20\mathrm{x}$  smaller than the number of sequential steps.
4.2.2 Pixel-space diffusion models
Next, we test out ParaDiGMS on pretrained diffusion models on LSUN Church and Bedroom, which perform diffusion directly on a 3x256x256 pixel space. In Fig. 5 in Appendix D, we plot the net speedup of 1000-step ParaDDPM by dividing the hardware efficiency by the algorithm inefficiency. We observe a similar trend of over 3x speedup when using multiple GPUs. Finally, we verify in Table 5 that ParaDiGMS maintains the same sample quality as the baseline methods as measured by FID score on 5000 samples of LSUN Church<sup>6</sup>.
$^{6}$ DPMSolver is not yet integrated with the LSUN model in the Diffusers library, so we omit its comparison.
Table 4: Evaluating CLIP score of ParaDiGMS on StableDiffusion-v2 over 1000 random samples from the COCO2017 captions dataset, with classifier guidance  $w = 7.5$ . CLIP score is evaluated on ViT-g-14, and sample speed is computed on A100 GPUs.
Stable Diffusion-v2	Sequential	ParaDiGMS	Speedup
Model Evalu	CLIP Score	Time per Sample	Tol.τ	Model Evalu	Parallel Iters	CLIP Score	Time per Sample
DDPM	1000	32.1	50.0s	1e-1	2504	50	32.1	14.6s	3.4x
DPMSolver	200	31.7	10.3s	1e-1	422	15	31.7	2.6s	4.0x
DDIM	200	31.9	10.3s	1e-1	432	15	31.9	2.6s	4.0x
DDIM	100	31.9	5.3s	5e-2	229	19	31.9	2.0s	2.7x
DDIM	50	31.9	2.6s	5e-2	91	17	31.9	1.1s	2.4x
DDIM	25	31.7	1.3s	1e-2	93	17	31.7	1.0s	1.3x
We highlight that 500-step DDIM gives a noticeably worse FID score than 1000-step DDPM, whereas using ParaDDPM allows us to maintain the same sample quality as DDPM while accelerating sampling (to be two times faster than 500-step DDIM). The ability to generate an image without quality degradation in 6.1s as opposed to 24.0s can significantly increase the viability of interactive image generation for many applications.
Table 5: Evaluating FID score (lower is better) of ParaDiGMS on LSUN Church using 5000 samples. Sample speed is computed on A100 GPUs. We use tolerance 5e-1 for DDPM and 1e-3 for DDIM.
LSUN Church	Sequential	ParaDiGMS	Speedup
Model Evalu	FID Score	Time per Sample	Model Evalu	Parallel Iters	FID Score	Time per Sample
DDPM	1000	12.8	24.0s	2583	45	12.9	6.1s	3.9x
DDIM	500	15.5	12.2s	1375	41	15.3	3.7s	3.3x
DDIM	100	15.1	2.5s	373	23	14.8	1.0s	2.5x
DDIM	50	15.3	1.2s	168	17	15.7	0.7s	1.7x
DDIM	25	15.6	0.6s	70	15	15.9	0.6s	1.0x
4.3 Related work
Apart from DDIM [29] and DPMSolver [18], there are a number of other fast sampling techniques for pretrained diffusion models [14, 17, 19, 40]. Most of these techniques are based on higher-order ODE-solving and should also be compatible with parallelization using ParaDiGMS. Other lines of work focus on distilling a few-step model [25, 21, 30] or learning a sampler [37], but these methods are more restrictive as they require additional training.
Besides diffusion models, many works have studied accelerating the sampling of autoregressive models using various approaches such as parallelization [33, 31], distillation [13], quantization [6], or rejection sampling [3, 16]. Of particular note is [31], which samples from autoregressive models by iterating on a system of equations until convergence. While these methods are not immediately applicable to diffusion models due to the differences in computational structure and inference efficiency of autoregressive models, there may be potential for further investigation.
Parallelization techniques similar to Picard iteration have been explored in theoretical works for sampling from log-concave [27] and determinantal distributions [1]. Our work is the first application of parallel sampling on diffusion models, enabling a new axis of trading compute for speed.
Limitations Since our parallelization procedure requires iterating until convergence, the total number of model evaluations increases relative to sequential samplers. Therefore, our method is not suitable for users with limited compute who wish to maximize sample throughput. Nevertheless, sample latency is often the more important metric. Trading compute for speed with ParaDiGMS
makes sense for many practical applications such as generating images interactively, executing robotic policies in real-time, or serving users who are insensitive to the cost of compute.
Our method is also an approximation to the sequential samplers since we iterate until the errors fall below some tolerance. However, we find that using ParaDiGMS with the reported tolerances results in no measurable degradations of sample quality in practice across a range of tasks and metrics. In fact, on more difficult metrics such as FID score on LSUN Church, ParaDDPM gives both higher sample quality and faster sampling speed than 500-step DDIM.
Discussion We present ParaDiGMS, the first accelerated sampling technique for diffusion models that enables the trade of compute for speed. ParaDiGMS improves sampling speed by using the method of Picard iterations, which computes multiple denoising steps in parallel through iterative refinement. Remarkably, ParaDiGMS is compatible with existing sequential sampling techniques like DDIM and DPMSolver, opening up an orthogonal axis for optimizing the sampling speed of diffusion models. Our experiments demonstrate that ParaDiGMS gives around 2-4x speedup over existing sampling methods across a range of robotics and image generation models, without sacrificing sample quality. As GPUs improve, the relative speedup of ParaDiGMS will also increase, paving an exciting avenue of trading compute for speed that will enhance diffusion models for many applications.
This research was supported in part by NSF (#1651565, #2045354, #2125511), ARO (W911NF-21-1-0125), ONR (N00014-23-1-2159, N00014-22-1-2293), CZ Biohub, HAI.
[1] Nima Anari, Yizhi Huang, Tianyu Liu, Thuy-Duong Vuong, Brian Xu, and Katherine Yu. Parallel discrete sampling via continuous walks. In STOC '23: 55th Annual ACM SIGACT Symposium on Theory of Computing. ACM, 2023.
[2] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
[4] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[5] Earl A Coddington, Norman Levinson, and T Teichmann. Theory of ordinary differential equations. Physics Today, 9(2):18, 1956.
[6] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8(): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
[7] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019.
[8] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[10] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[27] Ruoqi Shen and Yin Tat Lee. The randomized midpoint method for log-concave sampling. Advances in Neural Information Processing Systems, 32, 2019.
[28] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[29] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[30] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[31] Yang Song, Chenlin Meng, Renjie Liao, and Stefano Ermon. Accelerating feedforward computation via parallel nonlinear equation solving. In International Conference on Machine Learning, pages 9791-9800. PMLR, 2021.
[32] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[33] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018.
[34] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287-11302, 2021.
[35] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
[36] Hsiang-Chun Wang, Shang-Fu Chen, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. arXiv preprint arXiv:2302.13335, 2023.
[37] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
[38] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.
[39] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[40] Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. 2023.
[41] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, 2020.
A Proof of exact convergence
Proof of Proposition 1. Assume by induction that  $\pmb{x}_t^{k + 1} = \pmb{x}_t^\star$ . Then
$$
\begin{array}{l} \boldsymbol {x} _ {t + 1} ^ {k + 1} = \boldsymbol {x} _ {0} ^ {k} + \frac {1}{T} \sum_ {i = 0} ^ {t} s (\boldsymbol {x} _ {i} ^ {k}, \frac {i}{T}) \\ = \left(\boldsymbol {x} _ {0} ^ {k} + \frac {1}{T} \sum_ {i = 0} ^ {t - 1} s \left(\boldsymbol {x} _ {i} ^ {k}, \frac {i}{T}\right)\right) + \frac {1}{T} s \left(\boldsymbol {x} _ {t} ^ {k}, \frac {t}{T}\right) \\ = \boldsymbol {x} _ {t} ^ {k + 1} + \frac {1}{T} s (\boldsymbol {x} _ {t} ^ {k}, \frac {t}{T}) \\ = \boldsymbol {x} _ {t} ^ {k + 1} + \frac {1}{T} s \left(h _ {t - 1} \left(\dots h _ {2} \left(h _ {1} \left(\boldsymbol {x} _ {0}\right)\right)\right), \frac {t}{T}\right) \\ = \boldsymbol {x} _ {t} ^ {\star} + \frac {1}{T} s \left(\boldsymbol {x} _ {t} ^ {\star}, \frac {t}{T}\right) = \boldsymbol {x} _ {t + 1} ^ {\star}. \\ \end{array}
$$
B Total variation analysis
Proof of Proposition 2. A linear convergence rate with factor  $\geq 2$  ensures our error from the solution  $\pmb{x}_t^\star$  given by sequential sampling at each timestep  $t$  is bounded by the chosen tolerance:
$$
\| \boldsymbol {x} _ {t} ^ {K} - \boldsymbol {x} _ {t} ^ {\star} \| ^ {2} \leq \lim  _ {n \rightarrow \infty} \sum_ {j = K + 1} ^ {n} \| \boldsymbol {x} _ {t} ^ {j} - \boldsymbol {x} _ {t} ^ {j - 1} \| ^ {2} \leq \lim  _ {n \rightarrow \infty} \sum_ {j = K + 1} ^ {n} \frac {1}{2 ^ {j - K}} \| \boldsymbol {x} _ {t} ^ {K} - \boldsymbol {x} _ {t} ^ {K - 1} \| ^ {2} \leq \| \boldsymbol {x} _ {t} ^ {K} - \boldsymbol {x} _ {t} ^ {K - 1} \| ^ {2}.
$$
Then, for each timestep  $t$ , since the inference model samples from a Gaussian with variance  $\sigma_t^2$ , we can bound the total variation distance:
$$
\begin{array}{l} d _ {\mathrm {T V}} \left(\mathcal {N} \left(\boldsymbol {x} _ {t} ^ {K}, \sigma_ {t} ^ {2} \boldsymbol {I}\right), \mathcal {N} \left(\boldsymbol {x} _ {t} ^ {\star}, \sigma_ {t} ^ {2} \boldsymbol {I}\right)\right) \leq \sqrt {\frac {1}{2} \mathcal {D} _ {\mathrm {K L}} \left(\mathcal {N} \left(\boldsymbol {x} _ {t} ^ {K} , \sigma_ {t} ^ {2} \boldsymbol {I}\right) \parallel \mathcal {N} \left(\boldsymbol {x} _ {t} ^ {\star} , \sigma_ {t} ^ {2} \boldsymbol {I}\right)\right)} \\ = \sqrt {\frac {\| \pmb {x} _ {t} ^ {K} - \pmb {x} _ {t} ^ {\star} \| ^ {2}}{4 \sigma_ {t} ^ {2}}} \leq \sqrt {\frac {\| \pmb {x} _ {t} ^ {K} - \pmb {x} _ {t} ^ {K - 1} \| ^ {2}}{4 \sigma_ {t} ^ {2}}} \leq \frac {\epsilon}{T}. \\ \end{array}
$$
Finally, we make use of the data processing inequality, that  $d_{\mathrm{TV}}(f(P), f(Q)) \leq d_{\mathrm{TV}}(P, Q)$ , so the total variation distance  $d_t$  between the sample and model distribution after  $t$  timesteps does not increase when transformed by  $p_\theta$ . Then by the triangle inequality, we get that  $d_t \leq d_{t-1} + \epsilon / T$  giving a total variation distance  $d_T$  of at most  $T \epsilon / T = \epsilon$  for the final timestep  $T$ .
C Parallel Computation on Multiple GPUs
We experimented with two different approaches to implementing parallel computation on multiple GPUs in PyTorch: 1) using torch nn.DataParallel and 2) using torch multiprocessing. DataParallel is very easy to implement, but performed more poorly than multiprocessing. For multiprocessing, we use a producer/consumer design where we spawn a single producer process on GPU 0 to run the main loop in Algorithm 1, and  $N$  consumer processes (one for each of the  $N$  GPUs) to run Line 5 of Algorithm 1 in parallel. Empirically, this worked better than dedicating the producer process with its own GPU and using only  $N - 1$  consumer processes, since the producer process requires very little GPU resources.
D.1 LSUN Church and LSUN Bedroom
In Fig. 5 we plot wallclock speedup of ParaDDPM as a function of batch window size and number of GPUs on LSUN Church and LSUN Bedroom. These experiments on LSUN show a similar trend as the experiments on StableDiffusion-v2 in Fig. 4f.
(a) Lsun Church, close to 3x net wall clock speedup with 1000-step ParaDDPM using DataParallel
(b) LSUN Bedroom, over 3x net wall clock speedup with 1000-step ParaDDPM using DataParallel
Figure 5: Unconditional generation of  $256 \times 256$  images on diffusion models pretrained on the LSUN Church and Bedroom dataset, running ParaDDPM for 1000 steps on A100 GPUs. We plot the net speedup after dividing the hardware efficiency by the algorithm inefficiency as the batch window size increases. Note that Table 5 shows better speedups because for Table 5 we use a better parallel implementation with multiprocessing instead of DataParallel.
D.2 Ablation
In Table 6, we run an ablation study on the tolerance parameter  $\tau$  for 200-step ParaDDIM. A lower tolerance means the algorithm takes more parallel iterations and attains better sample quality, whereas a higher tolerance means the algorithm slides the batch window forward more aggressively leading to fewer iterations and faster sampling. We see in Table 6 that for 200-step ParaDDIM on StableDiffusion-v2, a good choice of  $\tau$  is 1e-1. Lower tolerance levels give less speedup without noticeable increase in CLIP score, and higher tolerance levels exhibit a drop in CLIP score.
Table 6: Ablation on the effect of error tolerance on sample quality and speed on StableDiffusion-v2. Samples are generated using ParaDiGMS with 200-step DDIM. CLIP score is computed over 1000 samples. Sample speed is computed on A100 GPUs.
StableDiffusion-v2	Steps	Tolerance τ	Parallel Iters	CLIP Score	Time per Sample	Speedup
DDIM	200	sequential	200	31.9	10.3s	-
DDIM	200	5e-3	36	31.9	7.4s	1.4x
DDIM	200	1e-2	28	31.9	5.1s	2.0x
DDIM	200	5e-2	19	31.9	3.3s	3.1x
DDIM	200	1e-1	15	31.9	2.6s	4.0x
DDIM	200	5e-1	13	31.5	2.1s	4.9x
DDIM	200	1e-0	12	31.5	1.9s	5.4x