Title: Plug-and-Play Fidelity Optimization for Diffusion Transformer Acceleration via Cumulative Error Minimization
Primary area: generative models
Keywords: Training-free acceleration, Diffusion transformer, Error correction

Abstract: Although Diffusion Transformer (DiT) has emerged as a predominant architecture for image and video generation, its iterative denoising process results in slow inference, which hinders broader applicability and development. Caching-based methods achieve training-free acceleration, while suffering from considerable computational error. Existing methods typically incorporate error correction strategies such as pruning or prediction to mitigate it. However, their fixed caching strategy fails to adapt to the complex error variations during the denoising process, which limits the full potential of error minimization. To tackle this challenge, we propose a novel plug-in acceleration strategy for cumulative error minimization, CEM, which can be seamlessly compatible with existing cache correction pipelines. CEM models the error distribution with the joint variation of the denoising timesteps and cache intervals. Guided by this prior, we formulate a dynamic programming algorithm with cumulative error approximation for strategy optimization, which achieves the caching error minimization. CEM is model-agnostic and exhibits strong generalization, which is adaptable to arbitrary acceleration budgets. It can be seamlessly integrated into existing cache correction frameworks and quantized models without introducing any additional computational overhead. Extensive experiments conducted on seven generation models across three tasks demonstrate that CEM significantly improves generation fidelity of existing acceleration models, even outperforms the original unaccelerated models. The code will be made publicly available.

TL;DR: A novel plug-and-play acceleration for diffusion transformers, which significantly improves generation fidelity on top of existing acceleration methods.