Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model

## Abstract

Abstract Diffusion-based image generation models excel at producing high-quality synthetic content, but suffer from slow and computationally expensive inference. Prior work has attempted to mitigate this by caching and reusing features within diffusion transformers across inference steps. These methods, however, often rely on rigid heuristics that result in limited acceleration or poor generalization across architectures. We propose E volutionary C aching to A ccelerate D iffusion models (ECAD), a genetic algorithm that learns efficient, per-model, caching schedules forming a Pareto frontier, using only a small set of calibration prompts. ECAD requires no modifications to network parameters or reference images. It offers significant inference speedups, enables fine-grained control over the quality-latency trade-off, and adapts seamlessly to different diffusion models. Notably, ECAD’s learned schedules can generalize effectively to resolutions and model variants not seen during calibration. We evaluate ECAD on PixArt- α 𝛼 \alpha italic_α , PixArt- Σ Σ \Sigma roman_Σ , and FLUX-1.dev using multiple metrics (FID, CLIP, Image Reward) across diverse benchmarks (COCO, MJHQ-30k, PartiPrompts), demonstrating consistent improvements over previous approaches. On PixArt- α 𝛼 \alpha italic_α , ECAD identifies a schedule that outperforms the previous state-of-the-art method by 4.47 COCO FID while increasing inference speedup from 2.35x to 2.58x. Our results establish ECAD as a scalable and generalizable approach for accelerating diffusion inference. Our code is available at https://github.com/aniaggarwal/ecad .

## 1 Introduction

Figure: Figure 1: We conceptualize diffusion caching as a Pareto optimization problem over image quality and inference time and propose ECAD to discover such Pareto frontiers using a genetic algorithm. Left: performance progression over generations for FLUX-1.dev. Right: example $1024{\times}1024$ results with corresponding speedups.
Refer to caption: extracted/6584413/plots/teaser_simpler.png

Diffusion has emerged as the backbone for state-of-the-art image and video synthesis techniques [1, 2, 3, 4].
Unlike prior methods involving deep learning, which would train a neural network to generate images in a single forward inference step, diffusion instead involves iterating over a prediction for many (20 to 50) steps [5].
This process is quite expensive, and many researchers and practitioners try to reduce the latency while preserving, or even improving, the quality [6, 7, 8, 9, 10].
Some of these strategies involve training some model that can perform the inference in 1 to 4 steps, particularly with model distillation [11, 9].
Other strategies do not train or tune any neural network weights, principally caching [6, 7, 12], where the diffusion model’s internal features are re-used across steps, allowing that computation to be skipped.

We introduce a new conceptual and algorithmic framework for diffusion caching by reframing the problem and replacing existing heuristic-based approaches with a principled, optimization-driven methodology that is generalizable across model architectures.
Existing caching methods typically offer a few discrete schedules, each with fixed trade-offs—for example, a 2x speedup with moderate quality loss, and a 3x speedup with greater degradation—without support for intermediate or more aggressive configurations.
However, real-world deployments often operate under variable latency or quality constraints, necessitating further flexibility.
We instead formulate caching as a multi-objective optimization problem, aiming to discover a smooth Pareto frontier that reveals a wide spectrum of speed-quality trade-offs. We show example frontiers we discover for FLUX.1-dev [13] in Figure [1](https://arxiv.org/html/2506.15682v2#S1.F1).

Such frontiers are very challenging to produce given how caching schedules are currently derived.
State-of-the-art approaches [8, 14, 15, 16] are motivated by heuristics, and key hyperparameters must be carefully hand-tuned by human practitioners based on performance on some set of key metrics.
We propose a different paradigm that does not rely on human-defined heuristics or hyperparameters, instead discovering effective caching schedules via genetic algorithm.

Our Evolutionary Caching to Accelerate Diffusion models (ECAD) requires two components: (i) some small set of text-only “calibration” prompts and (ii) some metric which computes image quality given a prompt and generated image (we use Image Reward [17]).
We formulate caching schedules such that the genetic algorithm can automatically discover which features to cache (in terms of blocks and layer types) and when (which timestep).
ECAD can be initialized with either random schedules or some set of promising schedules based on prior works [8, 15].
Thus, while ECAD presents a different paradigm compared to prior works, it can also build on their valuable findings.
ECAD takes these initial schedules and gradually evolves them according to the mating rules of a genetic algorithm, optimizing their “fitness” according to quality and computational complexity (measured in MACs, to be hardware-agnostic).

This strategy is extremely flexible.
While other methods are entirely designed around whether they cache entire block outputs, intermediate layer outputs (such as the output of an attention layer, or a feedforward layer), or even specific tokens, ours is orthogonal to all of these.
We offer a paradigm which can be used to optimize caching schedules according to any well-defined criteria.
We instantiate it with our criteria and schedule definitions in Section [3](https://arxiv.org/html/2506.15682v2#S3), but the general principles can be applied to arbitrary criteria and schedules to find Pareto-optimal caching frontiers.
For example, we could use other criteria to define fitness, such as human ratings of generated samples.
We could also change the caching schedule definitions to be more granular or more general, to focus on certain types of layers, or incorporate heuristics from other methods.
Furthermore, while ECAD involves some optimization, since we do not compute any gradients or update any weights, the memory requirements are quite low.
Additionally, there are no restrictions on batch size (allowing for use of single, small GPUs that would not be feasible for distillation), and the entire process can happen completely asynchronously.
Beyond this, schedules could be optimized for aggressively quantized diffusion models to further improve their acceleration and quality.

Figure [1](https://arxiv.org/html/2506.15682v2#S1.F1) showcases our method’s strong performance and highlights flexibility across resolutions. Although optimized for FLUX-1.dev at $256{\times}256$, the same schedule applied to $1024{\times}1024$ still outperforms SOTA methods in both speed and quality. At $256{\times}256$, ECAD matches or surpasses unaccelerated PixArt-$\alpha$ and FLUX-1.dev baselines with 1.97x and 2.58x latency reductions, respectively. At more aggressive 2.58x and 3.37x settings, quality slightly drops but remains competitive.

## 2 Related Work

### 2.1 Diffusion for Image and Video Synthesis

Diffusion models predict noise, given noised image inputs, to generate high-quality images [2, 18, 1] and videos [3, 19, 4].
To save time and reduce feature sizes, these computations are typically performed in the latent space [20] of a pre-trained variational autoencoder [21].
Although earlier works use U-Net backbones [22], more recent methods rely mainly on transformer-based models [23, 24, 25, 26], especially Diffusion Transformers (DiTs) [25], which dominant the current landscape due to their powerful scaling properties [27, 28, 4, 13].
Text-conditioning with multimodal models like CLIP [29], or extremely powerful text models like T5 [30], allows for more granular control over image content [31, 32, 33, 34, 35], not only in generative pipelines but especially for editing [36, 37, 38, 39, 40].

### 2.2 Accelerating Diffusion Inference

Training.
Many works focus on training or finetuning models for faster generation.
Some of these rely on model distillation [11], using the initial diffusion model to teach a model that uses less steps [41, 9, 42, 43, 10, 44, 45, 46].
Others train lightweight modules to predict skip connections [47], number of inference steps [48], features [49], or caching configurations [50].

Training-free.
Some works speed up diffusion without additional model training by caching and re-using features across steps during inference.
Works developed to cache U-Nets [12, 6, 7] do not easily transfer to DiTs [50], considering DiTs operate at a single resolution, have no encoder-decoder designation, and have only within-block skip connections.
Pioneering caching works for DiTs show promise, but some only cache entire blocks at fixed timestep intervals [8], which sacrifices some image quality, while others cache only attention layers [15], which mitigates potential speed-ups.
More recent works, including some concurrent works, use heuristics and carefully-tuned hyperparameters to allow for more dynamic and granular control over caching decisions [51, 14, 52, 53, 54, 55, 56, 16, 57].
Our method is most similar to these caching works, which do not tune any model parameters.
We overhaul the process of selecting an exact caching schedule and hyperparameters by replacing human-in-the-loop heuristic-based caching decisions and tuning with a genetic algorithm.

## 3 Methods

Figure: Figure 2: In the context of a transformer-based diffusion model, we describe how the transformer architecture allows for caching of attention and feedforward results separately (left). We then give a toy illustration of how our method might transition from one generation to the next, prioritizing mating for schedules with the best quality-speed trade-offs (right).
Refer to caption: x1.png

For an in-depth preliminary on diffusion for image generation, see Appendix [A.1](https://arxiv.org/html/2506.15682v2#A1.SS1).
Here, we give preliminaries for caching with diffusion transformers.
Then, we explain our method for conceptualizing caching in terms of Pareto frontiers on speed and quality, and our genetic algorithm which optimizes these frontiers on a customizable, per-model basis.

### 3.1 Preliminary: Caching Diffusion Transformers

Diffusion Transformers (DiTs) utilize a modified transformer architecture optimized for the diffusion denoising process.
A typical DiT block takes three inputs: a sequence of tokens $z^{\prime}$ representing the noisy image, a conditioning vector $c$ (e.g., text embeddings), and a timestep embedding $t$.

Caching in DiTs exploits temporal coherence between consecutive denoising steps. As the diffusion process proceeds from $z^{\prime}_{t}$ to $z^{\prime}_{t-1}$, the inputs to each block change gradually, creating an opportunity to reuse computed features from previous timesteps [6, 8].
We employ component-level caching within DiT blocks rather than caching entire blocks. For each transformer block, we selectively cache the outputs of specific functional components: self-attention ($f_{\text{SA}}$), cross-attention ($f_{\text{CA}}$), and feedforward networks ($f_{\text{FFN}}$).
Formally, for a component $f_{\text{comp}}$ in block $b$ at timestep $t$, we can decide whether to compute it directly or reuse its previously cached value:

$$ $f_{\text{comp}}^{b}(z^{\prime}_{t},t,c)=\begin{cases}\text{compute}(z^{\prime} _{t},c,t)&\text{if recompute}\\ \text{cache}[f_{\text{comp}}^{b},t+1]&\text{if cached}\end{cases}$ $$

When we choose to recompute, the new value is stored in the cache for potential reuse in subsequent steps. Figure [2](https://arxiv.org/html/2506.15682v2#S3.F2) demonstrates this for a DiT block with two components, self-attention ($f_{\text{SA}}$) and feedforward ($f_{\text{FFN}}$).
The DiT’s per-component skip connections allow features from the current inference step to be combined with cached features from previous steps.

This selective computation strategy can be represented as a binary tensor $S\in\{0,1\}^{N{\times}B{\times}C}$, where $N$ is the number of diffusion steps, $B$ is the number of transformer blocks, and $C$ is the number of cacheable components per block.
A value of 0 at position $(n,b,c)$ in $S$, which we show with shades of red in Figure [2](https://arxiv.org/html/2506.15682v2#S3.F2), indicates that we reuse the cached value of component $c$ in block $b$ at diffusion step $n$ rather than recomputing it.
The caching schedule directly impacts both computational efficiency and generation quality. Aggressive caching (more 0’s in $S$) reduces computation but may degrade output quality.
Our method focuses on finding caching schedules that offer an optimal trade-off between computational cost and generation quality by identifying which components can be safely cached for which blocks during which timesteps.

### 3.2 Genetic Algorithm as a Paradigm for Caching

Caching, as Pareto Frontiers.
The caching optimization problem inherently exhibits a trade-off between computational efficiency and generation quality. This can be formalized as a multi-objective optimization problem:

$$ $\min_{S}(C(S),Q(S))$ $$

where $C(S)$ denotes the computational cost function (lower is better) and $Q(S)$ represents the generation quality metric (lower is better, e.g., FID) for a caching schedule $S$.
This optimization operates directly on the binary caching tensor $S\in\{0,1\}^{N\times B\times C}$ introduced previously. Possible configurations for $S$ naturally induce sets of solutions that form Pareto frontiers – improving one objective necessarily degrades the other. However, this search space is intractable to exhaustively explore, even for small DiTs, given current compute. Prior acceleration methods have predominantly relied on fixed heuristics that typically provide only isolated operating points. By contrast, our proposed approach explores a greater search space and discovers Pareto-optimal configurations, enabling practitioners to select schedules based on application-specific constraints.

Figure: Algorithm 1 Evolutionary Caching to Accelerate Diffusion models (ECAD)

Evolutionary Caching to Accelerate Diffusion models (ECAD).
We introduce ECAD, an evolutionary algorithm-based framework for discovering efficient caching schedules for diffusion models, in Algorithm [1](https://arxiv.org/html/2506.15682v2#alg1).
Our approach’s key insight is that the optimal caching configuration can be discovered through a population-based search over the space of possible caching schedules, using a small set of calibration prompts to evaluate candidate solutions.
ECAD is a framework with 4 simple customizable components.

The practitioner may adjust granularity with the (1) binary caching tensor shape by adjusting $N$, $B$, and $C$ (the defaults we define for $S$ allow any component with a skip connection to be cached, on any block, for any timestep).
While it does not require any image data, ECAD needs (2) calibration prompts, which we instantiate with the 100 prompts from the Image Reward Benchmark [17].
The practitioner can also select their preferred (3) metrics, where ideally both can be computed quickly online.
We use Image Reward for quality, and Multiply-Accumulate Operations (MACs) for speed (to avoid hardware dependencies).
Then, we choose an (4) initial population of caching schedules, which should be diverse, and can be seeded based on prior knowledge (such as using FORA schedules) or initialized randomly.
We utilize NSGA-II [58] for our genetic algorithm due to its efficient non-dominated sorting approach and proven effectiveness in multi-criteria optimization problems.

With all these defined, the practitioner can run ECAD for the desired number of generations.
At each generation, images are generated for every binary caching tensor in the population, and the best tensors (in terms of quality and speed) evolve to form the next generation.
Each generation yields incrementally more optimal Pareto frontiers for caching the chosen model, with the chosen diffusion scheduler, for the chosen number of timesteps.

## 4 Experiments

### 4.1 Experimental Settings

#### Model Architectures

We provide experiments on three popular text-to-image DiT models: PixArt-$\alpha$, PixArt-$\Sigma$, FLUX-1.dev. Each model uses its default sampling method at 20 steps: DPM-Solver++ [5] for both PixArt models and FlowMatchEulerDiscreteScheduler [28] for FLUX-1.dev.
Guidance scales are 4.5 for PixArt models and 5 for FLUX-1.dev.
PixArt-$\alpha$ and PixArt-$\Sigma$ each employ 28 identical transformer blocks containing three components we enable caching for: self-attention, cross-attention, and feedforward network.
FLUX-1.dev, in contrast, implements an MMDiT-based architecture [28] with 19 “full blocks” and 38 “single” blocks. We enable caching for attention, feedforward, and feedforward context components in full blocks, and attention, MLP projection, and MLP output for single blocks.
Cacheable component selection is discussed in Appendix [A.2](https://arxiv.org/html/2506.15682v2#A1.SS2).
We calibrate ECAD for all models at $256{\times}256$ resolution, but present experiments with evaluations at both $256{\times}256$ and $1024{\times}1024$ resolution.

#### Evaluation Metrics

We evaluate performance using Image Reward [17], FID [59], and CLIP score [60] with ViT-B/32 [24] on the Image Reward Benchmark prompts set [17], the PartiPrompts set [61], MS-COCO2017-30K [62] (we use the same prompts and images as ToCa [14]) and MJHQ-30K [63].
On the Image Reward Benchmark prompts set, we generate each of 100 prompts at 10 different, fixed seeds for 1,000 total images.
For PartiPrompts we generate a single image for each of the 1,632 prompts.
To measure the speed of a particular caching schedule, we use two metrics: multiply-accumulate operations (MACs) and direct image generation latency. Except where otherwise stated, we utilize calflops [64] to measure MACs. We average end-to-end image generation latency using precomputed text embeddings on 1 NVIDIA A6000 GPU after discarding warmup runs; full details in Appendix [A.6](https://arxiv.org/html/2506.15682v2#A1.SS6).

Figure: Figure 3: PartiPrompt Image Reward vs. latency for PixArt-$\alpha$ (left) and FLUX.1-dev (right).
Refer to caption: x2.png

**Table 1: Main results, $\textbf{256}{\times}\textbf{256}$, 20-step text-to-image generation. We select schedules from our evolutionary Pareto Frontier and compare them to prior works for a variety of datasets and models in terms of Image Reward, CLIP Score, and FID. Despite being optimized only on Image Reward, only on the 100 calibration prompts, our method achieves superior results across other metrics and for unseen prompts.**
| Settings | Latency | Calibration | PartiPrompts | MS-COCO2017-30K | MJHQ-30K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | ms / img$\downarrow$ | Image | Image |  |  |  |  |  |
| Model | Caching | Setting | TMACs$\downarrow$ | (speedup$\uparrow$) | Reward$\uparrow$ | Reward$\uparrow$ | CLIP$\uparrow$ | FID$\downarrow$ | CLIP$\uparrow$ | FID$\downarrow$ | CLIP$\uparrow$ |
|  | None |  | 5.71 | 165.74 (1.00x) | 0.90 | 0.97 | 32.01 | 24.84 | 31.29 | 9.75 | 32.77 |
|  | TGATE | $m=15,k=1$ | 4.86 | 144.77 (1.14x) | 0.78 | 0.87 | 31.70 | 23.90 | 31.12 | 10.38 | 32.33 |
|  | TGATE | $m=10,k=5$ | 3.47 | 108.52 (1.53x) | -0.051 | -0.27 | 28.90 | 29.78 | 28.29 | 17.52 | 29.38 |
|  | FORA | $\mathcal{N}=2$ | 2.87 | 100.57 (1.65x) | 0.83 | 0.91 | 32.03 | 24.80 | 31.37 | 10.33 | 32.74 |
|  | FORA | $\mathcal{N}=3$ | 2.02 | 82.55 (2.01x) | 0.60 | 0.83 | 31.94 | 24.50 | 31.35 | 11.11 | 32.63 |
|  | ToCa | $\mathcal{N}=3,\mathcal{R}=60\%$ | 3.17^∗ | 90.71 (1.83x)^∗ | 0.71 | 0.76 | 31.46 | 22.05 | 30.99 | 12.01 | 32.37 |
|  | ToCa | $\mathcal{N}=3,\mathcal{R}=90\%$ | 2.13^∗ | 70.58 (2.35x)^∗ | 0.60 | 0.68 | 31.35 | 24.01 | 30.92 | 11.80 | 32.35 |
|  | DuCa | $\mathcal{N}=3,\mathcal{R}=60\%$ | 3.20 | 72.53 (2.29x)^∗ | 0.76 | 0.79 | 31.53 | 23.13 | 31.03 | 11.69 | 32.48 |
|  | DuCa | $\mathcal{N}=3,\mathcal{R}=90\%$ | 2.30 | 64.08 (2.59x)^∗ | 0.76 | 0.74 | 31.42 | 24.69 | 30.96 | 12.53 | 32.39 |
|  | Ours | fast | 2.13 | 84.09 (1.97x) | 0.96 | 0.99 | 31.94 | 20.58 | 31.40 | 8.02 | 32.78 |
|  | Ours | faster | 1.46 | 69.17 (2.40x) | 0.90 | 0.88 | 31.44 | 21.93 | 31.10 | 9.92 | 32.34 |
| PixArt-$\alpha$ | Ours | fastest | 1.18 | 64.24 (2.58x) | 0.81 | 0.77 | 31.53 | 19.54 | 31.28 | 8.67 | 32.24 |
|  | None |  | 5.71 | 167.62 (1.00x) | 0.85 | 1.08 | 31.90 | 24.63 | 31.11 | 10.53 | 32.65 |
|  | FORA | $\mathcal{N}=3$ | 2.02 | 82.12 (2.04x) | 0.65 | 0.81 | 31.91 | 27.69 | 31.16 | 12.70 | 32.28 |
|  | ToCa^† | $\mathcal{N}=3,\mathcal{R}=60\%$ | 3.17^∗ | 94.28 (1.78x)^∗ | 0.11 | 0.19 | 31.03 | 54.80 | 30.34 | 35.42 | 30.64 |
|  | ToCa^† | $\mathcal{N}=3,\mathcal{R}=90\%$ | 2.13^∗ | 73.03 (2.30x)^∗ | 0.07 | 0.14 | 30.89 | 56.48 | 30.25 | 36.53 | 30.55 |
| PixArt-$\Sigma$ | Ours | fast | 1.91 | 84.84 (1.98x) | 0.85 | 1.02 | 31.86 | 22.17 | 31.25 | 8.91 | 32.52 |
|  | None |  | 198.69 | 2620.09 (1.00x) | 0.69 | 1.04 | 31.88 | 25.76 | 30.95 | 17.77 | 31.06 |
|  | FORA | $\mathcal{N}=3$ | 69.80 | 1073.70 (2.44x) | 0.67 | 0.93 | 31.88 | 23.51 | 31.30 | 19.38 | 31.10 |
|  | ToCa | $\mathcal{N}=4,\mathcal{R}=90\%$ | 42.96 | 1576.97 (1.66x)^∗ | 0.63 | 0.93 | 31.81 | 23.78 | 31.26 | 21.59 | 30.88 |
|  | Ours | fast | 63.02 | 1016.59 (2.58x) | 0.83 | 1.04 | 32.24 | 21.61 | 31.58 | 16.14 | 31.69 |
| FLUX.1-dev | Ours | fastest | 43.60 | 778.17 (3.37x) | 0.69 | 0.89 | 32.27 | 26.66 | 31.63 | 21.43 | 31.67 |

### 4.2 Main Results

We optimize ECAD on three diffusion models: PixArt-$\alpha$, PixArt-$\Sigma$, and FLUX-1.dev.
We present results for select schedules in Table [1](https://arxiv.org/html/2506.15682v2#S4.T1).
For PixArt-$\alpha$ at $256{\times}256$ resolution with 20 inference steps, we run 550 generations with 72 candidate schedules per generation, where each candidate generates 1,000 images (10 per each of 100 Image Reward Benchmark prompts).
For FLUX-1.dev, we reduce the population to 24 schedules due to compute constraints and train for 250 generations under otherwise identical settings.
We initialize both using variants inspired by FORA and TGATE, detailed in Appendix [A.4](https://arxiv.org/html/2506.15682v2#A1.SS4).
For PixArt-$\Sigma$, we transfer 72 schedules from PixArt-$\alpha$’s 200th-generation Pareto frontier and run 50 additional generations, leveraging the models’ shared DiT architecture.

Across all models, ECAD achieves strong performance on Image Reward (which correlates well with human preference [17]) and FID.
On PixArt-$\alpha$, our ‘fastest’ schedule reduces FID by 9.3 over baseline and by 2.51 over ToCa’s best setting.
On PixArt-$\Sigma$ and FLUX-1.dev, ECAD schedules outperform prior work and baseline by a significant margin.
On FLUX-1.dev, our ‘fast’ schedule at 2.58x matches baseline Image Reward and the ‘fastest’ schedule at 3.37x maintains competitive quality.
For prompt-image alignment, measured via CLIP score, ECAD roughly matches prior works, which is expected as caching should not affect prompt-image alignment.

Figure: Figure 4: Qualitative results comparing our “fast” schedule for PixArt-$\alpha$ $256{\times}256$ with ToCa. “…” represent omitted text, see Appendix [A.10](https://arxiv.org/html/2506.15682v2#A1.SS10) for full prompts for the first and fifth columns.
Refer to caption: x4.png

We show full Pareto frontiers in Figure [3](https://arxiv.org/html/2506.15682v2#S4.F3) on unseen prompts.
ECAD discovers schedules that consistently outperform prior works across evaluation metrics while providing fine-grained control over the quality-latency tradeoff.
We provide some qualitative results which highlight ECAD’s good quality despite impressive speedups in Figure [4](https://arxiv.org/html/2506.15682v2#S4.F4).
We show the composition of the “fast” ECAD schedules for PixArt-$\alpha$ and FLUX.1-dev in Figure [5](https://arxiv.org/html/2506.15682v2#S4.F5), with more schedules in Appendix [A.9](https://arxiv.org/html/2506.15682v2#A1.SS9).

Figure: Figure 5: Figure that shows our “fast” schedule for PixArt-$\alpha$ (left) and FLUX-1.dev (right). Reds are cached components and grays are recomputed (for PixArt-$\alpha$, from left to right: self-attention, cross-attention, and feedforward). See Appendix [A.9](https://arxiv.org/html/2506.15682v2#A1.SS9) for more details.
Refer to caption: x5.png

Scaling Properties.
Unlike existing approaches, practitioners have the flexibility to run ECAD for as many generations as their time and compute constraints allow.
While competitive schedules emerge within a few iterations, continued optimization yields steady improvements. To illustrate this, we track the ‘slowest’ schedule throughout the genetic process for PixArt-$\alpha$ and report results in Table [3](https://arxiv.org/html/2506.15682v2#S4.T3). After just 50 generations, this schedule
outperforms the unaccelerated baseline and all prior methods on Image Reward for unseen PartiPrompts and MJHQ FID.
Further generations reduce latency at the eventual, but slight, cost in quality.
Figure [7](https://arxiv.org/html/2506.15682v2#S4.F7) shows the Pareto frontier for each generation on the calibration prompts; initial generations rapidly improve while later generations show incremental improvements.

**Table 2: Genetic scaling results. We show performance changes as we run more iterations (generations) of ECAD, in terms of latency, PartiPrompts Image Reward, and MJHQ-30K FID. We select the schedule with the highest TMACs for each generation.**
| # Gens | ms / img$\downarrow$ | Image | FID$\downarrow$ |
| --- | --- | --- | --- |
| (speedup$\uparrow$) | Reward$\uparrow$ |  |  |
| 1 | 145.09 (1.14x) | 1.00 | 9.40 |
| 50 | 92.76 (1.79x) | 0.98 | 7.97 |
| 150 | 87.11 (1.90x) | 1.00 | 8.11 |
| 300 | 86.62 (1.91x) | 0.99 | 8.04 |
| 500 | 76.52 (2.17x) | 0.96 | 8.49 |

**Table 4: FLUX-1.dev detailed transfer results, 1024 $\times$ 1024 resolution, 20-step text-to-image generation. We reuse our ‘fast’ schedule trained on FLUX-1.dev at 256x256 resolution, as well as an older, ‘slow’ schedule. We apply them for 1024 $\times$ 1024 image generation and compare them to prior works for a variety of datasets in terms of Image Reward, CLIP Score, and FID. Our results are competitive with prior work despite being evaluated at a different resolution than optimization.**
| Model Settings | Latency | Calibration | PartiPrompts | MS-COCO2017-30K | MJHQ-30K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Caching | Setting | TMACs$\downarrow$ | s/img$\downarrow$ (speedup$\uparrow$) | Image Reward$\uparrow$ | Image Reward$\uparrow$ | CLIP$\uparrow$ | FID$\downarrow$ | CLIP$\uparrow$ | FID$\downarrow$ | CLIP$\uparrow$ |
| None |  | 1190.25 | 18.30 (1.00x) | 0.68 | 1.14 | 31.98 | 25.45 | 31.08 | 14.63 | 31.99 |
| None | $40\%$ steps | 476.10 | 7.61 (2.41x) | 0.43 | 0.83 | 31.38 | 25.20 | 30.73 | 21.68 | 30.99 |
| FORA | $\mathcal{N}=3$ | 416.88 | 7.62 (2.40x) | 0.27 | 0.69 | 31.20 | 29.45 | 30.52 | 24.65 | 30.69 |
| ToCa | $\mathcal{N}=4,\mathcal{R}=90\%$ | 300.41 | 7.42 (2.47x)^∗ | 0.66 | 1.09 | 32.05 | 26.88 | 31.32 | 15.39 | 31.93 |
| Ours | $\mathrm{slow}_{256\to 1024}$ | 644.05 | 10.59 (1.73x) | 0.74 | 1.05 | 31.82 | 22.15 | 31.00 | 15.98 | 31.79 |
| Ours | $\mathrm{fast}_{256\to 1024}$ | 376.62 | 6.96 (2.63x) | 0.71 | 1.05 | 31.88 | 26.69 | 30.91 | 17.76 | 31.99 |

### 4.3 Emergent Generalization Capabilities

Model Transfer Results.
To demonstrate ECAD’s advantage over handcrafted heuristics, we transfer pre-optimized schedules between model variants.
In Table [3](https://arxiv.org/html/2506.15682v2#S4.T3), we select the “slowest” schedule from the Pareto-frontier across the first 200 generations of PixArt-$\alpha$ ECAD optimization and evaluate it on PixArt-$\Sigma$ as is, to demonstrate direct transfer results.
Then, we perform an additional 50 optimization generations on PixArt-$\Sigma$ using 72 schedules transferred from the PixArt-$\alpha$ ECAD frontier at 200 generations.
Although with direct transfer from PixArt-$\alpha$, PixArt-$\Sigma$ has higher latency than PixArt-$\alpha$ at 200 generations, after only 50 generations of optimization, it surpasses PixArt-$\alpha$’s speedup while improving calibration Image Reward and MJHQ FID.
By comparison, simply transferring the 250 generation PixArt-$\alpha$ configuration yields only a 1.79x speedup instead of 1.98x, and has worse calibration Image Reward and MJHQ FID.
This is a departure from recent caching innovations; for example, ToCa’s carefully tuned PixArt-$\alpha$ settings cannot be transferred to PixArt-$\Sigma$ (see Table [1](https://arxiv.org/html/2506.15682v2#S4.T1)), despite the similarities between the two models.

Resolution Transfer Results.
We present ECAD’s performance on FLUX-1.dev at $1024{\times}1024$ resolution after optimization on $256{\times}256$ in Table [4](https://arxiv.org/html/2506.15682v2#S4.T4), and highlight its superior performance compared to FORA and the “None” approaches.
We apply schedules as-is, with no further optimization of schedules at the higher resolution.
While it is likely preferable to optimize ECAD at the target evaluation resolution if sufficient compute is available, we show this is not necessary in practice.
In addition to the same ‘fast’ FLUX-1.dev schedule from Table [1](https://arxiv.org/html/2506.15682v2#S4.T1) at $256{\times}256$ resolution, we select a ‘slow’ model from just 50 generations of training at $256{\times}256$.
We find that even though ToCa was optimized for high resolution and ours for low resolution, our “fast” setting outperforms it in terms of Calibration Image Reward and COCO FID.

### 4.4 Ablation Analysis

Figure: Figure 6: ECAD evolution. ECAD iteratively improves quality/time trade-offs as it evolves across generations as measured by Image Reward (PixArt-$\alpha$ $256{\times}256$).
Refer to caption: x7.png

To better explore the evolutionary algorithm’s behavior, especially with respect to optimization time, we run three ablations with different hyperparameters on PixArt-$\alpha$ for 100 generations.
We separately vary the population size (from 72 to 24), the number of images generated per prompt (from 10 to 3), and the number of prompts used (from 100 to 33, selected randomly), each approximately reducing GPU time by 66%.
Figure [7](https://arxiv.org/html/2506.15682v2#S4.F7) shows that fewer images per prompt is the least harmful, while using a less diverse set of 33 prompts is the most harmful.
Notably, the reduced prompt setting performs comparably to the full setting at lower TMACs but fails to reach the same maximum quality.
This supports the intuition that larger, more diverse calibration sets enable stronger optimization and could yield better results than those presented here, motivating further exploration in future work.
Meanwhile, the frontier of the reduced population setting closely resembles that of earlier generations of the full population, as seen in Figure [7](https://arxiv.org/html/2506.15682v2#S4.F7).
Thus, we hypothesize that reducing the population size is akin to running the model for less generations.

## 5 Discussion

Limitations and Broader Impacts.
Optimizing our schedules on automatic metrics ties our performance to the quality of those metrics.
We use Image Reward for the sake of cost and time; however, if we replace it with ranking by human users, for example, results could improve.
Importantly, ECAD does not introduce new societal risks beyond those inherent to diffusion models. While reduced inference cost may increase potential for misuse, it also promotes broader image-generation accessibility and mitigates some environmental impact of image generation.

Conclusion.
In this work, we reconceptualize diffusion caching as a Pareto optimization problem that enables fine-grained trade-offs between speed and quality.
We provide a method, ECAD, which converts this problem into a search over binary masks, and can discover a best-case caching Pareto frontier.
With only 100 text prompts, our method runs asynchronously with much lower memory requirements than training or fine-tuning a diffusion model.
We achieve state-of-the-art results for training-free acceleration of diffusion models in both speed and quality.

## Acknowledgments and Disclosure of Funding

This work was partially supported by NSF CAREER Award (#2238769) to AS. The authors acknowledge UMD’s supercomputing resources made available for conducting this research. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF or the U.S. Government.

## References

- [1]
P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021.
- [2]
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
- [3]
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” Advances in Neural Information Processing Systems, vol. 35, pp. 8633–8646, 2022.
- [4]
Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao, L. He, and L. Sun, “Sora: A review on background, technology, limitations, and opportunities of large vision models,” 2024.
- [5]
C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,” 2023.
- [6]
X. Ma, G. Fang, and X. Wang, “Deepcache: Accelerating diffusion models for free,” 2023.
- [7]
F. Wimbauer, B. Wu, E. Schoenfeld, X. Dai, J. Hou, Z. He, A. Sanakoyeu, P. Zhang, S. Tsai, J. Kohler, C. Rupprecht, D. Cremers, P. Vajda, and J. Wang, “Cache me if you can: Accelerating diffusion models through block caching,” 2024.
- [8]
P. Selvaraju, T. Ding, T. Chen, I. Zharkov, and L. Liang, “Fora: Fast-forward caching in diffusion transformer acceleration,” 2024.
- [9]
C. Meng, R. Rombach, R. Gao, D. P. Kingma, S. Ermon, J. Ho, and T. Salimans, “On distillation of guided diffusion models,” 2023.
- [10]
A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach, “Adversarial diffusion distillation,” 2023.
- [11]
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015.
- [12]
S. Li, T. Hu, F. S. Khan, L. Li, S. Yang, Y. Wang, M.-M. Cheng, and J. Yang, “Faster diffusion: Rethinking the role of unet encoder in diffusion models,” 2023.
- [13]
B. F. Labs, “Flux.” [https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024.
- [14]
C. Zou, X. Liu, T. Liu, S. Huang, and L. Zhang, “Accelerating diffusion transformers with token-wise feature caching,” 2025.
- [15]
H. Liu, W. Zhang, J. Xie, F. Faccio, M. Xu, T. Xiang, M. Z. Shou, J.-M. Perez-Rua, and J. Schmidhuber, “Faster diffusion via temporal attention decomposition,” 2024.
- [16]
C. Zou, E. Zhang, R. Guo, H. Xu, C. He, X. Hu, and L. Zhang, “Accelerating diffusion transformers with dual feature caching,” 2024.
- [17]
J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong, “Imagereward: Learning and evaluating human preferences for text-to-image generation,” 2023.
- [18]
A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in International Conference on Machine Learning, pp. 8162–8171, PMLR, 2021.
- [19]
A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22563–22575, June 2023.
- [20]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2022.
- [21]
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” 2014.
- [22]
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015.
- [23]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
- [24]
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
- [25]
W. Peebles and S. Xie, “Scalable diffusion models with transformers,” 2023.
- [26]
F. Bao, S. Nie, K. Xue, Y. Cao, C. Li, H. Su, and J. Zhu, “All are worth words: A vit backbone for diffusion models,” 2023.
- [27]
J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y. Wu, Z. Wang, J. Kwok, P. Luo, H. Lu, et al., “Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis,” arXiv preprint arXiv:2310.00426, 2023.
- [28]
P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, D. Podell, T. Dockhorn, Z. English, K. Lacey, A. Goodwin, Y. Marek, and R. Rombach, “Scaling rectified flow transformers for high-resolution image synthesis,” 2024.
- [29]
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021.
- [30]
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023.
- [31]
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” in Advances in Neural Information Processing Systems (S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds.), vol. 35, pp. 36479–36494, Curran Associates, Inc., 2022.
- [32]
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022.
- [33]
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” 2022.
- [34]
N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22500–22510, June 2023.
- [35]
D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” 2023.
- [36]
B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, “Imagic: Text-based real image editing with diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6007–6017, June 2023.
- [37]
T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392–18402, June 2023.
- [38]
W. Sun, R.-C. Tu, J. Liao, and D. Tao, “Diffusion model-based video editing: A survey,” 2024.
- [39]
D. Ceylan, C.-H. P. Huang, and N. J. Mitra, “Pix2video: Video editing using image diffusion,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 23149–23160, 2023.
- [40]
W. Chai, X. Guo, G. Wang, and Y. Lu, “Stablevideo: Text-driven consistency-aware diffusion video editing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23040–23050, 2023.
- [41]
T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” 2022.
- [42]
S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao, “Latent consistency models: Synthesizing high-resolution images with few-step inference,” 2023.
- [43]
Y. Lee, K. Park, Y. Cho, Y.-J. Lee, and S. J. Hwang, “Koala: Empirical lessons toward memory-efficient and fast diffusion models for text-to-image synthesis,” 2024.
- [44]
J. Kohler, A. Pumarola, E. Schönfeld, A. Sanakoyeu, R. Sumbaly, P. Vajda, and A. Thabet, “Imagine flash: Accelerating emu diffusion models with backward distillation,” 2024.
- [45]
T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park, “One-step diffusion with distribution matching distillation,” 2023.
- [46]
Y. Xu, Y. Zhao, Z. Xiao, and T. Hou, “Ufogen: You forward once large scale text-to-image generation via diffusion gans,” 2023.
- [47]
Z. Jiang, C. Mao, Y. Pan, Z. Han, and J. Zhang, “Scedit: Efficient and controllable image diffusion generation via skip connection editing,” 2023.
- [48]
H. Zhang, Z. Wu, Z. Xing, J. Shao, and Y.-G. Jiang, “Adadiff: Adaptive step selection for fast diffusion,” 2023.
- [49]
M. Gwilliam, H. Cai, D. Wu, A. Shrivastava, and Z. Cheng, “Accelerate high-quality diffusion models with inner loop feedback,” 2025.
- [50]
X. Ma, G. Fang, M. B. Mi, and X. Wang, “Learning-to-cache: Accelerating diffusion transformer via layer caching,” 2024.
- [51]
P. Chen, M. Shen, P. Ye, J. Cao, C. Tu, C.-S. Bouganis, Y. Zhao, and T. Chen, “$\delta$-dit: A training-free acceleration method tailored for diffusion transformers,” 2024.
- [52]
Z. Yuan, H. Zhang, P. Lu, X. Ning, L. Zhang, T. Zhao, S. Yan, G. Dai, and Y. Wang, “Ditfastattn: Attention compression for diffusion transformer models,” 2024.
- [53]
F. Liu, S. Zhang, X. Wang, Y. Wei, H. Qiu, Y. Zhao, Y. Zhang, Q. Ye, and F. Wan, “Timestep embedding tells: It’s time to cache for video diffusion model,” 2024.
- [54]
Z. Liu, Y. Yang, C. Zhang, Y. Zhang, L. Qiu, Y. You, and Y. Yang, “Region-adaptive sampling for diffusion transformers,” 2025.
- [55]
J. Qiu, S. Wang, J. Lu, L. Liu, H. Jiang, X. Zhu, and Y. Hao, “Accelerating diffusion transformer via error-optimized cache,” 2025.
- [56]
W. Sun, Q. Hou, D. Di, J. Yang, Y. Ma, and J. Cui, “Unicp: A unified caching and pruning framework for efficient video generation,” 2025.
- [57]
J. Liu, C. Zou, Y. Lyu, J. Chen, and L. Zhang, “From reusing to forecasting: Accelerating diffusion models with taylorseers,” 2025.
- [58]
K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,” IEEE transactions on evolutionary computation, vol. 18, no. 4, pp. 577–601, 2013.
- [59]
M. Seitzer, “pytorch-fid: FID Score for PyTorch.” [https://github.com/mseitzer/pytorch-fid](https://github.com/mseitzer/pytorch-fid), August 2020.
Version 0.3.0.
- [60]
S. Zhengwentai, “clip-score: CLIP Score for PyTorch.” [https://github.com/taited/clip-score](https://github.com/taited/clip-score), March 2023.
Version 0.2.1.
- [61]
J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, B. Hutchinson, W. Han, Z. Parekh, X. Li, H. Zhang, J. Baldridge, and Y. Wu, “Scaling autoregressive models for content-rich text-to-image generation,” 2022.
- [62]
T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, “Microsoft coco: Common objects in context,” 2015.
- [63]
D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi, “Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation,” 2024.
- [64]
X. Ye, “calflops: a flops and params calculate tool for neural networks in pytorch framework,” 2023.
- [65]
L. Weng, “What are diffusion models?,” lilianweng.github.io, Jul 2021.
- [66]
J. Blank and K. Deb, “pymoo: Multi-objective optimization in python,” IEEE Access, vol. 8, pp. 89497–89509, 2020.
- [67]
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. G. Lopes, B. K. Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” in Advances in Neural Information Processing Systems, vol. 35, 2022.

## Appendix A Appendix

### A.1 Diffusion Preliminary

Diffusion models have emerged as powerful generative models capable of producing high-quality images. In this section, we provide a brief overview of the diffusion process, the denoising objective, and the specific formulation for Diffusion Transformers (DiT).

#### Basic Diffusion Process:

The diffusion process follows a Markov chain that gradually adds Gaussian noise to data. Given an image $x_{0}$ sampled from a data distribution $q(x_{0})$, the forward diffusion process sequentially transforms the data into a standard Gaussian distribution through $T$ timesteps by adding noise according to a pre-defined schedule. This forward process can be formulated as:

$$ $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{ I})$ (1) $$

where $\{\beta_{t}\in(0,1)\}_{t=1}^{T}$ represents the noise schedule [65]. We define $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$ for convenience. A key property arising from this process is that we can sample $x_{t}$ at any arbitrary timestep $t$ directly from $x_{0}$ without having to sample the intermediate states as:

$$ $x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$ (2) $$

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$. This property is particularly useful during training as it allows for efficient parallel sampling across different timesteps.

#### Denoising Objective:

The denoising process aims to reverse the forward diffusion by learning to predict the noise added at each step. This is typically accomplished by training a neural network $\epsilon_{\theta}(x_{t},t)$ to estimate the noise component in $x_{t}$. Its training objective is formulated as:

$$ $\mathcal{L}=\mathbb{E}_{t,x_{0},\epsilon}[||\epsilon-\epsilon_{\theta}(x_{t},t )||^{2}]$ (3) $$

where $t$ is uniformly sampled from $\{1,2,...,T\}$, $x_{0}$ from the data distribution, and $\epsilon$ from $\mathcal{N}(0,\mathbf{I})$.
During sampling, the noisy image is gradually denoised using various strategies. In the DDPM algorithm [2], the reverse process takes the form:

$$ $p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{t} ^{2}\mathbf{I})$ (4) $$

where $\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{\beta_{t}}{
\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)$. While effective, DDPM typically requires hundreds to thousands of denoising steps.
For more efficient sampling, DPM-Solver++ [5] (used in both PixArt-$\alpha$ and PixArt-$\Sigma$) reformulates the diffusion process as an ordinary differential equation of the (simplified) form below:

$$ $\frac{dx}{dt}=-\frac{1}{2}\beta_{t}\nabla_{x}\log p_{t}(x)$ (5) $$

DPM-Solver++ then applies high-order numerical methods to solve this ODE more efficiently. This leads to update rules that enable high-quality image generation in as few as 20 steps rather than the hundreds required by DDPM. However, each step still requires a forward pass through the noise prediction network, making the sampling process computationally intensive and a primary target for acceleration.

#### DiT-specific Processing

Diffusion Transformers (DiT) adapt the transformer architecture for diffusion models, offering improved scalability compared to conventional UNet architectures. The processing pipeline for DiTs follows several key steps:
first, the input image $x\in\mathbb{R}^{H\times W\times C}$ is encoded into a lower-dimensional latent representation using a pre-trained variational autoencoder (VAE): $z=\mathcal{E}(x)\in\mathbb{R}^{h\times w\times d}$, where $h$, $w$, and $d$ represent the height, width, and channel dimensions of the latent space, respectively.
The latent representation is then divided into non-overlapping patches and linearly projected to form a sequence of tokens $z^{\prime}=\text{Patch}(z)\in\mathbb{R}^{N\times d^{\prime}}$, where $N=\frac{hw}{p^{2}}$ is the number of patches with patch size $p\times p$, and $d^{\prime}$ is the embedding dimension of the transformer.
Additionally, timestep embeddings and class or text condition embeddings are incorporated into the model to condition the generation process.
Finally, the DiT model processes these tokens through a series of transformer blocks, each typically containing self-attention and cross-attention (or joint attention as in FLUX.1-dev), and feedforward network components.

### A.2 Cacheable Component Selection

To enable ECAD on an off-the-shelf model, one must first select which components are cacheable. Any computation whose output can be stored at one step and reused at another–while introducing only minimal, acceptable inaccuracy–can be considered for caching. The number of such components determines the value of $C$ in the binary caching tensor $S\in\{0,1\}^{N\times B\times C}$, introduced in Section [3](https://arxiv.org/html/2506.15682v2#S3). Since the search space grows linearly with $C$, careful selection is essential to ensure efficient and effective caching.

Note that the tensor notation is simplified for clarity. In cases where the model uses $k$ different types of DiT blocks, each with a different number of cacheable components, the caching tensor would instead take the form $S\in\{0,1\}^{N\times(\sum_{i=1}^{k}B_{i}\times C_{i})}$.

Table [5](https://arxiv.org/html/2506.15682v2#A1.T5) enumerates the computational complexity of each DiT block’s forward pass. We enable caching for the three most computationally expensive components per block, as they collectively dominate the total cost. Computations outside the DiT block’s forward pass (e.g., timestep and position embeddings) are not considered at this time as they contribute less than 1% of the total compute.

**Table 5: Computation breakdown of a single transformer block forward pass for PixArt-$\alpha$, PixArt-$\Sigma$, and FLUX-1.dev at $256\times 256$ resolution. We report GMACs and each component’s share of total block computation. Components marked as cache-enabled are those selected for caching in ECAD, as they dominate the computational cost. Components not selected are omitted for efficiency, not due to any fundamental limitation in their cacheability.**
| Model | Component | Cache-Enabled | GMACs | % of Block Total |
| --- | --- | --- | --- | --- |
|  | Feedforward | Yes | 5.440 | 53.6 % |
|  | Self-Attention | Yes | 2.720 | 26.8 % |
|  | Cross-Attention | Yes | 2.000 | 19.7 % |
|  | Ada Layer Norm Single | No | 0.000 | 0.0 % |
| PixArt-$\alpha$<br>and<br>PixArt-$\Sigma$ | Total: PixArt Transformer Block |  | 10.150 | 100 % |
|  | Feedforward (Context) | Yes | 77.310 | 44.39 % |
|  | Joint Attention (Mutli-stream) | Yes | 57.980 | 33.29 % |
|  | Feedforward (Regular) | Yes | 38.650 | 22.19 % |
|  | Ada Layer Norm Zero | No | 0.226 | 0.14 % |
|  | Layer Norm | No | 0.000 | 0.00 % |
|  | Total: Flux Transformer Block, Full |  | 174.170 | 100 % |
|  | Linear (MLP Input Projection) | Yes | 72.480 | 41.66 % |
|  | Linear (MLP Output Projection) | Yes | 57.980 | 33.32 % |
|  | Joint Attention (Single-stream) | Yes | 43.490 | 24.99 % |
|  | Ada Layer Norm Zero Single | No | 0.057 | 0.03 % |
|  | GELU | No | 0.000 | 0.00 % |
| FLUX-1.dev | Total: Flux Transformer Block, Single |  | 174.000 | 100 % |

### A.3 Genetic Algorithm Evolutionary Step in Detail

Given a population $P_{g}$ of size $n$ at generation $g$, ECAD employs the NSGA-II algorithm [66, 58] to produce the next generation $P_{g+1}$ through the following steps:

- 1.
Selection and Offspring Generation: An offspring population $Q_{g}$, also of size $n$, is generated from $P_{g}$ via binary tournament selection by repeating the following process until $Q_{g}$ is filled. Two pairs of candidates are randomly sampled from $P_{g}$. Within each pair, a tournament is conducted by first comparing candidates by Pareto rank, then breaking ties using crowding distance. The winners from each pair undergo crossover, followed by mutation, to generate offspring.
- 2.
Crossover: With a probability of 0.9, we apply 4-point crossover to the binary caching tensors of the parent schedules. Four distinct crossover points are randomly selected along the flattened tensor, and two offspring are created by alternating segments between parents. With probability 0.1, the offspring are direct copies of their respective parents.
- 3.
Mutation: Each candidate in $Q_{g}$ undergoes bit-flip mutation with a probability of 0.05. If selected, each bit in the binary tensor $S\in\{0,1\}^{N\times B\times C}$ is independently flipped with probability $\frac{1}{N\times B\times C}$.
- 4.
Non-Dominated Sorting: The union $P_{g}\cup Q_{g}$ (size $2n$) is sorted into Pareto fronts $F_{0},F_{1},\ldots,F_{d}$ based on dominance. For each candidate $c$, we compute $\text{Dom}_{c}(R)$, the number of candidates that dominate $c$ in some set of candidates $R$. Fronts are defined iteratively as:
$\displaystyle F_{0}$
$\displaystyle:=\{c\in P_{g}\cup Q_{g}\mid\text{Dom}_{c}(P_{g}\cup Q_{g})=0\}$
$\displaystyle F_{1}$
$\displaystyle:=\{c\in(P_{g}\cup Q_{g})\setminus F_{0}\mid\text{Dom}_{c}((P_{g}
\cup Q_{g})\setminus F_{0})=0\}$
$\displaystyle\vdots$
$\displaystyle F_{i}$
$\displaystyle:=\{c\in(P_{g}\cup Q_{g})\setminus\bigcup_{j=0}^{i-1}F_{j}\mid
\text{Dom}_{c}((P_{g}\cup Q_{g})\setminus\bigcup_{j=0}^{i-1}F_{j})=0\}$
Note candidates in front $F_{i}$ are said to be of Pareto rank $i$; lower rank candidates are ‘fitter’ solutions.
Each front $F_{i}$ contains candidates not dominated by any candidate in fronts of higher rank.
- 5.
Population Selection: The next generation $P_{g+1}$ is filled by sequentially adding complete fronts $F_{0},F_{1},\ldots$ until the population size $n$ is reached. If a front $F_{k}$ cannot be fully accommodated, it is sorted by crowding distance. The most diverse candidates—those with the fewest close neighbors—are selected to fill the remaining slots, always including the extrema to preserve frontier diversity.

**Table 6: Parameters used for latency evaluation. $W$ is the number of warm-up batches discarded, $N$ is the number of batches used to compute the average latency, and $B$ is the largest batch size that fits in memory on a single NVIDIA A6000 GPU. All values are empirically chosen to ensure stable and consistent measurements.**
| Model Name | Resolution | Warm-up ($W$) | Measured ($N$) | Batch Size ($B$) |
| --- | --- | --- | --- | --- |
| PixArt-$\alpha$ | $256\times 256$ | 1 | 5 | 100 |
| PixArt-$\Sigma$ | $256\times 256$ | 1 | 5 | 100 |
| FLUX-1.dev | $256\times 256$ | 1 | 10 | 18 |
| FLUX-1.dev | $1024\times 1024$ | 5 | 25 | 3 |

### A.4 Population Initialization

We initialize the first generation of schedules for PixArt-$\alpha$ using a diverse set of heuristic strategies informed by prior work. Each heuristic varies caching behavior based on step/block selection patterns:

- •
Cross-Attention Only: Cache cross-attention at $s$ evenly spaced steps. At each selected step, cache the cross-attention of $b$ DiT blocks, evenly spaced across the total 28 blocks.
- •
Self-Attention Only: Identical to the above, but cache only self-attention.
- •
Feedforward Only: Identical to the above, but cache only feedforward layers.
- •
Cross- & Self-Attention, All Blocks: Cache both cross- and self-attention for all blocks at every $n$th step.
- •
FORA-inspired: Following [8], cache cross-attention, self-attention, and feedforward layers for all blocks at every $n$th step.
- •
TGATE-inspired: Following the gating mechanism from [15], set gate step $m$ and interval $k$. After the first two warm-up steps, compute self-attention every $k$ steps, caching and reusing otherwise. After step $m$, self-attention is computed every step, while cross-attention is not recomputed and reuses the cached output from step $m$. Unlike TGATE, which averages the cross attention activation on text and null-text embeddings, we cache only the the result from the text embedding.

The resulting Pareto frontiers for these heuristics are shown in Figure [8](https://arxiv.org/html/2506.15682v2#A1.F8). From the complete set of generated schedules, we randomly select 72 to initialize ECAD’s first generation for PixArt-$\alpha$.

For PixArt-$\Sigma$, as summarized in Section [4.2](https://arxiv.org/html/2506.15682v2#S4.SS2), we initialize with 72 schedules randomly sampled from the Pareto frontier of PixArt-$\alpha$ after 200 generations of ECAD optimization.

For FLUX-1.dev, we start with a FORA-inspired schedule, apply a few rounds of mutation and crossover, and randomly select 24 candidates to initialize ECAD.

Figure: Figure 8: Pareto frontiers of Image Reward vs. computational cost for the handcrafted schedules described in Section [A.4](https://arxiv.org/html/2506.15682v2#A1.SS4), evaluated on the Image Reward Benchmark. Notably, caching a single component (e.g., cross-attention or feedforward) offers slight gains over baseline. Among all heuristics, FORA achieves the best trade-off, with slightly lower quality but superior efficiency.
Refer to caption: x9.png

### A.5 Comparison to Concurrent Works

While our method is rigorously compared against prior works, it is more challenging to compare against concurrent works.
Nonetheless, for completeness and given its strong reported results, we include a comparison with DuCa [16], a concurrent follow-up to ToCa [14].

At a slightly higher acceleration, our “faster” configuration outperforms DuCa in both human preference (as measured by Image Reward) and FID across MJHQ and COCO datasets. At matching speedups of 2.6x, our “fastest” configuration modestly improves on DuCa’s Image Reward and CLIP scores, while achieving notably lower FID with reductions of approximately 30% and 20% on MJHQ and COCO respectively.

### A.6 MAC and Latency Computations

Latency Setup:
Latency measurements are conducted on a single NVIDIA A6000 GPU for all models. For each model, we discard the first $W$ warm-up batches and compute the mean latency over the subsequent $N$ measured batches, using prompts from the Image Reward Benchmark. The reported per-image latency is obtained by dividing the average batch latency by the batch size $B$, except in the case of ToCa (see section below). Detailed configuration parameters are provided in Table [6](https://arxiv.org/html/2506.15682v2#A1.T6).

ToCa Latency Results:
The publicly available ToCa (and DuCa) implementation differs substantially from the infrastructure employed in our framework. While both methods use the same GPU (NVIDIA A6000) and identical warm-up and batch settings, ToCa consistently produces higher latency measurements. To enable fair comparison, we normalize ToCa’s reported latencies by computing the relative speedup of each ToCa setting over its own baseline, then applying this speedup to our unaccelerated baseline latency:

$$ $\text{Normalized Latency}_{\text{ToCa}}=\frac{\text{Latency}_{\text{ToCa}}^{ \text{cached}}}{\text{Latency}_{\text{ToCa}}^{\text{unaccelerated}}}\times \text{Latency}_{\text{Ours}}^{\text{unaccelerated}}$ $$

This procedure ensures that the reported values reflect performance improvements relative to each method’s own baseline, enabling direct comparison across implementations. See Table [7](https://arxiv.org/html/2506.15682v2#A1.T7) for details.

ToCa MAC Results:
Multiply-accumulate operation (MAC) counts for ToCa are derived using the analytical formulations provided in the original work [14], specifically Section A.4. The relevant expressions are:

$$ $\displaystyle\text{MACs}_{SA}$ $\displaystyle\approx 4N_{1}D^{2}+2N_{1}^{2}D+\frac{5}{2}N_{1}^{2}H$ $\displaystyle\text{MACs}_{CA}$ $\displaystyle\approx 2D^{2}(N_{1}+N_{2})+2N_{1}N_{2}D+\frac{5}{2}N_{1}N_{2}H$ $\displaystyle\text{MACs}_{FFN}$ $\displaystyle\approx 8N_{1}D_{FFN}^{2}+12N_{1}D_{FFN}$ $$

Here, $N_{1}$ and $N_{2}$ denote the number of image and text tokens respectively, $D$ is the hidden state dimensionality, $D_{FFN}$ refers to the dimensionality within the feedforward network, and $H$ is the number of attention heads. Results from DuCa [16], a concurrent method that builds upon ToCa, confirm that these approximations closely match empirical MAC counts.

**Table 7: Latency normalization details for ToCa and DuCa across different models and resolutions. “True ms / img” refers to direct latency measured from the official implementation. “Speedup” is computed relative to each method’s own unaccelerated baseline, and “Normalized ms / img” applies that speedup to our unaccelerated latency for fair comparison.**
| Model | Resolution | Implementation | Caching | Setting | True ms / img$\downarrow$ | Speedup$\uparrow$ | Normalized ms / img$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Ours | None |  | 165.736 |  |  |
|  |  | ToCa | None |  | 948.688 | 1.000x | 165.736 |
|  |  | ToCa | ToCa | $\mathcal{N}=3,\mathcal{R}=60\%$ | 519.258 | 1.827x | 90.715 |
|  |  | ToCa | ToCa | $\mathcal{N}=3,\mathcal{R}=90\%$ | 403.989 | 2.348x | 70.577 |
|  |  | DuCa | None |  | 981.263 | 1.000x | 165.736 |
|  |  | DuCa | DuCa | $\mathcal{N}=3,\mathcal{R}=60\%$ | 429.405 | 2.285x | 72.527 |
| PixArt-$\alpha$ | $256{\times}256$ | DuCa | DuCa | $\mathcal{N}=3,\mathcal{R}=90\%$ | 379.411 | 2.586x | 64.083 |
|  |  | Ours | None |  | 167.624 |  |  |
|  |  | ToCa | None |  | 925.024 | 1.000x | 167.624 |
|  |  | ToCa | ToCa | $\mathcal{N}=3,\mathcal{R}=60\%$ | 520.286 | 1.778x | 94.281 |
| PixArt-$\Sigma$ | $256{\times}256$ | ToCa | ToCa | $\mathcal{N}=3,\mathcal{R}=90\%$ | 403.038 | 2.295x | 73.035 |
|  |  | Ours | None |  | 2620.095 |  |  |
|  |  | ToCa | None |  | 3385.153 | 1.000x | 2620.095 |
|  |  | ToCa | ToCa | $\mathcal{N}=4,\mathcal{R}=90\%$ | 2037.433 | 1.661x | 1576.965 |
|  | $256{\times}256$ | ToCa | ToCa | $\mathcal{N}=5,\mathcal{R}=90\%$ | 1935.554 | 1.747x | 1499.949 |
|  |  | Ours | None |  | 18297.603 |  |  |
|  |  | ToCa | None |  | 34109.719 | 1.000x | 18297.603 |
| FLUX-1.dev | $1024{\times}1024$ | ToCa | ToCa | $\mathcal{N}=4,\mathcal{R}=90\%$ | 13832.082 | 2.466x | 7419.995 |

### A.7 Additional ECAD Optimization Plots

Figure [9](https://arxiv.org/html/2506.15682v2#A1.F9) illustrates the progression of ECAD optimization for PixArt-$\Sigma$ and FLUX-1.dev at $256{\times}256$ resolution. PixArt-$\Sigma$ converges rapidly, likely due to its initialization from pre-optimized schedules learned on PixArt-$\alpha$. FLUX-1.dev converges to a steeper Pareto frontier, with its resulting schedules substantially outperforming the unaccelerated baseline on the Image Reward benchmark. We hypothesize that this steep convergence is facilitated by an initial population with a relatively high mean acceleration. See Section [A.4](https://arxiv.org/html/2506.15682v2#A1.SS4) for additional details on population initialization.

Additionally, we include the Pareto frontier of PixArt-$\Sigma$ as measured by Image Reward on the unseen PartiPrompts set vs. image generation latency in Figure [10](https://arxiv.org/html/2506.15682v2#A1.F10). Our method achieves Pareto dominance over FORA but does reach the unaccelerated baseline’s level of performance.

Figure: Figure 9: Progress of ECAD optimization as measured by Image Reward and TMACs. Left: PixArt-$\Sigma$ optimized for 50 generations, initialized using 200 generations of PixArt-$\alpha$ optimization. Right: FLUX-1.dev optimized for 250 generations, initialized using basic heuristics.
Refer to caption: x10.png

Figure: Figure 10: PartiPrompt Image Reward vs. latency for PixArt-$\Sigma$. Note that ToCa is not optimized for PixArt-$\Sigma$ and its parameters are transferred from PixArt-$\alpha$. Our method achieves Pareto dominance with a significant margin, but does not reach baseline performance.
Refer to caption: x12.png

**Table 8: Calibration prompt set ablation. Comparison of ECAD performance when calibrated on the Image Reward set vs. DrawBench200. Metrics include Image Reward (IR) on both th calibration and unseen prompts, MJHQ-30K FID, CLIP score, and latency. Each result reflects the highest-TMACs schedule from the Pareto frontier after 100 generations.**
| Calibration Prompt Set | # of calibration prompts | # imgs per prompt | ms / img$\downarrow$ (speedup$\uparrow$) | Image Reward Benchmark IR$\uparrow$ | DrawBench200 IR$\uparrow$ | PartiPrompts IR$\uparrow$ | FID$\downarrow$ | CLIP$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Image Reward Benchmark | 100 | 10 | 100.68 (1.65x) | 0.94 | 0.77 | 1.00 | 8.18 | 32.88 |
| DrawBench200 | 200 | 5 | 99.53 (1.67x) | 0.87 | 0.79 | 1.00 | 8.90 | 32.93 |

### A.8 Ablations

We provide evolutionary progress plots for ablations that modify genetic hyperparameters (population size, images per prompt, and number of prompts), complementing the Pareto frontiers shown in Figure [7](https://arxiv.org/html/2506.15682v2#S4.F7). Each ablation is evaluated independently and visualized in the following figures.

Figure [11](https://arxiv.org/html/2506.15682v2#A1.F11) illustrates the impact of reducing the population size. This setting results in slightly noisier frontiers and slight performance degradation across all metrics: the MJHQ-30K FID worsens slightly and latency increases by 22 ms over the baseline–the largest increase among all ablations. Figure [12](https://arxiv.org/html/2506.15682v2#A1.F12) examines the effect of reducing the number of images per prompt from 10 to 3, while keeping 100 prompts and a population of 72. This configuration achieves the fastest latency at 100.30 ms, the highest calibration Image Reward of 0.96, and the smallest increase in MJHQ-30K FID. In Figure [13](https://arxiv.org/html/2506.15682v2#A1.F13), we reduce the number of prompts from 100 to 33 while maintaining 10 images per prompt. This setup exhibits the cleanest convergence behavior but significantly underperforms on calibration Image Reward and its final Pareto frontier is dominated by other settings. However, its PartiPrompts score remains competitive and it produces the best FID, suggesting the subset of prompts were challenging and remained suitable for generalization. Detailed results for the highest-TMACs schedule after 100 generations under each hyperparameter setting are shown in Table [9](https://arxiv.org/html/2506.15682v2#A1.T9).

We also conduct an ablation comparing two calibration sets under fixed hyperparameters. Both runs use a population size of 72. The baseline configuration employs the Image Reward calibration set with 100 prompts and 10 images per prompt. The alternative uses the DrawBench200 set [67], which includes 200 prompts and 5 images per prompt, preserving the same total image count. Table [8](https://arxiv.org/html/2506.15682v2#A1.T8) reports performance across key metrics, and Figure [15](https://arxiv.org/html/2506.15682v2#A1.F15) displays their Pareto frontiers after 100 generations. As expected, each ECAD schedule performs best on the evaluation set corresponding to its calibration set. Notably, while most metrics are similar, the schedule calibrated on Image Reward significantly outperforms the one calibrated on DrawBench200 in MJHQ-30K FID. Moreover, the Image Reward–calibrated schedule demonstrates superior generalization to DrawBench200 compared to the reverse scenario, indicating that Image Reward provides a more robust calibration foundation for cross-dataset deployment.

Figure: Figure 11: ECAD optimization progress and final Pareto frontier using a reduced population size of 24 (compared to the default of 72), with 100 prompts and 10 images per prompt. The resulting frontiers are noisier and exhibit slower convergence.
Refer to caption: x13.png

### A.9 Visualizing ECAD Schedules

To better understand how ECAD optimizes caching schedules under different constraints and settings, we visualize selected schedules using heatmaps. Each heatmap represents a schedule, where red shades indicate cached components and gray shades indicate recomputed components. For PixArt models, the component order left-to-right is self-attention, cross-attention, and feedforward. FLUX-1.dev uses two types of DiT blocks. Block numbers 0 to 18 are ‘full’ FLUX DiT blocks, whose components are multi-stream joint-attention, feedforward, and feedforward context. Blocks 19 to 56 are ‘single’ blocks with components single-stream joint-attention, linear MLP input projection, and linear MLP output projection.
Figure [16](https://arxiv.org/html/2506.15682v2#A1.F16) and Figure [17](https://arxiv.org/html/2506.15682v2#A1.F17) show representative schedules for PixArt-$\alpha$ and PixArt-$\Sigma$ used throughout the paper. Figure [18](https://arxiv.org/html/2506.15682v2#A1.F18) compares FLUX-1.dev’s ‘slow’ and ‘fastest’ schedules.
Furthermore, Figure [19](https://arxiv.org/html/2506.15682v2#A1.F19) visualizes how ECAD schedules evolve over time for PixArt-$\alpha$, comparing the highest-TMACs candidate at generations 50, 200, and 400. Finally, Figure [20](https://arxiv.org/html/2506.15682v2#A1.F20) presents the highest-TMACs schedules resulting from our genetic hyperparameter ablations, illustrating how variations in population size impact the structure of learned caching strategies.

Figure: Figure 12: ECAD optimization progress and final Pareto frontier using only 3 images per prompt (default is 10), with 100 prompts and a population size of 72. This configuration demonstrates stable convergence and achieves stronger overall performance.
Refer to caption: x14.png

### A.10 Further Qualitative Results

In addition to the PixArt-$\alpha$ $256{\times}256$ results shown in Figure [4](https://arxiv.org/html/2506.15682v2#S4.F4), we present further qualitative comparisons using FLUX-1.dev at $256{\times}256$ (Figure [21](https://arxiv.org/html/2506.15682v2#A1.F21)) and $1024{\times}1024$ (Figure [22](https://arxiv.org/html/2506.15682v2#A1.F22)). Notably, in prompts such as “I want to supplement vitamin c, please help me paint related food,” our method exhibits stronger prompt adherence than both the uncached baseline and ToCa. This behavior is likely influenced by ECAD’s optimization for the Image Reward metric, which emphasizes semantic alignment with the prompt.

Full Prompts from Figure [4](https://arxiv.org/html/2506.15682v2#S4.F4), from left to right:

- •
“Three-quarters front view of a blue 1977 Porsche 911 coming around a curve in a mountain road and looking over a green valley on a cloudy day.”
- •
“a portrait of an old man”
- •
“A section of the Great Wall in the mountains. detailed charcoal sketch.”
- •
“a still life painting of a pair of shoes”
- •
“a blue cow is standing next to a tree with red leaves and yellow fruit. the cow is standing in a field with white flowers. impressionistic painting”
- •
“the Parthenon”

Full Prompts from Figure [21](https://arxiv.org/html/2506.15682v2#A1.F21), [22](https://arxiv.org/html/2506.15682v2#A1.F22), from top-to-bottom:

- •
“Drone view of waves crashing against the rugged cliffs along Big Sur’s Garay Point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore.”
- •
“Bright scene, aerial view, ancient city, fantasy, gorgeous light, mirror reflection, high detail, wide angle lens.”
- •
“3d digital art of an adorable ghost, glowing within, holding a heart shaped pumpkin, Halloween, super cute, spooky haunted house background”
- •
“8k uhd A man looks up at the starry sky, lonely and ethereal, Minimalism, Chaotic composition Op Art”
- •
“I want to supplement vitamin c, please help me paint related food.”
- •
“A deep forest clearing with a mirrored pond reflecting a galaxy-filled night sky.”
- •
“A person standing on the desert, desert waves, gossip illustration, half red, half blue, abstract image of sand, clear style, trendy illustration, outdoor, top view, clear style, precision art, ultra high definition image”

Figure: Figure 13: ECAD optimization progress and final Pareto frontier using only 33 prompts (a random subset of the default 100), with 10 images per prompt and population size 72. Although convergence is relatively smooth, the final frontier is constrained by the reduced prompt diversity.
Refer to caption: x15.png

Figure: Figure 14: Illustrative example of per-generation and overall Pareto frontiers in ECAD. Points represent candidate schedules, with lines interpolated between them for visualization. Half-colored points lie on both the generational and overall frontiers. In this example, the frontier from generation $G$ appears to exceed the overall frontier, highlighting interpolation ‘artifacts’ that can occur between discrete candidate solutions.
Refer to caption: x16.png

Figure: Figure 15: ECAD calibration prompt set ablation. We show performance change when using the DrawBench200 prompts benchmark set for calibration instead of the Image Reward set. Performance is measured in Image Reward (IR) on the both calibration prompts, unseen PartiPrompts, and MJHQ-30K FID and CLIP. Latency is provided as well. The schedule with the most TMACs that lies on the Pareto frontier across all 100 generations is used in each instance.
Refer to caption: x17.png

### A.11 Clarifying Frontier Visualizations

Several frontier plots–such as Figures [11](https://arxiv.org/html/2506.15682v2#A1.F11), [12](https://arxiv.org/html/2506.15682v2#A1.F12), and [13](https://arxiv.org/html/2506.15682v2#A1.F13)–show both the Pareto frontier of individual generations (typically shown in color) and the overall frontier aggregated across all generations (typically in black). At first glance, it may seem that a generational frontier occasionally surpasses the overall frontier.
This apparent contradiction arises from interpolation between discrete candidate schedules. As illustrated in Figure [14](https://arxiv.org/html/2506.15682v2#A1.F14), the frontier from generation $G$ appears to extend beyond the overall frontier. However, the aggregated frontier integrates more finely sampled points, including high-performing candidates from earlier generations (e.g., generation $G{-}1$), which are not always aligned with the interpolated curves of later generations. The overall frontier, therefore, forms a tighter envelope of all known Pareto-optimal schedules, even if it may visually appear to be exceeded due to interpolation artifacts.

**Table 9: Genetic hyperparameter ablations. Performance of ECAD when varying population size, number of images per prompt, and number of calibration prompts. We report latency, Image Reward on calibration and unseen PartiPrompts, and MJHQ-30K FID. Each result corresponds to the highest-TMACs schedule lying on the Pareto frontier after 100 generations.**
| Population Size | # imgs per prompt | # of calibration prompts | ms / img$\downarrow$ (speedup$\uparrow$) | Calibration IR$\uparrow$ | PartiPrompts IR$\uparrow$ | FID$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| 72 | 10 | 100 | 100.68 (1.65x) | 0.94 | 1.00 | 8.18 |
| 24 | 10 | 100 | 122.96 (1.35x) | 0.93 | 1.00 | 8.92 |
| 72 | 3 | 100 | 100.30 (1.65x) | 0.96 | 0.99 | 8.60 |
| 72 | 10 | 33 | 110.44 (1.50x) | 0.85 | 0.99 | 7.52 |

Figure: Figure 16: ECAD schedules for PixArt-$\alpha$ from Table [1](https://arxiv.org/html/2506.15682v2#S4.T1): “faster” (left) and “fastest” (right). Despite being separate schedules with no guarantee of relation, the “faster” schedule has near identical structure to “fast”, with more caching along steps 6 and 16. Furthermore, it appears cross-attention matters less than self-attention and the feedforward network during steps 16 and 17 and can safely be cached.
Refer to caption: x18.png

Figure: Figure 17: ECAD schedule for PixArt-$\Sigma$ “fast” from Table [1](https://arxiv.org/html/2506.15682v2#S4.T1). Initial DiT blocks in steps 6, 9, and 12 are more important to recompute than the final blocks. Cross-attention has less of an impact than the other components in the final three steps, with it as the only component cached in step 17.
Refer to caption: x20.png

Figure: Figure 18: ECAD schedules “slow” (left) and “fastest” (right) for FLUX-1.dev from Table [4](https://arxiv.org/html/2506.15682v2#S4.T4) and Table [1](https://arxiv.org/html/2506.15682v2#S4.T1) respectively. Despite being almost 200 generations apart, both schedules share similar structures for the first 5 steps, particularly at step 2 for blocks 9 through 12.
Refer to caption: x21.png

Figure: Figure 19: Highest-TMACs schedules from generation 50 (left), 200 (center), and 400 (right) during PixArt-$\alpha$ ECAD optimization. While steps between 8 and 15 remain somewhat similar in structure, early and late steps change more.
Refer to caption: x23.png

Figure: Figure 20: Highest-TMACs schedules after 100 generations for PixArt-$\alpha$ under different hyperparameter ablations: (top-left) reduced population size; (top-right) fewer images per prompt; (bottom-left) fewer prompts; (bottom-right) baseline configuration. All configurations realize the cacheability of cross attention for steps where other components cannot safely be cached.
Refer to caption: x26.png

Figure: Figure 21: FLUX-1.dev $256{\times}256$ qualitative comparisons. Displayed left-to-right are generations from the uncached baseline, ToCa ($\mathcal{N}=5,\mathcal{R}=90\%$; 1.75x speedup), and our “fast” ECAD schedule (Table [1](https://arxiv.org/html/2506.15682v2#S4.T1); 1.97x speedup). ECAD consistently yields sharper images with improved prompt adherence.
Refer to caption: x30.png

Figure: Figure 22: FLUX-1.dev $1024{\times}1024$ qualitative comparisons. Outputs, top-to-bottom, are ToCa ($\mathcal{N}=4,\mathcal{R}=90\%$; 2.47x speedup), and our “fast” ECAD schedule (as shown in Table [4](https://arxiv.org/html/2506.15682v2#S4.T4); 2.63x speedup). Our method yields greater visual complexity with stronger prompt-alignment, despite higher acceleration.
Refer to caption: x31.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How computationally heavy is ECAD’s optimization stage in practice (e.g., total GPU-hours)? Is the cost justified by the inference savings for typical users?
    
2. What are the performance trade-offs if the number of calibration prompts is reduced further (e.g., to 10 or 20)? How stable is ECAD under smaller calibration sets?
    
3. Could ECAD be combined with other acceleration strategies (e.g., distillation or quantization)? Would the Pareto optimization remain consistent?
soundness: 3
strengths: 1. **Novel framing and methodological clarity.**  
    The paper introduces a principled optimization perspective (Pareto frontier with NSGA-II) that replaces heuristic-based caching rules. This reframing is conceptually elegant and well-motivated.
    
2. **Training-free and model-agnostic.**  
    ECAD operates without any weight updates or retraining, making it practical for various diffusion models—especially for users without access to large compute resources.
    
3. **Comprehensive experimental evaluation.**  
    Evaluations cover multiple high-profile models (PixArt and FLUX families), a range of datasets (COCO, MJHQ-30K, PartiPrompts), and metrics (FID, CLIP, Image Reward), showing consistent improvements.on prompts, and different reward functions
    
4. **Clear writing and structured presentation.**  
    The paper is well-organized, readable, and includes detailed algorithmic pseudocode and visualizations.
summary: This paper, “Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model (ECAD)”, addresses the problem of slow and computationally intensive inference in diffusion-based generative models, especially Diffusion Transformers (DiTs). Traditional caching methods reuse previously computed features during sequential inference steps to reduce redundant computation, but existing approaches rely heavily on rigid heuristics, hand-tuned hyperparameters, and limited trade-off flexibility. ECAD reframes feature caching as a multi-objective optimization problem over image quality and computational efficiency, solved via a genetic algorithm (specifically, NSGA-II). The algorithm evolves caching schedules—which specify what components (self-attention, cross-attention, feedforward outputs) to cache, and when—to form a Pareto frontier of quality–latency trade-offs. ECAD is model-agnostic, training-free, and learns these schedules using only a small set of “calibration prompts” and a quality metric (e.g., Image Reward). Experiments on three major diffusion models (PixArt-α, PixArt-Σ, FLUX-1.dev) show that ECAD provides higher-quality outputs at faster inference speeds compared to previous caching baselines such as FORA, ToCa, DuCa, and TaylorSeer. The method is also shown to generalize across resolutions (e.g., 256×256 to 1024×1024) and related model architectures without retraining.
weaknesses: 1. **Compute inefficiency during optimization.**  
    Although no network training is required, ECAD’s optimization process (hundreds of generations, thousands of images per prompt) might still be computationally expensive. The paper could better quantify this cost relative to real-world savings.
    
2. **Reliance on automatic quality metrics.**  
    The optimization depends on Image Reward or similar automatic metrics, whose correlation to human preference can be imperfect or domain-specific. The paper does not show any human evaluation.
    
3. **Potential overfitting to calibration prompts.**  
    While generalization is discussed, there’s limited theoretical grounding for why a small prompt set suffices to generalize to unseen data.
    
4. **Reproducibility complexity.**  
    The large-scale evolutionary optimization (e.g., 500 generations × 72 candidates) may be difficult for others to reproduce without substantial GPU time.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - what's the generalizability of a schedule optimized for a certain reward, e.g. Image Reward, when evaluated with a different metric?
- how stable are the Pareto frontiers across random seeds and initializations of population, versus being seeded with heuristic schedules?
- can you analyze the selected prompt in terms of how representative they are of real-world use, and how sensitive is the performance of the proposed method towards prompts with different length, granularity, etc?
soundness: 3
strengths: - framing diffusion caching as a multi-objective Pareto optimization problem to balance quality and efficiency is novel.
- the proposed method is lightweight, and can adapt across prompts, model variants, and resolutions.
- strong empirical performance that achieves state-of-the-art inference acceleration with competitive image quality across several models.
- the paper is well-written and the comprehensive ablation studies in the appendix provide supports for the design choices.
summary: This paper introduces ECAD, a method that uses genetic algorithms to learn optimal caching schedules for accelerating Diffusion Transformers. The authors demonstrate speedups on models like PixArt and Flux while preserving or even improving image quality across multiple benchmarks.
weaknesses: - the performance is dependent on the choice of the quality metric used during optimization, and the caching schedules may not generalize perfectly to all possible evaluation metrics.
- the experiment setup (e.g., 100 prompts) may not be representative of complex prompts from real-world applications.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: see weakness
soundness: 3
strengths: + The proposed evolutIionary caching framework is novel and effective.
summary: This paper introduces Evolutionary Caching to Accelerate Diffusion models (ECAD), a genetic algorithm-based framework designed to optimize caching schedules for diffusion models, significantly improving inference speed without compromising image quality. Unlike heuristic-based methods, ECAD formulates caching as a Pareto optimization problem, enabling fine-grained trade-offs between speed and quality. The method is scalable, generalizable across model architectures, and requires no modifications to network parameters. Experiments on PixArt-α, PixArt-Σ, and FLUX-1.dev demonstrate state-of-the-art performance, achieving up to 3.37x speedups while maintaining or improving image quality metrics like FID and Image Reward. ECAD also exhibits strong generalization to unseen resolutions and model variants.
weaknesses: - The authors emphasize in the introduction that "there are no restrictions on batch size." However, it is unclear whether they measure generation performance per image and acceleration ratio under different batch sizes. Doing so would provide strong evidence to support this claim.

- For the strongest model, FLUX, it would be more compelling to evaluate its performance on more comprehensive benchmarks, such as GenEval and DPG-Bench, rather than relying solely on toy datasets.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: No
soundness: 3
strengths: 1. This paper presents a highly novel approach to the cache-based accelerated sampling problem in diffusion models by leveraging concepts from traditional machine learning. I find the most compelling contribution to be the elegant design of the cache schedule and, in particular, the use of an evolutionary algorithm to efficiently search for the optimal schedule. This represents a creative and effective way to frame and solve the problem. 💡

2. The experimental validation in this work is exceptionally thorough and convincing.

- Comprehensive Evaluation: The authors employ a suite of evaluation metrics, including FID, CLIP Score, and ImageReward, ensuring a multi-faceted assessment of generation quality. Their method is rigorously tested on a diverse range of powerful, state-of-the-art models like PixArt-α, PixArt-Σ, and FLUX-1.dev.

- Insightful Ablations and Visualizations: The paper is further strengthened by thorough ablation studies and clear visualizations. The ablation on the number of evolutionary generations, for instance, provides a clear insight into the trade-off between quality (FID) and efficiency (acceleration ratio), showing how FID first decreases before increasing while the speedup consistently grows. Furthermore, the visualizations of the discovered cache schedules and the points on the Pareto frontiers provide strong, intuitive evidence for the effectiveness of the proposed search strategy.

Overall, the rigorous and well-executed experiments strongly substantiate the claims made in the paper.
summary: This paper proposes a training-free accelerated sampling algorithm for diffusion transformers. The method achieves its speedup by caching and reusing features across multiple inference steps, thereby avoiding redundant computations within the model.

The core innovation of this work is the introduction of the Pareto frontier to construct efficient, model-specific caching schedules. By leveraging evolutionary algorithms to search for this frontier, the authors identify optimal schedules that achieve a favorable trade-off between inference speed and generation quality, using only a minimal set of calibration prompts.

The authors evaluate their method, ECAD, on state-of-the-art models such as PixArt-α, PixArt-Σ, and FLUX-1.dev. Their experiments are conducted across diverse benchmarks (COCO, MJHQ-30k, and PartiPrompts) and assessed with multiple metrics (FID, CLIP Score, and Image Reward), demonstrating consistent performance improvements over previous approaches.
weaknesses: I find several points in the paper confusing, primarily as follows:

1. On the Behavior of Optimization Objectives (ImageReward vs. MACs). I have a question regarding the behavior of the evaluation metrics during the evolutionary search. The paper's objectives are to maximize ImageReward (quality) and minimize MACs (computational cost). I understand how the NSGA-II algorithm operates to find a Pareto frontier for this multi-objective problem. However, the results show that as the number of generations increases, the MACs consistently decrease, while ImageReward does not exhibit a monotonic improvement; instead, it appears to fluctuate. Could the authors clarify this dynamic? My understanding is that while reducing MACs can be achieved by simply caching more, this aggressive caching might negatively impact image quality. Is the fluctuation in ImageReward a result of the algorithm exploring this trade-off, where some generations might sacrifice quality for significant computational savings before finding a better balance? A brief explanation of this trade-off during the search process would be beneficial.

2. On the Representation of the Cache Schedule (S). I know $S \in \{0,1\}^{N,B,C}$, but I don't seem to see $C$ in the visualization. I'd like to understand what role $C$ actually plays.

3. I observe two implicit but critical constraints in the cache schedules: Initial Step Constraint: 

- The first inference step (n=0) must always be a full recomputation, as there is no prior cache to reuse. This is reflected in the visualizations. 

- Intra-Step Dependency: Within a single inference step n, if a block b is set to recompute, all subsequent blocks (b+1,b+2,…) in that same step must also recompute. A reuse decision for a later block would be illogical, as its input depends on the output of the recomputed block b, invalidating the stored cache.

My question is: how are these logical constraints enforced during the evolutionary search, specifically within the Crossover and Mutation operations? A random mutation or crossover could easily generate an invalid schedule that violates these rules. Do the authors employ constraint-aware genetic operators, or is there a "repair" function that corrects invalid schedules after they are generated? Detailing this mechanism would strengthen the paper's description of the algorithm.


Decision: Accept (Poster)