Title: Diffusion on Demand: Selective Caching and Modulation for Efficient Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative models

diffusion

inference acceleration
Abstract: Diffusion transformers demonstrate significant potential for various generation tasks but are challenged by high computational cost. Recently, feature caching methods have been introduced to improve inference efficiency by storing features at certain timesteps and reusing them at subsequent timesteps. However, their effectiveness is limited as they rely only on choosing between cached features and performing model inference. Motivated by high cosine similarity between features across consecutive timesteps, we propose a cache-based framework that reuses features and selectively adapts them through linear modulation. In our framework, the selection is performed via a modulation gate, and both the gate and modulation parameters are learned. Extensive experiments show that our method achieves similar generation performance to the original sampler while requiring significantly less computation. For example, FLOPs and inference latency are reduced by $2.93\times$ and $2.15\times$ for DiT-XL/2 and by $2.83\times$ and $1.50\times$ for PixArt-$\alpha$, respectively. We find that modulation is effective when applied to as little as 2\% of layers, resulting in negligible computation overhead.


Peer reviews:
comments: SUMMARY:
The study presents an approach that accelerates diffusion transformer inference by storing intermediate features and conditionally adjusting them using lightweight, learned linear mappings, all while maintaining generation quality and significantly cutting computational demands.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The method achieves a substantial reduction in FLOPs and inference latency (up to ~2.9×), while preserving generation quality across different diffusion transformers (e.g., DiT-XL/2, PixArt-α, SD3-medium), which demonstrates its strong generalizability and practical utility.

2. The authors provide detailed studies on modulation thresholds, activation cycles, and learned gate patterns, offering useful insights into how and where modulation is most effective (e.g., later timesteps, deeper layers)

3. The paper is clearly structured and effectively organized, making its content easy to understand and follow.


Weakness&Question:

1. While linear modulation is efficient, the paper does not explore or compare stronger alternatives like lightweight nonlinear modules, which might offer better accuracy-efficiency trade-offs.

2. Can this method be integrated with other diffusion model acceleration techniques, such as model distillation and low-precision arithmetic?

QUESTIONS:
See Weakness

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-25

comments: SUMMARY:
The paper proposes a cache-based framework for diffusion transformers that improves inference efficiency by reusing features and selectively modulating them through learned linear transformations, achieving comparable generation quality with significantly reduced computational cost.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The manuscript is well-written, exhibiting a clear and coherent structure that facilitates comprehension.
2. Comprehensive and effective experiments: The method is validated across multiple models and datasets, outperforming SOTA caching methods like FORA and ToCa with comparable or lower computational cost.
3. Building on existing feature caching methods, the paper introduces a novel linear modulation scheme that goes beyond the traditional choice between caching and full inference, enabling more flexible feature reuse.
Weakness:
1. Increased pipeline complexity: Although the training overhead is small, the introduction of modulators and gating mechanisms adds extra training and threshold tuning compared to the heuristic caching method.
2. Limited modulation capacity? Applying modulation to all layers degrades quality, possibly due to the limited expressiveness of linear transformations. Could stronger modulators like MLPs or attention improve performance in more complex scenarios

QUESTIONS:
As an acceleration method based on the caching approach, FasterCache[1] also explores the estimation of feature evolution between adjacent time steps. What are the relationships and differences between our method and theirs?

[1] FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a selective caching framework that speeds up diffusion transformers by recycling intermediate features that look alike across neighboring timesteps. When a cached activation is sufficiently close to what the network would recompute, the model skips the full block and, at most, applies a learned scale-and-shift modulation. A tiny gate, trained with the rest of the transformer frozen, decides when this lightweight adjustment is needed. Because fewer than two percent of blocks receive modulation, the extra operations are almost free, and the approach can be inserted into existing samplers without architectural changes.\n\nTests on several diffusion transformers confirm the efficiency gains. With DiT-XL/2 the method cuts FLOP by about 3× and wall clock time by 2×, yet keeps FID, sFID, and IS comparable. Similar savings are observed on PixArt-α, SD3-medium, and Sana-0.6B, including high-resolution 1024 × 1024 synthesis. Across these models it consistently surpasses prior caching techniques such as FORA and ToCa, and it outperforms simple step-reduction baselines at matched compute budgets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Across four backbones the cache scheme cuts floating point operations and wall clock time, yet FID, sFID, and IS stay comparable of the original samplers at all tested resolutions.\n2.  This method is easy to integrate. Only a light gating head is trained while the diffusion transformer stays frozen, so existing checkpoints can adopt the method without any retraining of the large backbone.\n3. The experiments are comprehensive and solid. The authors evaluate DiT-XL/2, PixArt-α, SD3-medium, and Sana-0.6B at 256, 512, and 1024 pixel outputs, report both FLOPs and wall clock time, and run ablations on cache depth, gate threshold, and modulation cost. Results are matched to the compute budget of ToCa, FORA, and step-reduction baselines, which makes the gains in speed and quality easy to verify.\n\nWeakness:\n1. The method section could be made more accessible.\n2. The ablation study uncovers a counter-intuitive trend: when the gate threshold is set low and many blocks are modulated, FID actually increases and image quality drops. This unexpected behaviour shows that the method is sensitive to the threshold and that excessive modulation can harm, rather than help, performance.\n\nQUESTIONS:\nN/A.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}