Faster Diffusion Through Temporal Attention Decomposition
Authors: Haozhe Liu
Wentian Zhang
Jinheng Xie
Francesco Faccio
Mengmeng Xu
Tao Xiang
Mike Zheng Shou
Juan-Manuel Perez-Rua
Jürgen Schmidhuber
Venue: Accepted by TMLR ()

Abstract: We explore the role of the attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. Self-attention, however, initially plays a minor role but becomes increasingly important in the second phase. These findings yield a simple and training-free method called TGATE which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experiments show TGATE’s broad applicability to various existing text-conditional diffusion models which it speeds up by 10-50%.  The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

Published in Transactions on Machine Learning Research (02/2025)
Faster Diffusion via Temporal Attention Decomposition
Haozhe Liu 1,4,6, Wentian Zhang JJineeng Xie 2,1, Francesco Faccio 1.3 Mengmeng Xu 4, Tao Xiang 4, Mike
Zheng Shou 2, Juan-Manuel Perez-Rua 4, Jürgen Schmidhuber
Center of Excellence for Generative AI, King Abdullah University Science and Technology (KAUST)
Show Lob, National University of Singapore (NUS)
Swiss Al Lab, IDSIA, USI SUPSI, Lugano" Meta Al
Reviewed on OpenReview:
Latency (s)
61.50
Baseline
60
46.65
w/ TGATE (CA) 34% uccelerate
40
20
14,85
TGATE
(SA,CA)
47% accelcrate
Total
Attentions
Others
(a) Latency Analysis
(b) Generated Samples
Figure 1: While generating a 1024 1024 image over 25 steps using well-known text-to-image diffusion model (Chen et al., 2024b), analyze the latency contributions of its various components. The major computational bot- tleneck is the attention mechanism, which exhibits a key pattern during inference: cross-attention is initially essential, but its significance decreases over time. Conversely, self-attention has minimal initial impact but becomes crucial over time. This allows for caching attention maps and reusing them when they are less crucial, therehy considerably speeding up inference with slight impact on generation quality, as illustrated in (b).
Abstract
We explore the role of attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross- cosssatttntionn outputs converge to fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images subsequent -iderii-i phase. Cross-attention essential the initial phase but almost irrelevant thereafter. However, self-attention initially plays a minor role but becomes crucial in the second phase. These findings yield simple and training-free free method known temporally gating the attention (TGATE), which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experimental results show when widely applied to various existing text- conditional diffusion models, TGATE accelerates these models by 10%-50% The code of TGATE available at https: hitthbb...../ cwm/ Hww.hhheii-S///-AAATTTTT
"Corresponding Author: haozhe. 1 kaust. edu. sa
Equul Contribution

---

Published in Transactions on Machine Learning Research (02/2025)
1 Introduction
A small leak will sink great ship."
Benjamin Franklin
Ho ct al. 2020) have been widely used for image generation. Featuring an attention mechanism (Vaswani et al. 2017 Schmidhuber 1992b). they align different modalities (Rombach et al. 2022), including text, to generate high-quality images and videos. Several studies highlight the importance of attention for spatial control (Xie al. 2023] Hertz et al. 2023 Chefer et al. 2023); however, only few have investigated its
Diffusion models (Jazzysski 1997 Neal
2001
role from temporal perspective during denoising.
The attention module generally comprises self-attention,, which processes the context across spatial positions, and cross-attention, which integrates signals from various modalities. Understanding the roles of these components during inference is key to comprehend the overall model behavior. We analyze their contributions at different time steps and gained three critical insights:
Cross-attention outputs convergence to fixed point in first several steps. Accordingly, the time point of convergence divides the denoising of diffusion models into two phases: i) an initial phase, during which the model relies on cross-attention to plan text-oriented visual semantics; this denoted as the semantic-planning phase, and ii) a subsequent phase, during which the model learns to generate images from previous semantic planning: this is referred to as the fidelity-improving phase.
Cruss-attention is redundant the fidelity-improving phase. During the semantics comantics-planning phase, cross- attention plays a crucial role in creating meaningful semantics. However, in the latter phase, it converges and has minor impact on image generation. Bypassing cross-attention during the fidelity-iproviing phase can indeed potentially reduce computational costs while maintaining the image generation quality.
Self-attention is largely redundant in the semantics-planning phase. Unlike pross-attention self- attentttn evidently plays a significant role the later phase. However, its contribution is limited the early semanties- planning phase. By selectively skipping self-attention during this phase, the inference process can be further accelerated with only minor impact generation.
Notably, the scaled dot product in the attention mechanism a quadratic complexity operation. As the resolution and token length in modern models increase, attention mechanism inevitably increases computational costs and becomes significant source of latency (Li et al. 2024). Thus, the role of attention mechanism must be re-evaluated; moreover, the aforementioned shortcoming inspires us to design simple, effective, and training-fres free method, i.e., temporally gating the attention (TGATE), to improve the efficiency and maintain the quality of images generated by off-the-shelf diffusion models. The principal observations with respect to TGATE are as follows:
TGATE increases efficiency by caching and reusing the cross-attention outcomes when they are rendered useless, thereby eliminating the calculation of redundant attention. This strategy does not affect the model performance, as the predictions of cross-attentinn converge and are potentially redundant.
TGATE is training-free and has broad applicability in text-to-image and text-to-video models, and supports U-Net and transformer-basee architectures. It is also orthogonal to different noise schedulers and acceleration methods.
TGATE can further accelerate diffusion models by dynamically caching and reusing self atffaattention predictions during the initial phase. In extreme cases, TGATE reduces the multiply-accumulad (MAC) operation of PixArt-Alpha from 107 T to 64 T and cuts its latency from 62 to 33 s on a 1080Ti commercial card, thereby enhancing the efficiency without considerably impacting the performance.
2 Preliminary
Diffusion technique has rich history, dating back nonequilibrium statistical physics (Jarzynski) 1997) and annealed importance sampling (Neal) 2001). This mechanism characterized by its scalability and stability, (Dhariwal & Nichol
2

---

Published in Transactions on Machine Learning Research (02/2025)
2021 Ho al. 2020), has been widely used in modern text-conditional generative models (Rombach al. 2022
Ramesh et al. 2022 2021 [Saharia et al. 2022 Chen et al., 2024c Brooks et al. 2024).
Learning Objective. Herein, the formulation introduced by LDM (Rombach et al. 2022 is used to construct a latent
diffusion model comprising four main components: an image encoder E()), an image decoder D(-), a denoising model c)))) and text embedding c. The learning objective for this model is defined as follows:
Le = Esp A. [[|c - (0(2, t, c)
(1)
where e-()) is designed to accurately estimate noise added to the current latent representation at time step / [1, that is conditioned on text embedding c. During inference, ea(2,, t, c) is called multiple times to recover 20 from 22. where 20 is decoded into an image x using D(20).
Inference Stage. In this stage, classifier-free guidance (CFG) (Ho Salimans 2022 commonly employed to incorporate conditional guidance as follows:
(, t, w. c) = co (ze, Ø)) wlea (Zt, t, (Zt, 4 Ø)),
(2)
where Ø represents the embedding of null text, i.e., "", w is the guidance scale parameter, and c. implicitly esti- mates p(c|z) & p(zlc)/p(z) to guide conditional generation p(=/e) x p(zlc)p" (c|z). In particular, V Vlo[(p(c(2)) x V. log(p(z|c)) - V. log(p(=)), which identical to Eq,
Attention Mechanism. the denoising model eg, each block extensively integrates the attention mechanism. Specif- ically, the self- self-attention module captures context across spatial positions, whereas the cross-attention module enables interactions with various input modalities, including text. The attention process is mathematically defined as follows:
Q O Vd
C Softmax
V.,
where Q' represents projection of zt. For cross-attention, K and V are projections of the text embedding c. However, in self-attention, they are derived from 2,. d denotes the feature dimension of K. This mechanism can be understood querying learnable key-value codes, where each token's prediction is derived from weighted sum of all values (V). The weights are determined by the similarity between the query (Q) and the corresponding keys (K) Despite its effectiveness,, the attention mechanism acts as a significant computational bottleneck when processing high-resolution input due to its quadratic computational complexity.
3 Temporal Analysis of Attention Mechanism
Here, the role and functionality of attention mechanism in the inference stage of well-trained diffusion model are discussed. First, empirical observation of cross- attention map convergence is discussed in Section 3.1 fol- lowed by systematic analysis of this observation in Sec- tion 3.2 Section 3.3 concludes with follow-up analysis of self-attention.
Cross-attention Difference
0.0015
0.0010
0.0005
0.0000
3.1 Convergence of Cross-Attention Map
10
15
20 25
Cross-attention mechanisms provide textual guidance at each step in diffusion models. However, the shifts in the noise input across these steps pose this question: Do the feature maps generated by cross-attention exhibit temporal stability, or do they fluctuate over time?
Inference Steps
Figure 2: Difference in cross-attention maps between two consecutive inference steps the MS-COCO dataset. Each data point in the figure is an average of 1,000 captions and all cross-attention maps within the model. The shaded area indicates the variance, whereas the curve demonstrates that the difference between con- secutive steps progressively approaches zero.
To find an answer, we randomly collect 1,000 captions from the MS-COCO dataset and generate images using
3

---

Published in Transactions on Machine Learning Research (02/2025)
pre-trained SD-2.1 mode which is based on DPM solver (Luu et al. 2022) with 25 inference steps. During inference, we calculate the L2 distance between C' and C where C' represents the cross-attention maps at time step t. The difference in cross- cooss-attention between the two steps is calculated by averaging L.2 distances among all input captions. conditions, and depths.
Fig. shows the variation in cross-attention differences across various inference steps. A clear trend is visible, showing gradual convergence of differences toward zero. Convergence always appears within 5- 5-10 inference steps. Therefore, cross-attention maps converge to fixed point and do not offer dynamic guidance for image generation. This finding supports the effectiveness of CFG with respect to cross-attention, demonstrating that despite varying conditions and initial noise, unconditional and conditional batches can converge toward single, consistent result (Castillo et al. 2023). We also track the cross- coos-attentionn differences across different blocks; refer to Appendix E for details.
This phenomenon shows that the impact of cross-attention during inference process is not uniform and inspires the temporal analysis of cross-attention.
3.2 Role of Cross-Attention in Inference
Analytical Tool. Existing analysis (Ma et al. 2024) shows that the consecutive inference steps of diffusion models have similar denoising behaviors. Inspired by behavioral explanation (Bau et al., 2020 Liu et al. 2023) ) we measure the impact of cross-attention by effectively "removing" it at a specific phase and observing the resulting difference in the image generation quality. In practice, this removal is approximated by substituting the original text embedding with a placeholder for null text, i.e., We formalize the standard denoising trajectory as a sequence as follows:
S = (eo(inn ()eec())- (.(=,),,,,.. cc(;
(4)
where we omit the time step / and guidance scale w for simplicity. The image generated from sequence S is denoted by x. This standard sequence is then modified by replacing the conditional text embedding c with the null text embedding @ over a specified inference interval, resulting two new sequences, S. and S)., based on a scalar m as follows:
s { c)2 c)( ... 2)}, S). (Ec(Zn) () D),... (e(=mm)); ,)),,,,,,,,(,,))))
(5)
Here, m serves as a gate step that splits the trajectory into two phases. In sequence S the null text embedding Ø replaces the original text embedding e for the steps from m + 1 to n. In contrast, in sequence st. the steps from I to m use the null text embedding Ø instead of the original text embedding c, whereas the steps from m to n continue to use the original text embedding c. The images generated from these two trajectories are denoted as of and 2/, respectively. To determine the impact of cross-attention at different phases, the differences in generation quality among 2, 2/,-- and z.M compared. If the image generation quality among x and ay are considerably different, it indicates the importance of cross-attention at that phase. If the quality does not vary considerably, the inclusion of cross-attention may not be necessary.
Herein, SD-2.1 is used as the model, and the DPM solver (Lu et al. 2022) is used for noise scheduling. The total inference step in all experiments is set as 25. The text prompt, "High quality photo of an astronaut riding a horse in space." is used for visualization.
Results and Discussions. Fig. ((a) shows the trajectory of the mean of predicted noise, which empirically shows that denoising converges after 25 inference steps. Therefore, analyzing the impact of cross-attention within this interval is difficult. As shown in Fig. (b), the gate step m is set to 10, which yields three trajectories: S, SE and st,, The visualization illustrates that ignoring the cross-attention after 10 steps does not influence the final outcome. However, a notable disparity is observed after bypassing cross-attention in the initial steps. As shown in Fig. B(c), the image generation quality (Fréchet inception distance, FID) considerably deteriorates in the MS-COCO validation set due to this elimination. The resulting quality is even worse than the weak baseline that generates images without CFG. We then generalize these assessments range of gate steps, inference numbers, noise schedulers, and base models. The experimental results consistently show that the FIDs of SF are slightly better than the bascline S and outperform
4

---

Published in Transactions on Machine Learning Research (02/2025)
(a)
(b)
Semantics-Plamnnn Phase
Sebelity-Improviggg Phase
Meun of Predictdd Noiso
Impporyyn Phase
Text Embeekiig
w/ Text Embedking
0.5
Semantics: Pemantis--Panmingg
0.29
Phase
w/ Text Embedding
w/ Nuil Text
0.00
10
20
(c)
Inference steps
50
w/ Nall Text
Text Embedding
40
30
Shi
20
19.352
38.194
10-
15
Baseline
Sm
S
Inference steps
Figure 3: Impact of cross-attention on the inference steps in a pre-trained diffusion model, i.e, stable diffusion 2.1 (SD-2.1). (a) The mean of the noise predicted at each inference step. (b) Images generated by the diffusion model at different inference steps. The first row, S, in (b) feeds text embedding to cross-attention modules for all steps, the second row, S only uses text embedding from the first step to the m-th step, and the third row, S inputs text embedding from the m-th to the n-th step. (c) Zero-shot FID scores based on these three settings on the MS- COCO validation set (Lin et al. 2014), with the baseline defined as conditional generation without CFG. Here, FID is calculated using the full COCO validation set (Lin et al. 2014).
S by wide margin. These empirical observations consistently underscore the broad applicability of the reported findings over different configurations. Appendix A details these results.
These analyses can be summarized as follows:
Cross-attention converges early during inference, which can be characterized by semantics-plannine and fidelity-mmrrviing phases. The impact of cross-attention is not uniform in these two phases.
Cross- attention in the mmnantics-planning phase is significant for generating semantics aligned with the text conditions.
The fidelity-iprrviing phase mainly improves the image quality without requiring cross-attention. FID scores can be slightly improved via null-text embedding this phase.
3.3 Role of Self-Attention in Inference
Analytical Tool Unlike cross-attention, the direct removal of self-attention during inference is not straightforward process, as the performance deteriorates considerably. Thus, determining the specific contributions of each time step is challenging. To address this issue, a novel analytical approach involving caching and reusing features is proposed. As our core premise, if the features of self-attention can be cached and reused across multiple steps without performance decline, they may be considered less critical. Building on this concept and drawing parallels with cross-attention analysis, two distinct trajectories, g and S are introduced.
In SF, self-attention features are cached and reused in the k. Specifically, after m inference steps, self-attention is reused for steps, and self-attention prediction is updated once for the next k-step reuse cycle. Contrarily, St skips self-attention during the initial semantics momantics-plannng phase for every step after several warm-up steps, typically set at 2. Then, self-attention is fully integrated into inference after m steps. The performance difference between S and Sh. indicates the contribution of self-atee atffaattention at different phases.
phase using gate step m and interval
5

---

Published in Transactions on Machine Learning Research (02/2025)
Sh
$4
Inference steps
Figure 4: Illustration of the impact of self-attention on inference in Stable Diffusion 2.1 (SD-221). The base trajectory, S. does not use eached self-attention features. In S these features cached and reused during the iimelityi phase. Conversely, S bypasses self-attention in the initial semantics-planning phase. The interval is set and the gate step to 10. The visualization shows that caching self-attention in the semantics-plannin phase does not significantly affect the generation result. The input prompt is "paisaje montañoso nevado".
Results and Discussions As shown in Fig. the visualization convincingly suggests that sell-attention is more important in the latter inference phases. For quantitative analysis, various experiments are conducted using different values of gate step m and interval k, which are detailed in Appendix
The observations from the analysis can be summarized as follows:
Unlike cross-attention, bypassing self-attention increases FID scores, indicating quality degradation. How- ever, selectively skipping during the semantics-plamiing phase results in a minor, manageable performance drop.
By increasing the interval for reusing the features, the efficiency be improved but the cost of perfor- mance, suggesting that it cannot be removed totally in the semantics-planning phase.
4 Proposed Method TGATE
Results of the empirical study show that self-attention and cross-attention in the initial and last inference steps, respec- tively, redundant. However, it nontrivial to drop/replace attention modules without retraining the model. To this end, effective and training-free free method is proposed herein: TGATE. This method caches the attention outcomes and reuses them throughout the scheduled time steps.
4.1 Skipping Cross-Attention in the Fidelity-lmproving Phase
Caching Cross-Attention Maps. Suppose is the gate step for the phase transition. In the m-th step and i-th cross- attention module, two cross-attention maps, Cmm.i and C can be accessed from CFG-based inference. The average of these two maps is calculated to serve an anchor and store it in fist-in-first-onn feature cache F. After traversing all the cross-attention blocks, can be written as follows:
F p= 2 (c
C. [1,4]}
(6)

---

Published in Transactions on Machine Learning Research (02/2025)
where 1 denotes the total number of cross-attentie modules.
Re-using Cached Cross-Attention Maps. In each step of the dilelit-iimproingg phase, when a cross-attention op- eration is performed during the forward pass, it is omitted from the computation graph. Instead, the cached F [i] is fed into subsequent computations. This approach does not yield identical predictions at each step. as a residual con- nection (Hochreiter) 1991 Srivastava et al. 2015; He et al. 2016) in the neural networks allows the model to bypass cross-attention.
Semantics- Planniing Stage
Fidelity-lmproving Stage
xk
xk
SA
SA
SA
SA
SA
SA
SA
SA
SA
CA
CA
CA
CA
CA
CA
CA
CA
CA
Warm Up
Iteratively Caching and Re-using SA
Totally Re-using CA
Figure 5: Pipeline of TGATE for accelerating inference. During the semantics planiinng phase, cross-attention (CA) continuously active, whereas self-attention (SA) applied every steps following an initial warm-up period to conserve computational resources. In the fidelity-improv/e phase, cross-attention is substituted with a caching mechanism, and self-attention remains operational.
4.2 Skipping Self-Attention in Cemantics-Planniine Phase
The analysis described in Appendix B demonstrates that self-attention contributes mainly to the second phase, sug- gesting a reduction in its usage in the first phase. However, unlike cross-attention, self-attention cannot be entirely bypassed without considerably degrading the capacity and performance of the model. An interval caching strategy is introduced to preserve the generation performance. steps, output from all blocks is cached and reused for every k step in the semantics-planning phase. In the fidelity- improving phase, self-attention is fully operational. The pipeline of TGATE is detailed in Fig.
particular, after activating self-attention with initial warm-up
5 Related Works
Herein, the role and functionality of cross-attention within diffusion trajectories are analyzed. These factors have been previously studied from different perspectives. Spectral diffusion (Yang et al., 2023) traces diffusion trajectory via frequency analysis and finds that the diffusion model restores an image from varying frequuency components at each step. T-stitch (Pan et al. 2024) shows that at the beginning of the inference, different models generated similar noise. This finding suggests that smaller model can produce the same noise as a larger one, thereby considerably reducing the computational costs. By analyzing prompt switching effects, eDiff-I (Balaji al. 2022) reveals that diffusion models respond to text prompts with distinct temporal dynamics, showing better comprehension of text signals in early denoising steps and diminishing in later ones. Adaptive guidance (Castillo et al. 2023) models diffusion trajectory as graph and applies neural architecture search (NAS) to automatically identify the importance of each step. This approach identifies CFG (Ho & Salimans| 2022) as redundant operator in some inference steps, suggesting the removal of unconditional batch for accelerating the generation speed. Building on similar foundations, recent studies (Kynkäänniemi et al.] 20244 Sadat et al.) 2023) suggest that strategically dropping or modifying the CFG scale can enhance generation performance. As per DeepCache (Ma et al. 2024), predictions from each block contain temporal similarities in consecutive time steps. Thus, reutilizing predictions from these blocks can improve the efficiency of the inference. Wimbauer et al. (Wimbauer et al. 2023) propose contemporary work for caching block features; however,
it requires resource-friendly training process.
To the best of our knowledge, this study is orthogonal to the existing studies. We observe that cross-attention and self-attention non-uniformly yet independently (almost complementarily) contribute to the final generated samples across various time steps. Therefore, attention outcomes can be selectively copied and reused in certain inference steps without affecting the generation performance. This may inspire several new studies toward developing faster diffusion models.
7

---

Published in Transactions on Machine Learning Research (02/2025)
wi TGATE
W/ TOATE
(CA)
w/ TGATE (SA, (SA,CA) CA)
(a) PixArt-Alpha
(b) SDXL
Figure 6: Samples generated from (a) PixArt-Alpha and (b) SDXL with or without TGATE given the same initial noise and captions. The visualization of PixArt is generated using two configurations, i.e., m=15 or (m=15,k=5) For SDXL, the configuration is m=10 and (m=10,k=5). Refer to Appendix E for visualizations with different hyper- parameters (m and k).
6 Experimental Results
The proposed method is integrated into several state-of-the-art diffusion models, including SD-series (Rombach et al.
2022 Podell et al. 2023 Blattmann et al. 2023). PixArt (Chen et al. 2024b), and OpenSora (Lab & etc. 2024).
2023 Li et al., 2023 Lab & etc. 2024), comprehensive experiments conducted using the MS-COCO (Lin al. 2014), MJHQ LLi al. 2023). OpenSora-Sample (Lab & etc. 2024) and DPG-Bench (Hu et al. 2024) datasets. The inference configuration, including the number of inference steps and the noise scheduler, follows the default settings for each model. Additionally, the proposed method is compared with other accelerating methods, such as the latent consistency model (Luo et (203)), adaptive guidance (Castillo et al. 2023). and DeepCache (Ma et al. 2024). Latency and MACs are considered the metrics for efficiency. MACs stand for Muuliply-Accumulile Operations per image, which is automatically generated using Calflops (xiaoju
Following established evaluation protocols (Podell
ye 2023). The generation performance evaluated using FID (Heusel et al. 2017), CLIP score (Radford et al. 2021 and DPG-Score (Hu et al.
2024). More details are given Appendix C
Table 1: Computational complexity, latency, and FID on the MJHQ-IOK (Li et al. 2023) using the base model of
PixArt-Alpha (Chen et al. 2024b). The latency of generating one image is tested on a 1080 Ti commercial card. FID
Heusel et al. 2017) is used to test the performance PixArt and CLIP score (Radford et al. 2021) for OpenSora
Lab & etc. 2024).
Inference Method
Caching Modules
MACs
Latency
Generation Performance (FID)
PixArt-Alpha
107. 077.031T
61.502s
9.653
PixArt-Alpha+ TGATE (m=10)
CA
70.225T
40.648s
11,268
PixArt-Alhha TGATE (m= 15)
CA
82,494T
47.599s
9.548
PixArt-Alpha TGATE (m=10 l=3))
CA.SA
65.355T
34.391s
11.789
PixArt-Alpha TGATE (m= 10, k=5)
CA.SA
64.138T
32.827s
12.738
PixAntAAphaa TGATE (m=15, 15, k=3)
CA.SA
73.9711
36.650s
10.289
PixArt-Alpha TGATE (m=115 k=))
CA.SA
71.536T
33.521s
11.298
Inference Method
Caching Modules
MACs
Latency
Generation Performance (CLIP)
OpenSora (250 inference steps)
8417.500T
178.913m
31.211
OpenSora TGATE (m=100)
CA
5696.500T
120.826m
30.779
OpenSora TGATE (m=1000 k=))
CA.SA
5370,7901
94.393m
30.766
OpenSora TGATE (m=1000, k=))
CA,SA
4298.410T
88.519m
30.683
OpenSora TGATE (m=1000, 1=
CA.SA
5246.710T
84.323m
30.161
8

---

Published in Transactions on Machine Learning Research (02/2025)
6.1 Improvement over Transformer-based Models
The proposed method is integrated into PixArt-Alpha (Chen al. 2024b), text conditional model based on trans- former architecture (Peebles & Xie 2023 Vaswani al. 2017) As shown in Table | TGATE considerably accel- erates the inference speed across various configurations. Notably, by setting m to 15, TGATE enhances the efficiency and slightly reduces the FIDs to 9.548. Additionally. configuring TGATE with m 10 and k 5 further reduces computational demands while only moderately impacting the performance. This configuration only requires 64.1 138T MACs to generate a single image, thereby reducing the latency by nearly in half-from 61. 61,502s to 32.827s. Notably, existing studies such as DeepCache (Ma al.) 2024) and BlockCache (Wimbauer et al.) 2023) rely on the skipping architecture of U-Net (Ronneberger al. 2015) to cache features for acceleration. However, these approaches are unsuitable for transformers due their different architectural demands. To address this research gap, TGATE decom- poses the contribution of attention mechanisms and reuses the features in redundant steps. This represents the first step to freely accelerate transformers by caching features.
Additionally, TGATE and its base models are qualitatively compared herein. Fig. shows the images generated by different base models with or without TGATE. Although TGATE increases FID scores some configurations, changes generated samples are nearly imperceptible and demonstrate the effectiveness of TGATE in maintaining performance. More visualizations available in Appendix H and more analysis, including frame consistency, text- image alignment and memory cost analysis, is provided in Appendices and
Table 2: Computational complexity, latency, and FID on the MS-COCO validation set using the base model of SD- 1.5, SD-2.1, and SDXL. MACs stands for DactiplyAAccumulate Operations per image. These terms are automatically generated using Calflops. The latency of generating one image tested on 1080 Ti commercial card. FID (Heusel
et 2017 used test the performance of text-to-image models (Rombach et al. 2022 Podell et al. 2023) and
CLIP score RRadford al. 2021) for video models (Blattmann et al. 2023).
Inference Method
MAC'
Latency
Generation Performance (FID)
SD-1.5
16.938T
7.032s
23.927
SD-1.5 + TGATE (m==))
9.875T
4.3138
20.789
SD-1.5 + TGATE (m=10)
11.641T
4.993s
23.269
SD-2.1
38.04IT
16.121s
22.609
SD-2.1 TGATE (m=5)
22.208T
9.878s
19.940
SD-2.1 TGATE (m=10)
16.166T
11.372s
21.294
SDXL
149.438T
53.187s
24.628
SDXL + TGATE ()==))
84.438T
27.932s
22.738
SDXL + TGATE (m=10)
100.688T
34.246s
23,433
SDXI. TGATE (m=5 k=3)
83,498T
27.412s
22.306
SDXL TGATE (m= 10 k=))
96.928
32.164s
22.763
SDXL + TGATE (m= 10 5=
95.988T
31.643s
23,839
Inference Method
MACs
Latency
Generation Performance (CI.IP
SVD
1609.250T
645.842s
31.322
SVD TGATE (m=5)
935.650T
408.485s
31.176
SVD + TGATE (m=10)
1104.050T
467.824s
31.334
SVD TGATE (m=5 lm3)
932.780T
402.044s
31.167
SVD TGATE (m=100 k=33)
1092.570T
442.060s
31.358
SVD TGATE (m= 10 k=5)
1089.700T
435.619%
31.343
6.2 Improvement over U-Net-based Models
TGATE can also be applied to U-Net-based models (Rombach et al. 2022 Podell et al. 2023). For all settings shown in Table TGATE enhances the performance of the base models in terms of computational efficiency and FID scores. In particular, TGATE works better when the parameter size of the hase model increases. In SDXI., TGATE can reduce
2 technology with rich history, which be dated back the principles of the 1991 unnormalized linear Transforner Schmidhuher 1992a
Schlag al. 202))

---

Published in Transactions on Machine Learning Research (02/2025)
the latency by half on the commercial GPU card (from 53.187 to 27.412 s). This indicates the effectiveness and
scalability of TGATE on U-Net-based diffusion models. Qualitative analysis is given in Fig. G and Appendix F and the evaluation of text alignment is discussed in Appendix H
6.3 Improvement over Acceleration Models
Table 3: Computational complexity, latency, and FID us-
Table 4: Comparison with Adaptive Guidance (Castillo et al. 2023) on the MS-COCO validation set based on SDXL and Pixart-Alpha.
ing the LCM distilled from SDXL Podell et al. (0033
and PixelArt-Alpha Chen et al. 2024b).
Inference Method
MACs
Latency
FID 4
Inference Method
MACs
Lateney
FID
L.CM (SDXI., n=4)
11,955T
3.805s
25.044
SDXL
149 438T
53. 187s
24.628
LCM (SDXL, n=
17.933T
5.708s
25.630
w/ Adaptive Guidance
104.606T
35.5384
23.301
LCM (SDXL, n=
23.910T
7.611s
27.413
w/ TGAG (m=55 k=3)
83.498T
27.412s
22.306
TGATE (m=l)
11.171T
3.533s
25.028
PixelAlpha
107.03 107.031T
61,502s
38.669
TGATE (m==2
11.433T
3.624s
24.595
w/ Adaptive Guidance
74.922T
42.684s
35.286
I.CM(iiAAr,, n=4)
8.563T
4.733s
36.086
w/ TGATE (m=88)
65.318T
37.867s
35,825
TGATE (m=1)
7.623T
4.448s
38.574
TGATE (m==0)
70.225T
40.648s
35.726
.GAAE (m=
7.9361
4.543s
37.048
w/TGATE (m=10.. 5)
64.138T
32.827s
38.415
Improvement over Consistency Model. TGATE is implemented using a distillation based method (Schmidhuber 1992b; Hinton et al. 2015), namely the latent consistency model (LCM). The LCM distilled from SDXL first used as the base model, and a grid search is performed for different inference steps. Table S reveals that the generation performance is improved in fewer inference steps (i.e., four). To incorporate TGATE into the LCM, the cross-attention prediction of the first or second step (m 1, 2) is cached and reused in the remaining inference steps. Due to limited inference steps, self-attention is not cached. Table 3 shows the experimental results of the LCM models distilled from SDXL and Aixhat-AlpheEl [ Although the trajectory is deeply compressed into few steps. TGATE functions well, and further decreases PixArt-based LCM computation. Thus, the MACS and latency are reduced by 10.98% and 6.02%, respectively, with comparable generation results. As TGATE does not incur any training costs, integrating it with consistency models is valuable and promising. As a reasonable blueprint. distillation- -bseed methods require sampling trajectories from the teacher model; contrarily, TGATE can enhance sampling efficiency, thereby accelerating the learning process. This aspect will be further discussed in future study. The visualization is provided in Appendix B
Comparison with Adaptive Guidance. Adaptive guidance (Castillo et al. 2023) offers a strategy for the early termination of CFG. The efficiency of TGATE surpasses that of adaptive guidance, as it innovatively caches and reuses attention. On terminating the CFG in PixArt-Alpha, adaptive guidance yields 2.14T MACs/step, whereas TGATE skips cross-attention and further reduces this value 1.83T MACs/step. This optimization moderately reduces the computational overhead, shown in Table 4 Notably, recent trends have shifted toward distillation- techniques (Song al.) 2023 Salimans & Ho| 2022) to accelerate inference. These methods compress the denoising process into fewer steps, often achieving single-digit iterations. The student model learns to mimic the CFG-based output during distillation; therefore, CFG decreases during inference, rendering adaptive guidance inapplicable. In contrast, TGATE can fill this gap and further accelerate the distillation basselttio--beeddddddd models, as shown in Table 3 Beyond current capabilities, the superior scalability of TGATE is compared with that of adaptive guidance, particularly with increasing input sizes. This scalability feature of TGATE is further explored Appendix D
Improvement over DeepCache. Table 5 compares the performance of DeepCache (Ma et al. 2024) and TGATE based on SDXL. Although DeepCache is more efficient, TGATE outperforms it in terms of generation quality. TGATE is integrated with DeepCache by reusing cross-attention maps, thereby yielding superior results: MACs and latency of 43.86 43,868 T and 14.666 s. Remarkably, DeepCache caches the mid-level blocks to decrease computational load, which is specific to the U-Net architecture. However, its generalizability to other architectures, such as the transformer- based architecture, remains underexplored. Beyond DeepCache, TGATE has wider applications and can considerably improve transformer-basse diffusion models, as shown in Table Owing to this adaptability, TGATE versatile
http:////huggnnnaaecccc///eeeess http:://wwggunnniccccociiiiaaaaa 1-844-M
10

---

Published in Transactions on Machine Learning Research (02/2025)
Computational complexity, latency, and FID on the AS-COOCO validation set using DeepCache (Ma ct al. 2024).
Table
Table 6: Zero-shot FIDs on the MS-COCO validation set using the base model of SDXL and different noise sched-
ulers (Karras et al., 2022 LU et al. 2022 Song et
202T
Inference Method
MACs
Latency
FID 4
SDXL
149.438T
53.187s
24.628
Scheduler
Base Model
TGATE(CA)
TGATE(CA,SA)
TGATE
83.498T
27.412s
22.306
FulerD
23.084
21.883
21.857
DeepCache
57.888T
19,931s
23.755
DDIM
21.377
21.783
21.821
DeepCache + TGATE
43.868T
14.666s
23.999
DPMSolver
24.628
22.738
22.306
and potent enhancement for various architectural frameworks. Self-attention is fully operational in the experiment conducted herein and aligned with the DeepCache strategy, it is not included in the first and last blocks.
Improvement over Different Schedulers. The generalizability of TGATE is evaluated on different noise schedulers. As shown in T three advanced schedulers (Karras et al. 2022 Lu et al. 2022 Song et al. 2021). are considered that could compress the generation process a diffusion model to 25 inference steps. Results show that TGATE could consistently achieve stable generation performance in all settings, further indicating its potential for a broad application.
Additional Comparison with other methods. Beyond the previously discussed methods, explore integrating our approach with other accelerated diffusion models: (1) SSD (Gupta al. 2024), which reduces model size via distillation from a larger model, and (2) ToMe (Bolya & Hoffman 2023). which accelerates inference by compressing token counts. Further details are provided in Appendix G
7 Conclusion and Discussion
The cross-attention and self-attention in the inference process of text-conditional diffusion models are empirically analyzed here, offering the following: i) the first few inference steps, cross-attention the primary contributor; however, its influence diminishes in later steps. ii) In contrast, self-attention plays a secondary role initially but gains significance denoising progresses. iii) By caching and reusing attention maps during scheduled inference steps, TGATE reduces computational demands while still achieving competitive outcomes. These findings encourage further analysis of the role of attention in text-conditional diffusion models.
Limitations We acknowledge the challenge of further improving a distilled diffusion model when working with few inference steps. In such cases, TGATE can only offer 10% reduction in computation cost (MACs) for models with highly compressed inference steps, such LCM. However, considering that model may be used billions of times daily, this reduction is significant. The proposed TGATE is particularly valuable since it does not require additional training costs or result in significant performance drops. Furthermore, TGATE can also be integrated into the distillation process, where it accelerates the generation process of the teacher model, potentially speeding up the training of the student model.
Empirical studies suggest that using TGATE may cause slight decline in text-image alignment performance (Sec, H) but generally improves the FID score (Sec. 66.I. Based visualizations, speculate that although TGATE- generated images are similar to those without it, they tend to produce simpler patterns and objects, akin to outcomes seen in other acceleration methods. To address this, we provide a set of parameters (k and m) to balance efficiency and performance, allowing adaptation to different application scenarios. Despite these trade-offs, TGATE reduces inference time by nearly half, making it an effective solution for most cases.
Broader Impacts
Positive Broader Impacts The text-conditional diffusion model, which may be used billions of times daily, typically requires extensive energy resources. Without incurring additional training costs, the computational demands of various base models can be reduced by 10% 50% using the proposed approach. Thus, an eco-friendly solution that considerably reduces the electricity consumption associated with Al technologies.
11

---

Published in Transactions on Machine Learning Research (02/2025)
Negative Broader Impacts This study is fundamental and not linked to specific applications. Therefore, the negative social impacts associated with TGATE are consistent with those of other text-conditional diffusion models and do not present unique risks that warrant a specific mention here.
Acknowledgment
We thank Dylan R. Ashley, Bing Li, Haoqian Wu, Yuhui Wang, and Mingchen Zhuge for their valuable suggestions. discussions, and proofreading.
Jinheng Xie and Mike Shou are only supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award No: 10E-122P20124-0022,
References
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Ait- tala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
David Bau, Jun Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(48): 30071-30078, 2020.
Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English. Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.15122 2023.
2023.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In CVPR, pp. 1398-4602, 2023.
Tim Brooks, Bill Peebles, Connor Homes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024. URL https://openai. com/research/iide- generaation-models-sss www.uuddllssssooooolsisnnaattts 2024.
Angela Castillo, Jonas Kohler, Juan C Pérez, Juan Pablo Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, and Ali Thabet. Adaptive guidance: Training-free acceleration of conditional diffusion models. arXiv preprint arXiv:2312.124877 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. TOG, 42:118:1-148:1,, 2023.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024b.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. In CVPR, 2024c.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, volume 34, pp. 8780-8794, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICMI., 2024.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Kumihiko Fukushima. Neural network model for mechanism of pattern recognition unaffected by shift in position- neocognitron. IEICE Technical Report, A, 62(10))658-666 1979.
Kunihiko Fukushima. Neocognitron: self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-20) 202, 1980.
Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progressive knowledge distillation of stable diffusion xl using layer level loss. arXiv preprint arXiv:2401.0267) 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to trompt-to-poompt image editing with cross-attention control. ICLR, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge to local nash equilibrium. In NeurlPS, volume 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint urXiv:1503.025:: 2015.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.125590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991. Advisor: J. Schmidhuber.
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with 1lm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135 2024.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In NeurlPS, volume 36, pp. 78723-78747, 2023.
Christopher Jarzynski. Equilibrium free-energy differences from nonequilibrium measurements: A master-equation
approach. Physical Review E, 5615),5018, 1997.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurlPS, volume 35, pp. 26565-26577, 2022.
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guid- ance in limited interval improves sample and distribution quality in diffusion models. In Neurips, 2024.
PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. URL https::/doi.orr/10..
10948109
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4): 541-551, 1989.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision- language learning by cross-modal skip-connections. arXiv preprint arXiv:2205. /2005, 2022.
Daiqing Li, Aleks Kamko, Ali Sabet, Ehsan Akhgari, Linmiao Xu, and Suhail Doshi. Playground v2, 2023. URL
(https: httpr//srwggggggaa.ooooooooro/rr
Yanyu Li, Huan Wang. Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang. Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. In NewrIPS, volume 36, 2024.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Haozhe Liu, Mingchen Zhuge, Bing Li, Yuhui Wang, Francesco Faccio, Bernard Ghanem, and Jürgen Schmidhuber. Learning to identify critical states for reinforcement learning from videos. In ICCV, pp. 1955-1965, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver+4: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221 2022.
Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consisteney models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In CVPR, pp. 15762-15772, 2024.
Radford M Neal. Annealed importance sampling. Statistics and computing, 11:125-139, 2001.
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anand- kumar. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching. arXiv preprint arXiv:2402. 14167, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pp. -19554205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp.
PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image seg- mentation. Medical image computing and computer-ussisted intervention 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads: Unleashing the diversity of diffusion models through condition-annealle sampling. arXiv preprint arXiv:2310.17347, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion mod- els with deep language understanding. In NewrIPS, volume 35, pp. 36479-36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint
arXiv:2202.00552, 2022.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In ICML, pp. 9356.9336. PMLR, 2021.
Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4 (1):131-139, 1992a.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-22) 1992b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. (rXxv:1505.00387, 2015.
arXiv preprint
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, volume 30, 2017.
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He. Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprint 4rXiv:2312.03209, 2023.
Xiaoju ye. calflops: flops and params calculate tool for neural networks in pytorch framework, 2023, URL https:
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffusion. In ICCV, pp. 7452-7461, 2023.
Xingyi Yang. Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In CVPR, pp. 22552-22562, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capabil- ity of clip. arXiv preprint ar37,.2403.15378, 2024.
Wei Zhang, Jun Tanida, Kazuyoshi Itoh, and Yoshiki Ichioka. Shift-invariant pattern recognition neural network and its optical architecture. In Proceedings of annual conference of the Japan Society of Applied Physics, volume 564. Montreal, CA, 1988.
15

---

Trajectory Published in Transactions on Machine Learning Research (02/2025)
A Additional Experiments for Temporal Analysis of Cross-Attention
Table Al: Zero-shot FIDs on the MS-COCO valida- tion set using the base model SD-2.1 1 with DPM-Solver al. 2022p. m is the gate step.
Table A3: Zero-shot FIDs on the MS-COCO validation set using the base model SD-2.1 with different noise schedulers. The total inference number is set as 50, and the gate step is 20.
Configuration
FID
SD-2.1(w/ CFG)
22.609
Configuration
S
S
S" (m 3)
29.282
EulerD (Karras al. 2022
22.507
21.559
47.108
S (m 5)
20.859
DPMSolver (Lu 2022
22.445
21.557
48.376
S (vn 10)
21.816
DDIM (Song et ul. 2021]
21.235
20.495
53.150
Table A2: Zero-shot FIDs on the MS-COCO valida-
Table A4: Zero-shot FIDs the MS-COCO validation
tion set using the base model SD-2.1 with DPM-Solver (Lu al., 2022). is the gate step and is the total inference number.
set using different models: SD-1.5, SD-2,1, 1, and SDXL.
The total inference number is set as 25, and the gate step 10.
Configuration
S
S
Configuration
S
S
n=15, m=
23.380
22.315
58.580
SD-1.5
Rombach ct al.
2022
23.927
22.974
37.271
n=25, m=10
22.609
21.816
53.265
SD-2.1
Rombach et al.
2022
22.609
21.816
53.265
n=50, m=20
22.445
21.557
48.376
SDXL
Podell et al. 2023
24.628
23.195
108.676
n=100, m=
22.195
20.391
26.413
Additional experiments are performed for different gate steps of (3,5,10] to support the analysis on cross-attention. As shown in Table [I] when the gate step is larger than five steps, the model that ignores cross-attention can achieve better FIDs. To further justify the generalization of these findings, experiments are conducted under various conditions, including range of total inference numbers, noise schedulers, and base models. Table A2 A3 and A4 that FIDs of S, SE and S on the MS-COCO validation set.
B Temporal Analysis of Self-Attention
Table A5: Zero-shot FIDs on MJHQ-10K using the base model PixArt-Alpha with different caching and re-using
strategies.
Total Inference Steps
FIDs
9.653
Inference steps in the semanties semantics-planning phase is less than that in the fidelity-improvee improving phase.
10
25
11.268
g."
19,205
gr.
11.789
S/
5
10
25
29, 29,507
84
10
25
12.738
Inference steps the crwuuntics-plannnn phase larger than that in the -improvig phase.
15
9.548
S'
3
15
11.436
84
15
25
10,289
Inference steps in the semantics planning phuse is equal that in the fidelity improving phuse.
10
10.105
g/
17.000
g4
10
11.482
Here, the functionality of self-attention in the denoising trajectories from pre-trained diffusion model is explored.
16

---

Published in Transactions on Machine Learning Research (02/2025)
For comprehensive study, we track the generation performance using different values of gate step m and interval k. As shown in Table A5 the empirical results convincingly show that self-attention plays vital role in the latter phases of the process. This is evidenced by consistently higher FID score for S than that for & across all tests.
C Implernentatinn Details
Base Models. Several pre-trained models are used in the experiments:
Stable Diffusion-1.5 (SD-1.5) (Rombach) PixArt-Alpha (Chen et al. 2024b). SVD Among them, the SD series are based on convolutional
et al. 2022), SD-2.1 (Rombach al. 2022), SDXL (Podell et al.
2023).
(Blattmann et al.
2023) and OpenSora Lab & etc. 2024).
neural networks ( Fukushima
1979
1980 Zhang et al.
1988
LeCun et al. 1989. Hochreiter 1991; Srivastava et al.
2015 He et al. 2016) (i.e., U-Net RRnneberger et al.
2015 Pixart-Alpha and OpenSora work on the transformer
(Vaswani et al. 2017) (i.e., Peebles Xie 2023 ). This experimental setting covers several conditional generation
tasks, including text-to-image, text-to-video, and image-to-vitn ti-eo tasks.
Acceleration Baselines. For convincing empirical study, TGATE is compared with several acceleration baseline methods: Latent Consistency Model (Luo et al. 2023), Adaptive Guidance (Castillo et al. 2023), DeepCache (Mal
al. 2024), and multiple noise schedulers (Karras et al. 2022 Lt 2022 Song et al.) 2021) TGATE is orthogonal
to existing methods used to accelerate denoising inference; therefore, it can be trivially integrated to further accelerate
this process.
Evaluation Metrics. Similar to a previous study (Podell et al. 2023). 10k images from the MS-COCO validation set (Lin al. 2014) are used to evaluate the zero-shot generation performance. The images are generated using 2022) with a predefined 25 inference steps and resized to 256 x 256 resolution calculate
DPM-Solver (Lu et al.
the FID (Heusel et al.
2017 TGATE is also tested on high-resolution dataset, namely MJHQ (Li et al. 2023) to
evaluate its aesthetic quality. The generated images are set at a resolution of 1024 x 1024, with a total of 10k samples. Following the protocol in ELLA (Hu et al., 2024), we utilize mPLUG-Large (Li et al. 2022) to score the generated samples based on predefined questions. For video generation, the prompts from SeeenSra-Sampaee (Lab & etc. 22024)
are used. videos per prompt are generated, and their performance evaluated based on the CL.IP score (Radford et al., 202Ty. As the text branch of SVD unavailable, SDXL is used to create image from a prompt, which is then input into SVD to produce the corresponding video. To evaluate the efficiency, Calflops (Xiaoju yel 2023) is used to count Multiple- AaltipleeAAcumulatt Operations (MACs) and the number of parameters (Params.). Furthermore, the latency per sample is assessed on a platform equipped with Nvidia 1080 Ti.
D Discussion on Scaling Token Length and Resolution
Table A6: MACs per inference step when scaling up the token lengths and image resolutions.
Tokens Scaling Factor
Resolution
Method
x
X 128
x 1024
x
4096
SID-2.1 (w/o CFG)
0.761T
1.011T
2.774T
8.820T'
768
Ours
0.73T
S13-2,1 (w/o CFG)
1.351T
1.601T
3.364T
9.410T
1024
Ours
1.298T
SD-2.1 (w/o CFG)
5.398T
5.648T
7.411T
13.457T
2048
Ours
5.191T
TGATE can improve efficiency by circumventing cross-attention, which motivated us to examine its contribution to the overall computational cost based on the input size. Specifically, TGATE is compared with SD-2.1 w/o CFG per step to determine the computational cost of cross-attention. Note that SD-2.1 w/o CFG is the computational lower bound for existing methods (Castillo et al. 2023), as it can stop CFG carly to accelerate diffusion process. As shown Table MACs are used as a measure of efficiency. In the default SD-2.1 setting, the resolution set as 768 with maximum token length of 77. Results show that cross-attention moderately contributes to the total computational load, which increases exponentially with increasing resolution and token lengths. By omitting cross-attention calculations, TGATE
17

---

Published in Transactions on Machine Learning Research (02/2025)
w/ TGATE
w/ TGATE
m= 10
w TGGTE (M=3)
on=ll (m 10 (eee)
wi TGAA
w/TGATE 5)
(m= 10,kk)
Figure Al: Generated samples by SDXL using the same initial noises and prompts, but with varying hyper- parameters. The configurations are arranged from top to bottom in the order decreasing latency.
18

---

Published in Transactions on Machine Learning Research (02/2025)
PixArt-a
w/O TGATE
W TOATTAT
W. TTGATE
(m- 10)
(m= k-3)
wi TGATE
(m- k-3)
w/ IGATE
(nn- (mn- S, S-)
WTGAAE
(on-1 (nn-10 k-
wIGATE
Figure A2: Samples generated by PixArt using the same initial noises and prompts and different hyperparame- ters. The configurations, from top to bottom, are ordered by decreasing latency.
19

---

Published in Transactions on Machine Learning Research (02/2025)
considerably mitigates its adverse effects. For example, in extreme scenario with the current architecture targeting an image size of 2048 and a token length of 4096 77, MACs can be decreased from 13.457 T to 5.191 T, achieving more than two-fold reduction in computation.
One may argue that existing models do not support such high resolutions or token lengths. However, there is an inevitable trend toward larger input sizes (Zhang et al. 2024 Esser al. 2024; Chen al. 2024a). Furthermore, recent study (Li al. 2024) has shown the difficulty computing cross-attention on mobile devices, underscoring the practical benefits of our approach.
down block. I Pransfommer...tt...
down block. tmansformerr Latm2
mid block. O transfoormerr....om
Closs-attcation Diffsence
0,0025
0 0.00015
0,0005
2.0
0,9054
0.0001
00005
0.0903
0.000
0.00000
0,0300
1P transformerrrreeeo attt2
up_ block. I transformer Latin2
1attm2
up. block. wwwnssormmee.....mmmmm attn2
Closs-ittattion Differeuce
0.50010
0.0000
0.50003
0.0000
1.4
0,50005
1.000
0.00044
0,00002
Figure A3: Illustration of cross-attention map differences between consecutive inference steps. We sample cross- attention maps from various blocks in SD-2.1 to monitor their convergence. The consistent convergence across differ- ent blocks supports the generality of our observations.
Semanties- Planning
Step
Step
Stop
Step
Step B
Fidelity- Improving
Step 10
Step 13
Step 16
Step 18
Step
Cross-Attention Feature Maps
Generated Sample
Figure A4: Illustration of cross-attention maps during different inference steps. We use bl_bl bl bloook. www.taasfooerrrblloo...o 2.ttcc in SD-2.1 representative module visualize its out- put for given prompt, "a delicious salad with beef, stock photo". During inference, the feature map converges rapidly and stabilizes, which aligns with our observations.
attn2
20

---

Published in Transactions on Machine Learning Research (02/2025)
Sunset over trunquil lake moountains.
Ncon-lit skyline of bustling future metropolis.
A coral roof busiling divense marine life.
Gears and sicam power an old- world cityscape.
Dusty trails leading a desert sunsel.
Red hot air balloon safari sunrise
Giant flowers under moon sky.
SD-2.1 (w/ CFG)
TOATE (m-10)
TOATE (m=5)
Figure A5: Samples generated by SD-2.1 (w/ CFG) and TGATE with two gate steps (i.e., 5 and 10) for the same initial noises and captions.
(a) LCM (SD-XL)
(b) LCM (SD-XL) w/ TGATE
(c) wise
Difference
Step
Step 2
Step =3
Step 4
Figure A6: Generated samples of (a) LCM distilled from SDXL and (b) LCM with TGATE given the same initial noise and captions. (c) represents the difference between (a) and (b).
E
Additional Visualization
support Sec. we further analyze the distribution of different blocks' attention modules. As shown Fig. A3
we sample the attention maps from down baockss mid_ mid_block- block, and up ap_blocks, and track their changes during different steps. The convergence rates are slightly different, but they share the same trend, which further indicates that this interesting phenomenon is common among different blocks. Mcanwhile, as shown in Fig. A4 we also visualize the feature map of po_block as the representative one during different inference steps.
We also provide visualizations of TGATE with different models and configurations. Fig. A5 shows the samples gener- ated using different gate steps. Results show that larger gate steps produce generation results more similar to those of the base models without TGATE. Moreover, the generated samples are visualized based on different steps. As shown in Fig. A6 the difference caused by TGATE invisible. Considering that different configurations have moderate impacts FIDs, the samples generated by SDXL and PixArt-Alpha are visualized under various settings, shown
21

---

Published in Transactions on Machine Learning Research (02/2025)
Although some configurations result in increased FIDs, the changes nearly imperceptible. These results demonstrate the effectiveness of TGATE.
in Fig. A1 and Fig. A2
Table A7:
Generation performance on MJHQ-I0K 10k (Li et al. 2023 and MS MS-COCO Lin et al, 2014) datasets with
different random seeds on PixArt (Chen et 2024b) and SDXL (Podell et 2023
Dataset
Caching Modules
Seed 1
Seed 2
Seed
Mean S.d.
PixArt
MJHQ-10K
9.653
9.642
9.539
9.611 0.051
TGATE
MJHQ- 1OK
CA
9.548
9.477
9.417
9.481 + 0.054
TGATE
MJHQ- 10K
CA. SA
10.289
10.108
10.138
10.178 + 0.079
SDXL
COCO-10K
24.628
24.164
24.661
24.484 + 0.227
TGATE
COCO-10K
CA
23.433
22,917
23.584
23.311 + 0.286
TGATE
COCO-10K
CA. SA
23.839
22.759
23.967
23.522 +: 0.542
F Error Bar
We provide error bars for the main experiments with configurations of m = 15, k = for PixArt and m = 10, k = for SDXL. Results in Table and Table are based on random seed
G Additional Comparison
We further incorporate our method into SSD-1B Gupta
Table A8: Generation performance of ToMe (Bolya
ct al. (2024). a lightweight model distilled from a larger diffusion model. As shown Table C, we test the models' performance on the COCO Validation Set in terms of la- tency and FID-10K Latency is defined the time required to generate one image in resolution of 768, and the results are collected platform using an Nvidia 1080ti. The re- sults demonstrate that our method is also compatible with models distilled from larger models.
& Hoffman, 2023) and SSD-IB (Gupta et al. 2024 with and without with TGATE on MS-COCO-10K (Lin
et al., 2014) dataset. The base mode for ToMe is SD- 2.1 (Rombach et al. 2022).
FID
Lutency
ToMe
24.763
12.240s
TGATE ToMc
24.383
8.731s
SSD-IB TGATE
28.564
30.097s
We also compare our method with ToMe Bolya Hoffman (2023). We ToMe's merging ratio 50% and use SD- 2.1 as the base model. The model utilizes the DPM-Solver with 25 inference steps as the noise scheduler. Our method achieves latency of 11.372 seconds per image, slightly outperforming ToMe. Given that our approach is orthogonal to ToMe, integrating TGATE with ToMe results in an improved latency of 8.731 seconds. This empirical study demonstrates the effectiveness of our method and its potential for broad application when combined with various acceleration techniques.
25.004
18.328s
H Evaluation on Text-Image Alignment
The performance of TGATE in text-image alignment is assessed based the protocol outlined by CLIPP score Radford et al. (2021). ELLA (Hu et al. (2024) and T21-Compbench (Huang et al. (023). Table A9 and Table A10 shows the competitive performance TGATE compared with the baseline across various evaluation dimensions, indicating its provide visualization samples Fig. A7 As noted in our limitations, increased inference accelera- tion can lead to greater deviations from the baseline. While these differences are often perceptually indistinguishable, in some cases, they may cause noticeable artifacts, such as the altered shape of the bowl slight distortion of the bracelet in Fig. A7 Given the acceleration achieved, this trade-off is acceptable in most cases that do not demand exceptionally high-quality samples.
effectiveness.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Table A9: Generation performance on DPG-Bench (Hu et al. 2024). CA and SA represent cross-attention and self- attention, respectively. For PixArt, parameters are m 15 and 3, whereas for SDXL, we utilize 10 and k Evaluation scores are obtained using mPLUG-Large (Li et al. 2022) with predefined questions.
Method
Globalt
Entityt
Attributet
Relation
Othert
Total Scoref
PixAn
78.81
79.46
80.61
78.22
80.24
72.53
TGATE (CA)
79.31
78.44
79.33
80.68
75.48
70.55
TGATE (CA, SA)
76.67
76.18
77.56
78.74
79.14
68.37
SDXL
81.87
82.50
82,46
83.50
61.45
74.92
TGATE (CA)
70.33
80.32
80.60
83.63
78.63
72.41
TGATE (CA, SA)
79.65
77.29
78.43
82.78
80.11
70.72
Table A10: Generation performance of SD-2.1
(Rombach et al.
2022 with TGATE on T21-Compbench (Huang et al.
2023).
We set 10 and lo as 5.
Color
Shape
Texture
Spatial
Non-Spatial
Complex
SD-2.1 SD-2.1 + TGATE (CA)
43.31
42.27
47.24
6.52
29.80
30.50
41.97
41.16
45.28
5.73
29.55
29.45
SD-2.1 + TGATE (CA,SA)
40.72
40.01
44.22
5.14
29.32
29.44
A cubic ice cube and spherical ice hucket
The blue howl top of the white placemat,
Baseline
TGATE (CA)
TGATE (CA, SA)
Baseline
TGATE (CA)
TGATE (CA, SA)
The green plant was right the white wall.
A diamond brooch and a teardrop bracelet
Figure A7:
Generated samples from (a) PixArt, (b) TGATE with CA cache, and (c) TGATE with both CA and SA cache. The caption is derived from 121-Coomphench.
Table A11:
CLIP score of TGATE on MJHQ LLi et al. 2023) and MS-COCO (Lin al., datasets PixArt (Chen
Table A12: Frame consistency on Open-Sora sample dataset LLab etc.] 2024). The frame consistency caleulated based the I.2 distance between the adjacent frames.
Table A13: Memory overhead caused by
TGATE. The base model PixArt (Chen) al. (024b)) and SDXL (Podell et al.) 2023) and the computational platform is single V100 GPU card with pytorch 2.2.
2014
et al.
2024bb and SDXL (Podell
et al.
2023).
Mothod
CLIP Score
Model
Frume Consistency
Memory Cost
SDXL
34,148
OpenSon
9,979
(m-10)
TOATE (m 100)
7.597
SDXL
8515
TOATE (10
32,710
TEATH tmn 100,
9.987
TOATE (m=10, k=5)
8531 MB
PixAnt-Alhh
34,233
SVD
30,686
PixArt-Alpha
13705 MB
TGATS (mm=15)
33,872
TGATE (m=15, k=3)
13707 MB
TGATE (m=11 k=3)
33.673
TeATE (m 10 = 5)
31,020
23

---

Published in Transactions on Machine Learning Research (02/2025)
TGATE Andy
wi TGNTE
TOATE
Figure A8: Generated samples based on SVD. SDXL generates the input frame with a caption of "A snowy forest landscape with dirt road rumning through it. The road is flanked by trees covered snow, and ground also covered snow. The sun is shining, creating bright and atmosphere. The road appears be empty, and there people animals visible in the video. The style of the vidco is natural landscape shot, with focus the beauty of the snowy forest and the peacefulness of the road.".
Evaluation on Frame Consistency
The performance of TGATE in video generation is evaluated based on the frame consistency. The L2 distance between different frames are obtained, where a smaller value indicates smoother change between frames. As shown in Table A12 TGATE does not cause significant differences this metric, confirming its effectiveness. We also provide a visualization in Fig.
J Evaluation on Memory Cost
We evaluate the memory overhead of TGATE, which accelerates inference by caching features and may increase GPU usage. As shown Table[A13] using SDXL and Pixart-Alpha as baselines for U-Net and transformer models, TGATE incurs only minimal, single-digit memory cost, which is negligible in most cases, demonstrating its feasibility.
K Discussion Anchor Feature.
This paper employs the average feature maps from both unconditional and conditional branches the anchor feature for caching, following CFG. We provide visualization samples using only branch instead of averaging. As shown in Fig, A9 the results suggest minimal impact performance. This aligns with our pilot study (Fig. ( where attention maps converge to fixed point after several inference steps, including for the conditional/uncconttiioaa branches in CFG.
L
Hyper-parameter Selection
TGATE is a simple method for training-free acceleration with two hyperparameters, and m. Their values can adapt to different inference steps achieve balanced trade-off. shown Fig. A10 m and h are as 3/5 and 1/5 of the total inference steps, respectively. Increasing inference steps minimally impacts the generated results, suggesting that l and m be reliably set fixed proportions of the inference steps for most scenarios.
24

---

Published in Transactions on Machine Learning Research (02/2025)
w/ TGATE
w/ TGATE (Go Ce)
w/ TGATE (Cp)
w/ TGATE (C2))
Figure A9: Generated samples of (a) PixArt and (b) TGATE reusing the averaged of cross-attention maps, (c) TGATE reusing the unconditional cross-attention maps, and (d) TGATE using text-conditional cross-attention maps.
w/o TGATE
w/ TOATE (CA) WTGAATE (CA, SA) TOAE w/ TGATE (CA)
W/TGATE (CA.
TOARR
TOATE (CA) W/IGATE (CA,SA)
25 steps
50 steps
100
Figure A10: Generated samples of PixArt are evaluated with 25, 50, and 100 inference steps. TGATE'S hyperparameter, set as ratios of inference steps (m 3 iinference-steps and innference-steps) ensure stable performance across different settings.
M Details of Prompt
We showcase all prompts used in this paper to ensure reproducibility.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Table A14: Figures and Corresponding Prompts
Figure
Caption
Figure
A desert oasis with a mirage of a city.'
A galaxy with colorful nebulae and star clusters."
An underwater coral reef with mermaids and treasure."
A magical duel in ruined castle.'
A painting of mountains, in the style of Monet.'
Figure 3
'High quality photo of an astronaut riding a horse in space."
Figure
Paisaje montañoso nevado."
Figure
(a)
'Beautiful lady, freckles, big smile, blue eyes, short ginger hair, dark makeup, wearing floral blue vest top, soft light, dark grey background." PProfessiona portrait photo of an anthropomorphic cat wearing fancy gentleman hat and jacket walk- ing in autumn forest,'
'Stars, water, brilliantly, gorgeous large scale scene, a little girl, in the style of dreamy realism, light gold and amber, blue and pink, brilliantly illuminated in the background.' A legendary figure in Cyberpunk City, rainy night, front, sunglasses.'
Figure
(b)
'Woman standing in front of a window with her hair blowing, modern anime, sunshine, highly de- tailed.'
"Create story or description inspired by this photo gray-haired year old Norwegian woman in front of the with subtle freckles and hazel eyes.'
A layer cake made out of stratographic cross-section of the Sonoran Desert.'
A teal flower in barren garden, bcautiful villa background, octane, redshift, highly detailed."
Figure A4
"a delicious salad with beef, stock photo.'
Figure
A5
"Sunset over tranquil lake in the mountains' 'Neon-lit skyline of bustling future metropolis.
A coral reef bustling with diverse marine life.'
'Gears and steam power old-world cityscupe."
Dusty trails leading
desert sunset."
'Red hot air balloon safari at sunrise"
Giant flowers under a moon sky"
Figure
A6
'Self-portrait oil painting, a beautiful cyborg with golden hair, 8k.
Figure A8
"A snowy forest landscape with dirt road running through it. The road is flanked by trees covered in snow, and the ground is also covered in snow. The sun is shining, creating bright and serene atmosphere. The road appears to be empty, and there are no people or animals visible in the video. The style of the video is natural landscape shot, with a focus on the beauty of the snowy forest and the peacefulness of the road.'
Figure A7
"A cubic ice cube and spherical ice bucket.'
'The blue bowl was on top of the white placemat.'
'The green plant was on the right of the white wall.'
A diamond brooch and teardrop bracelet'
Figure A10
'a chicken on the right of a suitcase.'
"professional portrait photo of an anthropomorphic cat wearing fancy gentleman hat and jacket walk- ing in autumn forest."
"ttars, water, brilliantly, gorgeous large scale scene, a little girl, in the style of dreamy realism, light gold and amber, blue and pink, brilliantly illuminated the background.'
26


Reviews:
[{"review_id": "c4ptTHP8ic", "reviewer": "Reviewer_s3sc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a comprehensive experimental analysis of the roles of cross-attention and self-attention at each denoising step in diffusion models. Leveraging these observations, the authors propose TGATE, a training-free acceleration method that achieves up to a 50% reduction in MACs and latency with minimal impact on generation quality.\n2. TGATE is extensively validated across diverse architectures (e.g., Transformer and U-Net), tasks (e.g., text-to-image and text-to-video), and sampling schedulers. Additionally, it demonstrates the ability to slightly accelerate latent consistency models, further showcasing its versatility.\n\n\nMajor Weakness:\n\n1. TGATE relies on the hyper-parameters gate step (m) and interval (k), which require careful tuning to achieve optimal performance across different use cases. Currently, k is often set to fixed values such as 10 or 15. A more systematic and flexible approach would be to relate k and m to the ratio of total sampling steps, allowing these parameters to scale with the specific inference settings. For example, the authors could provide a threshold based on the change in cross/self-attention differences to determine an appropriate ratio, thereby offering a more adaptive and generalizable framework for hyper-parameter selection.\n\n2. Several widely-used open-source models, such as SD3, Flux and its successors in video generation (e.g., CogVideo X), are built on the multi-modal DiT (MM-DiT) framework. This architecture employs attention mechanisms that may differ from the cross-attention + self-attention structure analyzed in this paper. The authors could enhance the work by including a discussion on how TGATE applies to MM-DiT and similar multi-modal architectures.\n\nMinor Weakness:\n\n* In Section 3.2, the conditional text embedding $c$ is replaced with the null text embedding $\\emptyset$, whereas in Section 4.1, the method caches and reuses the mean of the cross-attention maps $C_c$ and $C_{\\emptyset}$. Considering this gap, It would be worthwhile to explore the difference of reusing only $C_{\\emptyset}$ in fidelity-improving stage. \n* The definition of Multiply-Accumulate Operations (MACs), mentioned in the third line under Figure 5, should be introduced earlier in the paper for clarity.\n* Providing the prompts used to generate the main qualitative results would enhance reproducibility and help readers better understand the context of the examples.\n* Including qualitative results for video generation would offer a more comprehensive evaluation, especially considering the distinct challenges associated with temporal consistency in video synthesis.\n* There is a noticeable reduction in the properties evaluated by T2I-CompBench. To address this, additional qualitative results could be presented to illustrate the influence of TGATE on handling complex relationships, such as attribute-object interactions. This would provide deeper insights into TGATE’s impact on maintaining semantic consistency in challenging scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces TGATE (Temporally Gating the Attention), a method for accelerating text-conditional diffusion models by analyzing the roles of cross-attention and self-attention across different phases of the inference process. The authors observe that cross-attention becomes less critical after an initial “semantic-planning phase,” while self-attention grows more influential in a subsequent “fidelity-improving phase.” Leveraging this insight, TGATE selectively caches and reuses attention outputs, reducing computational overhead without retraining the model. \n\nThe authors demonstrate the efficacy of TGATE across multiple diffusion models, showing computational efficiency gains (up to 50%) without requiring additional training. They provide comprehensive evaluations across datasets, models, and configurations, establishing TGATE as a versatile tool for accelerating diffusion-based generative tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the \"Strengths and Weaknesses\" section.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SK1ZfktQ4t", "reviewer": "Reviewer_vkDp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Very interesting analysis of the role of Attention in text-to-image diffusion models.\n- The paper introduces a simple method to exploit the observations to make diffusion sampling more efficient.\n- Many comparisons to other acceleration methods, and also different results with different popular text-to-image diffusion models.\n- The method provides a nice acceleration (10-50%) with minimal impact in quality.\n\nWeaknesses:\n- No major weaknesses.\n\nComments:\n- Eq (6). Can you explain better how is that the average is used as anchor. I would expect that we need access to both of them (or one of them, but not the average). \n- It would be good to add more details in the body of the paper about the Self Attention experiments. Right now, the sections analyzing self-attention just point to the Appendix (e.g., Sec 3.3 and Sec 4.2).\n- It would be nice to see more discussion on the role of the sampler used in the Experiments (all the experiments and also the ones in Table 6).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes (empirically) the role of attention mechanisms in text-conditional diffusion models during the image generation process. The authors find that the cross-attention module is crucial in the initial *semantic planning* phase, where the model interprets the text prompt, while the self-attention module is more important in the later *fidelity-improving* phase, where the model generates the actual image details.  Based on this observation, the paper introduces a simple, training-free method called Temporally Gating the Attention (TGATE), which accelerates image generation by caching and reusing attention outputs at previous timesteps. In the fidelity-improving phase, the TGATE method caches and reuses the cross-attention output, while in the semantic-planning phase, self-attention predictions are cached and reused. The authors show that TGATE can accelerate various text-conditional diffusion models by 10-50% without significantly impacting the quality of the generated images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No major change is requested as the paper is well-written and clear. \n\nA few things I believe will make this paper better:\n\n- As mentioned, add more discussion about Self-Attention experiments in the main body of the paper.\n- Introduce the sampler for the experiments as early as possible. Right now, in Sec 3.1, there's no information about the sampler, just that convergence of cross-attention appears between 5-10 steps. All this, should be relative to the total number of steps, and the actual algorithm used. Note that DPM solver is introduced in Sec 3.2.\n- I would recommend adding more information about the Attention mechanism to make the paper more self-contained (Eq. 3). For example, you could give a little more information about K, Q, V.\n- Table 1. \"Generation Performance\" --> \"FID\" or \"Generation Performance (FID)\"\n- Page 9 - There's a line break in Appendix E (in latex: Appendix~E)\n- References. Check consistency in the format. Also some references are duplicated: Schmidhuber 1992b = Schimidhuber 1992c.", "broader_impact_concerns": "Limitations and Broader Impacts considerations are correctly discussed within the main body of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HGtYV0OWgG", "reviewer": "Reviewer_bq4G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths**\n\nMy overall impression of the paper is positive. The authors have presented a largely self-contained, well-motivated, and well-presented piece of work. The exposition and notation are clear and easy to follow. The initial experimental analysis establishing the biphasic behavior of cross- and self-attention during diffusion sampling is compelling, and the proposed method is well motivated by it. The method itself is quite flexible and appears to require little overhead of its own, and it does not require any retraining of the underlying models. The experimental results demonstrating the method show that it is effective and support the paper's core claims. The authors apparently plan to publicly release their code.\n\n### **Weaknesses**\n\n* Most issues with the paper are quite minor and could be corrected with an additional round of proofreading (e.g. a few punctuation errors, including doubled commas in one spot; an error in the Jarzynski citation at the beginning of Section 2, which includes the year twice; spelling out `log` in math mode versus using `\\log`, etc.). \n* The only major issue that I see as glaring is that one of the paper's main findings, namely about the biphasic behavior of cross- and self-attention, is already known in the literature but not cited. By current count, this earlier work [1] already has 630 citations, so it is an unfortunate omission. However, what the authors *do* with this knowledge in the current submission is a worthy and (I believe) novel contribution in itself, so this is not a fatal error. Nevertheless, this earlier work should absolutely be acknowledged.\n\n### **Additional Commentary**\n\nThere is some additional related work that may be worth discussing in the context of adaptive guidance. Reference [2] investigates the role of guidance during different stages of sampling and proposes applying guidance only within a limited interval. Reference [3] explores two strategies for applying guidance: The first weights the influence of guidance using a schedule, and the second anneals the condition itself with decreasing Gaussian noise during sampling. I would be interested in seeing the authors' perspective reconciling the findings reported in these papers with their analysis.\n\nOne of the biggest implicit questions all papers face is \"Who cares?\" I think the authors have done a good job acknowledging the limitations of their work by putting it in a global perspective. While an individual user may or may not notice a 10 percent reduction in computation cost, it is definitely something that adds up when one considers the sheer popularity of these models. I wanted to point out what I thought was a particularly good answer on the authors' part to this implied question.\n\nWhat may be the bigger limitation is also acknowledged by the authors, namely that of the poorer performance of their method on text-image alignment. I think the authors have provided a reasonable response here by introducing more flexibility into the model, but it would be interesting to have a bit more insight into this issue.\n\n### **References**\n\n[1] Balaji, Yogesh, et al. \"ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.\" arXiv preprint arXiv:2211.01324 (2022).\n\n[2] Kynkäänniemi, Tuomas, et al. \"Applying guidance in a limited interval improves sample and distribution quality in diffusion models.\" arXiv preprint arXiv:2404.07724 (2024).\n\n[3] Sadat, Seyedmorteza, et al. \"CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.\" arXiv preprint arXiv:2310.17347 (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents experimental results demonstrating that diffusion sampling breaks into two phases, the first of which is dominantly influenced by cross-attention and the second of which is dominated by self-attention. The authors leverage this biphasic behavior by proposing a method to reduce computational demands by periodically caching and reusing self-attention maps during the first phase of sampling and caching cross-attention feature maps (which converge after the early part of sampling) for use in the second phase. The described method is flexible, does not require model retraining, and can be integrated into a variety of popular frameworks. The authors appear prepared to publicly release the code for their method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The only absolute must in my opinion is to integrate some discussion of reference [1] into the paper. Another proofreading pass would be my next priority. The other things I mentioned above would be nice to have in the paper for completeness but should not be considered obligatory.", "broader_impact_concerns": "The authors have adequately addressed broader impact concerns in their statement in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision is based on its strong empirical analysis, novel insights into cross- and self-attention roles in diffusion models, and the practical efficiency improvements introduced by TGATE. Reviewers praised its clear motivation, solid experimental validation across models and tasks, and minimal quality loss despite acceleration. Minor concerns about hyperparameter tuning and applicability to MM-DiT were addressed satisfactorily. The reviewers provided constructive feedback, which the authors incorporated in the revision.
Decision: Accept (poster)