HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Authors: Yushi Huang
Zining Wang
Ruihao Gong
Jing Liu
Xinjie Zhang
Jinyang Guo
Xianglong Liu
Jun Zhang
Venue: ICML 2025 poster (2025)
Keywords: diffusion transformer
acceleration
feature caching
Primary area: applications->computer_vision
TL;DR: A learning-based feature caching method to accelerate diffusion transformer inference.

Abstract: Diffusion Transformers (DiTs) excel in generative tasks but face practical deployment challenges due to high inference costs. Feature caching, which stores and retrieves redundant computations, offers the potential for acceleration. Existing learning-based caching, though adaptive, overlooks the impact of the prior timestep. It also suffers from misaligned objectives-*aligned predicted noise vs. high-quality images*-between training and inference. These two discrepancies compromise both performance and efficiency.
To this end, we *harmonize* training and inference with a novel learning-based *caching* framework dubbed **HarmoniCa**. It first incorporates *Step-Wise Denoising Training* (SDT) to ensure the continuity of the denoising process, where prior steps can be leveraged. In addition, an *Image Error Proxy-Guided Objective* (IEPO) is applied to balance image quality against cache utilization through an efficient proxy to approximate the image error. Extensive experiments across $8$ models, $4$ samplers, and resolutions from $256\times256$ to $2K$ demonstrate superior performance and speedup of our framework. For instance, it achieves over $40\\%$ latency reduction (*i.e.*, $2.07\times$ theoretical speedup) and improved performance on PixArt-$\alpha$. Remarkably, our *image-free* approach reduces training time by $25\\%$ compared with the previous method. Our code is available at https://github.com/ModelTC/HarmoniCa.

Harmoni Ca:
Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Yushi Huang 1 Zining Wang 23*i Ruihao Gong?3 Jing Liu Jun Zhang'
Xinjie Zhang' Jinyang Guo 35
Xianglong Liu 36
Abstract
dreany 12 illnstration Floarrfillen neadow beneath deisting ciouds, bright bloasont in the foreground and distant cottage perched on gent le slope, humana vigible."
Diffusion Transformers (DiTs) excel in generative tasks but face practical deployment challenges due to high inference costs. Feature caching, which stores and retrieves redundant computa- tions, offers the potential for acceleration. Exist- ing learning-based caching, though adaptive, over- looks the impact of the prior timestep. also suf- fers from misaligned objectives-alignee predicted noise vs. high-quality images-between training and inference. These two discrepancies compro- mise both performance and efficiency. To this end, we harmonize training and inference with a novel learning-based caching framework dubbed HarmoniCa. first incorporates Step- Wise De- noising Training (SDT) to ensure the continuity of the denoising process, where prior steps can be leveraged. In addition, Image Error Proxy- Guided Objective (IEPO) is applied to balance image quality against cache utilization through an efficient proxy to approximate the image er- ror. Extensive experiments across models, 4 samplers, and resolutions from 256 x 256 to 2K demonstrate superior performance and speedup of our framework. For instance, it achieves over 40% latency reduction (i.e. 2.07x theoretical speedup) and improved performance on PIXART- a. Remarkably, our image-free approach reduces training time by 25% compared with the previ- ous method. Our code is available at https: / //github. www./ooee//eeeeeeeeeoooooeeeeeee Www.eee///aamooo.....
(a) PIXART-E
(b) HarmoniCa (1.73x)
Figure 1. High-resolution 2048 2048 images generated using PIXART-E (Chen et al., 2024a) with a 20-step DPM-Solver++ sampler (Lu et al., 2022b). Our proposed feature caching frame- work achieves a substantial 1.73x speedup with minimal visual difference. More visualization results can be found Sec.
1. Introduction
Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021) have recently gained increasing popularity in variety of generative tasks, such as image (Saharia et al., 2022; Esser et al., 2024) and video generation (Blattmann et al., 2023; Ma et al., 2024c), due to their ability to produce diverse and high-quality samples. Among different backbones, Dif- fusion Transformers (DiTs) (Peebles & Xie, 2023) stand out for offering exceptional scalability. However, the tensive parameter size and multi-round denoising nature of diffusion models bring tremendous computational overhead during inference, limiting their practical applications. For instance, generating one 2048 x 2048 resolution image using PIXART-E (Chen et al., 2024a) with 0.6B parameters and 20 denoising rounds can take up to 14 seconds on a single NVIDIA H800 80GB GPU, which is unacceptable.
"qqual contribution Work done during intersship at Sense- Time Resaerch iComAl Lab, Hong Kong University of Science and Technology 'SenseTime Research 'SKCCSS,, Beihang Uni- versity *ZIP Lab, Monash University 'IAI, Beihang University OSCSE, Beihang University. Correspondence to: Ruihao Gong <gongruihao@ @ buaa. edu.en, Jun Zhang <eejzhang @ us..
In light of the aforementioned problem, previous methods emerge from two perspectives: reducing the number of sampling steps (Liu et al., 2022a; Song et al., 2020b) and de- creasing the network complexity in noise prediction of each step (Fang et al., 2023; He al., 2024). Recently, new
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
branch of research (Selvaraju al., 2024; Yuan et al., 2024; Chen et al., 2024b) has started to focus on accelerating sam- pling time per step by the feature caching mechanism. This technique takes advantage of the repetitive computations across timesteps in diffusion models, allowing previously computed features to be cached and reused in later steps. Nevertheless, most existing methods are either tailored to the U-Net architecture (Ma et al., 2024b; Wimbauer al., 2024) or develop their strategy based on empirical observa- tions (Chen et al., 2024b; Selvaraju et al., 2024). Therefore, there is a lack of adaptive and systematic approaches for DiT models. Learning-tor to-Chhhh (Ma et al., 2024a) introduces a learnable router to guide the caching scheme for DiT models. However, we have found that this method induces discrepan- cies between training and inference, which always leads to distortion build-up (Ning et al., 2023; Li et al., 2024b; Ning et al., 2024). The discrepancies arise from two main factors: (1) Prior Timestep Disregard: During training, the model directly samples timestep and employs the training images with manually added noise akin to DDPM (Hu et al., 2021). This pattern ignores the impact of the caching mechanism from earlier steps, which differs from the inference process. (2) Objective Mismatch: The training objective minimizes noise prediction error of each timestep. Differently, the in- ference goal aims for high-quality final images, causing misaligmment in objectives. We believe these inconsisten- cies hinder effective and efficient router learning.
We identify two key discrepancies in existing learning- based caching methods: (1) Prior Timestep Disregard, where training neglects the impact of earlier timesteps, and (2) Objective Mismatch, which minimizes intermedi- ate output errors rather than final image errors. These is- sues hinder performance and acceleration improvements.
We propose HarmoniCa, a framework that resolves these discrepancies by (1) Step-Wist Denoising Training (SDT), which captures the full denoising trajectory to account for earlier timesteps, and (2) Image Error Proxy-Guided Optimization Objective (IEPO), which aligns the training objective with inference by approximating image error.
Extensive experiments on NVIDIA H800 80GB GPUs with 8 models, 4 samplers, and resolutions demonstrate the efficacy and universality of our framework. For ex- ample, it achieves a 6.74 IS increase and 1.24 FID re- duction on DiT-XL/2, surpassing previous tfate-of-the-ar (SOTA) methods with a higher speedup ratio. Moreover, our image-free framework offers greater efficiency and applicability at significantly lower training costs.
2. Related Work
Diffusion models. Diffusion models, initially conceptual- ized with the U-Net architecture (Ronneberger et al., 2015), have achieved satisfactory performance in image (Rombach et al., 2022; Podell et al., 2023) and video generation (Ho et al., 2022). Despite their success, U-Net models struggle with modeling long-range dependencies in complex, high- dimensional data. In response, the Diffusion Transformer (DiT) (Peebles Xie, 2023; Chen et al., 2023; 2024a) is introduced, leveraging the inherent scalability of Transform- ers to efficiently enhance model capacities and handle more complex tasks with improved performance.
To alleviate the above discrepancies effectively, we harmo- nize training and inference with HarmoniCa, novel cache learning framework featuring a unique training paradigm and a distinct learning objective. Specifically, to mitigate the first disparity, we design Step-Wise Denoising Training (SDT). It aligns the training process with the full denoising trajectory of inference using a student-teacher model setup. The student utilizes the cache while the teacher does not, effectively mimicking the teacher's ' outputs across all con- tinuous timesteps. This approach maintains the reuse and update of the cache at earlier timesteps, similar to inference. Additionally, to address the misalignment in optimization goals, we introduce the Image Error Proxy-Guider Objec- rive (IEPO). This objective leverages a proxy to approximate the final image error and reduces the significant costs of di- rectly utilizing the error to supervise training. IEPO helps SDT efficiently balance cache usage and image quality. By combining the two techniques, extensive experiments show the promising performance and speedup of HarmoniCa, e.g., over 1.69x speedup and much lower FID (Nash et al., 2021) for non-accelerated PIXART-a (Chen et al., 2023). In addi- tion, HarmoniCa eliminates the requirement of training with a large number of images and reduces about 25% training time compared to the existing learning-based method (Ma et al., 2024a), further enhancing its applicability.
Efficent diffusion. Diverse methods have been proposed to reduce the generation overhead for diffusion models. These techniques fall into two main categories: reducing the number of sampling steps and decreasing the computa- tional load per denoising step. In the first category, several works utilize distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021) to obtain reduced sampling iterations. Fur- thermore, this category encompasses advanced techniques such as implicit samplers (Kong & Ping, 2021; Song et al., 2020a; Zhang et al., 2022) and specialized differential equa- tion (DE) solvers. These solvers tackle both stochastic differential equations (SDE) (Song et al., 2020b; Jolicoeur- Martineau et al., 2021) and ordinary differential equations (ODE) (Lu et al., 2022a; Liu et al., 2022a; Zhang & Chen, 2022), addressing diverse aspects of diffusion model opti- mization. In contrast, the second category mainly focuses on model compression (He et al., 2025; Yang et al., 2024; Guo et al., 2024). It leverages techniques like pruning (Guo
Our contributions are summarized as follows:
2

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Leamable
Froeet
Backwwrd Data Flow
cache
[a
P]
Do
DiT
0n
Reuter
[02
81
cache
0.
04 0.
DIT O D
0.5
Forward Data
ROUTer
Flow
by
by
0
Update cache
Figure 3. One training iteration of I Learning-to-Cacee Cache. This method uses the noisy image æt as the input at t. Ci.ee denotes the loss function. "*" "DiT (+)" represents the timestep.
Use Cached
(=3
Figure 2. Generation process from random Gaussian noise ay to an image using feature caching (T N = 3). We omit the sampler and conditional inputs.
et al., 2020; Zhang et al., 2024: Wang et al., 2024b;c) and quantization (Shang et al., 2023; Huang et al., 2024a; He et al., 2024; Huang et al., 2024b; Lv et al., 2024) to reduce the workload in a static way. Additionally, dynamic infer- ence compression is also being explored (Liu et al., 2023; Pan et al., 2024), where different models are employed at varying steps of the process. In this work, we focus on the urgently needed DiT acceleration through feature caching, a method distinct from the discusdie ones.
3. Prelimilaries
Caching granularity. The noise estimation network of DiT (Peebles & Xie, 2023) is built on the Transformer block (Vaswani, 2017), which is composed of an Attention block and a ferd-forward network (FFN). Each Attention block and FFN is wrapped up in a residual connection (He et al., 2016). For convenience, we sequentially denote these Attention blocks and FFNs without residual connections as (bu. , bx-1), where N is their total amount. Fol- lowing the existing study (Ma et al., 2024a), we store the output of b, in cache as C₁. The cache, once completely filled, is represented as:
Feature caching. Due to the high similarity between ac- tivations (Li et al., 2023b; Wimbauer et al., 2024) across continuous denoising steps in diffusion models, recent stud- ies (Ma et al., 2024b; Wimbauer et al., 2024; Li et al., 2023a) have explored caching these features for reuse in subsequent steps to avoid redundant computations. No- tably, their strategies rely heavily on the specialized struc- ture of U-Net, e.g., samsainling blocks (Ma et al., 2024b) or Spat aartiialllaaa aaaaaaaarasssooo former blocks (Li al., 2023a). Be- sides, FORA (SeIvaraju et al., 2024) and DDDiT (Chen et al., 2024b) further apply the feature caching mechanism to DIT. However, both methods select the cache position and lifes- pan in handcrafted way. Learning-to- to-ch (Ma et al., 2024a) introduces a learnable cache scheme but induces dis- crepancies between training and inference. In this work! ! we design a new learning-
ca cache [co, CI .1...., CN-1)
(1)
Caching router.
The caching scheme for DiT can be for-
mulated with a pre-defined threshold T (0 < t < 1) and a customized router matrix:
Router = [Fr.]]SISST.SIISNIII E RTX,, (2)
where 0 < Ft, S 1 and T is the maximum denoising step. At timestep t during inference, the residual branch
corresponding to b, is fused with o, defined as follows:
(b, cs),
Et > T
framework to alleviate the discrepancies between the training and inference, which further enhances the performance and acceleration for DIT.
0 =
Ci:
rt. ST
(3)
where h, is the image feature and cs represents the condi- tional inputs 2. Specifically, Σt,; > 7 indicates computing by (h,, cs) as o,- This computed output also replaces o, in the cache. Otherwise, the model loads C; from cache without computation. A naive example of the caching scheme is depicted in Fig. 2. To be noted, err, is set to [1] [ by default to pre-fill the empty cache.
In addition, token caching (Zou et al., 2024; Lou et al., 2024), a granular way to reduce computation, recently emerged. It can be seen as an extremely fine-grained feature caching. Although compatible with our work, we only focus on feature caching with block-wise granularity as below.
It is worth noting that we focus on the real-time speedup ratio in this paper instead of the theoretical upper bound in some existing works (Zou et al., 2024; Chen et al., 2024b).
Cache usage ratio (CUR). In addition, we define cache
usage ratio (CUR) formulated as Σ L. in this paper represent the reduced computation from reusing cached features. In Fig. 2, CUR is roughly equal to 33.33%.
NxT
FFor example, represents the time condition and textual condition for text-to-image (T2I) generation.
3

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
(a) Step-Wise Denoising Training
(b) Image Error Proxy-Guided Objective
Teacher
1201 (2.08%) (24.
DIT
DIT
DIT
MSU
1.2
Studert
Dil
cacle
Uhr Garbed Poyher
Router
ON
DIT 03
Figure 5. The Mean Square Error (MSE) of xt for DiT-X1/2 256 x 256 (Pee- bles & Xie, 2023) induced by differ- Figure 4. Overview of HarmoniCa. (a) Step- Wise Denoising Training (SDT) mimics the multi- ent feature caching methods. 2; is the timestep inference stage, which integrates the impact of prior timesteps at t. (b) Image-Erron noisy image obtained at timestep + 1. For
Cache
Routerr M tD
"LIC" denotes Learning-to-Cace an efficient proxy (0) which is updated through gradient-free image generation passes every C fair comparison, < is employed training iterations. (M masks the Router to disable the impact of the caching mechanism for SDT. CUR is marked in the brackets.
Trixe-Guided Objective (IEPO) incorporates the final image error into the learning objective by
t. @ denotes the element-wis multiplication. See detailed algorithms in Sec. A.
4. HarmoniCa
leverages the following learning objective at timestep t:
In this section, we first observe that the existing learning- based caching shows discrepancies between the training and inference (Sec. 4. Then, we propose framework named HarmoniCa to harmonize them for better feature caching (Sec. 4.2). Finally, it shows better efficiency and applicabil- ity than the previous thining-based method (Sec. 4.3).
c = MSE + 3 E Te.i:
(4)
where ,3 is coefficient for the regularization term of the Router;: and cit) noise of DiT with and without feature caching at I.
MSE represents MSE between predicted
In contrast, the target for inference is to obtain a high-quality image To. Thus, Lit) induces a target mismatch due to bypassing direct image optimization. This potentially results in optimization shift (Rezatofighi et al., 2019) and severe object distortion, as shown by Fig. 6 (a) vs. (b).
4.1. Discrepancy between Training and Inference
Most previous approaches (Selvaraju et al., 2024; Chen et al., 2024b) for DiT set the Router in a heuristic way. To be adaptive, Learning-to-Cactt (Ma et al., 2024a) employs learnable Router* However, we have identified two discrepancies between its training and inference as follows.
4.2. Harmonizing Training and Inference
Existing studies (Ning et al., 2023; Li et al., 2024b; Ning et al., 2024) on diffusion models also validate that discrepan- cies between training and inference can lead to error accumu- lation and result in performance degradation. To solve the problem, we introduce HarmoniCa, a new learning- based caching framework that harmonizes training and inference through the following two techniques.
Prior timestep disregard. As illustrated in Fig. 2, the in- ference process employing feature caching at timestep is subject to the prior timesteps. For example, at timestep t = 1, the input x1 has the error induced by reusing the cached features Co and 61 at preceding timestep & 2. Fur- thermore, reusing and updating features at earlier timesteps also shape the contents of the current cache.
Step-wise denoising training. To mitigate the first dis- crepancy, shown in Fig. 4 (a), we propose a new train- ing paradigm named Step-Wise Denoising Traning (SDT), This strategy completes the entire denoising process over T timesteps, thereby accounting for the cache usage and update from all prior timesteps. Specifically, at timestep T, we randomly sample a Gaussian noise ØT and perform a single denoising step to pre-fill the cache. Over the fol- lowing T timesteps, the student model, which employs feature caching, step-wise removes noise to generate an image. Concurrently, the teacher model executes the same task without utilizing the cache. Requiring the student to mimic the output representation of its teacher, we compute the loss function and perform back back-propagation to update Router, r,; at each timestep f. To ensure that each Fe, is dif-
However, Learning-to-CC -oanche is unaffected by prior denois- ing steps during training. Specifically, for each iteration, as depicted in Fig. 3, it first uniformly samples a timestep akin to DDPM (Ho et al., 2020). It then pre-fills an empty cache at and proceeds to train Router at subsequent t - Without being influenced by the caching mechanism from T to f 1, this pattern incurs significant error accumula- tion (Arora et al., 2022; Schmidt, 2019) as demonstrated by the trends of red polyline in Fig. 5.
Objective mismatch. Moreover, we also find that Learning- to-Cache (Ma et al., 2024a) solely aimed at aligning the predicted noise at each denoising step during training. It
F in the ecuter is learnalle parameter.
4

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
(a) DIT-XL/2
(b) SDT+ LTC (1.40x)
(c) HarmoniCa (1.44x)
Figure Random samples for DIT-XI/2 256 x 256 w/ and w/o feature caching (T = 20). We mark the speedup ratio in the brackets.
ferentiable during training (Ma et al., 2024a), distinct from Eq. (3), we proportionally combine the directly computed feature with the cached one to obtain o,:
where kj, denotes the index of MM E R As depicted in Fig. 4 (b), 26g and m0 are final images generated from randomly sampled Gaussian noise ær using feature caching guided by (Upper) Router and (Lower) Router element- multiplied by _M respectively. Then, we can formulate X as:
o T2 (b, cs) (1 re,))c+-
(5)
Similar to inference, we also update in the cache with (,, cs) when Σt, > т. To improve training stabil- ity (Wimbauer et al., 2024), we fetch the output from the student as the input to the teacher for the next iteration. We repeat the above T learning iterations during training.
(8) where || || denotes the Frobenius norm. To adapt to the training dynamics, we periodically update all the co- efficients {x())... ((())))))(T)))))) every C iterations, where mod T = O, instead of employing static ones.
X 200 - x 11:
As depicted in Fig. 5, by incorporating prior denoising timesteps during training, SDT significantly reduces cumulated error at each timestep and obtains a much more accurate To, with lower computation, compared to LTC.
Fig. 6 shows that LIPO achieves significant performance improvement and yields accurate objective-level traits at higher speedup ratio than Lire. The study in Sec. justi- fies that employing Lire incurs the optimization deviation from minimizing the error of oo-
c(t)
Image error proxy-guided objective. For the second dis- crepancy, a straightforward solution to align the target with inference involves using the error of the final image Dg caused by cache usage directly with regularization term of Router our training objective. However, even for DIT-XL/2 256 x 256 (Peebles & Xie, 2023) with small training batch size, this requires approximately 5x GPU memory and 10x time compared to SDT combined with (!t) as detailed in Sec. B, making it impractical. There- fore, we have to identify proxy for the error of the image Ty that can be integrated into the learning objective.
4.3. Efficiency Discussion
Training efficiency. HarmoniCa incurs significantly lower training costs than the previous learning-based method. In Tab. 1, HarmoniCa requires no training images (i.e., image- free), whereas Learning-to- to-Chhh utilizes original training datasets. Thus, is challenging to apply Learning-to- co-Cache to models like the PIXART-a (Chen al., 2023) family, which are trained on large datasets, limiting its applicability. Moreover, while dynamic update of (*) incurs approxi- mately 10% extra time overhead, HarmoniCa requires only á training hours compared to Charning-to-Cachhe which needs to pre-fill fill the cache for each training iteration.
Based on the above analysis, we propose an Image Er- ror Proxy-guided Objective (IEPO). defined at each timestep follows:
c/1 C X() (t)
MSE* EN-1 It.
(6)
3
where (1)) is our final image error proxy treated as a coeffi cient of (1) caused by the cache usage at t. With a large ((), ((t) prioritizes reduction of the output error at f. This tends to decrease the cached feature usage rate at the corresponding timestep, and vice versa. Therefore, our proposed objec- tive considers the trade-off between the error of wo and the cache usage at a certain denoising step.
MSE- This proxy represents the final image error
Table 1. Training costs of lcarning- based feature caching for DìT- XL/2 256 256 (T 20). We train with all methods for 20K iterations using global batch size 64 on 4 NVIDIA H800 80GB GPUs. We set 500. Learning -oo to---oo-CChhhcc Cache uses the full ImageNet training set (Russakovsky al., 2015) as its original paper.
Method
#images
Timech)
Memory/GB//PP)
Leaming to
1.23M
2.15
SDT++400
0
1.47
38.25
Here, we detail the process to obtain X For a given Router, a mask matrix is defined to disable reusing cached features and force updating the entire cache at as:
HarmanCa
1.63
33.25
Inference efficiency, For inference, our method with a pre- learned Router has little computational overhead during
M
1,
j i
j j=t"
41 S J S T and S k s -
5

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
runtime. Additionally, less than 6% extra memory over- head ' is induced by cache for DiT-XI//2 256 x 256. There- fore, the introduced cost is controlled at a small level. For PIXART-a, HarmoniCa achieves over 2.07 x theoretical speedup 6 and x.69x real-world speedup with hugely im- proved performance than the non-accelerated one.
2019) as reference metrics. For PIXART-a, gauge the compatibility of image-caption pairs, we calculate CLIP score (Hessel et al., 2022) using ViT-B/32 (Dosovitskiy et al., 2020) as the backbone. To evaluate the inference effi- ciency, we measure CUR 7 and inference latency. In detail, we sample 50K images adopting DDIM (Song et al., 2020a) for DiT-XL/2, and 30K images utilizing IDDPM (Nichol & Dhariwal, 2021). + (Lu et al., 2022b), and SA-Solver (Xue al., 2024) for PIXART-a. All of them use classifier-free guidance (cfg) (Ho & Salimans, 2022).
We provide more training and inference costs in Sec. D.
5. Experiments
5.1. Implementation Details
More implementation details can be found in Sec. E, and the qualitative experiments are available in Sec. N. In addition, we apply the trained Router to a different sampler from training during inference in Sec. L., The robustness of our methods has also been validated in Sec. M.
Models and datasets. We conduct experiments on two dif- ferent image generation tasks. For class-conditional task, we employ DiT-XL/2 (Peebles & Xie, 2023) 256 x 256 and 512x x 512 pre-trained on ImageNet dataset (Russakovsky et al., 2015). For text-to-image (T2I) task, we utilize PIXART-O (Chen al., 2023) series, known for its outstand- ing performance. These models, including FIXART-XL/2 at resolutions of 256 x 256 and 512x 512, along with PIXART- XL/2-1024-MS at higher resolution of 1024 x 1024, are tested on the MS-COCO dataset (Lin et al., 2015).
5.2. Main Results
Class-conditional generation. We begin our evaluation for DÝT-XL/2 on ImageNet and compare HarmoniCa with current SOTA the approach employing fewer timesteps. The results in Tab. 2 show that our method surpasses all baselines. No- tably, with higher speedup ratio for a 10-step DiT-XL/2 256 x 256, HarmoniCa achieves an FID of 13.35 and an IS of 151 151.83, outperforming Learning-tor to-Cache by 1.24 and 6.74, respectively. Moreover, the superiority of our Har- moniCa increases as the number of timesteps decreases. We conjecture that it is because the difficulty to learn Router rises as the timestep goes up. Additionally, we further con- duct experiments with a lower CUR for this task in Sec. H.
torniingtt--Cchhe (Ma et al., 2024a) and
Training settings. Following (Ma et al., 2024a), we set the threshold 7 as 0.1 for all the models. Each of them is trained for 20K iterations employing the AdamW opti- 4 NVIDIA H800 80GB GPUs. The learning rate is fixed at 0.01, C is set to 500, and global batch sizes of 64, 48, and 32 are utilized for models with increasing resolutions. Additionally, we collect 1000 MS-COCO captions for T2I training.
mizer (Loshchilov & Hutter, 2019)
Baselines. For class-conditional experiments, we choose the current SOTA Learning- Maaiiingg---Caaaa Cache (Ma et al., 2024a) as our baseline. Due to the high training cost mentioned in Sec. 4.3, we employ FORA (Selvaraju et al., 2024) and A-DiT (Chen et al., 2024b), excluding Searning-to-Cace for the T2I task. The results of these methods are obtained either by re-running their open-source code (if available) or by using the data provided in the original papers, all under the same conditions as our experiments. We also report the performance of models with reduced denoising steps. For fair comparison, we exclude methods (Zou et al., 2024; Lou et al., 2024) employing highly different caching granularity, which is not the focus of the work, from our baselines.
Table 2. Accelerating generation on ImageNet for the DiT-XL/2 We highlight the best score in bold.
|sst
Pro..
Becall
CURTENT
Laxzariol
(ofi 1.D)
DOIM SSeep 2000ll
241337
2.27
4.25
80.25
50.37
1,55
DOIM /Sng al, 20000)
78
237.41
3.37
4.52
80.23
59.31
1411
258.4
2.36
4.34
$6.57
20.68
23.65
1.36 22/2
DOMM
20000
224.37
4.96
18,47
58.35
06611
DOIM Soog al., 2000))
201.33
5.77
55.06
(466
Lemning Cadde of cl, 2024
20
201.37
5.34
6.36
75.04
56.00
35.60
(.68
Hummeela
4442
DOii Sung el al, Iodik
10
19659
15.16
11.31
67.00
50.17
DOIM al. 2000G;
16.51
14.44
53.63
50.06
0.29
La (Ma et 20340
145.04
11.50
11.58
54/03
52.06
Evaluation. To assess the generation quality, Fréchet In- ception Distance (FID) (Nash et al., 2021), and sFID (Nash et al., 2021) are applied to all experiments. For DiT/XL- 2, we additionally provide Inception Score (IS) (Salimans et al., 2016), Precision, and Recall (Kynkäänniemi et al.,
HomseiCa
10
151.33
13.35
11:13
66.22
53.15
22.86
6.77)
Da 512
(12
1.5g
DOIM
54.50
33356
DOIM Soog in al, 2000;;
173.31
6.47
6.67
$1.10
10
2440 0043
Haoming-n--aaee (Ma 2134
17615
K1.21
53.30
2157
2017
HomselCa
2.574, 8813
'The cache occupics 0.49 GB GPU memory with batch size of 8 and original inference takes 8.18 GB GPU memory. 'The thcoretical specdup is based on floating-poin operations (FLOPs), and the real-world speedup is shown in Fig. 3.
T2I generation. We also present PixArt-a results in Tab. 3, comparing our HarmoniCa against FORA (Selvaraju et al.,
Definition can be found Sec. 3.
6

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Table Accelcrating generation MS-COCO for PIXART-a. The results of PIXART-2 (Chen et al., 2024a) family are available Sec. F. including generation with resolution of 2048 x 2048.
Table 4. Evaluation with additional metrics for PIXART-O. Mehhd
DINO:
HPSSaT
Plcasenet
CURRRI
Lawecyoo
PIXART-O 206 x (00 4.5)
Mechod
CUPT
HIDL
CUR
Leescyis)
DPM Schvree al., 20120)
20
0,5092
25.91
27.59
0.533
DPM Sclvern (La et... 20220)
15
0.2522
27.98
23.02
0.418
PIXART-O 250
250
(cigg
4.5)
FORA (Selvaajja et al. 2024)
0.2212
25.11
22.44
50.00
0.364: shx)
Solvere- (Lx al. 2021b)
20
33.66
27.08
0.553
HanoonCa
20
0.3335
26.72
26.65
56,91
0.546
DFM Solvere- (La al. 20236)
15
30.77
31.68
38,92
0,418).22x
POXART-o 512 x (12 4.5)
FORA cSelvangg al, 2024)
20
$1.10
37:42
$7.98
50.00
DPM Schvree cla eial. 20220)
20
0.5339
30.53
28.52
1,799
HammaiCC
31.13
26.33
56,01
0.346 nex1
DPM IPM--Sctrreee etal. 2012h)
0/5177
78:19
22.04
[_a9X]
IDDPM (Nichol Dharwoll 2021)
31.25
26.15
33.65
2.572
FURA (Seharaga al. 2024)
20
0.3059
29.82
21,96
50.0
1.150
IDOM (Nichol Dharwal 2021)
75
51.25
14.17
33,73
1.80
Hamonitss
20
IL3289
36.38
27,47
54.64
FORA (Schanga al, 3024)
100
31.25
25.16
33.62
50,00
1.558
Harmoni'ss
100
M1.17
23.33
1.523 stee
Sotere (Nus atal. 2024))
25
31.M
16.78
SA-Suleer (Naz al. 2024)
20
31.23
27:45
39.01
0,665
in LFM (Dao et al., 2023). As shown in Tab. 5 (T' = 100,
Hormonit's
25
31.27
27.07
38.62
$4.19
0,56 sex)
Euler Solver, and a batch size of 8), it achieves substantial speedup ratios without performance degradation.
PoxAat 612
532
(c.Eg
4.5)
DPM-Sullare- (Ia et al. 20220)
20
31.30
13.96
40.54
1,750
DFM Solvere- (L. et al. 20226)
15
31.29
25:12
40.57
Harmonitas SA Solver (oee at. 2024)
31.29
24N1
1.77 2220
25
31.25
25.43
39.84
Table 5. Accelerating generation for LFM (Dao et al., 2023).
Salver (Xuz at. 2024)
20
31.19
25.85
40.08
Hammonttas
25
$1.30
25.74
39.99
$4.24
1.20)
Dutaeet
Model
HD
Liteacytsa)
PINART-O 1024
1024(cc0
4.5)
LSUN Bedroom 256 (50 2016)
DIT
5.22
1,76
HarmoniCa
DIT-L/2
5,13
1.09
DPM-Sollare- (Ia at al. 2022h)
20
$1.10
25:01
37.00
9.470
CalebuHQ 206 i 200 (Kurms al., 2018)
DIT-L/2
5.42
1.76
DPM Solvere- (I. et at. 20226)
15
$1.07
25:77
42.50
7,041
w/ HarmoniCa
DIT-E/2
5.41
1.07
Hs
20
31.09
13012
ImugeNNrr 256 x 256 (Russkkonsky et al. 2015)
DIT 8/2 (ede- - 1.5)
5.06
1.37
Solver (iunetal all 2034)
31.06
15.05
36.12
11.931
SA-Saber ctal, 2024)
20
$1.00
23.88
30.41
p,000
HarmoniCaa
DIT-8/2 (edy 1.50
5.04
0.87 tex)
Hormanics
31.0
13.77
53,98
Comparison with the increase in speedup ratio. To emphasize the significant advantage of our method over Learning-to-Cache, we present the IS and FID results as the speedup ratio increases for both Learning-to-Catt our methods in Fig. 7. As the speedup ratio grows, the gap between Learning- to- co-Cache and our approach widens substantially. Specifically, with a speedup ratio of approxi- mately 1.6, HarmoniCa achieves substantially higher IS and lower FID scores, 30.90 and 12.34, respectively, compared to Learning-to Loaiiin-tt-Cache.
2024) and the method using fewer timesteps. HarmoniCa outperforms these baselines across all metrics. For example, with the 20-step DPM- PIXART-o 256 x 256 em- ploying HarmoniCa achieves an FID of 26.33 and speeds up by 1.60x, surpassing the non accellrreered model's FID of 27.68. In contrast, DPM- Solver++ with 15 steps and FORA only achieve FIDs of 31.68 and 27.42, respectively, with speed increases under x.52x. Notably, HarmoniCa also cuts about 41 % off processing time without dropping performance when using the IDDPM sampler, while FORA results in over a 1.10 FID increase compared with the non- accelerated model. Overall, our method consistently deliv- ers superior performance and speedup improvements across different resolutions and samplers, demonstrating its effi- cacy. Additionally, in Sec. I, HarmoniCa also significantly outperforms DiT (Chen et al., 2024b).
and
35-
30
-12.34
2 120
25-
100
+30.00
20-
Louning
15*
12
1.5
1.
11
1.2
13
14
1.3
1.
Specdup Katio
Speedup Ratio
Besides the above evaluation, we have also conducted ex- periments with more metrics (e.g., Ieage-Reward (Xu et al., 2024), LPIPS (Zhang et al., 2018), and PSNR), which are provided in Sec. K. Here, we present DINO (Caron et al., 2021) and human evaluations (e.g.. HPSv2 (Wu al., 2023) and PickScore (Kirstain et al., 2023)). As shown in the Tab. 4, our method outperforms baselines and achieves com- parable performance with non-accelerated models.
Figure 7. IS/FID with the increase of the speedup ratio for different methods. We employ DIT-XL/2 with a 10-step DDIM sampler on ImageNet 256 256.
Comparison with additional feature caching methods. To highlight HarmoniCa's 's advantages, we compare it with DeepCache (Ma et al., 2024b) and Faster Diffusion (Li et al., 2023a) on a single A6000 GPU. Due to the par- tial
Results for flow-based models. HarmoniCa can be effectively applied to rectified flow models (Liu et al., 2022b). We employ the pretrained models and evaluation
of the compared methods and the lack of implementation details, we directly report their results
7

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
from Learning-to-Cache. HarmoniCa shows a
As demonstrated in Tab. 6,
Table 8. Results of combining our framework with an advanced quantization method: EfficientDM (He et al., 2024). ISt is for DLL and CLIPt is for PIXART-Q the table. Experimental details here can be found in Sec. G.
negligible < 0.05
FID increase with a 1.65x speedup ra-
Table Comparison between different caching- based approaches. We use U- ViT (Bao et al., 2023) on ImageNet x56x here.
Mathad
18
Finj
sFID]
(URRAR)
Latreybl)
tio, outperforming both methods. No- tably, DeepCache constrained by the U-shaped structure, making it unsuit- able for DiTs.
DET-XL22
296
256 (cti
1.5)
Efficien1000 (Tle er al., 2009)
172,70
6.10
4.55
Methed
FIDJ Latasyoo
w/ HammaroCa (1 - demlly
165.16
6.48
4.32
36:25
0,47311
Soll-S 2032g)
7.50
PIXARI- 336 x cotg
= 0.5
Poster Dittasionn (LA 201360
5,9% mex)
FfficinelMM 2004)
10.9
DeepChohe (Ma et 2004))
20 2.70
Bemoniias
4,60 46.3
HammoniCa
30.15
34.96
30.55
53.34
0.250 ss::
re.)
PIXART-I 5/2 x M124
1.5)
EffiicudDM (Hic es al., 2004)
30.71
25.82
41.64
n (1
0.59
w/ Hammoriaa
30.75
26.15
41.99
53.11
0,281
0.99;
Table 7. Comparison between different acceleration approaches. We use DiT-XI/2 on ImageNet 256 here. "*" denotes the la- tency was tested on one NVIDIA A100 80GB GPU. Experimental details are presented in Sec. G.
our SDT significantly decreases FID by 10 compared to that of Seariingtto-Cache For the learning objective, our IEPO achieves nearly a 40 IS improvement and 3.13 FID reduction for SDT compared with _ 4irc- Moreover, both SDT and IEPO can help significantly enhance performance for the counterparts in the table. For a fair comparison, we modify the implementation of Soominggtt-Cachhee to train the entire Router in Tab. 9. A detailed discussion of this can be found in Sec. J.
Michod
Is;
FIDJ
.FTD]
Lateneyto)
Lateneyto]
DDIM (Zhang et al., 2002)
20 224.77
3.52
4.96
0,658
1,217
#MeiznDMM al. 2014)
20
172.70
6.10
4.55
0.59% mX)
0,842 48x)
PTQDDIT (Wu et al., 2024)
20
17.06
71.82
23.16
0.577
0.839 40xd
Dift-raninn (Fing ol., 2023)
20
168.10
8.22
6.30
0,458
0
HarmoniCs
20
4.80
(.4k)
( mlx)
Comparison with pruning and quantization. As shown in Tab. 7, we compare our HarmoniCa with advanced quan- tization and pruning methods. Our method significantly outperforms these methods, demonstrating the substantial benefit of feature caching for accelerating DiT models. It is important to note that the specdup ratio for quantization is partially determined by hardware support, which we do not rely on, and the current customized CUDA kernel often lacks optimization we believe the significant performance drop of PTQ4DiT results from small sampling steps. A 50/250-step DDPM sampler is used in the original paper.
Table 9, Effect of different components. The first row denotes the model w/o feature caching. The green and blue rows denote Learning-to-Cade and HarmoniCa, respectively.
Triniin Partign
Leaming Objectve
Ist
FIDJ
SFID
CURIS)1
Lusescy'sL
Leaming to Cache
SDT
Li
224.37
3.52
4.6
0.688
115.00
32.68
(.483
203.41
5.20
6.07
36.70
0,458
H800's Hopper architecture. Here,
34.20
4.88
591
0.456 443)
Effeet of itera- tion interval C. As illustrated in Fig. 8, we carry out experiments to evaluate the impact of differ- ent values of c on updating ((t) in Eq. (8). De- spite the similar speedup ratios, employing an extreme C value leads to no- table performance drops. Specifically, a large C means the proxy (4) fails accurately and timely reflect the caching mechanism's 's effect on the final image. Conversely, a small C results in overly frequent updates, complicating training convergence. Hence, we choose a moderate value of 500 as C in this paper based on its superior performance.
Combination with quantization. We conduct experiments to show the high compatibility of our HarmoniCa with the model quantization technique. In Tab. 8, our method boosts considerable speedup ratio from 1.18x to 1.85x with only a 0.12 FID increase for PIXART-Q 256 x 256. In the future, we will explore combining our HarmoniCa with other ac- celeration techniques, such as pruning and distillation, to further reduce the computational demands for DIT.
23
840
150
4013
800
185
000
Inserval
Figure 8. Ablation results of different iter- ation interval C. "ø" denotes the employ- ing ((t) as loss function.
5.3. Ablation Studies
In this subsection, we employ a 20-step DDIM (Song et al., 2020a) sampler for DIT-XL/2 256 x 256.
Effect of different components. To show the effectiveness of components involved in HarmoniCa, we apply different combinations of training techniques and show the results in Tab. 9. For the training paradigm, equipped with c(t) LrC.
8

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Table 10. Ablation results of different metries for ( The first and second columms represent the model w/o feature caching and SDT+ SDT+ respectively. (-)..)) denotes Kullback-Leibler (KL)
Table 11. Performance of Harmon-
iaa across different values of
divergence. We employ 3 the paper to obtain X()
[0, 1). T is the threshold for the
+
Els - =
| - =
Pxc(xa. )
1 MS- ASS-SSI(( =0))
LPIPS x6
Router described in Sec. 3.
1St
224.37
166.65
172.08
206.57
205.91
204.72
205 83
T 15- FID, .FID)
FID]
3.52
8.01
6.95
5.25
491
a1
10
15188
13.85
11.13
0.200 (00)
xFID.
4.96
7.62
5.91
5.51
5.83
5.57
05
10
151.00
13443
11.09
(.209
CUR(S))
34.20
34.82
37.50
36.79
37.68
37.32
09
10
151.8
1330
11.08
0.200
Latency())
0.173/..003)
(.456
4.458
(.456
0.456
Effeet of coeffi-
Table 12, Comparison between SDT and teacher forcing. Both employ LEPo as their loss functions.
cient B. We also explore the trade- off between infer- ence speed and performance for various values of in Eq. (6). As shown in Fig. 9, a higher B leads to greater acceleration but the cost of more pronounced performance degradation, and vice versa. Notably, perfor- mance declines gradually when 3 < 8e- and more sharply outside this range. This observation suggests the potential for autonomously finding an optimal 33 to balance speed and performance, which we aim address in future research.
Oydeel
Mehual
T
BS1
FIDJ
.FID.
Lanancytil
DIT 236
296
SDT
4.88
206.57
444
Teacher forting
4.87
207.12
6,02
Figure 9. Ablation results of different co- efficient 8. "ø" denotes the model w/o feature caching.
DIT-XL/2 x (12g
1.5)
SDT
20
5.73
179.84
6.61
2.57
Teucher foscing
5.74
178.95
2.777
6. Conclusions and Limitations
In this research, we focus on accelerating Diffusion Trans- formers (DiT') through a learning- caching mecha- nism. We first identify two discrepancies between training and inference of the previous method: (1) Prior Timestep Disregard in which earlier step influences are neglected dur- ing training, and (2) Objective Mismatch, where training only focuses on intermediate results. To solve these, we introduce a novel caching framework named HarmoniCa, which consists of Step-wise Denoising Training (SDT) and an Image Error Proxy-Guides Objective (TEPO). SDT cap- tures the influence of all timesteps during training, while IEPO introduces an efficient proxy for image error. Exten- sive experiments show that HarmoniCa achieves superior performance and efficiency with lower training costs than the existing training-based method. In terms of limitations, we focus on block-wise caching and image generation in this work. However, we believe our work easy to expand to any caching granularity and video/audio/3D generation.
Effect of different metrics for A()) In Tab. 10, we con- duct experiments to explore the effect of (0) with different metrics. Both || ||2 and (- lead to notable perfor- mance enhancements compared to using only the output error (i.e., (+) = 1) at each timestep. Due to the insensitiv- ity to outliers, Σl - is generally less effective for image reconstruction and inferior to the others in the table. We fur- ther test MS-SSIM (Wang et al., 2003) and LPIPS (Zhang et al., 2018), which are designed to evaluate natural image quality as metrics for ((). These metrics exhibit similar performance compared with ||- -12..
Effect of different threshold т. we conduct study on dif- ferent values of caching threshold in Tab. 11. The results demonstrate our method is robust w.s variations in r. Thus, set to 0.1 for all the experiments in this work.
A Acknowledgement
SDT vs. teacher forcing. Teacher forcing, which uses the outputs of a non-accelerated teacher model as the input data (i.e., replace e(!)' by e(t) in the 13th row of Alg. 1) for the next iteration of training, may further help mitigate cumula- tive errors. In Tab. 12, SDT shows comparable results with teacher foreing, which indicates that using (!)) would not lead to potential error accumulation compared with ett) A more detailed theoretical analysis is planned for future work.
We thank Chengtao Lv and Yuyang Chen for their insights and feedback, and Yifu Ding for her help with diagrams. This work was supported by the Hong Kong Research Grants Council under the Areas of Excellence scheme grant AoE//-601/22-R and NSFC/RGC Collaborative Research Scheme grant CRS HKUS6603/22. It was also supported by the Beijing Municipal Science and Technology Project (No. Z23 100010323002) the National Natural Science Foundation of China (Nos. 62306025, 92367204), CCF. Baidu Open Fund, and Beijing Natural Science Foundation (QY24138).
$ AleeNett (Krizhevsky et al., 2017) used extract features.

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
Impact Statement
Dhariwal, P. and Nichol, A. Diffusion models beat gans
image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner. T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
References
Arora, K., El Asri, L.,, Bahuleyan, H., and Cheung, J. Why exposure bias matters: An imitation learning perspective of eгrоr accumulation in language generation. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 700-710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10118655/1/222/ findings-acl.58. URL https: /atsss//allaaae /aaaaaaaaannoooooooooooooooooooa org/ htgg2022.o0 wwndings-nnnnnnnnn.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Fang, G., Ma, X., and Wang, X. Structural pruning for diffusion models, 2023. URL https::/arxxv..rrtt abs/ 220..2355.110 1094..
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, All are worth words: A vit backbone for diffusion mod- cls, 2023. URL https: / 12152.
Gong, R., Liu, X., Jiang, Li, Hu, P., Lin, J., Yu, F., and Yan, J. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
Gong, R., Yong, Y., Gu, S., Huang, Y., Lv, C., Zhang, Y.. Tao, D., and Liu, X. LLMC: Benchmarking large language model quantization with a versatile compres- sion toolkit. In Demoncourt, F., Preofiuc-Pietro, D., and Shimorina, A. (eds.), Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Pro- cessing: Industry Track, pp. 132-152, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653//
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers, 2021. URL https: //arxiv. org/ aww.////......../p 14294.
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-)alpha, Fast training of diffusion transformer for photorealistic text- to-image synthesis. arXiv preprint arXiv:2310.0042., 2023.
12. URL https: https://aallaathhlogggeo wrge 2024. emnlp- heillp-industr..co 12/.
Guo, J., Ouyang, W., and Xu, D. Multi-dimensionn pruning: A unified framework for model compression. In CVPR, 2020.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L.,, Ren, X., Wang, Z Luo, P.. Lu, H., and Li, Z. Pixart- Fixart-'signa sigma: Weak-to-strong training of diffusion transformer for 4k text-to-ing eot-tt--mmaee generation. arXiv preprint arXiv:2403. 04692, 2024a.
Guo, J.. Wu, J., Wang. Z., Liu, J., Yang, G., Ding, Y., Gong, R., Qin, H., and Liu, X. Compressing large language mod- els by joint sparsification and quantization. Forty-first International Conference on Machine Learning, 2024.
Chen, P., Shen, M., Ye, P., Cao, J., Tu, C., Bouganis, C.-S., Zhao, Y., and Chen, T. 5-dit: A training-free free acceleration method tailored for diffusion transformers, 2024b. URL https:/ / /arxivivvv..////a//aaaa//aaaaavv org/ atch/bbs/20000 hbt.//24.....1..
He, C., Ding, Y., Guo, J., Gong, R., Qin, H., and Xianglong, L. Da-kd: Difficulty knowledge distillation for ef- ficient large language models. In Forty-first International Conference on Machine Learning, 2025.
Chen, T., Xu, B., Zhang, C., and Guestrin, C. Training deep nets with sublinear memory cost, 2016. URL https: //a //arxiii taaxii..a org/ abs/ / 604 06174.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Dao, Q., Phung, H., Nguyen, B., and Tran, A. Flow match- ing in latent space. arXiv preprint arXiv:2307.08695 2023.
10

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
He, Y., Liu, J., Wu, W. Zhou, H., and Zhuang, B. Effi cientdm: Efficient quantization-aware fine-tuning of low- bit diffusion models, 2024. URL https: //axiv.. ht5/abs/2330 0327C.
Kong, Z. and Ping, W. On fast sampling of diffusion proba- bilistic models. arXiv preprini arXiv: 106.00/32, 2021.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Commun, ACM, 60(6):84-90, May 2017. ISSN 0001- 0782. doi: 10. 145/3065386 URL https:/ /doi.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.,, and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning, 2022.
org/10 1 1144/30653388
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium, 2018. URL https: /tta://arxxiv aariv.iiivvv// org/ httg/bb/////e 1706. 08500.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. Advances in neural information processing systems, 32, 2019.
Ho, J. and Salimans, T. Classifier-free free diffusion guidance. arXiv preprini arXiv:2207. 12598, 2022.
Li, D., Kamko, A., Akhgari, E., Sabet, A., Xu, L., and Doshi, S. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024a. URL https: www.aaavvvv........ httpa/axvvv..iraaaeea/aa wtt.://rrgg/bbb/2...222222
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6855 2020.
Li M., Qu, T. Yao, R., Sun, W., and Moens, M.-F. Allevi- ating exposure bias in diffusion models through sam- pling with shifted time steps, 2024b. URL https: // //arxiv. org/ /as/////////// / 2200 15583.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models, 2022. URL https:/ arxiv. .rrg aa///bs//// 2204 03444
03458.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: arrxiv. org/abs/ 2106 6. 09685.
Li, S., Hu, T., Khan, F. S., Li, L.,, Yang. S., Wang, Y., Cheng, M.-M., and Yang, J. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv preprint mrXhi:2312.0900 2023a.
Huang, Y. Gong, R., Liu, J., Chen, T., and Liu, X. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7362- 7371, 2024a.
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing dif- fusion models. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp. 17535-17545, 2023b.
Huang, Y., Gong, R., Liu, X., Liu, J., Li, Y., Lu, J., and Tao, D. Temporal feature matters: A framework for diffusion model quantization. arXiv preprint 2xXV:22407.19544, 2024b.
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L.., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. Microsoft coco: Common objects in con- text, 2015. URL https: /tass://aaxiii /ttp/vxivvvv///ii///a/////aaavva wiexiivooo org/ abs/1405. 0312.
Jolicoeur-Martineau, A., Li, K., Paile-Taillefer, R., Kach- man, T., and Mitliagkas, I. Gotta go fast when gen- erating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Liu, F.,, Ning, X., Lin, Z.., Yang, H.,, and Wang. Y. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. In International Conference on Machine Learn- ing. pp. 21912 21936. PMLR, 2023.
Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of gans for improved quality, stability, and varia- tion, 2018. URL https: /ttp/aaxxvvv./////////////.a org//bs//// 10196.
Liu, L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numeri- cal methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022a.
Kerr, A., Merrill, D., Demouth, J., and Tran, J. Cutlass: Fast linear algebra in cuda c++. NVIDIA Developer Blog, 2017.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learn- ing to generate and transfer data with rectified flow, 2022b. URL https: arxiv wweeiiv.orree orgg aas//b///// / 22092 03003.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL https://arxits wtt./oog////aooooooo 01569.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, 2019. URL https: /aaxii. /bbs/ 1711.05100

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
Lou, J., Luo, W., Liu, Y., Li, B., Ding, X., Hu, W., Cao, Li, Y., and Ma, C. Token caching for diffusion transformer acceleration. arXiv preprint arXiv:2409 18523, 2024.
Pan, Z., Zhuang, B., Huang, D.-A., Nie, W., Yu, Z., Xiao, C., Cai, J., and Anandkumar, A. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching, arXiv preprint WxXiv:2402.14100 2024.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neurul Information Processing Systems, 35:5775-5787, 2022a.
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu. J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022b.
Podell, D., English, Z., Lacey. K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for nigh-resolution image synthesis, 2023, URL https: http:///axxvv.rrr. abs/ 2307.01955
Luhman, E. and Luhman, T. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02386 2021.
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Lv, C., Chen, H., Guo, J., Ding, Y., and Liu, X. Ptq4sam: Post-training quantization for segment anything. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattrn Recognition, pp. 15941-15951, 2024.
Ma, X., Fang, G., Mi, M. B., and Wang. X. Learning- to-cache: Accelerating diffusion transformer layer caching, 2024a. URL
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
www.aiiwww..oooaeeoooooooooeeeee abs/
2406.00 01733.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmentation, 2015.
Ma, X., Fang. G., and Wang, X. Deepeache: Acceler- ating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 15762-15772, 2024b.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115: 211-252, 2015.
Ma, X., Wang, Y.. Jia, G., Chen. X., Liu, Z., Li, Y.-F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03046 2024c.
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang. J., Denton, E. L.. Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., and Blankevoort, T. A white paper on neural network quantization, 2021. URL https: //arxiv.org/ abs /216. 08295.
Nash, C., Menick, J., Dieleman, S., and Battaglia, P. W. Generating images with sparse representations. arXiv preprint arXiv:2 103.0381,, 2021.
Salimans, T. and Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005122 2022.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-811 8171. PMLR, 2021.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Ning, M., Sangineto, E., Porrello, A., Calderara, S., and Cucchiara, R. Input perturbation reduces exposure bias in diffusion models, 2023. https:// /atpix.xxvvv...//..a.... abs/ 2tt..231..oo 11706.
Schmidt, F. Generalization in generation: A closer look at exposure bias. In Birch, A., Finch, A., Hayashi, H., Konstas, I., Luong, T., Neubig, G., Oda, Y., and Sudoh, K. (eds.), Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 157-167,
Ning, M., Li, M., Su, J., Salah, A., and Ertugrul, O. Elucidating the exposure bias in diffusion models, 2024.
URL httph:tt///eee. /ttss://ariii org/ /200/ 15321.
12

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
Hong Kong, November 2019. Association for Compu- tational Linguistics. doi: 10.18653//1/D11--5611 URL https: http://caannthohooeo org/D19- 100gg/D99-5500 5616.
Wu, J., Wang, H., Shang. Y., Shah, M., and Yan, Y. Ptq4dit: Post-training quantization for diffusion trans- formers, 2024. URL https: /trpsv////avvva///aeeeeeeee org oogoorggg///....ooooo.ooo abs/ 2405.16000
Selvaraju, P.. Ding, T., Chen, T., Zharkov, I., and Liang,
L. Fora: Fast-forward caching in diffusion transformer acceleration. arXiv preprint arXiv:2407.01425. 2024.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of eext-to-image synthe- sis, 2023. URL https: /tttps://arxvv. org/ abb/23b6... 09341.
Shang, Y., Yuan, Z., Xie, B., Wu, B., and Yan, Y. Post- training quantization on diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1972-1981, 2023.
J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang. J., and Dong, Y. Imagereward: Learning and evaluating hu- man preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint Mxii:20111.13456 2020b.
Xue, S., Yi, M., Luo, W., Zhang, S., Sun, J., Li, Z., and Ma, Z.-M. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., and Romero-Soue Soriano, A. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions, 2024. URL https: /ttp///axvvvv//v///////eee/aaaaa org/ abs/ 2312. 08578.
Yang, G., He, C., Guo, J., Wu, J., Ding, Y., Liu, A., Qin, H., Ji, P., and Liu, X. Llmebench: Benchmarking large language model compression for efficient deployment. NeurIPS, 2024.
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop, 2016. URL https: /aa//aavvvv/////i///eee/e/eeaeve org/ abs /5006. 03365.
Vaswani, A. Attention is all you need. Advances in Neurul Information Processing Systems, 2017.
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., and Wei, F. Deepnet: Scaling transformers to 1,000 layers. IEFE Transactions on Pattern Analysis and Machine In- telligence, 2024a.
Yuan, Z., Lu, P., Zhang, H., Ning, X., Zhang, L.,, Zhao, T. Yan, S., Dai, G., and Wang, Y. Ditfastattn: Attention compression for diffusion transformer models. arXiv preprint arXiv:2406. 08552, 2024.
Wang, K., Chen, J., Li, H., Mi, Z., and Zhu, J. Sparsedm: Toward sparse efficient diffusion models, arXiv preprint arXiv:2404.10445 2024b.
Zhang, D., Li, S., Chen, C., Xie, Q.. and Lu, H. Laptop-diff: Layer pruning and normalized distillation for compress- ing diffusion models. arXiv preprint arXiv:2404.1 1098, 2024.
Wang, Z., Simoncelli, E. P., and Bovik, A. C. Multiscale structural similarity for image quality assessment. In The SSveethSevetth Asilomar Conference on Signals, Sys- tems Computers, 2003, volume 2, pp. 1398-1402. leee, 2003.
Zhang, Q. and Chen, Y. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.1399 2022.
Wang. Z., Guo, J., Gong, R., Yong, Y., Liu, A., Huang, Y., Liu, J., and Liu, X. Pisbench: A comprehensive post-training sparsity benchmark towards algorithms and models. Proceedings of the 32nd ACM International Conference on Multimedia, pp. 5742-5751, 2024c.
Zhang, Q. Tao, M., and Chen, Y. gddim: Generalized denoising diffusion implicit models. arXiv preprint WKKv:2206.005644 2022.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang. O. The unreasonable effectiveness of deep features as perceptual metric, 2018. URL https: /arxiv.orgg//////a......... abs/ / ///1/1111.10 008880119292 03924.
Wimbauer, F., WU, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., et al. Cache me if you can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6211 1-2200 2024.
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojamazeri, H., Ott, M., Shleifer, S., Desmai- son, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan,
13

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Expe- riences on scaling fully sharded data parallel, 2023. URL https: /arxiv.iivvvv.///////.e///aaaa org/ abs//bb///// 2 11277.
Zou, C., Liu, X., Liu, T., Huang, S., and Zhang, L.. Ac- celerating diffusion transformers with token-wise feature caching. arXiv preprint arXir:2410.05317, 2024.
14

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
A. Alogrithm of HarmoniCa
As described Alg. 1, we provide a detailed algorithm of our HarmoniCa. For clarity, we omit the pre-fill stage (i.e., denoising T), where Routerr: is forced to be {+ The conds for T2I tasks and class-conditional generation are pre-prepared text prompts and class labels, respectively.
Algorithm HarmoniCa: the upper snippet describes the full procedure, and the lower side contains the subroutine for computing the proxy of the final image emror.
fune HARMONICA((s, Es, iters, conds, 3,
Require: o(.) difflusion sampler
c*() DiT model
iters amount training iterations
conds conditional inputs
threshold
constraint cocfficient
maximum denoising step
iteration interval
1: Initialize Router with normal distribution
2: cacne = =0
Initialize cache
for to itess do:
4: FT ~ ((,
5:
if if = then
6:
"A"",...,AA7)) . = gen.proxy(e. e0, ær, conds(), Router)
7:
end if
8:
for in T to do:
(") es c conds Ro cache)
D Fig.
10:
e11) co(0,, conds((]))
11:
LIPPO "le" 1 3 r]
> Eq. (6)
12:
Tune Router, by paaekppropgattoon
13:
T--1 t,
end for
15: end for
16: return Router
func gen.proxy(ó, Eg, ær, cond, t, Router)
b Wrapped by stopgradient operator
1: cache =
Initialize cache
2: Employ feature cache guided by Router to generate wi
3: for in 7 to do:
4: Generate (9)
> Eq. (7)
5: Employ feature cache guided by Router & ( to generate x.
(8)
6: X() || =
b Eq. (8)
7: end for
return (1 12) 2033
B. Image Error with Router Regularization Term as Training Objective
In Tab. 13, SDT+ ((t) requires additional denoising passes per training iteration at to compute the error of Ep. Consequently, this ap- proach consumes about 9.73 GPU hours compared to SDT+ Due the extensive intermediate activations stored from timestep t for propaarop,,,,,,n, also costs 4.90x GPU memory. This estimation is conducted with small batch sizes and limited iterations. Therefore, SDT+ c(t) is less feasible for models with larger latent spaces or higher token counts per image, such as DiT-XL/2 512 x 512, particularly in large- batch, complete training scenarios. Additionally, the network effectively becomes T N stacked Transformer blocks under this strategy, making it difficult (Wang al., 2024a) to optimize
Table Training costs estimation across different methods for DiT-XL/2 x56 (Peebles & Xie, 2023)
(T = 20). We only employ iterations with a global batch size denotes the loss function replacing LMSE in Eq. (4)
NVIDIA H800 80G GPUs. Cit)
with the final image error.
Method
#Tmages
Time(h)
Memory(GB/GPU)
SDT (0
0
1.46
65.36
SDT+ 12C
0.15
13.33
15

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
the Router with even moderate T value, such 50 100.
C. Optimization Deviation
FID: 8.01, CUR(%): 34.20
107
20
35
(a) SDT+ LTC
FID: 4.88, CUR(%): 37.50
107
Tpach
Dpack
Taach
X0 Rock
600 Dpcck
600 Spct
20
Ag Soy
(b) HarmoniCa
Figure 10. (Left) Variations of Sse and (*) for SDT+ (Right) Rout er visualization different methods. The gray grid (t, i) represents using the feature in cache at t without computing o,. The white grid indicates computing and updating cache. We also mark their FID (Heusel al., 2018) and CUR. All the above experiments employ DiT-XL/22 256 256 (T 20, N 56).
To generate high-quality 3o and accelerate the inference phase, we believe only considering the output error certain timestep can cause deviated optimization due to its gap w.z/ the error of zo. To validate this, we plot the values of 1 MSE Eq. (4) and (1) in Eq. (8) during the training phase of SDT+ in Fig. 10 (Left). Comparing ((t) LSSE and ( across different denoising steps, their results present a significant discrepancy. For instance, MSE / t= 14 is several orders of magnitude smaller than that at
during the entire training process, and the opposite situation happens for (4)) Intuitively, this indicates that we could increase the cache usage rate at t = 1, and vice versa at t = 14 for higher performance while keeping the same speedup ratio according to the value of the proxy ( However, only considering the
output error at each timestep (i.e., c(t) < can optimize towards shifted direction. In practice, the learned Router with the guidance of (4) in Fig. 10 (Right) (b) caches less in large timesteps like t = 14 and reuses more in small timesteps as t == compared to that in Fig. 10 (Right) (a) achieving significant performance enhancement.
D. Training and Inference Costs for PIXART-a
We provide the computation costs of training and inference for PIXART-a (Chen al., 2023) in Tab. 14. It is worth noting that we only use naive distributed data parallel (DDP) training without any advanced strategies like gradient checkpointing (Chen et al., 2016) and FSDP (Zhao et al., 2023). Thus, the requirements of computation can be further decreased in practice.
Table 14, Training and inference costs,
Mathod
Infer. Men. (GRUPUU)
Trais (GBMPUU
Time
PIYARTsa 255 750 4.5)
DPM-SSleeree (La 2022h)
20.56
Hormonil's
21:21
70.05
1.34
PIXART- 512 312
4.5)
DEM DEM.Subeer Sulvere (La d., 202b)
24.11
Hormonit's
24,1
77.30
2:2
16

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
E. More Implementation Details
In this section, we present more details on the implementation of our HarmoniCa, First, following (Ma et al., 2024a), we also perform sigmoid function 9 each Et,; before it is passed the model. Moreover, unless specified otherwise, the hyper-parameter in Eq. (6) for all experiments given in Tab. 15; any exceptions are noted in the relevant tables.
Table 15. Plyper-parameed | for training the Router.
Model
DiT-XL/2
PIXART-O
PIXART-E
Resolution
256
256
512 x 512
256
256
512 x 512
1021 1021
512
512
1021
1924
2018 2018
T
10
20
50
20
20
100
25
20
20
20
20
20
8 Te
5e-
de
1e-3
8e-
8e-1
Se-+
Se-1
1e-1
8e-4
8c-
F. Results for PIXART-E
In this section, we present the results for the PIXART-E family, including PIXART-E-XL2-1024-N and PIXART-E-XL/? 2K-MS. For the latter one, we test by sampling 10K images. Additionally, we train the Router with batch size of 16 and measure latency using batch size of 1. All other settings consistent with those described in Sec. 5.1.
As shown in Table 16, HarmoniCa achieves a 1.56x speedup along with improved CLP scores compared to the non- accelerated model for PIXART-> 2048 x 2048. Notably, this is the first time for the feature caching mechanism to accelerate image generation with such a super-high resolution of 2048 x 2048.
Table 16. Accelerating image generation MS-COCO for the PIXART-E.
Maled
CLIP
FID,
sFID
CUR(S)
Laines6a))
PIXARI-E, 512 M12 4.5)
DPM Solvecte (Lu et 2022h)
20
31,20
26.81
42.79
1912
DPM Sulveres (Lu et 2022b)
15
31,23
25.99
42.05
1,35gy
HannoniCaa
201
31,25
24.64
41.58
53.45
L.74
PINART-Y: 1091 x 1921 (ofgg
4.5)
DPM Solveree (Lu at... 20226)
20
31.37
20.95
27.47
9.467
DPM Solverrs (Lu ata.. 2022b)
15
31.34
21.63
25.68
7.100
HammoniCa
20
31.50
20.53
27.05
52.74
5,852
PINART-3: 2018
2018 (ergg
1.5)
DPM-Salverss (Lu ctal. 20221)
31.19
23.61
14.198
Solvecee (Lu 2022h)
31.25
24.40
9,782 (53)
Hamonniaa
31.51
24.09
9/081; asx)
G. Experimental Details for Quantization and Pruning
For EfficientDM (He et al., 2024), we employ 8-bit channel-wise weight quantization and 8-bit layer-wise activation quantization (Gong et al., 2024; 2019) for full-precision (FP32) DiT-XL/2 and half-precision (FP16) PIXART-a. The former uses a 20-step DDIM sampler (Song et al., 2020a), while the latter employs a DPM-Solver++ sampler (Lu et al., 2022b) with the same steps. More specifically, we use MSE initialization (Nagel et al., 2021) for quantization parameters. For the quantization-awar fine-tuning stage, we set the learning rate of LoRA (Hu al., 2021) and activation quantization parameters to 1 and that of weight quantization parameters to 1e respectively. Additionally, we employ 3.2K iterations for D (Peebles & Xie, 2023) and 9.6K iterations for PIXART-a (Chen et al., 2023) on a single NVIDIA H800 80G GPU. Other settings are the same as those from the original paper (He et al., 2024). Leveraging NVIDIA CUTLASS (Kerr et al., 2017) implementation, we evaluate the latency of quantized models employing the 8-bit multiplication for all the linear layers and convolutions. For PTQ4DiT, we implemented the DDIM sampler and re-run the open-source code, which originally only supported DDPM. For Diff-Pruning. we re-implement the method for the DIT model (which originally only supported U-Net models) and follow the settings specified in the original paper.
T+e-x
17

---

outperforms Learning-to-Cann CaChe regardless of HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
H. Comparison between Learning-to-Cacte and HarmoniCa with Low CUR %)
In this section, we compare HarmoniCa with Learnin-to-Cacche (Ma et al., 2024a) at a relatively low CUR())) As shown in Tab. 17, both methods achieve similar speedup ratio and even better performance than non-accelerated models. Therefore, we employ higher CUR in Tab. 2 to show our pronounced superiority.
Table 17. Comparison results between I Learning-ton tooning-----aacre Cache and HarmoniCa for the D with a low CUR(%)).
Mebod
ISt
FID!
sPIDU
Prect
Recalf
CUR
DIT-XL/2 250 256 (orig L.5)
(Song 2000g)
334.37
DDIM (Sung 3000a)
4.17
5.54
56.30
0.56k 17x)
Leurning Cache al., 2024g)
4.66
59.10
Hammonn's (9 Se-1)
30
228.79
3.51
4.26
79.43
59.32
21.07
0.547
DIT-X122 512
5 (12
1.55
DDIM (Song et 200000
20
184.47
5:30
5.79
81.77
54.50
3,306
DDM (Sung al., 2000a)
190.06
5.62
6:13
81.37
53.90
3.03 33x0
Leizning Cache et at., 20246)
183.57
5.45
6.05
54.90
2,927 1940
Hamment's
I. Comparison between A-DiT and HarmoniCa
In this section, we compare HarmoniCa with A-DIT (Chen et al., 2024b). Given that the code and implementation details of A-DiT 10 are not open source, we report results derived from the original paper. Additionally, we evaluate performance sampling 5000 images as used in that study. As depicted in Tab 18, our framework further decreases 9.3% latency and gains 3.35 FID improvement compared with A-DiT for PIXART-a with a 20-step DPM-Solver++ sampler (Lu al., 2022b).
Table 18. Comparison results between A-DiT and HarmoniCa on on MS-COCO for PIXART-a 1024 X 1024,
Method
CUP?
FTD
1S1
CUR(S)
Speedup1
PIXART 1024
1024 4.5)
DPM Solv-raa (La et al, 2022b)
20
31,07
31.98
41.30
DPM Selveree (La al., 2022b)
31,04
33.29
39.15
1.56x
4 DIT (Chen al.. 2024b)
30.40
35.88
32:22
57.89
1.49x
HormoniCa le-1)
40.36
J. Comparison between Learning-to-Cache and HarmoniCa with Different Sampling Strategies
For the implementation details II. Learning-to- Coarning-to-Cachte taChe uniformly samples an even timestep & during each training iteration 12 as opposed to sampling any timestep from the set (1...., T, as mentioned in Alg. 1 of its original paper. Consequently, according to Fig. 3, only rx,., where an odd timestep, is learnable, while the remaining values are set to one. We compare Learning-to- Cache under different sampling strategies (i.e., sampling an even timestep or without this constraint for each training iteration) against HarmoniCa. As shown in Tab. 19, our framework-whetter training the entire Router only parts of it (similar to the Learning-to-Cache implementaaiion the sampling strategy.
should be noted that the experiments in Sec. 5, with the exception of those in Tab. 9, use an implementation that uniformly samples an even timestep during each training iteration. This approach achieves significantly higher performance compared to sampling without constraints.
18

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
Table 19. Comparison results hetween L.aaming-t- with different sampling strategies and HarmoniCa for the DiT-XI1/2 256 x 256. denotes that only parts of the Router corresponding to odd timesteps are learnable and the remaining values are set to one (i.e.. disable reusing cached features).
Mathad
FIDU
sFD]
Proc1
Recalft
CUR(S)
Laiesy(0)
DIT-XL/2
350
236 (ctig 1.5)
DDIM (Song et 2020))
20
224.37
3.52
4,96
78.47
55.33
0.655
Leaminn to Cache (Ma et 3024a)
20
115.00
18.57
16.18
00:35
02.99
32,08
p
LaaminglosCeehee (Ma 20046)
20133
5334
75.04
35,60
0.46%
HamoniCa (3 3.5e"))
206
0.40
HamaniCa
246.7
75.20
0.456 4633
Table 20. Accelerating image generation on MJHQ-30K (Li al., 2024a) and sDCI (Urbanck et al., 2024) for the PIXART-a. We sample 30K images for MJHQ-30K and 5K images for sDCI. "IR" denotes Image Reward.
MITQ
JDC1
Methed
Qualkty
Similariy
Quality
Sinilarity
Laieaey (33
pMn
IR1
LPPS)
PSNRT
HID.
IRT
CLIP
LPPSS]
PSNK1
PIXART-1
(otg
1.5)
Solvere-lla ctal.
0.947
26.04
11.47
0.994
25.22
DPM DPM--odoone-dla 2022b)
0.800
26.02
0.136
21.11
11.55
0.86
25.19
0.178
1.391 2exo
HammoriCas
0.944
26,02
0.130
21,08
11.52
0.933
25.20
0.175
$9.91
1.872
PIXART-O 1024
1024 (ctg
4.5)
Sul stal,
6.24
0.966
26.23
10.96
0.966
25.56
Solvere-cla stall 20220)
5.49
0.021
26.18
0.107
23.08
11.22
0.942
25.51
0.186
18.44
7.341
Hammoritas
20
6.40
26.19
IL102
25.66
10.99
11969
25.53
0.183
20:23
5.786 sx.)
K. Results of T21 Generation on Additional Datasets and Metrics
In addition to the evaluations on ImageNet and MS-COCO, we conducted further tests using the high-quality MJHQ-30K (Li al., 2024a) and sDCI (Urbanek al., 2024) datasets with PIXART-a models. We added several metrics, including Image Reward (Xu et al., 2024), LPIPS (Learned Perceptual Image Patch Similarity) (Zhang et al., 2018), and PSNR (Peak Signal-to-Noise Ratio). The results, summarized in Tab. 20, demonstrate that HarmoniCa consistently outperforms DPM-Solver across all metrics on both the MJHQ and sDCI datasets. For instance, at the 512x512 resolution, HarmoniCa achieves an FID of 7.23 on the MJHQ dataset, which is lower than the 7.45 FID of DPM-Solver with 15 steps, indicating better image quality. Additionally, under the same configuration, HarmoniCa achieves a PSNR of 21.98, compared to DPM-Solver's 21.41 with 15 steps, reflecting better numerical similarity.
L. Apply the Trained Router to a Different Sampler from Training During Inference
As shown in Tab. 21, the Router trained with one diffusion sampler can indeed be applied to a different sampler, such as 9PM-Solver+++>33 -oott+-S---oovee (6th row) and IDDPM
(10th row). However, the performance of these trials is much worse than the standard HarmoniCa. We believe this is due to the discrepancies in sampling trajectories and noise scheduling between the two samplers, which need to be accounted for during the Router training. In other words, the sampler used for training should match the one used during inference to improve the performance.
M. Results with Different Seeds
We conducted five independent runs with different random seeds on the following setting: PIXART-a 256 x 256 (Chen et al., 2023), using DPM-Solver++ (Lu et al., 2022b) (20 steps) and evaluating on 5000 images. The results in Tab. 22 show high consistency across runs, and more importantly, our models in all evaluation metrics, confirming our method's 's robustness and effectiveness.
models consistently outperform the aon-accelerated
A-DiT presents the speedup ratio based on multiply-ccuumulat operates (MACs). Here we report the results according the latency that study.
"Let 7 be an even numher here.
https:: : // thub. com http::///./.eeeeeeeeee//ee/////a http:///aaaaaaaaaaaoooooooooaaaa 4241244412227
19

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
Table 21. Results of applying the trained Router different sampler from training during inference. "A-+B" denotes the Router trained with the sampler "A" is directly used during inference with the sampler "B".
Method
CLIF
Loteneyts)
PIXART-a 256 (c6 4.5)
SA-Solve (Kue 2024)
31,23
27.45
39.01
0.665
SA-Solver (Kue al., 2024)
0.483
HarmoniCas
31,20
27.98
39.26
0.420, (.08-)
HarmoniC's (DPMM-Sohve--444 SA-Solver)
0.51 0.845)
DPM Solvere+ et 20226)
31.30
25.01
35.42
2.701
DPM-Saleres et 20226)
2.005 3666
HarmoniCa
100
31.29
25.00
35.38
(.377
-DMM-Salverss)
1.648
Table 22. Results with different seeds. soeds."ø/y" in the table denotes the results that come from non-accclerated models and HarmoniCa. We use the same settings Tab, the main text.
Sced
IST
FIDI
sFID]
37.31/35.44
94.78/92.10
16
33.61/33.62
37,43/35:46
94,74/9,11 11
33.59/33.60
37,55/35.48
95.01/92.32
33.65/33.65
37.11/35.34 34
94,87/92.13
40
33.64/33.64
37.05/35.29
94.88/2.119
deviation
0.138/0.144
0.18810.073
0.093/0.08:
N. Qualitative Comparison and Analyses
As shown in Fig. 11 and 12, we provide qualitative comparison between HarmoniCa and other baselines, e.g. Learning- to- Cache (Ma et al., 2024a), FORA (Selvaraju et al., 2024), and the fewer-step sampler. Our HarmoniCa with a higher speedup ratio can generate more accurate details, e.g., 2nd column of Fig. 12 (d) vs. (b) and objective- traits, e,g,, 2nd column of Fig. /1 (d) vs. (c).
20

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
20-step DDIM sampler
(b) 14-step DDIM sampler (1.41x)
(c) Learning-toe Ceanninggtt--Caaee to-Cache (1.41)
(d) HarmoniCa (1.44x)
Figure 11. Random samples from DiT-XL/2 256 x 256 (Chen et al., 2023) with different acceleration methods. The resolution of each sample 256 256. We employ cīg here for better visual results. Key differences highlighted using rectangles with various colors.
O, Visualization Results
As demonstrated in Figures 13 to 16, we present random samples from both the non-accelerated DiT models and ones equipped with HarmoniCa, using a fixed random seed. We employ the same settings as those the main text or more aggressive caching strategies. Our approach not only significantly accelerates inference but also produces results that closely resemble those of the original model. For detailed comparison, zoom in to closely examine the relevant images.
21

---

HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration
styulzed paperesc ond dop icting the casogy ot minnorest, tapara hold Blune, cokordul pacrotr pacched asong Lomk, people around.
"Ao axey beronter Earopeat caset 2ixed vata paatei-ociene latnnay the ruflectiog the acdely cased cacades, do people possent."
wildlebe prodo ginache stoeptog to drine anadowa stowrching sararne, undiatucbed by hasan presutoe."
acyliped vector ciccccaation on or tale wind sithoupted againse acldidee coaming with no buease visible."
codkleation
(a) 20-step DPM-Solver
(b) 15-step DPM-Solver (1.36x)
(c) FORA (1.53x)
(d) HarmoniCa (1.64x)
Figure Random samples from PIXART-a 512 512 (Chen al., 2023) with different acceleration methods. The resolution of cach sample is 512 x 512.
(a) DIT-XL/22
(b) HarmoniCa (1.44x)
Figure 13, Random samples from (a) acceleceeeete and (b) accelcrated DIT-XL/2 256 256 (Chen et al., 2023) with a 20-step DDIM sampler (Song et al., 2020a). The resolution each sample 256 256. We mark the speedup ratio in the brackets.
22

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
(a) DiT-XL/2
(b) HarmoniCa (1.30x)
Figure 14. Random samples from (a) non-accelerated and (b) accelerated DiT-XL/2 512 512 (Chen al., 2023) with 20-step DDIM sampler (Song et al., 2020a). The resolution of each sample is 512 x 512.
sortay phocogrape capsuzang at the edge of reffectio sirrored toe sciad sty."
doanty vatecoscoz plaaa 1003 spottcan nechllaze still
"A uitrant accpice postzajal feeld
sasb passcing aiiling
parsted cliff.
arages semma
againat the cooos
wilaclonecs, sazzos
tower
virdows muspentiny
thooagt the reate, lapaled axply
sucaes, adresd any
peopie
conapanca sichss, jet
outside,
(a) PIXART-a
(b) HarmoniCa (1.60x)
Figure 1.5. Random samples from (a) non-accelerated and (b) accelerated PIXART-a 256 x 256 (Chen et al., 2023) with a 20-step DPM- Solver++ sampler (Lu et al., 2022b). The resolution of each sample is 256 x 256. Text prompts exhibited above the corresponding images
23

---

HarmoniCa: Harmonizing Training and Inferenee for Better Feature Caching in Diffusion Transformer Acceleration
minineiist Nateresior corporition teaturing Jonely gentio b1.1, againet graatent eky, agcrounding tigures objecte."
expression painting secluded cabio by wioding river, pines cnnletting
on the water's suzface, a faint eiow from the cabin windors hinting at life within, yet no one is showa outside."
"An acrylic paint of trangui] lagoon filled with 1ily pads and reed fringed chores, brilight retlecting absent
(a) PIXART-E
(b) HarmoniCa (1.73x)
Figure 16. Random samples from (Left) non-accelerated and (Right) accelerated PIXART-E-2K (Chen et al., 2024a) with 20-step PPM--olver++ sampler (Lu et al., 2022b). The resolution of each sample 2048 x 2048.
24


Reviews:
[{"review_id": "DA2Iej4354", "reviewer": "Reviewer_DPj7", "summary": "This paper proposes HarmoniCa, a learning-based caching framework designed to accelerate Diffusion Transformersby reusing transformer block features during inference. HarmoniCa builds on the Learning-to-Cache framework by improving the training mechanism for the caching router. Extensive experiments demonstrate the effectiveness of this method in class-conditional image generation and text-to-image (T2I) generation.\n\n\n## Post-Rebuttal Comments\n\nI thank the authors for the rebuttal. While most of my initial concerns have been addressed, I note that two reviewers have raised valid points regarding the limited performance improvements over prior methods (e.g., CT5R, 5nDy). Given this, I maintain my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s constructive comments.\n\n> **Claims And Evidence**\n> \n- **Concerns regarding acceleration rates and resolution scaling.** Thanks for the insightful question. We agree that the reason why the acceleration rates of HarmoniCa decline ($1.67\\times\\rightarrow1.56\\times$) with the resolution increase ($256\\times256\\rightarrow2048\\times2048$) is due to memory bottlenecks. Specifically, HarmoniCa achieves similar CUR (i.e., cache usage ratio) for these resolutions, but requires a cache size increase from 936 MB (BS=8) to 4320 MB (BS=1). Thus, we believe preloading and asynchronous saving can help resolve the problem and will explore this in the future.\n\n> **Methods And Evaluation Criteria**\n> \n- **Sampler consistency during training and inference.**  The sampler used during training is not required to match the one used at inference. We have already included the discussion in Sec.  K. However, mismatched samplers can lead to a discrepancy in sampling trajectories. This does affect performance as reflected in our experiments. Therefore, we suggest that the sampler should be the same for optimal performance.\n- **The evaluation metrics rely on FID for text-to-image generation.**  We have conducted experiments with more metrics (e.g., Image-Reward, LPIPS, and PSNR), which are provided in the Appendix (Sec. J). Here, we also consider DINO and human evaluations (e.g., [HPSv2](https://arxiv.org/pdf/2306.09341) and [PickScore](https://arxiv.org/pdf/2305.01569)) as suggested. As shown in the following tables, our method with a higher speedup ratio far outperforms baselines and achieves comparable performance with non-accelerated models.\n    \n    *We use the configurations and models in Tab. 2 in the manuscript to generate images for evaluation.*\n    \n    | Model | DINO $\\uparrow$ | HPSv2 $\\uparrow$ | PickScore $\\uparrow$ | Latency (s)$\\downarrow$ | Speedup$\\uparrow$ |\n    | --- | --- | --- | --- | --- | --- |\n    | PixArt-$\\alpha$ $256\\times256$ | 0.3082 | 28.91 | 27.89 | 0.553 | - |\n    | + 15 steps | 0.2582 | 27.98 | 23.02 | 0.418 | $1.32\\times$ |\n    | + FORA | 0.2712 | 28.11 | 22.44 | 0.364 | $1.52\\times$ |\n    | + HarmoniCa | **0.3235** | **28.72** | **26.65** | **0.346** | **$1.60\\times$** |\n    | PixArt-$\\alpha$ $512\\times512$ | 0.3339 | 30.53 | 28.52 | 1.759 | - |\n    | + 15 steps | 0.3127 | 29.79 | 22.03 | 1.291 | $1.36\\times$ |\n    | + FORA | 0.3099 | 29.82 | 21.98 | 1.150 | $1.53\\times$ |\n    | + HarmoniCa | **0.3289** | **30.28** | **27.47** | **1.072** | $1.64\\times$ |\n\n> **Experimental Designs Or Analyses:**\n> \n- **Compute cost for PixArt experiments.** Thank you for the question. We provide the computation costs of training and inference in the following table. It is worth noting that we only use naive DDP training without any advanced strategies like gradient checkpointing and FSDP. Thus, the requirements of computation can be further decreased in practice.\n    \n    *The settings of training can be found in Sec. 5. We employ BS=1 here to obtain the memory consumption of inference.*\n    \n    | Model | Infer. Mem. (GB/GPU)$\\downarrow$ | Train. Mem. (GB/GPU)$\\downarrow$ | Train. Time (h)$\\downarrow$ |\n    | --- | --- | --- | --- |\n    | PixArt-$\\alpha$ $256\\times256$ | 20.86 | - | - |\n    | + HarmoniCa | 21.21 | 79.05 | 1.84 |\n    | PixArt-$\\alpha$ $512\\times512$ | 24.11 | - | - |\n    | + HarmoniCa | 24.61 | 77.30 | 2.92 |\n- **Absence of standard deviation in experimental results.** Thank you for the suggestion.  We conducted five independent runs with different random seeds on the following setting: PixArt-$\\alpha$ $256\\times256$, using DPM-Solver++ (20 steps) and evaluating on 5,000 images. The results show high consistency across runs, and more importantly, our caching-accelerated models consistently outperform the non-accelerated models in all evaluation metrics, confirming our method's robustness and effectiveness.\n    \n    *”x/y’’ in the table denotes the results that come from non-accelerated models and HarmoniCa. We use the same settings in Tab. 2 in the manuscript.*\n    \n    | Seed | IS$\\uparrow$ | FID$\\downarrow$ | sFID$\\downarrow$ |\n    | --- | --- | --- | --- |\n    | 8 | 33.28/33.27 | 37.31/35.44 | 94.78/92.10 |\n    | 16 | 33.61/33.62 | 37.43/35.46 | 94.74/92.11 |\n    | 24 | 33.59/33.60 | 37.55/35.48 | 95.01/92.32 |\n    | 32 | 33.65/33.65 | 37.11/35.34 | 94.87/92.13 |\n    | 40 | 33.64/33.64 | 37.05/35.29 | 94.88/92.19 |\n    | deviation | 0.138/0.144 | 0.188/0.073 | 0.093/0.081 |", "claims_and_evidence": "Most claims in the paper are supported by empirical evidence, but some require further clarification:  \n\n1) **HarmoniCa accelerates Diffusion Transformers while maintaining high-generation quality.** Tables 1 and 2 show speedup improvements compared to the Learning-to-Cache framework and SA-Solver, demonstrating effectiveness in class-conditional image generation and text-to-image (T2I) tasks.  \n\n2) **Concerns regarding acceleration rates and resolution scaling.** Why do acceleration rates of Harmonica decline as resolution increases in PixArt-$\\alpha$ and PixArt-$\\Sigma$ (Table 2 [25 steps], and Table 12 in the supplementary material)? Is this due to memory bottlenecks associated with caching longer sequences?", "methods_and_evaluation_criteria": "1) **Sampler consistency during training and inference.** Should the sampler used during training match the one used at inference, given that the caching router is trajectory-dependent? If so, this introduces a limitation that should be clearly discussed.\n\n2) **The evaluation metrics rely on FID for evaluating text-to-image generation which is concerning.** As FID relies on Inception model features trained on ImageNet-1K at 299×299 resolution, it may not be the best fit for evaluating PixArt models. Did the authors explore alternative evaluation metrics such as FID_{DINO} or human evals for a more comprehensive assessment?", "theoretical_claims": "The paper does not introduce formal theoretical claims or proofs.", "experimental_designs_or_analyses": "1) Compute cost for PixArt experiments: Can the authors provide the computational requirements for the PixArt experiments?\n2) Absence of standard deviation in experimental results: While it may be computationally expensive to repeat all experiments, reporting standard deviations for at least one key setup would improve result reliability of the proposed caching method.", "relation_to_broader_scientific_literature": "The paper builds on existing work in feature caching for diffusion models.", "essential_references_not_discussed": "I don't see this work missing any essential references.", "other_strengths_and_weaknesses": "Please see the above sections for strengths and weaknesses: Claims And Evidence, Methods And Evaluation Criteria, and Experimental Designs Or Analyses.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the above sections for strengths and weaknesses: Claims And Evidence, Methods And Evaluation Criteria, and Experimental Designs Or Analyses.\n\nAuthors please consider addressing the weaknesses during rebuttal.\n\nIn my opinion, the weaknesses of this paper slightly outweigh the strengths. But I’m willing to change my opinion based on the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "soasU9G5gQ", "reviewer": "Reviewer_CT5R", "summary": "This paper introduces a method for improving the cache mechanism in diffusion transformers. The proposed method including two parts:\n* A step-wise denoising training. It changes the objective in Learning-to-cache from single step to a all step objective. \n* An image error proxy-guided objective. The authors propose to use the matching of the final image as an extra guidance for the caching router. \n\nExperiments show that it surpasses Learning-to-Cache and FORA at a high CUR setting. The authors also conduct extensive analysis experiments to compare their results with diffusion compression and sampling methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s constructive comments.\n\n> **Methods And Evaluation Criteria**\n> \n\n**Contribution insufficiently novel.** While our work builds upon the foundational concept (i.e., learn a caching strategy) introduced by Learning-to-Cache (L2C), our contributions and improvements are both significant and distinct:\n1. **Identifying Key Problems**: We first identify two critical discrepancies between training and inference overlooked in L2C as follows：\n    1. **Prior timestep disregard**: Ignore the error caused by reusing cached features at prior timesteps during optimization.\n     2. **Objective Mismatch:** Focus on optimizing the intermediate output during the denoising process.\n        \n     They both significantly affect the overall performance of L2C.\n        \n2. **Novel Training Strategies**: To address these problems, we propose two novel training techniques(i.e., SDT+IEPO). **Our entire optimization processes differ fundamentally** compared with L2C:\n     1. For the training paradigm, L2C uses random timestep sampling similar to DDPM, focusing solely on the current sampled timestep. In contrast, we implement SDT for iterative denoising from random noise, which accounts for all previous timesteps, mirroring the inference process.\n      2. For the learning objective, L2C focuses on intermediate noise and cache usage. We further optimize the final image error and cache usage with our IEPO.\n3. **Reduced Training Cost**: For our method, **no training images** are needed, and **training time is much shorter** than L2C. This makes it applicable to large-scale models like PixArt.\n4. **Substantial Performance Improvements:** Our method consistently outperforms L2C. For instance, Tab. 2 shows **+1.24** FID and **+6.74** IS on DiT-XL/2. Fig. 7 shows **+30.90** IS and **−12.34** FID at $1.6\\times$ speedup.\n\n> **Experimental Designs Or Analyses**\n> \n- **The performance improvement of Learning-to-Cache is quite limited.** First, both methods without re-training the model gain comparable performance, which is also much better than that of the non-accelerated models in Tab. 13. Thus, it is reasonable to show our superiority with experiments at more challenging settings (i.e., higher CUR). For example, in Fig. 7, HarmoniCa can achieve substantially higher IS and lower FID scores, 30.90 and 12.34, under $1.6\\times$speedup. Moreover, our method consumes less training time in these experiments, since we employ the same training settings (e.g., iterations and batch size) as Learning-to-Cache (analysis can be found in Sec. 4.3).\n- **The results in the tables do not match those in the original papers.** For the performance and speedup gaps of Learning-to-Cache between this work in Tab. 2 and the official paper, we have to clarify that we re-run [the official implementation of Learning-to-Cache with the provided arguments](https://github.com/horseee/learning-to-cache/tree/main/DiT) without any modification.  However, we conducted training procedures on 4 H800 GPUs and evaluation phases on 8 H800 GPUs, contrasting with the original experimental setup employing 8 A6000 GPUs for both stages.\n- **The results in Tab. 9 are inconsistent with Tab. 2.**  As mentioned in Sec. I, the implementation in the code of Learning-to-Cache is inconsistent with that in Alg. 1 in the official paper. The former only employs the caching strategy in non-adjacent steps, and the latter utilizes the caching strategy across all steps (the same as ours). The trick in the former can help mitigate the impact of accumulative error by the no-caching steps (4th row vs. 5th row in Tab. 15). To make a fair comparison, we implement Alg. 1 in the official paper and report the results in Tab. 9.", "claims_and_evidence": "The claim and the motivation are correct, but I believe the experimental validation is insufficient to fully support their claims. Please refer to section Experimental Designs or Analyses.", "methods_and_evaluation_criteria": "Yes, the method proposed by the authors is reasonable, but it is incremental compared to learning-to-cache. \n\nThe motivation stated in the paper is to address two inconsistencies between training and testing in the design of learning-to-cache algorithms: (1) only considering a single step and (2) optimizing for the noise in a single-step prediction rather than the final image.\n\nThe proposed solution can be broadly summarized as transforming the single-step learning-to-cache approach into an optimization objective that considers the entire denoising process, taking into account how cached results from earlier steps affect later steps. This involves moving from sampling single steps, as in learning-to-cache, to running a full denoising process, computing the loss at each step, performing backpropagation for each loss, and incorporating a proxy loss derived from the final image.\n\nI find this contribution insufficiently novel.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "(1) The performance improvement of the proposed method over learning-to-cache is quite limited. \n\nIn the low CUR setting (Table 13), there is almost no performance gain, with improvements only observed in high CUR scenarios. Moreover, the results presented in the tables do not match those reported in the original papers. In Table 2, for example, at 50 timesteps, the original paper reports **an FID of 2.27 with a 1.3× speedup**, while the authors report Learning-to-cache result with **an FID of 2.62 with a 1.25× speedup**. The proposed method here achieves **an FID of 2.36 with a 1.3× speedup**, which is even worse than the results in learning-to-cache. For 20 timesteps, Figure 4 in the learning-to-cache paper shows that at a 1.4× speedup, the FID is below 4.5. However, the authors report an FID of 5.34 for learning-to-cache, while this method achieves 4.88. \n\n(2) The results in the ablation study are inconsistent with Table 2. \n\nIn Table 9, the authors report that the learning-to-cache with $L_{LTC}$ achieves an FID of 18.57 (If I understand correctly this would be the same as the algorithm of learning-to-cache), which is inconsistent with results for learning-to-cache in Table 2 (FID = 5.34). It is unclear what the authors are basing their claim on Line 387, where they state that their method (SDT) achieves an FID improvement of 10 over the learning-to-cache.", "relation_to_broader_scientific_literature": "An improved method for learn-to-cache", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "My main concern is the novelty and the experimental design of the method. Please refer to previous section.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amHQFTTLn8", "reviewer": "Reviewer_u3jx", "summary": "This paper introduces HarmoniCa, a learning-based caching framework that addresses two key discrepancies between diffusion training and inference: ***prior timestep disregard*** and ***objective mismatch***. First, it presents Step-wise Denoising Training (SDT), which adopts a student-teacher setup to consider full denoising trajectories during diffusion training. The student employs a caching system to store the denoising trajectory, while the teacher leverages the student to indirectly incorporate this trajectory. Second, it introduces an Image Error Proxy-Guided Objective (IEPO) to supervise diffusion models to approximate the final image error, allowing SDT to balance cache usage and image quality. Extensive experiments verify the effectiveness of HarmoniCa.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the reviewer’s constructive comments.\n\n> **Claims And Evidence**\n> \n- **Comparison with rectified flow models.** We clarify that our approach is not based on pretraining, unlike flow-based models. A direct comparison is not entirely appropriate, as our method merely trains a small tensor (i.e., the Router) and only focuses on improving inference efficiency in a post-training manner. Thus, we would like to emphasize that our methods can bring significant benefits compared with other post-training methods (e.g., quantization, pruning, and other caching methods in Sec. 5). Moreover, HarmoniCa can be effectively applied to rectified flow models. We employ the pretrained models and evaluation in [LFM](https://vinairesearch.github.io/LFM/). As shown in the following table ( *$T=100$*, Euler Solver, and $BS=8$), our method achieves substantial speedup ratios without performance degradation.\n    \n    \n    | Datasets | Models | FID$\\downarrow$ | Latency (s)$\\downarrow$ | Speedup$\\uparrow$ |\n    | --- | --- | --- | --- | --- |\n    | LSUN-Bedroom $256\\times256$ | DiT-L/2 | 5.22 | 1.76 | - |\n    | + HarmoniCa | DiT-L/2 | **5.13** | **1.09** | **$1.62\\times$** |\n    | CelebaHQ $256\\times256$ | DiT-L/2 | 5.42 | 1.76 | - |\n    | + HarmoniCa | DiT-L/2 | **5.41** | **1.07** | **$1.65\\times$** |\n    | ImageNet $256\\times256$ | DiT-B/2 (CFG=1.5) | 5.06 | 1.37 | - |\n    | + HarmoniCa | DiT-B/2 (CFG=1.5) | **5.04** | **0.87** | **$1.58\\times$** |\n- **Relationship between the discrepancy and the caching system.** We further explain the relationship as below (also can be found in Sec. 4.1):\n    1. **Prior Timestep Disregard**: In previous research, such as Learning-to-Cache, the training process employs a raw image with manually added noise as input $\\mathbf{x}_t$(see Fig. 3), which is unaffected by any cached feature reuse. However, during accelerated inference, the input (i.e., a noisy image) is disrupted by cached feature reuse in earlier timesteps (as illustrated in Lines 187-191). Moreover, the cache context in current timestep is shaped by the caching strategy in earlier timesteps (e.g., $t +1,\\ldots, T$), which is unconsidered as shown in Fig. 3. This discrepency of disregarding the caching effects of prior timestep can lead to significant error, which is caused by cache usage, accumulation (see Fig. 5). To address this, SDT (Sec. 4.2) ensures that the model learns to incorporate the prior timestep's effect on the cache, aligning the training process with the inference stage and reducing error accumulation.\n    2. **Objective Mismatch**: Equipped with the caching strategy, the training objective in previous research typically minimizes each timestep noise prediction error caused by caching usage. However, inference aims to generate high-quality images, the same as those from non-accelerated models. This mismatch between the two processes leads to an optimization deviation (Sec. C) and image distortion (Fig. 6). By integrating IEPO (Sec. 4.2), we ensure that the training process considers the final image error, which mitigates the optimization shift and improves the quality of generated images.\n\n> **Relation To Broader Scientific Literature**\n> \n\nWe appreciate the reviewer’s insight. While Learning-to-Cache introduced the concept, HarmoniCa extends this by addressing the discrepancies between training and inference, and also requires fewer training resources (Sec. 4.3).  Moreover, HarmoniCa achieves much higher efficiency and performance across all metrics compared with previous methods, including caching, quantization, and pruning. Regarding the concern about the marginal improvements, we acknowledge that the gains in some small models may appear modest. However, the benefits become more pronounced in large-scale models (e.g., Figs. 3 and 12). Besides, we emphasize that HarmoniCa can be effectively applied to rectified flow models (see **Claims And Evidence**). \n\n> **Other Strengths And Weaknesses**\n> \n- **Lacks a clear logical argument.** Regarding the concern, we clarify our general and natural logical argument process as follows. First, we have carefully analyzed the two discrepancies between training and inference. Then, we proposed our framework to solve the problems. Both parts include detailed logical analyses and experimental validation. These processes provide the rationality to support the large number of experimental findings in Sec. 5. We will make the paper logic clearer in the revision.\n- **Hyperparameters limit the adaptability.** We would like to clarify that we have three hyperparameters in total: interval $C$, threshold $\\tau$, and regularization coefficient $\\beta$. The superior performance and speedup across all experiments validate the efficacy of using fixed $C$ and $\\tau$. Therefore, the only one that requires adjustment is $\\beta$, which determines the trade-off between acceleration and image quality. We will explore the automatic methods to determine $\\beta$ in the future.", "claims_and_evidence": "The authors address the claims in ***prior timestep disregard*** and ***objective mismatch*** with the previously introduced caching framework [1]. However, their claims appear to be limited to diffusion models. Recently, several rectified flow models [2, 3, 4] have been proposed, which train on straight ordinary differential equation (ODE) paths that naturally mitigate these issues. In these models, the denoising trajectory follows a nearly linear interpolation between source and target distributions, leading to minimal error accumulation and interference. Given the complexity of both training and sampling pipelines compared to rectified flow models, HarmoniCa contributes only marginal improvements in diffusion models, and I am not fully convinced that their claims are well-substantiated. \n\nFurthermore, the relationship between the discrepancy and the efficient caching system remains unclear. The authors have to provide a claim on this relationship.\n\n[1] Ma et al., Learning-to-cache: Accelerating diffusion transformer via layer caching, NeurIPS 2024. \\\n[2] Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, ICLR 2023. \\\n[3] Lee et al., Improving the Training of Rectified Flows, NeurIPS 2024. \\\n[4] Liu et al., InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, ICLR 2024.", "methods_and_evaluation_criteria": "The proposed methods are well-aligned with their claims, where SDT effectively handles the outputs from earlier timesteps as caches and IEPO makes SDT more efficient. Extensive experiments across various models, samplers, and image resolutions validate the effectiveness of HarmoniCa.", "theoretical_claims": "No theoretical claims and proofs.", "experimental_designs_or_analyses": "The authors provide in-depth analysis and various metrics in evaluating image quality and inference efficiency.", "relation_to_broader_scientific_literature": "In my opinion, a caching system itself holds significant potential for application across various generative models. However, the concept of a learnable caching system was previously introduced in Learning-to-Cache. Moreover, this approach appears to offer only marginal improvements within diffusion models and does not seem applicable to recent rectified flow models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "* **Potential strengths.** I believe the comprehensive experiments in HarmoniCa demonstrate its high potential for application across various generative models.\n\n* **Potential weakness.** However, it lacks a clear logical argument to support its extensive experimental findings. Moreover, the expSeveral hyper-parameters limit the adaptability of HarmoniCa.", "other_comments_or_suggestions": "As HarmoniCa addresses a new training framework, the authors have to provide some analysis or experiments regarding its effectiveness compared to other powerful generative modeling, such as rectified flow models.", "questions_for_authors": "My concerns are summarized into:\n\n1. The authors have to provide some analysis or experiments regarding its effectiveness compared to other powerful generative modeling, such as rectified flow models.\n\n2. The relationship between the discrepancy and the efficient caching system remains unclear. The authors have to provide a claim on this relationship.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8v3FBBoxuO", "reviewer": "Reviewer_5nDy", "summary": "This study introduces a novel caching framework to enhance inference efficiency in diffusion transformers. The core idea is to employ a learnable router that determines optimal caching dynamics during the generation process. The primary contribution lies in developing a new learning paradigm that alleviate the error accumulation in previous method (which utilizes a skip-step training strategy). Besides, a dynamic loss reweights the training objective of the proposed learning paradigm. Guided by final output quality, the reweighting strategy, as the authors stated, can minimizing the discrepancies between cached and non-cached generateion pathways.\n\n## Update After Rebuttal\n\nThe rebuttal clarified the authors’ perspective on the methodological novelty of the proposed SDT and IEPO training strategies, which they position as innovations. However, key disagreements persist regarding the substantiation of these claims. My primary concerns are as follows:\n\n1. **Questionable novelty of SDT/IEPO**:​​ The empirical results do not convincingly demonstrate that the performance gains stem uniquely from SDT/IEPO rather than the baseline L2C framework. For instance, the reported acceleration improvements show negligible divergence from those achieved by L2C alone, suggesting that the observed benefits may originate from the underlying L2C methodology rather than the proposed strategies.\n\n2. **Insufficient empirical validation of methodological impact**: The rebuttal’s discussion of teacher forcing—where the authors acknowledge that strict teacher forcing yields comparable results to their proposed strategy—fens the case for SDT’s significance. If varying training strategies (including their own) produce similar outcomes, the necessity and distinct contribution of SDT remain unsubstantiated.\n\nBesides, I would like further clarify my **score justification** in the rebuttal: I mean **either (not both)** technical innovation or theoretical breakthrough is required for an ICML paper. This paper mainly provides technical innovation which should be rigorously validated through measurable performance gains or demonstrate methodological superiority. In this work, neither criterion is met: the strategies lack clear differentiation from existing approaches (e.g., L2C), and the marginal empirical improvements fail to justify their novelty. (**That is also why I am suggesting exploring theoretical foundations**: I find that it may be hard to explain why the gains compared to L2C is marginal.)\n\nOverall, the paper does not meet the threshold for technical novelty required for ICML. Thus, I maintain my recommendation for rejection.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the constructive comments.\n\n**Literature Review and Citations**\n\n- **Missing work.** We appreciate the suggestions and will include ToMe, TokenFusion, and attention-driven efficiency methods in the revision.\n- **Citation updates.** We will replace arXiv references with conference/journal versions where available.\n\n**Experiments**\n\n- **Baselines.** We have already included $\\Delta$-DiT in Sec. H, showing our method outperforms it in both accuracy and efficiency. For [1], it was accepted by CVPR after the ICML deadline and appeared on arXiv after Oct. 2024. According to [ICML policy](https://icml.cc/Conferences/2025/ReviewerInstructions), it qualifies as a concurrent work. Moreover, its code is not available, making comparison infeasible within the rebuttal window.\n- **Performance Significance.** We respectfully disagree since a larger batch or model size can show larger absolute speedup. While some latency results are based on small models (e.g., DiT-XL/2 with batch size 8) for controlled ablations or comparisons, they still show consistent speedups without sacrificing quality (e.g., Tab. 2). On larger models like PixArt-$\\Sigma$ $2048\\times2048$, our method yields much larger absolute gains and higher acceleration ratios (Sec. E).\n- **Failure Cases.** Across all tested settings (i.e., models, datasets, and samplers), our method shows robust and consistent performance. We did not observe clear failure cases that warrant separate analysis. Moreover, such analysis is uncommon in related works and not part of standard practice in this field.\n\n**Methods**\n\n- **Novelty Concerns.** While our work builds upon the foundational concept (i.e., learn a caching strategy) introduced by Learning-to-Cache (L2C), our contributions and improvements are both significant and distinct:\n    1. **Identifying Key Problems**: We first identify two critical discrepancies between training and inference overlooked in L2C as follows：\n        1. **Prior timestep disregard**: Ignore the error caused by reusing cached features at prior timesteps during optimization.\n        2. **Objective Mismatch:** Focus on optimizing the intermediate output during the denoising process.\n        \n        They both significantly affect the overall performance of L2C.\n        \n    2. **Novel Training Strategies**: To address these problems, we propose two novel training techniques(i.e., SDT+IEPO). **Our entire optimization processes differ fundamentally** compared with L2C:\n        1. For the training paradigm, L2C uses random timestep sampling similar to DDPM, focusing solely on the current sampled timestep. In contrast, we implement SDT for iterative denoising from random noise, which accounts for all previous timesteps, mirroring the inference process.\n        2. For the learning objective, L2C focuses on intermediate noise and cache usage. We further optimize the final image error and cache usage with our IEPO.\n    3. **Reduced Training Cost**: For our method, **no training images** are needed, and **training time is much shorter** than L2C. This makes it applicable to large-scale models like PixArt.\n    4. **Substantial Performance Improvements:** Our method consistently outperforms L2C. For instance, Tab. 2 shows **+1.24** FID and **+6.74** IS on DiT-XL/2. Fig. 7 shows **+30.90** IS and **−12.34** FID at $1.6\\times$ speedup.\n- **Generalization.** Our router is static once trained and does not change per sample during inference. The method is also image-free during training without tuning the weights, reducing the risk of overfitting. Moreover, we can observe strong generalization across multiple datasets and model sizes in this paper.\n- **Theoretical Foundation.** The reviewer suggests using a teacher-forcing strategy—this is exactly what SDT employs during training. Our experiments (e.g., Fig. 5, Tab. 9) validate its effectiveness in reducing error accumulation. A more detailed theoretical analysis is planned for future work, as we mainly focus on practical speedup while maintaining the performance in this work.\n\n**Questions for Authors**\n\n- Prompts in Figs. 12, 15, and 16 are randomly generated using GPT-4.\n- The router structure is identical to L2C, with all the details in Sec. 5.1 and Sec. D.\n\n[1] You et. al. Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformer. Arxiv 2024.12", "claims_and_evidence": "Somewhat convincing. Though experimental results exhibit overall marginal performance boost, I still do not believe that the proposed strategy (SDT and IEPO) can solve the error accumulation problem. (See Other Strength and Weakness)", "methods_and_evaluation_criteria": "Somewhat convincing.", "theoretical_claims": "N/A. Providing formal analysis will significantly strengthen the theoretical foundations (while theoretical validation is not explicitly addressed). I find some directions that may help strengthen the theoretical foundations: \n\n+ The error. Analysis the error propagation and quantify the error accumulation rate and demonstrate why SDT is better than skip-step strategy.\n+ The reweighting strategy. Analysis on the convergence dynamics of reweighting strategy, especailly how this strategy alters the convergence properties of Eq.4.", "experimental_designs_or_analyses": "While the experimental framewok adopts conventional benchmark, there are some concerns about the efficacy of the proposed method. (See Other Strength and Weakness)", "relation_to_broader_scientific_literature": "The key contributions of this paper (SDT and IEPO) can be seen as adjustments to the Learn2Cache. Learn2cache is a sample-specific caching method that alters the inference caching pathway according to each generated samples.", "essential_references_not_discussed": "While the submission adequately cites foundational works in caching-based method (e.g. learn2cache), some merging-based works are missing such as token merge. (See Other Strength and Weakness)", "other_strengths_and_weaknesses": "## Strengths\n\nThe topic is very practical and enjoys a wide range of appications.\n\n## Weaknesses\n\n1. (Minor) Literature Review and Citations\n\n+ **Some key works are missing.** The discussion overlooks recent advancements in token merging (which is also an important method in efficient diffusion model), such as ToMe [1], TokenFusion [2], and attention-driven efficiency method [3].\n\n+ **Citation.** Multiple references cite preprint versions despite peer-reviewed alternatives being available. Please update these to converence/journal citations to strengthen scholarly rigor.\n\n2. (Major) Experiments\n\n+ **Limited Baseline.** Some baselines should be included such as $\\Delta$-DiT [4]. Besides, leveraging a router to determine the inference caching pathway is not a fresh idea. Thus, [5] is also suggested as a baseline. (This is optional as it is publicly avaiable within 4 mounths. According to the Reviewer Instructions, it is not a concurrent work.)\n\n+ **Performance Significance.** The reported latency improvements (less than 0.1s for most experiments) lack practical significance, particularly for application-oriented claims.\n\n+ **Limitations and Failure Cases.** The manuscript omits scenarios where the method underperforms. Including qualitative examples of failure modes would clarify its operational boundaries and foster reproducibility, providing valuable insights on the scope of application of the proposed method.\n\n3. (Major) Methodology\n\n+ **Novelty Concerns.** The core methodology appears incremental, building primarily on Learn2Cache with refinements (i.e. the SDT and IEPO). However, these caching-based methods are quite crowded. A sufficient efforts have been made to explore the layerwise redundancy. Given this, this paper does not provide clear insight to the community.\n\n+ **Generazation Risks.** (not for this paper only, but for all training-based methods) The router’s sample-specific computation flow raises questions about robustness. For example, could certain inputs bypass acceleration entirely due to routing decisions? Empirical validation on out-of-distribution or edge-case samples is absent. \n\n+ **Theoretical Foundations.** The theoretical justification for SDT’s error reduction remains unclear. For instance, while SDT purportedly stabilizes training invariance, its dependency on cached features (ε') risks error propagation—a concern unaddressed in the current analysis. (Maybe a teacher-forcing like strategy is better?)\n\n[1] Daniel Bolya and Judy Hoffman. Token merging for faststable diffusion. CVPR 2023\n\n[2] Kim et. al. Token fusion: Bridging the gap between token pruning and token merging. WACV 2024\n\n[3] Wang el. al. Attention-driven training-free efficiency enhancement of diffusion models. CVPR 2024\n\n[4] Chen et. al. $\\Delta $-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. Arxiv 2024.06\n\n[5] You et. al. Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformer. Arxiv 2024.12", "other_comments_or_suggestions": "See Weaknesses.", "questions_for_authors": "+ What is the prompt dataset utilized in Figure 12, 15 and 16?\n\n+ What is the architecture of the router? Is this architecture identical to the one described in learn2cache? If so, more detailed configurations are expected. If not, the extra computational overhead should be quantified and provided.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received three reviews that positively assessed the work and recommended acceptance. The fourth review recommended a strictly negative rating. During the rebuttal period, there was healthy engagement between the authors and the reviewers, during which many concerns were addressed, and the scores either remained the same or improved. However, the authors were not able to change the opinion of Reviewer 5nDy. They composed a message to the AC in which they outlined their disagreements with Reviewer 5nDy. While the AC believes that certain statements made by 5nDy have merit, the AC feels that their score may be overly critical of the work. The other reviewers also raised concerns regarding differences with L2C; however, they accepted the arguments provided by the authors.

Hence, the AC believes that the present paper is a good piece of work and recommends acceptance.
Decision: Accept (poster)