Let Features Decide Their Own Solvers: Hybrid Feature Caching for Diffusion Transformers
Authors: Shikang Zheng
Guantao Chen
Qinming Zhou
Yuqi Lin
Lixuan He
Chang Zou
Peiliang Cai
Jiacheng Liu
Linfeng Zhang
Venue: ICLR 2026 Oral (2026)
Keywords: Generative models
Efficient ML
Diffusion Transformer Acceleration
Feature Caching
Primary area: generative models

Abstract: Diffusion Transformers offer state-of-the-art fidelity in image and video synthesis, but their iterative sampling process remains a major bottleneck due to the high cost of transformer forward passes at each timestep. To mitigate this, feature caching has emerged as a training-free acceleration technique that reuses hidden representations. However, existing methods often apply a uniform caching strategy across all feature dimensions, ignoring their heterogeneous dynamic behaviors. Therefore, we adopt a new perspective by modeling hidden feature evolution as a mixture of ODEs across dimensions, and introduce \textbf{HyCa}, a Hybrid ODE solver inspired caching framework that applies dimension-wise caching strategies. HyCa achieves near-lossless acceleration across diverse tasks and models, including 5.55$\times$ speedup on FLUX, 5.56$\times$ speedup on HunyuanVideo, 6.24$\times$ speedup on Qwen-Image and Qwen-Image-Edit without retraining.

Published as a conference paper at ICLR 2026
LET FEATURES DECIDE THEIR OWN SOLVERS: HYBRID FEATURE CACHING FOR DIFFUSION TRANSFORMERS
Shikang Zheng
Guantao Chen'
Qinming Zhou
Yuqi Lin'
Lixuan He
Chang Zou't Peiliang Cail
Jiacheng Liu'
Linfeng Zhanglt
Shanghai Jiao Tong University
South China University of Technology TTinghua University
ABSTRACT
Diffusion Transformers offer state-of- of-e-of-theeart fidelity in image and video synthesis, but their iterative sampling process remains a major bottleneck due to the high cost of transformer forward passes at each timestep. To mitigate this, feature caching has emerged as training-free free acceleration technique that reuses hidden representations. However, existing methods often apply a uniform caching strategy across all feature dimensions, ignoring their heterogeneous dynamic behaviors. Therefore, we adopt new perspective by modeling hidden feature evolution as a mixture of ODEs across dimensions, and introduce HyCa, a Hybrid ODE solver inspired caching framework that applies dimension achieves near-lossless acceleration across diverse tasks and models, including 5.55 specdup on FLUX, 5,56x speedup on Hunyuan Video, 6.24x speedup on Qwen-Imasn and Owen-Image-Eil Edit without retraining, Our Project Page.
caching strategies. HyCa
INTRODUCTION
Figure 1: Images generated on Qwen- Image using HyCa combined with distillation at 12x speedup.
Diffusion Transformers (DiTs) have recently achieved impressive success across image and video generation tasks, demonstrating strong modeling capacity and generation quality. However, the iterative nature of diffusion sampling presents a significant bottleneck, each output demands multiple transformer passes. This high computational cost hinders deployment in scenarios with strict latency or resource constraints, driving the ongoing research on efficient inference methods.
To address this challenge, two primary acceleration directions have emerged: reducing the total number of sampling steps via algorithmic advancements (Lu et al. 2022a), and lowering the cost of each step through architectural optimization (Yuan et al. 2024 Zhao et al. 2024). Among these,
Corresponning author.

---

Published as a conference paper at ICLR 2026
All Across Prosyts AlU Acruse Timesiops
ARl Across Prosays All Acruo Timeriepa
Principal Componennt
Principal Component
Smooth and
Continuous Trajectory.
Sample Count
Sample Count
Oscillatory and
Hunyuan Video
Qven-Image
Pluctuant Trajectory
Cluster
Principal Component (4)
Principal Component
Adjusted Rand Index
Adjusted Rand Index (d)
(b)
(c)
Figure 2: Feature trajectory clusters and stability of assignments. (a-b) Cluster shows oscillatory trajectories while Cluster 2 shows smooth ones. (c-d) ARI distributions Hunyuan Video and Owen-Image exceed 0.8 in most cases, confirming stable and consistent cluster assignments across prompts and timesteps. An ARI above 0.8 indicates strong agreement and high clustering reliability.
training-free free feature caching has emerged promising solution. It exploits the temporal coherence of hidden representations by reusing features, thereby reducing redundant computation. Early works such DeepCache (Ma et al.) 2024) demonstrated the feasibility of this idea in U-Net backbones, recent methods such as FORA (Selvaraju et al. 2024), ToCa (Zou et al. (2024a). TaylorSeer (Liu al. 2025a) extended caching to transformer-bass
architectures and showed that feature caching can be effectively viewed as solving the temporal evolution of hidden features. Despite progress, current approaches are still limited in critical ways.
Existing methods implicitly assume that all hidden dimensions evolve under a single, unified system. However, this assumption is untenable in DiTs, where the feature space is high-dimensional and exhibits complicated behaviors. Such complexity is unlikely to be captured by single process. To further investigate, we analyze how each feature dimension changes over timesteps and group them into clusters based on their dynamics. As shown in Fig. 2a), some dimensions fluctuate sharply with oscillatory patterns, indicating stiffness or multimodal behavior, while others evolve smoothly and predictably, reflecting stable dynamics Fig. 2)b). These observations suggest that the feature space of DiTs is better described as a complex system, where different groups of dimensions follow distinct temporal patterns, highlighting the need for tailored solvers rather than one-size-fits-all approach.
Therefore, we introduce HyCa, a hybrid caching framework that models hidden feature evolution as mixture of ODEs and applies suitable solvers for every dimension. Hyca begins with unsupervised clustering, grouping dimensions with similar dynamic behaviors, and modeling them into shared ODE. Then, HyCa assigns the most suitable solver each cluster. Normally, identifying the best solver would require running inference on large set of images and comparing quantitative metrics. However, surprisingly, we found that cluster assignments are highly stable across resolutions, timesteps, and even prompts. As shown in Fig. 2 (d), this invariance allows us to evaluate solver performance on single prompt at a single timestep to reliably identify the best solver, achieving results comparable to large-scale evaluation. Thus, with "One-Time Choosing" " performed offline for each model, "All-Time Solving" becomes possible without any additional cost during inference.
HyCa provides robust and adaptive feature prediction across diverse tasks and architectures. Without retraining, achieves loss-lsss acceleration of 5.56x on FLUX and Hunyuan Video, 6.24x on Image and Owen-Image Edit. Moreover, it is also fully compatible with distillation, reaching while maintaining strong image quality.
up to 24.4x x speedup on FLUX and 12.2x on In summary, main contributions are:
Heterogeneous Feature Dynamics. We show that feature dimensions in DiTs do not follow a single unified system but exhibit heterogeneous dynamic behaviors that are better described as a mixture of ODEs. Through dynamics clustering analysis across multiple settings, we further reveal that these cluster's distributions are consistent and input-invariant.
HyCa Framework. Inspired by hybrid ODE solvers in numerical analysis, we propose HyCa, training-free framework that groups feature dimensions by their dynamics and automatically assigns the most suitable solver to each group, with minimal overhead.
Outstanding Performance. We evaluate HyCa across diverse architectures and tasks, including Drawbench on FLUX and Qwen-Image, Vbench on Hunyuan Video, GEdit-Bench on Qwen- Image- Edit, and even distilled models. In all settings, HyCa delivers state-of tfte-of-thheart performance.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Diffussio models (Sohl-Dickstein et al. 2015 Ho et al. 2020) have achieved strong image/video generation quality. Early U-Net backbones Ronneberger et al. Diffusion Transformers (DiT) (Peebles Xie 2023b) alleviated.
2015) faced scaling limits that enabling rapid progress across modalities and resolutions (Chen et al. 2024b|a Zheng et al. 2024 Yang et al. 2025). Nevertheless, the iterative nature of sampling remains a key inference bottleneck. Two complementary research lines thus emerge: (i) reducing the number of steps and (ii) reducing the cost per step. Beyond speed, a central challenge is maintaining stability and fidelity under aggressive acceleration, especially when feature dynamics are heterogeneous across dimensions and timesteps.
2.
SAMPLING TIMESTEP REDUCTION
DDIM (Song et al. 2021 introduced deterministic few-step sampling that preserves perceptual quality. Higher-order ODE solvers (DPM-Solver and variants) (Lu et al.) 2022a bj Zheng al.) 2023) improve accuracy-com cost trade-offs via multi-step/mmultisttae
discretizations with carefully
controlled local truncation error. Rectified Flow (Liu al. 2023) shortens transport paths, while distillation (Salimans& & Ho. 2022) compresses long trajectories into compact generators. Consistency models (Song et al. 2023) enable few-step synthesis by learning a direct noise-to-clean mapping.
2.2 DRNOISING NETWORK ACCELERATION
Model Compression Acceleration. Pruning (Fang et al.
2023 Zhu al.
2024), quantization (Li
ctt al. 2023b Shang ct al. 2023 Kim et al. 2025). distillation (Li et al.
2024). and token reduc-
tion (Bolya & Hoffman 2023 Kim al.] 2024 | et al. 2024 2025 Cheng et al. 2025 reduce
compute with limited runtime overhead. While effective, they typically require additional training and may degrade robustness under domain shifts if the compression too aggressive.
Feature Caching Acceleration. Feature caching reuses activations to avoid redundant computation.
Early U-Net methods (Li al., 2023a Ma et al. 20244) inspired DiT-specific designs: FasterCache (Lv
et al. 2025). FORA (Selvaraju et al. 2024). A-DIT (Chen et al. (024c). TeaCache (Liu et al. (022).
and FoCa Zheng al.) (2025). Dynamic updates (ToCa/DuCa) (Zou al.] 2022a b). unified
cache-prune pipelines (Sun et al., 2025), and region-adaptive sampling (Liu ct al. 2025cc) further
improve efficiency. Among these advances, TaylorSeer (Liu et al., (2025a) exemplifies the cache-then-
forecast paradigm by polynomial extrapolation from cached neighbors.
3 METHOD
3.1 PRELIMINARY
Diffusion Models. Diffusion models (Ho al. 2020 Song al. 2021 generate structured data by progressively refining random noise through a series of denoising steps. At each timestep t, the model predicts conditional Gaussian distribution over hz-1 given 2,, where both the mean and variance are parameterized. This generative process can be formulated as:
Pa ( (x)) = /V (
VI - To(L,,)) 311
1-a at
Ft
V Ooz
where A denotes normal distribution, Or, and Br are noise schedule parameters, and To(2,, t) denotes the model's estimate of the noise component. Sampling begins from a pure noise vector and proceeds by repeatedly drawing samples from these intermediate distributions until a clean image is produced.
Diffusion Transformer Architecture. The Diffusion Transformer (DiT) (Peelles & Xie 2023a)
adopts hierarchical design, expressed as composition of modules g = 9) 92 0 gr. Each module yi consists of a self-attention layer (F cross-attention layer (Fxx), and a feedforward MLP (aL.). These components are dynamically modulated across timesteps accommodate the evolving noise levels during generation. The input x, = { H is represented as a sequence of patch tokens. Each module includes residual update of the form F(x) x AdaLN fx), where AdaLN (adaptive layer normalization) conditions the normalization parameters on the noise timestep. allowing for more effective denoising across varying noise scales.
3

---

Published as a conference paper at ICLR 2026
Festure Space of 1 Taken
Custening
Temporal Dynomics
(usteri(C1)
C1
difference.
Compore
Dimonetone
dfferrncce
Clustening
D7
D6
(uster2(C2)
Cluster
V5.
Energy
D8
Runge lafts (M4)
57
Curneture rotio.
Spectrol Hlstness
Zerk
(luster3(C3)
timestep (eg. (-1)
For each
0tfe
Adere-Nochss (AN)
<g.. Detais of I2
Real Festure
Prodicted Fanture
8K
AM
TF
(a) Pipeline during Offline Preprocessing
For Cluster!
Cluater2
For D is Cluster3
t-3
[-]
2-2
2-3
+-]
6-2
Compute
Ship
Compute
Compute
Predictor
A. Predictor
TF Predictor
(b) Pipeline during Inference
Figure 3: HyCa Framework. (a) Offline Preprocessing: feature dimensions are first analyzed and clustered with temporal indicators (e.g., differences, curvature). For each cluster, candidate solvers generate predicted features, then compared against real computed features; the solver with minimum error is then assigned to that cluster. (b) Inference: once assigned, each cluster consistently reuses its solver, enabling efficient prediction by skipping redundant computations while maintaining accuracy.
Feature Caching. Feature caching aims to reduce the cost of diffusion sampling by avoiding repeated computation of hidden features across timesteps. At each timestep the model produces hidden features F; = (=)) and a caching function C(FA, k) estimates features Fg at future timestep using cached features. A common strategy is to reuse features from the last computed step:
Fx C(F,, Fe, Vk (t, + 1],
(2)
which provides up to (n-1) speedup. Recent methods improve reuse by forecasting future features, yet their reliance uniform prediction strategy across all dimensions often proves unstable in DiT's complex hidden feature space. In this work, we propose hybrid approach that assigns suitable solvers for every dimension according their dynamic behaviors.
3.2 FEATURE CACHING HYBRID ODE SOLVING
During reverse-time denoising in diffusion models, the hidden features evolve across timesteps. Let a(T) be the latent variable at continuous time 7, and let F(x(r)) denote the hidden feature extracted from it. Since the generative model is differentiable and zz()) follows a continuous reverse-timg trajectory, the composite feature map T + F(x(T) is also differentiable. By the chain rule and the probability flow ODE governing ær s()), the feature dynamics satisfy:
F(+(T)) go (F(x(r)), r),
where ge captures the implicit time- deeeddepnddnnt vector field induced by the underlying network weights and structure. Although go is not directly accessible, we can sample the trajectory (F(x(T))) on a discrete timestep grid, enabling numerical integration using only cached feature values. This perspec- tive naturally casts feature caching as a numerical ODE solving problem. Instead of performing full forward computation at every (discrete) sampler step, we aim solve the next feature value using prior ones:
fitl A Solver( F..
(4)
where the solver is applied locally to the residual output of each transformer block at skip steps. To accommodate diverse local feature dynamics, from smooth near-linear segments to rapidly varying regions, we adopt hybrid solver strategy. Concretely, we define predictor pool S that includes both explicit and implicit numerical solvers with different stability and accuracy properties, including Runge-Kun Kutta (RK), Adams-Bel Bashforth (AB), Taylor Formula (TF), Backward Differentiation Formula (BDF) and Moulton (AM). This diverse solver set enables HyCa to assign methods tailored to local feature dynamics. Please refer A.2.2 and A.6 detailed implementation.
4

---

Published as a conference paper at ICLR 2026
3.3 HYCA FRAMEWORK
Building on this foundation, HyCa is designed as feature caching framework that models hidden dynamics mixture of ODEs and automatically assigns the most suitable solver each cluster through one-time optimization procedure. HyCa begins by analyzing the temporal dynamic behavior of each feature dimension. During a probe pass on single prompt at the first few timesteps. we extract descriptor vector da E IR for each feature dimension d E (1..... D}, capturing dynamic indicators such as Jerk ratio and curvature ratio. Then, we apply k-means clustering to obtain a partition {c(d)) where each dimension d is assigned to a cluster E (1...., C}. These cluster assignments remain stable across prompts, timesteps and resolutions, thus reused throughout inference.
The resulting clusters represent groups of dimensions that share similar temporal behaviors, enabling solver assignments to be conducted independently for each cluster. Given solver pool S, HyCa selects the optimal solver st E S for each cluster by minimizing the average next-step prediction error across all dimensions in that cluster:
E c
Σ
-
1
min (
dec
where fload) denotes the predicted feature for dimension at timestep + using solver sc. This formulation enables per-cluster solver selection via one-time probing pass, ensuring that HyCa combines efficiency with the adaptability hybrid solvers.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Model Configurations. We conduct experiments four representative diffusion-based models: the text-to-image models FLUX. 1-dev (Labs) 2024) and Qwen-Image (Wu et al. 2025), the text-to-video model Hunyuan Video (Sun al.)), and the image editing model Qwen-Image-Bil Edit (Wu et al. 20225). To further assess compatibility with model compression techniques, we also evaluate our method on distilled models: FLUX. --schnell and Laan---mag---itttiii All models are evaluated under official or recommended configurations on standard public checkpoints.
Evaluation and Metrics. For text-to-image generation, we follow the DrawBench (Saharia et al.) 2022) protocol and evaluate all models a fixed set of 200 prompts. We evaluate images using Rewaed (Xu et al. (023) for photorealism, CL.IP Score (Hessel et al. (02T) for text-image alignment, and PSNR, SSIM, LPIPS for fidelity. For text-to- toxt-to-vidoo generation, Vidco on VBench (Huang et al. 2023), which provides milti-dimensionall auman-aligned assessments of motion quality, visual appearance, and semantic consistency. For image editing tasks, we GEdit-Bench (Liu et al. 20255) to evaluate model performance across a diverse set of edit types and prompts. Unless otherwise specified, all evaluations are conducted using fixed random seeds and default inference settings. Additional implementation details please refer to A.7
evaluate Hunyuan-
Table 1: Quantitative comparison of text-to-image generation
Acceleration
Quality Metries
Perceptual Metries
Method
Latency(s) I
Speed
FLOIS() L
Speed
Imuge Reward
CLIP
PSNR
SSIM 1
LPIPS
Original: 50 steps
74.91
1.00x
12917.56
1.00x
1.2547 (600009)
35.51
00
1.000
0.000
50% steps
37.73
1.99
6458.78
2,00x
1,2048
(309999))
35.31
30.85
0.798
0.249
20% steps
2583.51
(284426)
35.17
0.504
TVylorSeertVV = 3)
36.90
2.03x
4646.60
2.78
1.0685
34.76
28.29
0.504
0.628
HyCaLN
35.33
4646.60
(1466)
34.97
0.763
0.247
FORAIN = 5)
21.71
3.45x
2585.46
5.00x
0.7767 (38.099)
34.47
24.55
0.553
0.556
ToCalA/
60.62
1.24%
2991.34
4.32x
0.9673
34.83
29.00
0.643
0.417
DuCaiN
24.83
3.02
13
4.37
0,8213
(063783
34.69
28.42
0.582
0.531
TuylorSeentA
24.61
3,04x
2585.46
5.00%
0.9483
34.76
28.29
0.504
0.628
HyCalV
21.58
3,47%
2584.46
5.00%
L.999
(484%%
34.87
29.65
0,709
0.320
FORALN 6)
17.89
4.19x
2323.30
5.56x
0.4781
(6LD0R))
28.50
28.38
0.546
0.597
ToCaIV/ =
52.72
1,42%
2406.20
5.37%
0,5593
(554499)
35.92
28.72
0,589
0.519
DuCaiA/=12)
21,82
3.43x
2171,56
5.98
0,5225
33.97
28.37
01576
0.593
TaylorSeert
21.88
2323.30
HyCatN 8
13.92
5.38%
2066.81
6.25%
1.0111
(13884)
34.75
28.89
0.600
0.433
5

---

Published as a conference paper at ICLR 2026
Table 2: Quantitative comparison text-to-image generation on FLUX. TIXX..1des 1-dev.
Method
Efficient Attention
Acceleration
Image Reward
CLIP Score
FLUX.1
Latency(s)
Speed
Idev]: 50 steps
25.82
1.00x
3719.50
1.00x
0.9898 (100000)))
32.404
(1000005))
60% steps
16.70
1.55x
2231.70
1.67x
0,9663
(23719))
32.312
(0.33/4)
99459
60-09933
A-DIT CN =
13.02
1.98
1686.77
2.21x
0.8721
(11,900)
32. 102
(0,9399)
DBeache
16.85
1.53x
2384.29
1,56x
1,0069
(1725)))
32.550
(10.3599))
TaylorSeer (W -
-099990
(+00329))
FoCa (V -3
9.28
2
1327.21
2.80x
0.9890
(0.0919)
32.577
(+0.5359))
HyCa (N
8.09
3
967.91
3.84x
10182
(+238886)
32,671
(-HLR225)
steps
9985)
(0.99993
Chipmenk
12.72
2,02X
1505 87
2.47%
0.9936 (843))
32.54K (+0.46495)
FORA LV
10.16
2.54x
1330,07
2.82
0,9776
(3296)
32.266
(D.425R)
Toca
0.9802
(009900))
Duca
8.18
3.15
978.76
3.80x
0.9955
(.05566))
32. 24.241 (005009))
TaylorSeer (N
4,0
9.24
2
967.1
1.84%
0.9857
(0 41499
32.413
(+0103755
FoCa
(+0.4145)
Clasca =,0=2
9.25
x
1045.58
3.56
0.9850
(0.4855)
32.441 (VOL1169)
HyCa (N
7.62
3.38 X
893.54
4,16x
1,0066
(+1.200%)
32,693
(-4L80055)
(995)
(0.5093)
Toca (AV
X
11.36
2
784.34
4.74x
0.9451
(4-51499)
31,993
(-122719)
Duca (V' 7)
6,74
3.83
760.14
4.89%
0.9757
(1) 4MM9
32,096
(-1000933
TeCCche
TavlorSeer (V 5,
7.46
3,46X
893.54
4.16x
0.9768
(1.3149)
32.467
(+0L1945)
FoCa (V
7.54
3.42x
745.39
4,99x
0.9713
87999
32.922
(+1.60095)
Speca (N
8.,.
Chasca
0.9718
81855
(0.707533
HyCa (V =
6.81
3.79
744,81
5.00%
1.0014
(1.1633))
32,483
(H0.24499)
0.7760 (21.629)
ToCa (NV WW=11
X
7,93
3.25x
714.66
5.20x
0.7155 (21,209)
31.808
(1.5399)
Duca (V'
1.27
3.55
690.25
5.39x
0,8382
(-15.379)
31.759
999993
ReCache [1
TaylorSeer (V -
6.77
3.81x
671.39
5.54
0.9698
(220020))
32.128
(0USS19)
Clusca (.V =6,0= 1.K
7.13
3.62X
748-48
4,97x
0.9704 (1099)
32.217
(.05779))
HyCa (N=
6,43
4,01X
670.44
5.55x
0.9895
(40830%)
32,520
(40.3889))
4.2 RESULTS ON TEXT-TO-IMAGE GENERATION
As shown in Table HyCa achieves the best overall trade-off Qwen-Image across all compression levels. -33, it matches TaylorSeer in speed (2.78x) but yields higher quality (ImageReward 1.2363 vs. 1.0685, and highest PSNR 30.42). At =6 HyCa remains strong (1.1939, 29.65), outperforming TaylorSeer (0.9483), ToCa (0.9673), and FORA (0.7767). Even at =88, it sustains good visual quality (1.0811 6.25x), while others drop sharply (ToCa 0.6326, FORA 0.4781). These results highlight HyCa's Ca's robustness under high acceleration while preserving visual fidelity.
On Table HyCa consistently achieves the best speed speed-qualiny quality trade-off on FLUX.1-dev. moderate accelcration (W=5), it reaches ImageReward of 1.0066 with 4.16x FLOPs reduction, surpassing TaylorSeer (0.9857 at 3.84x) and DuCa (0.9955 at 3.80x). Even under aggressive settings, maintains superior quality: 1.0014 at x.00 and even 0.9895 at 5.55 x, closely matching the original model (0.9898) while other baselines degrade (TeaCache 0.8683, ToCa 0.7155). Visual comparison on Fig, further confirms its advantage in image quality under high compression.
Table 3: Quantitative comparison of text-to-video generation on Hunyuan Video.
Method
Efficient
Acceleration
VBench f
Hunyuan Video
Attention
Latency(s)
Speed
FLOPs(T) | Speed
Score( %
Original: 50 steps
145.00
1.00x
29773.0
1.00x
80.66
(40.0%)
22% steps Te 0.4)
31.87
4.55x
6550.1
4.55x
78.74 (22.4%)
30.49
4.76x
6550.1
4.55
79.36 (+1.6%)
FORA(N =
34.39
4.22x
5960.4
5.00x
78.83 (:2.3%)
ToCa (N 5, R 90%)
38.52
3.76x
7006.2
4.25x
78.86 18.86 (-2.2%
(N= W=5, R 90%)
31.69
4.58x
6483.2
4.48
78.72 (-2.4%)
TaylorSeer (N' =5,50
34.84
4.16x
5960.4
5.00
79.93 (.0.9%)
Speca (Num Niiis
34.58
4.19x
5692.7
5.23x
79.98
Clusea (N = 5,0 K 16)
35.37
4,10x
$373.0
5.54x
79,96 (-099)
FoCa (N= =5)
34.52
4.20x
5966.5
4.99
79.96 (.0.5%)
HyCa (N
28.48
5.09
5359.1
5.56x
80.25 (40.5%)

---

Published as a conference paper at ICLR 2026
Original
HyCa x5.35
TeaCache
FORA x5.35
ToCa x5.20
DuCa x5.20
TaylorSeer x5.35
x1.00
x5.55
Figure 4: Visual comparison of 5.5x accelerated FLUX.
4.3 RESULTS TEXT-TO-VIDEE GENERATION
As shown in Table 3 our method delivers the best performance on Hunyuan Video. With V=6, it achieves the highest acceleration (5.56x FLOPs reduction) while maintaining strong VBench score (80.25), marginly lower than the original (80.66) full 50-step inference. In contrast, TaylorSeer reaches only 4. 4.6 with 79.93, TeaCache drops 79.36, and DuCa/ToCa degrade further. This demonstrates a superior gpeed-qualing trade-off and strong generalization to video generation.
Table 4: Quantitative comparison of image editing Qwen-Image-Ediin
Acceleration
GEdit-CN (Full)
GEdit-EN (Full)
Method
Latency(s) |
Speed
FLOPS(T)
Speed
SC1
PQ 1
08
SC1
PQ 1
OS
Original: steps
1.00x
28190.88
1.00x
7.68
7.51
7.54
7.54
50% steps
143.29
1.99
14095.4 44
2.00x
7.70
7.53
7.44
7.77
7.52
7.47
20% steps
58.45
4.87
5638.11
5.00x
7.65
7.42
7.35
7.73
7.46
7.44
FORALN
63.15
4.51x
5643.13
5.00x
7.60
7.31
7.25
7.34
7.28
DuCa (V' G, =
70.95
4.01x
$897.67
4.78x
7.63
7.44
7.44
7.68
7.42
7.39
TaylorSeen(V = 6)
65.66
4.33
5643.13
5.00x
7.25
7.09
6.92
7.26
7.14
6.89
HyCa (N
62,89
4.52x
5642.24
5.00%
7.76
7.49
7.50
7.77
7.47
7.45
FORAIN 77
$2,20
5.45x
4515.74
6.24x
7.42
7.13
7.06
7.43
7.19
7.06
DuCa (N
95%)
4.76x
5158.45
5.46
7.50
TaylorSeer(A'
53.92
5.28
4515.74
6.24x
6.61
6.65
6.31
6.67
6.63
6,31
HyCa (N' 8)
51.09
5.57
4514.48
6.24x
7.74
7.41
7.44
7.80
7.36
7.42
SC denotes Semantic Consistency Gedit Bench, denotes Perceptual Quility, OS denotes the Overall Sonte.
4.4 RESULTS ON IMAGE EDITING
Our method also performs strongly Qwen-Image-Edlis as shown in Table 4 At W--6, it achieves the best overall scores (7.50 CN 7.45 EN), surpassing TaylorSeer (6.92/ 6.89), FORA (7.25 7.28), and even the original model's (7.41 CN). A it remains leading (7.44 CN 7.42 EN) even exceeding the original model, while other baselines drop sharply (TaylorSeer 6.31 / 6.31). Visual comparisons in Fig. further confirm HyCa's superiority across diverse editing tasks.
7

---

Published as a conference paper at ICLR 2026
Reference
Prompts
Original
HyCa x6:31
10 Steps 500
FORA
DuCa XL495
TaylorSeer x5W
XCAB
Use abstract color blocks and lines express the composition.
Generate collage-style artwork.
Change the weather SHOWL
Convort the image to a Jupanese mange style.
Figure 5: Visual comparison of different caching method on Qwen-Image-Ediin
4.5 RESULTS ON DISTILLED MODELS
To examine compatibility with distillation, we evaluate HyCa on FLUX..1-chnnel and Qwen-Image-
On FLUX.I-schnell, HyCa cuts latency from 2.34s to 1.16s for the 4-step distilled model x24.4x over the original 50-step model) while improving quality: ImageReward increased to 0.9592 and excellent perceptual metrics (PSNR 34.37, SSIM 0.928, LPIPS 0.056). On it reduces latency from 13.35s(8-step distilled baseline) to 6.68s (12.2x) with quality largely preserved (ImageReward 1. 1.2201, CLIP 35.07) and perceptual metrics maintained (PSNR 30.97, SSIM 0.754, LPIPS 0.189). These results confirm that HyCa complements distillation, delivering extreme speedups with equal or even better quality.
Lightning, as shown in Table 5
Deee--maae-LLghtning
Table 5: Quantitative comparison of Distilled Model on Flux and Qwen-Image.
Acceleration
Quality Metrics
Perceptual Metrics
Latency
Speed
FLOPs(T)
Speed
ImageReward!
CLIP
PSNR
SSIM
LPIPS]
Iddex. 50 steps
25.82
1.00x
3719.50
1,00x
0.9898
32.40
FLUUX.fschhnel]: steps
297.60
12.50x
00
TaytorSSer (N 2):4 steps
1.5%
16. 34x
209.70 70
17:74x
09191
33.76
29.13
0.746
0.249
TeaCnche (1 0.0): 4 steps
1.26
20.49x
163.78
22.71x
0.9023
33.87
28.01
0.379
0,734
HyCa
sleps
152.32
24.42%
(558296)
34.37
Owen-Image: 50 steps
74.91
1.00x
12917.56
1.00x
1.257
35.51
Owen Owee--mage-LLghhhee Lightning: 8 steps
13.35
5.61
2113.67
6.11x
L
35.32
L.000
0.000
TuylorSeer (V 2) steps
8.11
9.24x
1320.81
9.78%
1.0418
34.44
29.49
0.62
0.377
TaylorSeer
31: steps
x1.07x
1057.08
12.22x
0.2644
27.98
0.672
HyCa (N' - 2): 8 steps
8:20
9.13x
132088
9.78%
1.2478
(83385)
35.27
32.52
0.837
0.119
HyCa (V- 3): 8 steps
6.68
11,21x
1057.08
12.22x
1.2201
(2.522)
35.07
30.97
0.754
0.189
PSNR, SSIM, LPIPS of IlyCa are computed with respect oulputs corresponding distilled huseline models.
5 DISCUSSION
Ablation Study. We conduct our ablation study on FLUUX, as shown in Fig. (d),, HyCa consistently achieves higher ImageReward and lower prediction error than any individual solver baselines from solver pool under the same conditions. These results confirm that HyCa benefits from combining diverse solvers rather than relying on a single integration strategy.
Why Dimension- Wise Assignment? A central design in HyCa is to assign caching strategies at -iiturr-dimmesion level rather than token level primarily due to its better stability, as shown in Fig clustering results in feature space remain nearly invariant once identified. Thus, solver assignments can be reused extensively, reducing both computational and data requirements. On the contrary, token-wiss assignment varies with prompt and resolution, requiring frequent re-selection during
8

---

Published as a conference paper at ICLR 2026
"Two dugs on the street. 1024 Step
"A pink culored car." 800 640 Step
"A panda making latte art."
"A pizza cooking an oven."
$12*1140 Step
1664-1140 Step
Clattrr
Clster
Chaier
Clattr
Principal composet?
Cealare
Chater2
Principal compincal
Chaer1
Crntes
Princippal compancet?
* Carres
Chair
Principal compoacal
x Crates
Glater2
Silhowite Scane: 0,72
bithommt Score: 0,72
Sillemettee Score: 8.73
"Siifoiiet Sconet 0.75
Principal component 1
Principal component
Principal composcat
Principal component
Oaster1
Chowe Onier3 Cusii
Ohater Chear2 x Crntes
Chiter Clatte x Contes
Principal component
Cheter2 Centes
Prited cumponent 2
Principal companentt
Principal componcnt
Silhowette Score:
Silkomette Score: 0.73
Sithouette Score:
Sithwsstte Scuvne 0.72
Principal component
Principal component "A bird souring gracefially the sky."
Principal component
Princigal companent "A bus accelerating to gain speed" 640 Step
"A caw bending down to drink water from
horse on the right of sheep, front
450 Step 45
720"480 36
480"640 Step 17
Top row: Clustering results from FLUX; Bottom row: Clustering results from Hunyuan Video. The clustering assignments remain highly consistent across various prompts, resolutions and timesteps, suggesting stable and robust geometric structure in the feature space.
Figure 6:
inference. Empirically, Fig. (a)(b) confirms that our dimension-wise assignment outperforms both token-wise (ToCa, DuCa) and mne-siz--itt--le (FORA. TaylorSeer) baselines.
Compatibility with Distillation. Feature caching is traditionally difficult to adapt to distilled models, distillation drastically reduces sampling steps (e.g., from 50 to 4 or 8), making feature trajectories more discrete and oscillatory. Prior caching methods rely heavily on smooth temporal evolution and thus fail in this setting. In contrast, HyCa remains effective: its solver pool includes implicit methods suited for discrete or oscillatory dynamics, and solvers are assigned per cluster for each model. This flexibility makes HyCa fully compatible with distillation, achieving substantial acceleration while preserving generation quality.
1.0
Mean Prediction Error
TF
1890
AB
ImageReward
RengeRReaat Imagekewe
ImageReward
160
RK
Flux
HyyCa
Owen-Image
HaC's
Taylefer
Ilp's
[lyCaa
120
-20 Larger
PORA
TeCa
100
DvCo
TesCadk
DaCa
10
Speedup Ratio
Speedup Ratio
Speedup Ratio
Skipping Interval
(a)
(d)
Figure 7: Overall and ablation results of HyCa. (a b) HyCa consistently outperforms token-wisst
(ToCa, DuCa) and one-size-fitss (FORA, Taylorseer) baselines on FLUX and Qwen-Image. (c-d) Ablation on FLUX shows HyCa surpasses all single-solver baselines in the solver pool, maintaining lower enror and better quality. Confirming that HyCa benefits from combining diverse solvers rather than relying on single integration strategy.
Compatibility with LoRA Fine-Tuning HyCa remains effective under LoRA fine-tuning, As LoRA only updates small subset of parameters through low-rank adapters, most feature dynamics and their clustering structures should remain stable. We evaluated this on XLabs-AL Art LoRA and Anime LoRA of FLUX (Xlabs-AI] 2024), shown in Fig. the clustering results before and after

---

Published as a conference paper at ICLR 2026
fine-tuning remain consistent. We further quantify this similarity by computing the Adjusted Rand Index (ARI) between the LoRA cluster assignments and the original model's. both cases, the mean ARI exceeds 0.8, indicating strong agreement and confirming that HyCa's 's clustering results largely invariant under LoRA adaptation. Therefore, we believe that for models sharing the same backbone, switching between different LoRA does not require re-clustering or solver re-selection in most practical cases. Even if re-computation is needed in rare cases, HyCa's offline cost remains minimal (about 1s). This demonstrates HyCa's robustness and efficiency in real-world deployment scenarios where multiple LoRA adapters are used for personalization or domain adaptation.
"A brown colored giraffe."
"A black colored dog."
"A pink colored car."
"A car and a white sheep."
Principal component
Chier
Chatar
Choer3
FLUX.I-dev
FLUX.I-dev
FLUX.I-dev
FLUX.I-dev
Principal component
Charrt
Chaor
Clanri
Cloke2
Art LoRa
Art LoRa
Art LoRa
Art LoRa
Principal component
@ Caseri
Clanri
Clanri
e Clanci
Anime LoRa
Anime LoRa
Anime LoRa
Anime LoRa
Principal component 2
Principal component 2
Principal component 2
Principal component 2
Figure Clustering consistency between original and LoRA-tuned models. Each subplot visualizes the clustering results under different prompts for FLUX.1-de (top row), and its two LoRA variants, the Art LoRA (middle row) and Anime LoRA (bottom row). Despite fine-tuning, the cluster boundaries remain highly consistent across prompts and models, indicating that HyCa's solver assignments remain stable and reusable even after LoRA adaptation.
CONCLUSION
In this work, we revisited feature caching in DiTs as hybrid solving ODEs. We revealed that the evolution of hidden features is not governed by single, unified system, but rather by a mixture of ODEs, where different dimensions follow heterogeneous dynamics. Building on this insight, we introduce HyCa, training- faiming-free free framework that clusters feature dimensions and assigns tailored solvers from a hybrid solver pool. Analysis shows that the cluster structures in feature space are input-invariant, enabling "One-Time Choosing" and "All-Time Solving" with negligible overhead. Extensive experiments across text-to-image, text-to-video, and image-editing tasks demonstrate that HyCa achieves substantial acceleration without sacrificing quality, including 5.55 acceleration FLUX, 5.56 on Hunyuan Video, 6.24x on Qwen- Image, and even up to 22 speedup on distilled variants. These results highlight its broad applicability to diverse architectures and the orthogonality to distillation. Looking forward, we envision extending this mixture-of -DD perspective to other generative models and exploring learning-basee caching strategies to further enhance efficiency.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This project is sponsored by CCF-Tencent Rhino-Bird Funds.
ETHICS STATEMENT
We acknowledge and adhere to the ICLR Code of Ethics in all aspects of this research. This work presents HyCa, a training-fre free acceleration framework for diffusion transformers that aims to improve computational efficiency without compromising generation quality. We address several ethical considerations relevant to our contribution:
Responsible Use and Dual-Use Considerations. While method improves the efficiency of diffusion models used for creative and productive applications (text-to-image, text-to-video, image editing), acknowledge that accelerated generative models could potentially be misused for creating misleading or harmful content such as deepfakes. However, our contribution is technical acceleration method that is agnostic to content generation- does not introduce new capabilities for harmful content creation but merely makes existing capabilities more efficient. The responsibility for ethical application remains with end users and deploying organizations.
Data and Privacy. Our method operates entirely model activations and requires no additional training data or model modifications. We evaluate on standard public benchmarks (DrawBench) VBench, GEdit-Bench) without collecting processing personal data. The caching mechanism operates on intermediate neural network representations and does not store or expose original input content, maintaining privacy during inference.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of our results, we have made extensive efforts document all imple- mentation details and experimental configurations. Our code is provided supplementary material, and will be released on Github.
All experimental protocols are thoroughly documented in A.1 including specific model configurations, evaluation datasets (DrawBench, VBench, GEnct.Bench), resolution settings, and random seed specifications (fixed seed=0) for all experiments). Our feature clustering methodology, including the descriptor vector construction using indicators such as Second-Order Change d2, Curvature Ratios n, Jerk Ratios к and Spectral Flatness sf, can be found in A.2.1 The mathematical formulations for our hybrid ODE solver framework are provided in detail in A.2.2 with complete derivations each numerical method in our predictor pool.
REFERENCES
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4599-4603, 2023.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation, 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024b.
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. 5-dit: A training-fres acceleration method tailored for diffusion transformers. arXiv preprint 44Xiv:2406.01 125, 2024c.
Xinle Cheng, Zhuoming Chen, and Zhihao Jia. Cat pruning: Cluster-aware token pruning for text-to-image diffusion models, 2025.
http:////aaaai..or///////500.000 00433

---

Published as a conference paper at ICLR 2026
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. arXiv preprint arXiv:2305. 100244 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv22144008/11 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 23:6840-6855 2020.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing
Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu, VBench: Comprehensive Benchmark Suite for Video Generative Models, November 2023. URL penttt::/aarxee
org/ abs/ 2311 17982 arXiv:2311.1798 [es].
Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1383-1392, 2024.
Sungbin Kim, Hyunwuk Lee, Wonho Cho, Mincheol Park, and Won Woo Ro. Ditto: Accelerating diffusion model via temporal value similarity. In Proceedings of the 2025 JEEE International Symposium on High-Perie Perforrefo Computer Architecture (HPCA). IEEE, 2025.
Black Forest Labs. Flux. bttps:///ithabee com
flux 2024.
Senmao Li, Taihang Hu, Fahad Shahbaz Khan, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and Jian Yang. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv preprint 23v:22312.0960, 2023a.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17489-17499, 2023b. doi: 10.1100//CCV55107.222233 01608.
Yanyu Li. Huan Wang. Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024.
Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model, 2024.
Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorseers, 2025a. URL hattps://arxtt 06923
Shiyu Liu, Yuan Han, Zhiwei Wang, Qingsong Ma, Xiaolong Li, Xianfang Zeng. Xiao Ma, Chen Zheng, Hongxin Ma, and Zhiwei Liu. Steplx-edit: A practical framework for general image editing. arXiv preprint arXiv:2504.1 17761, 2025b.
Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference Learning Representations, 2023.
Ziming Liu, Yifan Yang, Chengruidong Zhang, Yiqi Zhang, Lili Qiu, Yang You, and Yuqing Yang. adgion-ad sampling for diffusion transformers, 2025c. URL
abs 2t55/2502.133
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 2211:2211.00999 2022b.
12

---

Published as a conference paper at ICLR 2026
Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan-Yar Yee K. Wong. Fastercache: Training-free free video diffusion model acceleration with high quality, 2025. URL
https://arxiv.co org/ abs 2410 19355
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15762-15772, 2024.
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers, March 2023a. URL arXiv:2212.09748 [cs].
http://arxiv/ org/ http://Aarav.oorr//////22222.222
William Peebles and Saining Xic. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023b.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicul image computing and compuler-assisted InterventionMMICAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 2.1. Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Wang, Karim Ghasemipour, Raphael Gontijo Lopes, Burcu Lee, Ekin Gontijo Lopes, Jonathan He, and et al. Photorcalistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511 2022.
Pratheba Selvaraju, Tianyu Ding. Tianyi Chen, Ilya Zharkov, and Luming Liang. Fora: Fast-forward caching in diffusion transformer acceleration. arXiv preprint vrXiv:2407.01425 2024.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1772-1981, 2023.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 3221 -22222.. PMLR, 2023.
Wenzhang Sun, Qirui Hou, Donglin Di, Jiahui Yang, Yongjia Ma. and Jianxun Cui. Uniep: A unified caching and pruning framework for efficient video generation, 2025. URL https //arxiv.org/att 220.. 04393
Xingwu Sun, Yanfeng Chen, Huang, et al. Hunyuan-large: An open-source MoE model with 52 02212
billion activated parameters by tencent.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang. Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng. Lin Qu, Minggang Wu, Peng Wang. Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yonggiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report, 2025. httttp::/aaxdd..cg//abb../
02324
ALa Xlabs-ai / flux-lora-collection, 2024. XLabs - AI htr///fuuxx-ra-ooo www.-
URL https://nnggiggfaee
Jiazheng Xu, Xiao Li, Guoli Xu, Yuan Zhang, Xinyu Zhang, Qishuo Zhou, Yanan Wang, Qiming Liu, Yunjie Zhang, Yuqing He, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint 055777 2023.
13

---

Published as a conference paper at ICLR 2026
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan.Zhang Weihan Wang, Yean Cheng, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference on Learning Representations, 2025. http::ttts////peeeeeeee
ew net/ wttee//forumme@1112.6TTT..
Zhihang Yuan, Hanling Zhang. Pu Lu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. arXiv preprint arXiv:2406.0888 08552, 2024.
Evelyn Zhang, Bang Xiao, Jiayi Tang. Qianli Ma, Chang Zou, Xuefei Ning, Xuming Hu, and Linfeng Zhang. Token pruming for caching better: 9 times acceleration on stable diffusion for free, 2024. URL https::///// pettps://arxi>> wwwaaaaivvvooo... abs/ 00375
Evelyn Zhang, Jiayi Tang, Xuefei Ning, and Linfeng Zhang. Training-free and hardware-friende acceleration for diffusion models via similarity- -baed token pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408, 12588, 2024.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. DPM-solver-v3: Improved diffusion ODE solver with empirical model statistics. In Thirry-evvntth Conference Neural Information Processing Systems, 2023. URL
?id= WWWKExmKal
Shikang Zheng, Liang Feng, Xinyu Wang, Qinming Zhou, Peiliang Cai, Chang Zou, Jiacheng Liu, Yuqi Lin, Junjie Chen, Yue Ma, and Linfeng Zhang. Forecast then calibrate: Feature caching as ode for efficient diffusion transformers, August 2025. URL
org/abs 2508
16211
arXiv:2508. 16211 [cs.CV].
Zangwei Zheng, Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. UtLttttss: github.. wwtp//acaaatccch/opeee.ena Sooaa
Haowei Zhu, Dehua Tang, Ji Liu, Mingjie Lu, Jintu Zheng, Jinzhang Peng. Dong Li, Yu Wang, Fan Jiang. Lu Tian, Spandan Tiwari, Ashish Sirasao, Jun-Hai Yong, Bin Wang, and Emad Barsoum. Dip-go: A diffusion pruner via few-step gradient optimization, 2024.
Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang. Accelerating diffusion transformers with wise feature caching. arXiv preprint 13X0v.2410.03317, 2024a.
Chang Zou, Evelyn Zhang, Runlin Guo, Haohang Xu, Conghui He, Xuming Hu, and Linfeng Zhang. Accelerating diffusion transformers with dual feature caching, 2024b. URL https:
//a raaxiv. www.v.orggbba///eo /ta.ab//24111 2412. 11.4112..1991 18911
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DETAILED EXPERIMENT SETTINGS
Model Configurations and Evaluation Protocols. Our experimental evaluation encompasses six distinct generative models across three primary tasks: text-to-image generation, text-to-video generation, and image editing. For text-to-image generation, we evaluate four different models: FLUX.1-dev, Qwen-Image Flux-Schnell, and standardized benchmarks and protocols to ensure fair comparisons. For text-to-video generation, utilize the Hunyuan Video model, which is evaluated using the VBench framework capture various aspects of video quality and coherence. Finally, for image editing tasks, we assess the Qwen- Image-Edit model using the GEdit-Bench benchmark, which provides a comprehensive suite of editing scenarios and evaluation metrics, Detailed configurations for each model, including resolution settings, prompt selections, and evaluation metrics, can be found the subsequent sections.
Ban--mmage-LLghhniine
Each model is assessed using
A.1.1 CAKT-TO-IMAGE GENERATION
FLUX.1-dev. We evaluate FLUX, 1-dev using the standard DrawBench protocol, which provides a diverse set of 200 prompts spanning multiple categories including animals, colors, conflicting, and fine-grained details. All images are generated at a resolution of 1024 x 1024 pixels. maintaining consistency with the model's 's optimal operating parameters. We assess generation quality using two metrics: ImageReward (IR) for evaluating photorealism and overall image quality and CLIP Score for measuring text-image semantic alignment.
model, we utilize the DrawBench dataset as well, which contains 200 prompts designed for evaluating image generation capabilities. The model generates images at higher resolution of 1328 x 1328 pixels. We not only assess generation quality using ImageReward (IR) and CLIP Score, but also traditional fidelity metrics including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) for pixel-level and perceptual similarity assessment.
Qwen-Image- For the
FLUX.1-schnell. FLUX. 1-schnell,, step-distillation model from FLUX.I-dex is designed for rapid image generation. It evaluated using the standard DrawBench protocol with 200 prompts at 1024 x 1024 resolution. Despite its focus on speed optimization, we maintain the same evaluation standards, assessing both generation quality through ImageReward and CLIP Score, as well as fidelity through PSNR, SSIM, and LPIPS metrics. This allows us to quantify the trade-offs between generation speed and output quality.
the step-distilled variant of Qwen-Image is evaluated using 200 DrawBench prompts to maintain consistency with other text-to-image models, while operating at the model's native high resolution of 1328 x 1328 pixels. We also apply the same evaluation framework as above to assess the quality of the generated images.
A.1.2 TEXT-TO- VIDEO GENERATION AND IMAGE EDITING
Hunyuan Video. For text-to-video generation, we evaluate Hunyuan Video using the comprehensive
VBench framework, which provides multi-dimensional human-aligned assessments across 946 diverse prompts sourced from the VBBech-ffll-info.pppp resolution with 65 frames each, providing substantial temporal content for thorough evaluation. VBench evaluates 18 critical aspects including but not limited to motion quality, visual appearance consistency, temporal coherence, semantic alignment with text prompts, and overall video quality through human-correlated metrics.
file. Videos are generated at 480 x 640
Qwen-Image-Fdiis For image editing capabilities, we evaluate Owen-Image-Faae using the GEdit- Bench framework, which contains 1212 carefully designed editing prompts covering 11 diverse edit types, including but not limited to background change, color alteration, material alteration, motion change, and pose manipulation. The benchmark provides comprehensive assessment across multiple editing scenarios without fixed resolution constraints, thus no specific resolution is enforced during evaluation. We utilize both the GEdit-CN and GEdit-EN subsets for evaluation, employing metrics such as Success Rate (SC) to measure the effectiveness of edits, Perceptual Quality (PQ) to assess visual fidelity, and Overall Score (OS) for holistic evaluation of editing performance.
15

---

Published as a conference paper at ICLR 2026
Reproducibility and Consistency Measures. All other settings, such batch sizes, efg-scale and model-specific parameters, are kept consistent with the original configurations recommended by the model developers. The standardized evaluation protocols ensure that observed performance differences reflect genuine model capabilities rather than experimental artifacts. All implementation details and hyperparameter settings carefully documented facilitate reproduction of our experimental results.
A.2 DETAILED MATHEMATICAL FORMULATIONS
A.2.1 CLUSTERING INDICATORS
In this section, we provide detailed mathematical definitions for the statistical feature indicators used in our clustering- based solver assignment strategy. These indicators capture various aspects of the temporal dynamics of feature dimensions, enabling effective clustering and solver selection.
First-Order Difference. The first-order change indicator measures the velocity of feature evolution, representing the Euclidean distance between current and previous time steps:
dj lIf fx-1ll
(6)
Where fú denotes the feature value timestep k. This metric captures the immediate rate of change feature values and serves as a fundamental indicator of temporal dynamics.
Second-Order Difference. The second-order change indicator quantifies acceleration or curvature in the feature trajectory:
d2 1 2 fu-21
This represents the Euclidean norm of the second-order difference, providing insights into the smoothness and stability of feature evolution patterns.
Third-Order Difference. The third-order change indicator measures jerk in the feature trajectory:
d3
f 3f4-1 3 fx-all
(8)
This metric captures the Fuclidean norm of the third-order difference, indicating abrupt changes in acceleration and helping identify highly dynamic or unstable features.
Curvature Ratio. The curvature ratio provides normalized measure of curvature relative to velocity:
d2
17
+
where is a small constant to prevent division by zero. This ratio helps distinguish between features with different curvature characteristics while accounting for their overall rate of change.
Jerk Ratio. The jerk ratio measures the relative magnitude of jerk compared to curvature:
d3
(10)
d2 +
This indicator helps identify features with sudden acceleration changes and is particularly usefuu for detecting instabilities that require specialized numerical treatment.
Direction Consistency. The direction consistency indicator measures the consistency of directional changes in feature evolution:
(Jr fk-1) -fx-1- fk-2)
cos =
(11)
Ilfa fk-1 flllllff----
-2i1-fx-211 fk-2ll +
This metric quantifies how consistently features change direction over time, with values close to indicating consistent direction and values close to -1 indicating frequent direction reversals.
Relative Change Rate. The relative change rate normalizes the first-order change by the magnitude of the previous time step:
d
(12)
16

---

Published as a conference paper at ICLR 2026
This ratio provides features with different magnitude scales.
information about the rate of change, enabling comparison across
Energy. The energy indicator represents the magnitude of the current feature vector:
c f
(13)
This Euclidean norm provides information about the overall scale and importance of feature dimen- sions, helping to weight their contribution to clustering decisions.
Low Frequency Ratio. The low frequency ratio quantifies the proportion of power concentrated in low-frequency components:
Σ P
lfr
(14)
Σ1=0 P.
where P. represents the power in the i-th DCT frequency component and q corresponds the first 20% of frequency components. This metric helps identify features with predominantly smooth, low-frequency evolution patterns.
Spectral Flatness. The spectral flatness measures the uniformity of the frequency distribution:
exp( + Σラ P3)
sf
(15)
M Py
where M is the number of frequency components. This ratio of geometric mean to arithmetic mean of the DCT power spectrum indicates whether the feature evolution has flat peaked frequency profile, helping distinguish between noise-like and structured temporal patterns.
A.2.2 PREDICTOR POOL FORMULATIONS
In this section, we provide the detailed mathematical formulations for the predictor pool S used in our HyCa framework. The predictor pool consists of both explicit and implicit numerical solvers, each with different stability and accuracy properties. For a general ordinary differential equation (ODE) of the form:
dy = f(t y). y(to) yo
(16)
we define the following numerical methods for advancing from time step fn to tn41 tn + h, where h is the step size.
Runge-Kutta Methods (RK). The family of Runge- Kutta methods provides high-order accuracy through multiple function evaluations per step. The general s-stage explicit Runge-Kutte method is formulated as:
= tn + cch, yn h Σ
aiskis
i= 1,
(17)
j=1
yn+1 Un + h
bik:
(18)
where asj. b;, and q are the Butcher tableau coefficients that define the specific Runge- Kutta method.
These coefficients satisfy consistency conditions:
Σ
b=
(19)
The order of accuracy is determined by the number of order conditions satisfied by these coefficients.
Methods (AB). These explicit multistep methods utilize information from previ- ous time steps to achieve higher accuracy. The k-step Adams-Bas -ashffoth method is given by:
ya+1 Un + h Σ
8tf(tn-ji vn-1)
(20)
17

---

Published as a conference paper at ICLR 2026
where By are the Adams-Bashfer coefficients for the k-step method, which can be computed using the formula:
(-1)
k-1
Bj j!(A 111k-1-93 j}!
II
s s i)
(21)
Taylor Series Extrapolation (TF). This method extends the Taylor series expansion to higher orders for improved accuracy:
h2
yn+1 = Un + hLn +
2! Sn 3."
In
p1
(1--1)
(22)
p!
where f denotes the k-th derivative of f evaluated at (In; Vn.) For practical implementation, compute derivatives using automatic differentiation:
0f
0f
f
+
f
(23)
0t
dy
02
021
8 oy²
of dy
f/ 0t
+
8 +
f
(24)
Otdy
Backward Differentiation Formula (BDF). These implicit methods provide excellent stability properties for stiff equations. The k-step BDF method is formulated
(25)
where the coefficients aj and 3 are determined by the requirement that the method is exact for polynomials of degree k. The coefficients can be systematically computed through the differentiation of the interpolating polynomial through the points (fn+1-j, (--11,))+)1)) for j = 0, 1......, k.
Adams-Moulton Methods (AM). These implicit multistep methods offer improved stability com- pared to their explicit counterparts:
k-1
Vn+1 = Un
B (tt++14:.
(26)
where the coefficients B, are obtained by integrating the interpolating polynomial through the points Ma+1++::n++1: for = 0,.0....... k 1, resulting in k-step implicit method with order k + 1.
Solver Selection Criteria. Each method the predictor pool S exhibits distinct stability and accuracy characteristics. Explicit methods including Runge-Kutta, Adams-Bashforte and Taylor extrapolation offer computational efficiency but may suffer from stability constraints. contrast, implicit methods such BDF and Adams-Moulton provide superior stability for stiff problems at the cost of solving nonlinear equations. Higher-order methods achieve better accuracy but require more computational resources, while multistep methods utilize historical information efficiently but require special initialization procedures.
The adaptive selection mechanism in HyCa evaluates local feature dynamics and assigns the most appropriate solver from based on stability requirements, accuracy demands, and computational constraints each spatial location.
A.3
VISUAL QUALITY COMPARISONS
A.3.1 TEXT-TO-IMAGE GENERATION RESULTS
Figure 9 provides qualitative comparisons between HyCa framework and baseline methods across different tasks. Each figure highlights the visual fidelity and detail preservation achieved by HyCa under high acceleration ratios.
18

---

Published as a conference paper at ICLR 2026
Hall's Wct
FLUX.11
[schnell] x12.50
Holli Ro1l
Hallo
HyCa x24.42
Figure 9: Comparison between HyCa and the original FLUX.1 1-Schnell model. HyCa achieves much higher acceleration ratio while producing images that almost indistinguishable in quality from the originals.
A.3.2 TEXT-TO- VIDEO GENERATION RESULTS
Figure 10 illustrates the superior temporal coherence and visual fidelity of videos generated by HyCa compared to TaylorSeer under high acceleration ratios. TaylorSeer exhibits noticeable artifacts and temporal inconsistencies, while HyCa maintains smooth motion and consistent details throughout the video sequence. Artifacts are highlighted with red boxes along with text clarifications.
"A
"A rusning herd of kind."
HyCa
Tay lorser
x5.00
"A couple formal evening waar pring home gat cought in havvy downpaur with unbrellas, black and while."
spoon,
Ca
X5.56
Muives ovverapping
Unibrllla mndsing
Tay BorSeer
x5.00
"A panda drisking coffee in safe in Paris, animuste style?
"A person shooting good (inoccr))"
Cy
x5.36
ans massing
TaylorSour
x5.00
Figure 10: Comparison between HyCa and TaylorSeer on Hunyuan Video. HyCa maintains superior temporal coherence and visual fidelity under higher acceleration ratio, while TaylorSeer exhibits noticeable artifacts and temporal inconsistencies.
19

---

Published as a conference paper at ICLR 2026
A.4 CLUSTERING ANALYSIS AND SOLVER ASSIGNMENT
Our HyCa framework operates a fundamental principle that feature dimensions with similar temporal dynamics should be grouped together and assigned the same numerical solver. This clustering-based approach ensures computational efficiency while maintaining prediction accuracy. The methodology follows a three-stage logical progression: (1) identifying feature dimensions with similar temporal behavior through clustering analysis, (2) characterizing these clusters using trajectory-based descriptors, and (3) assigning optimal solvers to each cluster based on their dynamic properties.
The clustering similarity is first established by analyzing the temporal evolution patterns of individual feature dimensions across multiple timesteps. Features exhibiting similar rates of change, curvature characteristics, and stability properties naturally form coherent clusters. Once these clusters are formed, we extract trajectory-bases metrics such as second-order differences and curvature ratios to quantify the dynamic behavior within each cluster. These trajectory features serve as descriptors that capture the essential characteristics of feature evolution within each group. Finally, the solver selection process leverages these trajectory descriptors to match each cluster with the most suitable numerical method from our predictor pool, ensuring that explicit methods are assigned to stable, smooth regions while implicit methods handle stiff or rapidly varying dynamics.
button ARl Villes
Ditiibution Values
Desribution
Proppt
Recolubions Beeckation
Timentep
Promptt
Trompt
Pharpt dite
limalap
Tirestop
Timestep KDB
Tinestpp 4DE
Tirastop KDC Ph:
FLUX
Owen-Image
Hunyuan Video
ULUX
03
14
kandlisded (NR)
Adjuded kand Iode 11-1
Adsusoo Ford
Normalized Cluster Drift Time Series Festure 2: Normekzed Drit
Feeture 1: Normelzed Dritt
Feeture 3, Normelzedd Dritt
Quearo
Cloke 9
0.75
Ouwor
Ca 1
NomooFood D/i V/
0.60
2:50
Normall foed Cvf Phklee
0.5
0.55
2.20
0.55
0.00
0:00
0.02
4.1
4.5
4.0
-130
Prompt Indix
Ficrest Indox
Prompt Index
Figure 11: The first row shows, from left to right, ART comparison plots of FLUX, Qwen-Image, and Hunyuan Video under different prompts and timesteps, with the last plot illustrating ART comparisons of FLUX across different resolutions. can be seen that ARI distributions exceed 0.8 in most cases, confirming stable and consistent cluster assignments across prompts, timesteps and resolutions. An ARI above 0.8 indicates strong agreement and high clustering reliability. The three plots in the second row depict the intra-cluster metric shifts between clusters formed by indicators from every two adjacent prompts of FLUX. The normalized value is obtained by dividing the inter-cluster shift of given indicator by the average value of that indicator within the cluster. It can be seen that most values fluctuate between -0.02 and 0.02, indicating excellent consistency of the metries.
A.5 COMPATIBILITY WITH LORA TUN-TUNING
HyCa is also robust under LoRA fine-tuning. We evaluate HyCa on XLabs-Al AI Art LoRA and Anime LoRA of the FLUX.11dde model and analyze whether clustering and solver assignments need to
20

---

Published as a conference paper at ICLR 2026
"A brown giraffe."
black colored dog."
"A pink colored car."
and white sheep."
Principal component
Cunrri
Chaner
Chars i
e Chater
dary
FLUX.I-deY
F-Ux.I-dev
FLUX.I-dev
FLUX.I-der
Principal component
Clatar
Cletr
Onier
Cherr
Clekr2
Clarr2
Art LoRa
Art LoRa
Art LoRa
LoRa
Principal component
Cwarr
Clanr
Clnor
Chor2
Anime LoRa
Anime LoRa
Anime LoRa
Anime LoRa
Principal component
Principal component
Principal component
Principal component
Clustering consistency between original and LoRA-tuned models. Each subplot visualizes the clustering results under different prompts for FLUX. (top row), and its two LoRA variants, the Art LoRA (middle row) and Anime LoRA (bottom row). Despite fine-tuning, the cluster boundaries remain highly consistent across prompts and models, indicating that HyCa's solver assignments remain stable and reusable after LoRA adaptation.
Figure 12:
Promp:
Timestop
ARI Across Prompts
ARI Across Prompts
Sample Count
Prompt
Sample Count
Sample Count
Timestop
03
FLUX.11de
Art LoRA
Anime LoRA
0.
02
0.6
0.6
0.8
1.0
0.
0.2
0.4
2.6
2.
10
Adjusted Rand Index
Adjusted Rand Index
Adjusted Rand Index
(a)
(b)
(c)
Figure 13: ARI between clustering assignments from FLUX.1-dev and its LoRA variants. (a) ARI distributions across different prompts and timesteps for FLUX.I-des show clustering consistency within the base model. (b) and (c) compare the original model's clustering with that from Art LoRA and Anime LoRA, respectively. In both cases, the ARI values remain high (most above 0.8, marked by the red dashed line), confirming strong agreement between the LoRA and original clustering results. This further supports that solver assignments in HyCa can be reused across LoRA variants.
21

---

Published as a conference paper at ICLR 2026
be re-computed. Since LoRA only updates small subset of parameters through low-rank adapters, most feature dynamics and their clustering structures should remain stable.
We extract feature trajectories from both LoRA-tuned models and compare their clustering assign- ments with those from the original FLUX.I-dew model. As shown Fig. 12 the clustering results remain visually consistent. We further quantify this similarity by computing the Adjusted Rand Index (ART) between the LoRA cluster assignments and those of the original model. As shown in Fig. 13 b-c), in both cases, the mean ARI exceeds 0.8, indicating strong agreement and confirming that HyCa's clustering results are largely invariant under LoRA adaptation.
Therefore, believe that for models sharing the same backbone, switching between different LoRA does not require re-clustering or solver re-selection in most practical cases. Admittedly, it is infeasible exhaustively test all possible LoRA adapters: however, even in rare cases where HyCa may require changes, the offline process only needs about 1 second
re-clustering due to drastic adapter-induce to assign new solvers for the updated LoRA. This demonstrates HyCa's 's robustness and efficiency in real-world deployment scenarios where multiple LoRA adapters are used for personalization or domain adaptation.
A.6 DEMONSTRATION OF RUNGE- KUTTA METHOD
Our use of Runge- Kutta (RK) follows the standard ODE formulation and is then adapted to the discrete setting of feature caching.
Classical Runge-Kutta method. Consider single feature dimension following an ODE
dy [(T, v),
y(Tx) = V/u
dr
with step size = Tn+1 Tn. A standard second order Runge- Kutta method computes:
k f(Tn, Bn), 22 [(Tn + h, + hki),
(27)
(28)
and updates:
11 (hn k2)
Un+11 Yn +
(29)
Intuitively, Runge-Kutta averages the slope at the beginning and (predicted) end of the step to obtain second order accurate approximation of the trajectory.
Adapting RK2 to discrete feature caching. In our setting we do not have direct access to f(r, u). because evaluating it would require additional forward passes of the diffusion transformer. In- stead, for each timestep index tn and feature dimension d, we only have the cached activations p!l)) .p((d) rrdd) t from previously computed steps. To construct an RK predictor that uses only cached features, we replace the derivatives in equation 29 with finite differences.
With uniform step size h, we approximate the instantaneous slopes by backward differences:
((d)
F
(d) in
(d) En
A
h
Here A, plays the role of s(Tn while an extrapolated slope
An + (A, An-
approximates f(T, h, yn. hk1) in equation 29 Substituting hı A,, and k2 Ant1 into
equation 29 yields the following discrete RK2 predictor in feature space:
p(d) + (An + An+1)
(30)
p(()) + h (An,
An + (An, An-1))
(31)
p(d) +
3h
An
h
An-1
(32)
2
2
3 (d)
fn
1 ((d)
2 in-2"
(33)
22

---

Published as a conference paper at ICLR 2026
This is exactly the form in HyCa for RK2 solver: it operates purely cached features and does not invoke any extra network forward passes. During the offline "one-time choosing" stage, such discrete RK2 predictors (along with AB, BDF, Taylor, etc.) instantiated per cluster, their next-step prediction errors are measured on recorded trajectories, and the solver with the lowest error is assigned to that cluster. At inference time, the chosen predictor equation [33 is then applied to forecast Fi..,, from {Ft,, : Ft providing a concrete and fully specified example of how RK2 is used for feature prediction in our framework.
In practice, we instantiate P(.)) as the residual output of specific DiT block timestep 4,, rather than as the full diffusion latent. At compute step, we evaluate the original block function and obtain the exact residual update, which is then cached as Ft.. At skip step, we bypass the expensive block computation and instead use discrete solver (e.g., the RK2 rule in equation 33) to predict the next residual from cached outputs, dimension-wise:
Pe RK2( Fr.,, Pt,, Pt,, 2t..-2):
analogously for AB, BDF, Taylor, and other solvers in pool. The predicted residual Fr... then injected back into the network via the standard residual connection of that block, i.c., the block output is obtained by adding Fr.., to the block input hidden state, exactly as in the original DIT architecture. Chaining this operation over all blocks yields the final hidden representation timestep fa+1. Therefore. HyCa only substitutes the per-block residual computations at skip steps with solver-based predictions on cached features, while the residual pipeline and the outer diffusion sampler remain unchanged.
A.7 CLUSTERING STABILITY UNDER EXTREME SETTINGS
Beyond the standard configurations reported in the main paper, we further evaluate the robustness of our clustering procedure under several extreme settings. Our goal is verify that the "one-time choosing" assumption continues to hold even when the generation conditions deviate significantly from common practice.
Extreme resolutions. We first examine clustering stability under highly atypical resolutions. In particular, we test very high resolutions (e.g., 2048 x 2048) as well as extremely narrow aspect ratios (e.g., 128 x 1024). For each setting, extract feature trajectories and compute clustering assignments using the same indicators and hyperparamcters as in the default setting. Across all tested resolutions, the resulting cluster structures remain highly consistent: the same dimensions are assigned to the same clusters with only negligible variation, and the ARI between extreme and standard resolutions stays well above 0.8 which consider stable, shown in Fig 14 a).
Resolution 128x 1024
Resolution 2048x2045
Adversarial Prompts
Nonsensical Prompts
Acroes Prompts
20
ARI Acroos Prompts
Across Frumpts
Prompis
Sample Count
Adjusted Rand Index
Adjusted Index
Adjusted Index
Adjusted Index
(a)
(b)
(c)
(d)
Figure 14: (a-b) The left two figures show that clustering structures remain stable under extreme resolution settings. (c-d) The right two figures show that clustering is also consistent for nonsensical prompts and closely matches those from normal, well-formed prompts.
Adversarial and nonsensical prompts. We also probe stability with respect input text by using diverse collection of prompts, including grammatically incorrect descriptions, deliberately ambiguous instructions, and even semantically nonsensical phrases. Despite the irregular semantics, the induced feature trajectories still yield cluster assignments that closely match those from normal, well-formed prompts. Example clustering results shown in Fig 14 b).
23

---

Published as a conference paper at ICLR 2026
Dlocke Colton
Stop 49
Dhaeer Celare
Step
Stip
Stop
Prine cappinar
Prieepah compeemnt
X Curon
Prtecipal comppesst
2t Cartr
Prissigal nongearut
Princppl oompasnnt
Prindppl cospencet
(b)
Figure 15: (a) Although the clustering results at Step 49, Step 48, and Step 47 are inconsistent, the assignment Step 47 already aligns closely with that at Step 35, suggesting that instability limited to the first few timesteps in rare cases. (b) Despite this early-stage instability, the final generated image remains unaffected, the first few steps are fully computed by the model and not involved in feature caching. Most predictors require at least three historical features, and caching typically begins Step 4 or after cluster assignments have stabilized.
Other than the extreme cases discussed above, further investigate how different timesteps influence the clustering behavior. Our analysis shows that clustering can exhibit mild instability only at very early denoising steps. The noticeable deviation occurs rarely in small subset of cases at the very beginning of the denoising process. Specifically, we observe that the cluster assignments during the first 3-3 timesteps be less stable in
cases shown in Fig T5fa). However, this does affect the final generation quality. In actual inference, the first steps are always computed fully by the model without caching, since most of our predictors require at least three historical features to make accurate forecasts. As result, even if cluster assignments are slightly unstable during the initial steps, they have no impact on the quality or correctness of cached predictions. Feature caching actually begins after the first few compute steps, by which time the clustering results have already stabilized, therefore the generation quality is ensured as shown in Fig b)..
24


Reviews:
[{"review_id": "UoTizAyQ7W", "reviewer": "Reviewer_byp7", "summary": "The paper propose a method for feature caching/predicting in Diffusion Transformers (DiT). The main contribution of the paper upon TaylorSeer [1] are the introduction of feature dimension clustering which allows to use different estimation methods (i.e., ODE solvers) for different cluster group. Importantly the proposed method is able to accelerate even distilled models, improving both speedup 5.2 Db in PSNR compared to the runner-up baselines [1].\n\n[1] Liu, Jiacheng, et al. \"From reusing to forecasting: Accelerating diffusion models with taylorseers.\" arXiv preprint arXiv:2503.06923 (2025).", "questions": "1. In appendix A.2.1 does $v_k$ stands for the features?\n\n2. Could the authors please provide explicit example of how does the Runge-Kutta method is applied to predict features?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The method is validated across multiple tasks such as image generation, image editing, and video generation showing very strong results with significant speedups.\n\n2. Importantly, the method is able to accelerate the a state-of-the-art (SOTA) distilled flow model FLUX.1 Schnell [2] by a factor of 2x while maintaining  SOTA performance (measured by CLIPScore and ImageReward) as we as achieve 34.37 Db in PSNR w.r.t. to ground truth generated by the original FLUX.1 Schnell model.\n\n3. The method is training free.\n\n4. Fast inference of diffusion/flow model is of interest for many researcher. Therefore, given the two points mentioned above, this paper has the potential to make a significant impact.\n\n[2] FLUX.1 Schnell (Black Forest Labs, 2024).", "weaknesses": "1. The presentation is somewhat lacking. Adding explicit examples illustrating how different solvers are applied to predict future feature would significantly help the reader understand the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQ5hHIVcrE", "reviewer": "Reviewer_K9wt", "summary": "This paper proposes HyCa, a training-free acceleration framework for Diffusion Transformers. The authors observe that different hidden feature dimensions in DiTs exhibit distinct temporal dynamics—some smooth, others highly oscillatory—making uniform caching suboptimal. HyCa reformulates feature caching as a mixture of ODE solving problem, where each feature cluster automatically selects its best numerical solver. During an offline stage, clusters are formed based on dynamic indicators and matched with optimal solvers; at inference time, each cluster reuses its solver to skip redundant computations under a “one-time choosing, all-time solving” mechanism. Experiments on image, video, and editing tasks demonstrate 5–6× acceleration with negligible quality loss. The method is simple, generalizable, and practically useful.", "questions": "1. Under what conditions does the clustering become unstable? How do prompt, resolution, or timestep variations affect stability? Could you show a few failure cases?\n\n2. Please provide details on the offline clustering and solver-selection cost—runtime, data scale, and scalability for larger models.\n\n3. If a LoRA or fine-tuning is applied on top of the same base model, can the previous configuration still be reused, or must the offline process be repeated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel idea of modeling feature caching as a mixture of ODE solving process, offering a dynamic and interpretable perspective on diffusion sampling. The approach is simple, training-free, and practical.\n\n2. The experiments are comprehensive, comparing against strong baselines while maintaining image quality under significant acceleration. Ablation studies and visualizations clearly support the proposed design, and the framework generalizes across different models.", "weaknesses": "1. **Limited validation of robustness and generalization.** The assumption that feature clusters remain stable across prompts, resolutions, and timesteps seems optimistic. The presented evidence is limited and lacks failure or stress cases.\n\n2. **Missing analysis of error accumulation.** The method minimizes one-step prediction error, but inference involves multi-step extrapolation, potentially causing cumulative errors. No analysis or long-horizon stability evaluation is provided.\n\n3. **Unclear offline cost.** The paper does not report the computational cost or scalability of the offline clustering and solver-selection process, which is critical for reproducibility and practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Goq2BTlkVH", "reviewer": "Reviewer_731c", "summary": "This paper presents Hybrid Caching (HyCa), a general acceleration framework for Diffusion Transformers (DiTs) that interprets feature caching as a hybrid ODE-solving process.\n\nInstead of applying a single extrapolation rule across all timesteps or layers, HyCa clusters latent features by temporal dynamics and assigns each cluster a numerical solver (e.g., RK, AB, BDF, AM, or Taylor).\n\nThis design enables feature-adaptive extrapolation with improved stability and reduced computation.\n\nExperiments on three DiT applications—text-to-image (Qwen-Image, FLUX), text-to-video (Hunyuan-Video), and image editing (Qwen-Image-Edit)—show 1.6–2.3× acceleration with minimal quality degradation, suggesting broad applicability and practical efficiency.", "questions": "1. How much time does the offline clustering step require\n2. Can you include a single-solver ablation to isolate the hybrid solver’s benefit", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Breaking the “Unified Caching Strategy” Assumption — Aligning with the Dynamic Nature of DiT Features**  \n   Existing feature caching methods (e.g., FORA, TaylorSeer) typically assume that all feature dimensions follow a *single dynamic system*, adopting a unified caching/prediction strategy that overlooks the *heterogeneous dynamic behaviors* inherent in high-dimensional DiT features:\n   - **New dynamic modeling perspective:** It models feature evolution as a *multi-dimensional ODE hybrid system*, revealing through clustering that DiT features exhibit two typical dynamics — oscillatory trajectories (Cluster 1, requiring stable solvers) and smooth continuous trajectories (Cluster 2, predictable with efficient solvers).  \n   - **Clustering stability verification:** Using the Adjusted Rand Index (ARI), the authors demonstrate consistent clustering stability (ARI > 0.8) across prompts, time steps, and resolutions, supporting the claim that *dimension-wise dynamics are input-invariant* and thus enabling *“once-off offline clustering, lifelong online reuse.”*  \n   This design fundamentally overcomes the “one-size-fits-all” issue that causes *sharp quality degradation at high acceleration*, which explains its strong performance gains.\n\n2. **Near-Lossless Acceleration with Superior Speed-Quality Trade-off**  \n   Across three major tasks — text-to-image generation, text-to-video generation, and image editing — HyCa achieves an optimal balance between speed and quality **without retraining**:  \n\n3. **Seamless Integration with Distilled Models — “Stackable Acceleration”**  \n   Traditional caching methods struggle to handle distilled models (where sampling steps shrink from 50 to 4–8, making features more discrete and harder to predict). HyCa addresses this with a *hybrid solver pool* (including explicit/implicit methods — RK, AM, TF, etc.) that adapts to varying dynamics.\n   This property enables deployment in *low-latency scenarios* (e.g., mobile, real-time generation), broadening HyCa’s application scope.\n\n4. **Unifying Feature Caching through the ODE Solver Perspective — High Interpretability**  \n   HyCa formalizes *feature caching* as an *ODE numerical solving* problem.  \n   By deriving $\\frac{d}{dt}F(x_t) = g_\\theta(F(x_t), t)$ (where $g_\\theta$ is an implicit vector field), it shows that caching essentially *numerically integrates past features to predict future ones*.  \n   Building on this, HyCa’s solver pool (explicit RK/AB and implicit AM/BDF) selectively matches dynamics: smooth trajectories use efficient TF (Taylor formula) prediction, while oscillatory ones use stable AM (Adams-Moulton) solving. This provides both clear interpretability and a foundation for future extensions (e.g., adding new solvers for specific dynamics).", "weaknesses": "**Incomplete Computational Complexity Analysis**  \n   The paper only reports *end-to-end latency*, omitting details on offline clustering and solver overheads:  \n   - **Offline Phase:** For high-dimensional models, k-means complexity is ma. Such large-scale clustering may require substantial preprocessing time, yet this is undocumented.  \n   - **Online Phase:** Different solvers have varying computational loads (e.g., implicit AM requires nonlinear equation solving, explicit TF only polynomial evaluation). However, the paper does not quantify each solver’s latency impact, leaving it unclear whether further speed gains are possible through solver re-selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Rg0OnYn9Z", "reviewer": "Reviewer_VNQd", "summary": "Diffusion Transformers (DiTs) offer state-of-the-art fidelity in image and video synthesis, but their iterative sampling process remains a major bottleneck due to the high cost of transformer forward passes at each timestep. To mitigate this, feature caching has emerged as a training-free acceleration technique that reuses or forecasts hidden representations. However, existing methods often apply a uniform caching strategy across all feature dimensions, ignoring their heterogeneous dynamic behaviors. Therefore, this paper adopts a new perspective by modeling hidden feature evolution as a mixture of ODEs across dimensions, and introduces \\textbf{HyCa}, a Hybrid ODE solver inspired caching framework that applies dimension-wise caching strategies. HyCa achieves near-lossless acceleration across diverse domains and models, including 5.56× speedup on FLUX and HunyuanVideo, 6.24× speedup on Qwen-Image and Qwen-Image-Edit without retraining.", "questions": "1. What's $g_\\theta$ mentioned in eq.3 ?\n2. I am still confused about ODE modeling of the feature cache. Since there are discrete feature cache sets $[t_i, F(x_i, t_I)]_{i=0}^j$, estimating the next feature $F(x_l, t_I)$ constitutes a classic extrapolation problem, as stated in TaylorSeer [1].\n\n3. Alternatively, if we treat the feature difference $F(x_i, t_i) - F(x_{i-1}, t_{i-1})$ as the subject of the ODE, a closed-form solution exists, which can be simplified to a classic extrapolation problem.\n\n4. Line 206-207: 'sample the trajectory on a discrete timestep grid, enabling numerical integration using only cached feature values', Why? \n\n Finally, I would be willing to raise the score if my concerns (even partially) can be addressed.\n\n[1] Liu, Jiacheng, et al. \"From reusing to forecasting: Accelerating diffusion models with taylorseers.\" arXiv preprint arXiv:2503.06923 (2025).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Performance is solid. The experiments are detailed, and the visualizations are rich.\n2. Achieves high-speedup gains (specifically 5.56× and 6.24×) across diverse domains and multiple models, including FLUX, HunyuanVideo, Qwen-Image, and Qwen-Image-Edit.  \n3. Maintains synthesis fidelity close to that of the original model while delivering acceleration.", "weaknesses": "1. The most important part of the ODE modeling should be elaborated in detail.  (see Question 4)\n2. Missing related works with bespoke solver[1, 2, 3], which also searches the optimal solver parameters of a pretrained diffusion model.\n\n[1] Xue, Shuchen, et al. \"Accelerating diffusion sampling with optimized time steps.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wang, Shuai, et al. \"Differentiable Solver Search for Fast Diffusion Sampling.\"  International Conference on Machine Learning (ICML) 2025. \n\n[3] Shaul, Neta, et al. \"Bespoke solvers for generative flow models.\" arXiv preprint arXiv:2310.19075 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)