HiCache: A Plug-in Scaled-Hermite Upgrade for Taylor-Style Cache-then-Forecast Diffusion Acceleration
Authors: Liang Feng
Shikang Zheng
Jiacheng Liu
Yuqi Lin
Qinming Zhou
Peiliang Cai
Xinyu Wang
Junjie Chen
Chang Zou
Yue Ma
Linfeng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Acceleration
Efficiency ML
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: HiCache: a training-free, Hermite-based cache-then-forecast with dual-scaling that drop-in replaces Taylor to accelerate diffusion transformers while improving quality.

Abstract: Diffusion models have achieved remarkable success in content generation but suffer from prohibitive computational costs due to iterative sampling. While recent feature caching methods tend to accelerate inference through temporal extrapolation, these methods still suffer from severe quality loss due to the failure in modeling the complex dynamics of feature evolution. To solve this problem, this paper presents HiCache (Hermite Polynomial-based Feature Cache), a training-free acceleration framework that fundamentally improves feature prediction by aligning mathematical tools with empirical properties. Our key insight is that feature derivative approximations in Diffusion Transformers exhibit multivariate Gaussian characteristics, motivating the use of Hermite polynomials, the potentially theoretically optimal basis for Gaussian-correlated processes. Besides, we introduce a dual-scaling mechanism that ensures numerical stability while preserving predictive accuracy, which is also effective when applied standalone to TaylorSeer. Extensive experiments demonstrate HiCache's superiority: achieving \$5.55\times\$ speedup on FLUX.1-dev while exceeding baseline quality, maintaining strong performance across text-to-image, video generation, and super-resolution tasks. Moreover, HiCache can be naturally added to the previous caching methods to enhance their performance, e.g., improving ClusCa from \$0.9480\$ to \$0.9840\$ in terms of image rewards. Our code is included in the supplementary material, and will be released on GitHub.

Accepted at ICLR 2026
HICACHE: A PLUG-IN SCALED-HERMITE UPGRADE FOR TAYLOR-STYLE SION ACCELERATION
DIFFU-
Liang Feng 1.2.t, Shikang Zheng 1.s.t Jiacheng Liu', Yuqi Lin', Qinming Zhou 14 Peiliang Cai', Xinyu Wang'. Junjic Chen', Chang Zou', Yuc Ma", Linfeng Zhang'
Shanghai Jian Tang University
Fudan University
South China University Technology
4 Tsinghua Universit
G Hong Kong University of Science Technology
ABSTRACT
Diffusion models have achieved remarkable success in content generation but suffer from prohibitive computational costs due to iterative sampling. While recent fea- ture caching methods tend to accelerate inference through temporal extrapolation, these methods still suffer from severe quality loss due the failure in modeling the complex dynamics of feature evolution. To solve this problem, this paper presents HiCache (Hermite Polynomial-basee Feature Cache), a training- free acceleration framework that fundamentally improves feature prediction by aligning mathemat- ical tools with empirical properties. Our key insight is that feature derivative approximations in Diffusion Transformers exhibit multivariate Gaussian charac- teristics. motivating the use of Hermite polynomials- the potentially theoretically optimal basis for Gaussian-correlaan scaling mechanism that ensures numerical stability while preserving predictive accuracy, which is also effective when applied standalone to TaylorSeer. Exten- sive experiments demonstrate HiCache's s superiority: achieving 5.55x speedup on FLUX. while exceeding baseline quality, maintaining strong performance across text-to-image, video generation, and super-resolution tasks. Moreover, Hi- Cache be naturally added to the previous caching methods to enhance their performance, egg., improving ClusCa from 0.9480 to 0.9840 in terms of image rewards. Code: httpps: //gi thub. com feng 9 Cache
processes. Besides, we introduce dual-
Hello World
openai
Figure 1: Teaser of HiCache using the FLUX.I-dev model, with a 6.24X FLOPs speedup
INTRODUCTION
Diffusion Models (DMs) (Ho et al. 2020 Nichol & Dhariwal 2021 Dhariwal & Nichol 2021 I)
especially when implemented with the scalable Transformer architecture (DiT) (Peebles & Xie
2023) have set a new standard for
content generation (Chen et al. 2023 Rombach
al. 2022 Karras et al. 2022) Saharia et 2022
Balaji al. 2022). This success, however,
Equal contribution. Emails: wi Anndbright 80gma commm
2hang in feng@s jou. edu.
This project is sponsored by CCF-Baidu Open Fund.

---

is tempered by the substantial computational overhead inherent to the diffusion process. The iterative sampling, requiring hundreds of sequential forward passes through large models, results in high latency that critically hinders practical deployment, especially on diffusion transformers. Consequently, developing acceleration methods for diffusion models has become a hot research topic.
Among various acceleration strate- gies, feature caching has emerged as a particularly effective and training- free approach. Motivated by the ob- servation that diffusion models ex- hibit similar features in the adjacent timesteps, the original feature caching seeks to directly reuse features in the adjacent timesteps (e.g., Deep-
Orrbal 1re8
35-
Podeetic Marm
Tiam Stop
Tieos Stop
Cache (Ma et al., 2023), FORA (Sel- varaju et al. 2024). ToCa (Zou et al.
Figure 2: Trajectory prediction comparison of Taylor and HiCache methods FLUX. "Full trajectory" indicates the feature trajectory from the original Flux (Esser et al., 2024) in the 14th and 28th layer. The y-axis denotes the most principal component of the features in diffusion models.
2025), ClusCa a (Zheng et | (2025))
which leads to significant genera- tion quality loss when the accelera- tion ratios increase. Recently, the paradigm has advanced to a "cache- then-forecast" mechanism as Tay- lorSeer (Liu et al., 2025a), where Taylor series expansion used to extrapolate features in the future timesteps, significantly reducing the error from feature caching. TaylorSeer significantly reduces the error introduced by feature reusing, but still faces fundamental theoretical limitations: The standard polynomial busis of a Taylor series is suboptimal for modeling the complex, non-monotonic trajectories of feature evolution in DMs.
Figure 2 visualize the features of the original diffusion models (i.e., ground-truth) and the predictive features from TaylorSeer. It is observed that faylor-based extrapolation exhibits significant deviations from ground truth trajectories, particularly turning points where its monotonic nature fails to capture the underlying dynamics. These deviations indicate a modeling mismatch: the monomial Taylor basis poorly fits the non-monotonic, ourning-point behavior of feature trajectories, causing approximation error to grow rapidly with prediction horizon and order (Proposition T]; this, in turn, caps the attainable acceleration before quality degradation becomes evident.
To overcome this limitation, in this paper, we propose HiCache (Hermite Polynomial-based Feature Cache), a framework that improves the feature forecasting by aligning the predictive basis with the intrinsic statistical properties of the feature dynamics. Our approach is based on a key empir- ical observation: the derivative approximations of feature vectors in DiTs consistently exhibit multivariate Gaussian distribution (details in Section 3.3.1). Motivated by this finding, HiCache replaces the standard polynomial basis with scaled Hermite polynomials. According to approxi- mation theory (Szegö 1975). Hermite polynomials are the potentially optimal orthogonal basis for representing Gaaussia-coorelltee processes, making them theoretically better suited for this task. Besides, to address the numerical challenges of Hermite polynomials at large extrapolation steps, we further introduce a dual-scaling mechanism that simultancously constrains predictions within the stable oscillatory regime and suppresses exponential coefficient growth in high-order terms through single hyperparameter. This principled design enables both accurate and numerically stable feature prediction. As shown in Figure the feature trajectory predicted by HiCache closely matches the original FLUX, demonstrating its effectiveness. Importantly, HiCache naturally serves as a drop-in replacement for Taylor-based predictors in existing foreeett frameworks, requiring only the substitution of the polynomial basis while preserving the same predictor form and computational structure with negligible overhead.
Extensive experiments demonstrate the effectiveness of HiCache in class-to-image generation with DIT, text-to-image generation with FLUX.IIdes image-to-image generation with Inf-DiT for image super-resolution, and text-to-video generation with Hunyuan Video (Kong et al., 2024), showing improvements over previous methods. For instance, on FLUX. 1-dev, HiCache achieves 5.55x acceleration with 4.25% improvement on ImageReward over the second-best method. This superiority extends to its use as a component upgrade: replacing ClusCa's 's Taylor predictor with our Hermite basis yields IewaeReward improvements from 0.9480 to 0.9840 at N 7
2

---

In summary, main contributions follows.
Hermite Polynomial-bare Feature Caching. We propose a training-Fin free feature extrapola- tion/cache predictor grounded in the empirically Gaussian differeeefe statisties of diffusion features; instantiated via KL-optimal scaled Hermite polynomials under Gaussian correlation, better captures non-monotonic trajectories while preserving the Taylor-style form and compute.
Dual-scaling mechanism. We introduce single-hyperprraeet dual scaling (input contraction + coefficient suppression) that stabilizes predictions and suppresses high-order growth without accuracy loss; it also works standalone for TaylorSeer.
Blug-and-play generality. HiCache is model-agnostic and training-fires by only swapping the polynomial basis, it drops into any Taylor-style teehe-tthe-forecas pipeline (e-g.. TaylorSeer, ClusCa) with unchanged predictor form and negligible compute overhead only few per-step scalar mnsis-function evaluations and no extra matrix multiplications.
Broad empirical gains. Across text-to-image, text-to-video, class-conditional, and super- resolution, we achieve equal-or- -igher speedups with better quality: e.g., on FLUX.11dev, 5.55x speedup with IewgeReward improving from 0.9872 to 0.9979; zero-extra FLOPs basis swap lifts ClusCa from 0.9480 to 0.9840.
RELATED WORK
deterministic sampling with large steps, while DPM-Solver (Lu al. 2022) formalized introduced
Sampling Step Reduction aims to reduce denoising iterations. DDIM (Song et al. 2021
ODE
with high-order numerical methods. Advances include knowledge distillation (Meng et al. 2023)
and Consistency Models (Song al.) 2023) for few-step generation. Other approaches include
cascaded diffusion models (Ho et al. 2022a) and improved sampling techniques (Karras et al. 2022).
Text-to-image diffusion models show progress (Saharia al. 2022 Balaji al.) [022] Ramesh et
2022 Podell et al., 20233 Zhang et al. 2023), while video generation (Singer et al. 2022 Ho et al. 2022b presents additional challenges.
Denoising Network Compression focuses on reducing per-step computational cost. Model compression methods include neural network pruning (Fang et al., (023) and quanti- zation (Li et al., 2023) have applied the traditional model compression methods diffusion models.
Besides, token merging (Bolya & Hoffman| 2023) has been introduced reduce the number of
tokens computed in each timestep. However, these methods tend to lead clear loss in generation quality, and sometimes rely on retraining the diffusion model.
Feature Caching has been recently proposed to offer training- craniin-ffree alternative. Motivated by the great similarity of features in the adjacent timesteps, these methods aim to skip the computation in some timesteps based the historical features, which
be roughly divided into two paradigms: (1) Cache-then-RRute directly reuses features from adjacent timesteps (DeepCache (Ma et al. (2023). FORA (Selvaraju al. 2024). ToCa (Zou al. (2025). L2C (Ma al. (024)), but suffers quality predicts future features using historical data. TaylorSeer (Liu al., 2025a) pioneered this approach with Taylor series expansion, significantly improving quality at high acceleration ratios. However, its standard polynomial basis remains
degradation at large intervals: s (2) Cache-then-Foree
suboptimal for complex feature dynamics, motivating our Hermite polynomial based approach.
3 METHOD
3.1 PRELIMINARIES
Diffusion Transformers (DiT) Peebles & Xie 2023) apply transformer blocks iteratively dur- ing denoising, with each block updating features via Fout MLP(AdaLN((.t)) The timestep-conditions normalization and nonlinearities create complex, non-monotonic feature trajectories that challenge existing acceleration methods.
Fmn + MHSA(AdaLN(Fmm,
3

---

3.2 TAYLOR EXPANSION BAASED FEATURE PREDICTION
To accelerate sampling, existing methods like TaylorSeer extrapolate future features using Taylor series. We first formalize the finite difference framework:
Definition 1 (Finite Difference Operator). The i-th order backward difference operator (0) feature trajectory {Fi} } is defined recursively:
on
F Novenal
F
with
=
(I)
Ninteval
Taylor expansion- ppaase prediction then takes the form:
m A'F
F Fe + Σ
(2)
Proposition 1 (Limitation of Monomial Basis), For feature trajectories with bounded variation but containing turning points, the Taylor prediction error grows as:
km+1 (m +11 1)! (E(t-AKA)
S -Fi-kl=
sup
| ( (6)1
(3)
where the supremum can be arbitrarily large at trajectory inflection points.
This limitation motivates search for basis that naturally captures non-monotonic behaviors.
3.3 HICACHE: HERMITE-BASED FEATURE CACHING
Our key insight is that feature differences in diffusion transformers exhibit Gaussian properties (empirically validated in Section 5 which theoretically motivates the use of Hermite polynomials as optimal basis functions.
Proposition 2 (Gaussianity of Feature Differences). Feature differences AF;
F( t)
F A) in diffusion transformers exhibit approximate Gaussian behavior through: (i) local linearization yielding conditional Gaussianity AF, z N(u,, Ex) for small A, and (ii) ag- gregation effects across network components invoking CLT with Berry-Esseen bound Omj.). See 4ppendix44fom the complete statement and proof.
Corollary (Optimal Basis Selection). When the temporal correlation can be approximated by Gaussian kernel K ((x exp(-(s--))' /2)",, the Karhunen-Loove expansion yields (scaled) Her- mile functions eigenfunctions. Thus, under such Gaussian kernel correlation, Hermite polynomials form the optimal orthogonal basis in the weighted L (r) sense. For general Gaussian processes, they remain approximately optimal (see Appendix 4 Theorem 2 for precise conditions).
3.3. HERMITE POLYNOMIALS AS BASIS FUNCTIONS
den e Abramowitz Stegun 1964) satisfying the recurrence Hn+1 (x) = (xHH((x)) 2nH (2)). (4). Unlike monotom cally growing Taylor basis functions, Hermite polynomials exhibit oscillatory behavior (as shown in Figure 3b). This oscillatory nature provides implicit regularization (van der Westhuizen & Lasenby
Given the Gaussian nature, we adopt Hermite polynomials H2 (x) = (-1)=c=² d"
2018). enabling accurate capture of non-monotonic dynamics in neural network optimization.
Unlike monotonic Taylor basis, Hermite polynomials' oscillatory nature captures turning points. However, they face numerical instability for large arguments, motivating our scaled version:
Definition (Scaled Hermite Basis). Scaled Hermite polynomials with contraction factor E E0,1 are defined as:
Hr(x)= = o
This scaling provides dual stabilization: (i) input scaling oæ constrains predictions within the stable oscillatory regime, and (ii) coefficient scaling o" suppresses exponential growth in high-order terms.
Remark. The scul-scaling mechanism can also applied as a standalone enhancement to TaylorSeer, providing measurable gains without changing its basis.
4

---

Srued Trats
KGale
Toyla
-CC
Copacte
af (-aa)
Order
Order-
1-30
2-36-
-A A'Fizua)
((())))
1-30
HSD Ach--ii
11,01 100"-1p0--
17 Hol--1a
Xa.araa
X
Protetie Steps
Prolctee Skapss
(a) Overview of HiCache compared with TaylorSeer.
(b) Comparison of basis functions.
Figure Method overview and basis function comparison. (a) TaylorSeer (orange) predicts features using power basis, whereas HiCache (green) keeps the functional form but swaps the basis for scaled Hermite polynomials. (b) Hermite's s oscillatory behavior (e.g.. H2(z) negative offset) captures non-monotonic evolution better than monotonic Taylor growth.
Algorithm HiCache: Hermite-based Feature Caching
Require: Interval Nitevaal order Nonters contraction o
1: Initialize last +- T (Track last activation step)
2: for timestep = 7 to
3: if't mod Nintevval 0 then
4: Compute Fr = 4 (xx) {Full model forward]
5:
Update cache: (A*F, Y
Nante via Def. with Athis N interval
6: binst UUpdte last activation step
7:
else
8:
A + h (Distance from last activation)
9:
Predict Fr using Eq. (5 with
10:
end if
11:
end for
3.3.2 HICACHE ALGORITIIM
Proposition 3 (HiCache Feature Prediction). Given cached derivative approximations (AFF) under Gaussian feature dynamics, a theoretically-motivvei predictor is given by:
Nave
Nander A
Ft-k
Hi(-k)
(5)
where K {L...., Neterval 1} is the prediction horizon and Hi is the scaled Hermite polynomial
from Definition 2 When coefficients are obtained via weighted least squares, this achieves projection- optimality in L (T).
The direct use of F provides an efficient approximation.
HiCache reduces cost by factor (Nimerval 1)/N interval with O(Noder) overhead. The total error Exotal Exuncation + Empproximation Enumerical satisfies:
Emaa 0
+ O(Aini VN) + O (Emmchine)
(6)
VIN 1
Under 222 < 1, the Hermite truncation (o (c22) 1)! can be smaller than Tay-
lor's As N+ N+1/(N + 1)! due to a N+1 suppression, yielding conditional superiority (see Appendix 4h.
5

---

Table 3: Quantitative comparison on the text-to-image generation task with the FLUX.1-der model.
We also report Hi-ClusCa (ClusCa + HiCache) to highlight plug-and-play generality.
Method
Lateney(s)
Speed
FLOPPx() L Speed1
Imagee Reward
PSNR 1 SSIM
LPIPS L
FLUX
[dev]-
s0p
17.12
1,00x
3719.50
1.00x
0:9872
10090
0.0000
FLUX.1 [der] seps
8.77
1.95
1859.
2,00x
0.9691
29.555
0.7327
0.3115
FLUXX [dex]- sepss
1487
2,62x
29.122
0.6983
0.3599
FLUX
[ex] s17
6.10
2881
3.13x
0:9147
28,882
0.6767
0.3912
Chipounn
21 (34% sieps)
12.72
2,02
1505.87
2
0.9936
FORA
4.22
4.08%
670.44
5.55x
0.7418
25.315
0.5870
0.5409
ToCa - 10 90%)
7.93
714.66
5.20x
0.8384
28.761
0.6068
0.4887
TeaCache
5.00%
0.6360
0.4773
- 4,
1.08
1,39x
816.65
4.56%
0.8245
ClusCa =T)
4.87
3.52x
674.21
5.52
0,9480
28.630
0.6210
0.4560
TNglatSeerrN-7... -)
4.84
3.54x
670.44
5.55x
0.9572
28.634
0.6237
D.4520
Hi-ClusCa LV -
674.21
0.6561
0.40:10
HiCache (V -V7.00
4.84
3,54x
670.44
5,55X
0,9979
25.932
0.6572
0.3982
FORA (V=
3,90
4.42x
$96.07
6.24%
0.5457
28.233
0.5613
0,5860
ToCa 12 90%)
7.34
2.34x
644.70
5.77%
0.7155
28.575
0.5677
0.5500
TesCache (f] - 1.2)
4,45
3,85x
669.27
5.56x
0.7394
28.151
0,4744
D.6765
3,56
CluCCa =9)
4.53
3.78x
599.93
6.20x
0.5850
0,5120
TaclorSeer =9, -9,0 2)
4.50
3,80x
596.07
6.24x
0.8562
28.359
0.5882
0,5088
Hi-ClusCa (V - 11000--000000
4,53
3.78%
599.93
6.20x
0.8960
28.630
0.6380
0.4460
HiCache CV 0.0 2. 0.5)
4,50
3,80
596.07
6:24X
0,9113
28.647
0.6443
0.4374
FLUX [schnell]" stps
1,10
25 00
28,053
0.4721
0.6743
[schnell] stops
2.20
7.79x
12,50x
0.6708
HiCache+ FLUX.1 [schmell] (0 1. 0.3"
1.10
15.58K
198.81
21.99%
0,9975
28.057
0.4507
0.6757
denotes the baseline (FLUX.1 [dev] -50 50 steps). Best results are in bold and second best are underlined. Hi Hi-CCusCa: ClusCa with its Taylor predictor replaced by HiCache's Hermite predictor.
FLUX.1 I [schnell] is distilled variant of FLUX.I.
HiCache+ FLUX.1 [schnell]: Two full computation steps followed by two accelleacce steps.
Table
Comparison on text-to-
Table 2: Ablation study with FLUX.I-dev. o = 1.0
video generation with Hunyuan Video on
means no contraction factor is applied. Hi-Taylor indicates TaylorSeer upgraded with HiCache.
VBench.
Ouginal (50-
145000
1,00
23775
1,00x
Method
Speed 1
Image Reward 1
PSNK 1
SSIM 1
LPIPS :
IRCaahe (N - 7, m=0.4)
5.55k
0.683
29.058
0.6812
0.3914
32.40
4.76%
6050
4.85%
15356
HiCazite (N 7. ()
5.55
0.9979
28937
0.6572
0.3982
FORA
HiCache (V = -7 m=077
5.55x
0.0623
25.651
0.6268
0.4479
38:52
4.5
1,26x
7086
0833
482x
4.35x
13.66
15722
HBCache (NV 7. 1.00
Ii Taylor (N 7,0=0.5)
5.55%
535K
0.7586
0.9624
29.080
28.087
0.6571
0.3682
0.7208
0.3998
HiCaley 6,
31:71
4.38x
3059
3355
19300
Tiyle (N
5.58x
0.0572
25.634
0.6237
0.4520
TakeSaan 7,0m
25.2
1/8
478
15.40
19.01
FORA 7)
ToCa 18. N 95%)
5.77
5.55k
0.5418
0.7155
23.585
28.315
0.5871
0.5498
0.5409
Tadestann 7,0
2830
1/0x
-706
15.35
C& 12,N 9655)
0,3322
BiCedbeN -10--
25.6
4705
1966
4
EXPERIMENTS
4.1 1 EXPERIMENT SETTINGS
We conduct comprehensive experiments across four representative generative tasks using state-of-the- art diffusion models: text-to-image generation with FLUX. 1-dev (Esser al. (024)), text-to-video generation with Hunyuan Video (Kong et al. 2024), class-conditional image generation with DiT- XL/2 (Peebles & Xie 2023), and image super-resolution with a modified Inf-DiT (Yang et al. 2024). Each task employs standard evaluation protocols and widely-adopted benchmarks to ensure fair comparison with existing methods. More detailed setup is provided in Appendix 4
4.2 EXPERIMENT RESULTS
Text-to- Generation. Table 3 shows HiCache maintains superior quality at high acceleration ratios. At 5.55x speedup (N = = 2), its ImageReward (0.9979) exceeds both competitors and the unaccelerated baseline (0.9872). When integrated with ClusCa by (Zheng al.) (025), HiCache substantially improves ImageReward: from 0.9480 to 0.9840 (N = 7) and from 0.8440 to 0.8860 (V 9). Figures 4 and 5(a) (a) demonstrate HiCache's 's visual advantages. As shown in Figure 5 b).
HiCache maintains stability across acceleration factors from 1.00x to 9.00x, while other feature caching methods exhibit significant degradation in high acceleration ratios.
Text-to-Videe Generation. Table 3.3.2 shows HiCache outperforms recent methods. At higher orders (A 7, O = =2), HiCache's 's advantage over TaylorSeer becomes more pronounced, validating

---

HiCache Original
openai
x1.00
openal
x5.54
TaylorSeer x5.54
Oponvoo
opeiniin
ToCa x5.20
FORA
x5.55
Figure Qualitative comparison on the text-to-image task across diverse prompts.
Original x1.00
HiCache x5.54
TaylorSeer x5.54
DuCa x5.49
ToCa x5.20
FORA x5.55
TeaCache x5.55
(a) High- frgh-frequency detail preservation.
x1.00
x3.33
x4.17
x5.00
x5.56
x5.25
x9.00
HiCache TaylorSeer
(b) Consistent style and clean backgrounds.
Figure 5: Detail retention and style consistency. (a) Superior detail retention. (b) Greater stability than TaylorSeer under higher acceleration, with consistent style and clean outputs.
our theoretical analysis. Figure 6 shows HiCache's superior temporal consistency compared to methods terms of both temporal consistency and frame generation.
Class-Cor Conditional Image Generation Table that HiCache outperforms recent accelerators on ImageNet with XLT-XL/2 across speedups, with larger relative gains compared with TaylorScer higher acceleration about 3. 3.2% at 5.0x, 4.5% sFID at 6.2x, and 4.9%/6.0% FID/sFID 7.1x-while remaining close to the unaccelerated baseline under aggressive acceleration.
Supee-Resolution. HiCache achieves 5.93x theoretical and 2.43x wall-clock speedup image super-renolution with Inf-DiT (Yang et al. 2024). following NTIRE 2025 protocol (Chen et al. 2025). Figure shows consistent improvements over TaylorSeer in both PSNR and SSMM. At this
7

---

HiCache
Tay Serreer
DDIM-22%
TeaCache
Toca
<5.00
+5.00
Jul 55
xi.55
H8.25
0.56
pereen
handazing
PSNR (UB)
0.52
0.50
poliery making
person
A
2J
PSNR (HLCldde)
PSNK (laylorSeer)
(Teylodeen))
0.76
PLOPs Speedup
Figure 6: Qualitative comparison of text-to-video
Figure 7: Comparison of HiCache and
generation methods under significant acceleration.
TaylorSeer image super-resolution.
Table 4: Quantitative comparison on class-to-image generation on ImageNet with DiT-XL/2.
Method
Latency(s) 1 FLOPs(T)
Speed
FID
sFID
Inception
DDIM-50 steps'
0.995
23.74
1.00x
2.32
4.32
241.25
DDIM-20 steps
0.406
9.49
2.50x
3.81
5.15
221.43
DDIM-8 steps
0,189
3.80
6.25x
23.13
19.23
120.58
A-DIT (N =
0.173
16.14
1.47x
3.75
207.57
1.2 (NFH 30)
0.281
11.55
2.05
2.61
237.83
SmoothCache 0.22)
0.251
8.57
2.77x
4.15
231.71
FORA (N 6)
0,427
5.24
4.98
14.84
171.33
ToCa (N = N.N -
1.016
6.34
3.75
7.10
189.53
TaylorSeer (N
4.76
4.98
3.11
6.35
223.85
HiCache = 6, 4,0 0.75)
0.193
4.76
4.99
6.32
223.68
FORA (N 77)
0.405
3.82
6.22
12.55
18.63
148.44
ToCa (N = 13, 95%)
1.051
4.03
5.90x
21.24
19.93
116.08
TaylorSeer (N = 8, 0 =
0.190
3.82
6.22
4.40
7.34
205.00
HiCache (N = 8, O 4, 0.75)
0.172
3.82
6.21
4.33
7.01
205.00
FORA (N N 95%)
0.405
3,34
7.10x
15.31
21.91
136.21
ToCa (N = 13,
98%)
1.033
3.66
6.48
22.18
20.68
110.91
TaylorSeer (N = 9, O =
0.179
3.34
7.10x
4.58
7.30
201.12
HiCuche (N 9, 0, 0 0.75)
0.161
3.34
7.11x
4.31
6.86
203.49
denotes the baseline (DDIM-50 steps). Best results in each block are in bold and second best are underlined.
acceleration, the restoration quality degrades only mildly: PSNR drops by Rs 2.24% (from 30.87 to 30.18) and SSIM by z 1.44% (from 0.832 0.820) relative the interval= 1 baseline. Detailed latency/FLOPs and restoration metrics for each interval are provided in Table
4.3 ABLATION STUDY
Table dissects our contributions. Without scaling (a 1.0), HiCache underperforms TaylorSeer due to numerical instability. With optimal scaling (a 0.5), performance improves dramatically. The Hi- Tayloo variant (TaylorSeer with HiCache's 's dual-scaling) validates that both our scaling mechanism and Hermite basis contribute independently to HiCache's superiority.
5 DISCUSSION
Empirical Validation of Proposition] 2 We validate the Gaussianity of finite differences posited in Proposition Using energy tests (Székely Rizzo 2005) across to 5th order feature differences in five key FLUX modules (Esser et al. 20044) the tests confirmed Gaussianity with maximum confidence (p-value = 1.0) across all 25 configurations, providing strong empirical support for our theoretical framework and, in turn, Corollary which motivates Hermite polynomials for Gaussian- correlated processes.
The appendix provides theoretical support through two complementary mechanisms: (i) local lin- carization yielding conditional Gaussianity for small steps, with residual controlled by second-orde -odder bounds, and (ii) aggregation effects invoking multivariate CLT with Berry-Esseen convergence rates.
8

---

These mechanisms align with transformer architectures' residual connections and normalization layers that maintain required regularity. Moreover, the robustness analysis shows Hermite bases retain advantages even under approximate Gaussianity, with efficiency degrading most O(e) for e-perturbations from exact Gaussian (Appendix 4 Proposition [). This theoretical foundation, com- bined with empirical energy test validation, justifies modeling finite differences approximately Gaussian and motivates the Hermite basis choice in Corollary [ To further address key concerns beyond the main-page budget, Appendix provides targeted evidence on three fronts: validity of the Gaussian finiee-dfferrnne assumption (Appendix Table [8. stability under aggressive scaling via adaptive per-layer a (5a 55), and transfer/complementatyy across backbones and accelerators (Table and Appendix Table
4
Additional qualitative trajectory evidence is provided in Appendix 4
Prediction Simulation Framework To fairly evaluate HiCache against Taylor expansion, we established a simulation framework using real feature trajectories from the FLUX model with activation interval Ninerval = 6, where derivative approximations are computed using sparsely sampled data from previous activation steps. Our simulation eliminates cumulative error by initializing each prediction task with ground-truth history. This isolates cascading inaccuracies and enables fair quantitative comparison of Hermite and Taylor bases.
Empirical Verification Oscillatory Advantages Appendix FigurePprovides empirical evidence for the theoretical advantages of Hermite polynomials' oscillatory properties (discussed in Section 3.3.2). Using the non-cumulative error framework on representative FLUX features, we observe that while both methods perform similarly order 1, Taylor expansion increasingly eeeeexttrapoltees at higher orders, especially at trajectory turning points- ppee-ielecyyyyee confirming our intuition about monomial limitations. HiCache's 's oscillatory Hermite basis, stabilized by the contraction factor (Definition 2, maintains accurate fitting across all orders (see Appendix 4 for details).
Quantitative Validation of Error Bounds To empirically verify the tighter error bounds discussed in our methodology, we quantitatively compared HiCache against Taylor expansion using the simula- tion framework. We define the relative error ratio R Taylor MSE MSE/Hermite (where R indicates superior HiCache performance) and analyzed 25 configurations (5 FLUX modules x 5 polynomial orders); Figure 8 summarizes the cumulative error ratios.
Timestep Analysis:
Relative Cumulative Error Ratio by Timestep
Order
Order
Order 3
Order
Order
Equal Performance
Relative Cumulative Error
Bayer14-IMC ATTN
4-INGAATTN
Layer14-IMG MLP
Layer28-TOTAL
10a
10a
10a
(Taylor/ HiCache))
102
10a
10a
101
10
10
10a
10a
10
2
Prediction Tamestep
Prodiction Timestep
Prediction Timestep
Figure 8: Cumulative error ratios (Taylor MSE/ / HiCache MSE) across 5 FLUX modules.
The results empirically confirm HiCache's consistent advantage (R > 1.0 across all modules), with the performance gap widening for larger prediction steps and higher polynomial orders. These results confirm HiCache's consistent advantage our parameter regime. In particular, when the contraction condition a 2/2. | < holds and for moderate N, the empirical truncation behavior aligns with conditional analysis in Section
Additional Evidence on Stability and Generality To further address key concerns beyond the main-page budget, we summarize three supplementary findings. First, energy-distance Gaussianity tests on FLUX finite differences show strong support across depth and diffusion phases: 75/75 tests pass over all timesteps, 75/75 in the early high-noise phase, 73/75 in the mid transition phase, and 75/75 in the late low-noise phase (Appendix Table (8). This indicates that the Gaussian-motivvnn motivated Hermite basis is not tied to narrow operating regime.

---

Second, adaptive per-layer scaling confirms the practical value of our stability condition. Under interval = 7 and order 2, fixed a = 1.0 unstable (SSIM 0.362, LPIPS 0.725, ImageReward 0.736), whereas adaptive per-layer at comparable initial scale recovers strong quality (SSIM 0.648, LPIPS 0.415, ImageReward 0.957), close to the robust fixed-a Ri 0.5 regime (Table
Table 5:
Adaptive per-layer a (HiCache-Adaptive) on FLUX.1-de under interval = 7, order = (200 prompts, 1024 1024). Here lixed/ max denotes the global for fixed HiCache and max for HiCache-Adaptive.
Method
0 fised/
Init.
CLIP
PSNRT
SSIM
LPIPS
ImageReward"
Baseline
TaylorSeer
27.41
28.63
0.621
0.456
0.951
Comparison at 22 1.0
HiCache-Fixed
1.0
27.19
28. 10
0.362
0.725
0.736
HiCaive-Adaptive 1.40
1.0
& 1.0
27.80
29.14
0.648
0.415
0.957
Comparison at = 0.5
HiCache-Fixed
0.5
27.62
28.93
0.655
0,404
0.974
HiCache-Adee -ddapiive 0.70
0.7
83 0.5
27.87
29.11
0.646
0,415
0.971
Third, cross-backbone and accelerator results show both transfer and complementarity. On Qwen- Image at higher intervals (e.g.. / = 8), HiCache improves ImageReward from 0.750 to 0.844 and reduces LPIPS from 0.591 0.531 versus TaylorSeer (Table
Table 6: Qwen-Image results (1328x 1328, 50 steps, 200 prompts) comparing TaylorSeer and HiCache at different intervals, with approximate FLOPs speedup.
Interval
Mode
FLOPsx
CLIP
ImageReward1
PSNRT
SSIM1
LPIPS
Taylor
2.78
28.96
1.214
30.44
0.800
0.208
HiCache
2.78
28.99
1.216
30.92
0.795
0.198
Taylor
5.00
28.09
1.012
28.52
0.601
0.481
HiCache
5.00
28.62
1.070
28.72
0.613
0.422
Taylor
5.56
27.73
0.895
28.34
0.564
0.538
HiCache
5,56
27.87
0.944
28.50
0.579
0.478
Taylor
6.25
27.01
0.750
28.23
0.519
0.591
HiCache
6.25
27.72
0.844
28.37
0.539
0.531
On Chipmunk-Fluxx HiCache improves ImageReward from 0.845 to 0.938 over the Taylor variant at similar latency (3.5s vs, 3.3s per image) (Appendix Table Detailed supplementary analyses are provided in Appendix 4 and additional qualitative trajectory evidence is provided in Appendix
CONCLUSION
We introduce HiCache, training- acceleration strategy that overcomes the limits of Taylor-series feature caching in diffusion models. Grounded in the empirical observation that feature derivatives follow multivariate Gaussian statistics, we replace monomials with Hermite polynomi- als-the Karhumen--Lovve-ppiine dual-scaling mechanism that stabilizes coefficients while retaining the oscillatory behavior needed to model non-monotonic trajectories. Across text-to-image, text-to-video, and super-resolution benchmarks, HiCache consistently matches or surpasses prior methods, delivering large speedups without visible quality loss. Because it only swaps the predictive basis, HiCache drops into existing cache-then-foreet pipelines and requires no retraining or architectural changes, offering a principled and practical path to fast diffusion sampling.
basis under Gaussian correlation-and pair them with a simple
10

---

ETHICS STATEMENT
This work does not introduce new data collection or involve human subjects. All datasets used are publicly available under their original licenses; personally identifiable sensitive information processed. Generated content may reflect biases present in the underlying datasets; we report standard evaluation metrics and release code facilitate auditing and reproducibility. By reducing the number of expensive model evaluations, HiCache lowers the compute and energy consumption relative to baselines. We discourage misuse of accelerated generative models for deceptive harmful purposes and will release our implementation with appropriate usage guidelines.
REPRODUCIBILITY CHECKLIST
General Paper Structure
1.1. Includes a conceptual outline and/or pscudocode description of Al methods introduced (yes/par- tial/no/NA) yes
1.2. Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results (yes/no) yes
1.3. Provides well-marked pedagogical references for less familiar readers gain background necessary to replicate the paper (yes/no) yes
Theoretical Contributions
1. Does this paper make thcoretical contributions? (yes/no) yes
If yes, please address the following points:
2.2. All assumptions and restrictions are stated clearly and formally (yes/partial/no) yes
2.3. All novel claims are stated formally (e.g., theorem statements) (yes/partial/no) yes
2.4. Proofs of all novel claims are included (yes/partial/no) yes
2.5. Proof sketches or intuitions are given for complex and/or novel results (yes/partial/no) yes
2.6. Appropriate citations to theoretical tools used are given (yes/partial/no) yes
2.7. All theoretical claims are demonstrated empirically to hold (yes/partiallnoo/AAA) yes
2.8. All experimental code used climinate disprove claims is included (yes/no/NA) yes
Dataset Usage
3.1. Does this paper rely on one or more datasets? (yes/no) yes
If yes, please address the following points:
3.2. A motivation is given for why the experiments conducted the selected datasets (yes/partial/no//NA)) yes
3.3. All novel datasets introduced in this paper are included in a data appendix (yes/par- tial/no/NA) NA
3.4. All novel datasets introduced in this paper will be made publicly available upon publication of the paper with license that allows free usage for research purposes (yes/partial/no//NA)
NA
3.5. All datasets drawn from the existing literature (potentially including authors' own previously published work) are accompanied by appropriate citations (yes/no/NA) yes
11

---

3.6. All datasets drawn from the existing literature (potentially including authors' own previously published work) are publicly available (yes/partial/o/NN) yes
3.7. All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives scientifically satisficing (yes/partial/no/NNA) NA
4. Computational Experiments
4.1. Does this paper include computational experiments? (yes/no) yes
If yes, please address the following points:
4.2. This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting (yes/partial/no/NA) yes
4.3. Any code required for pre-processing data is included in the appendix (yes/partial/no) yes
4.4. All source code required for conducting and analyzing the experiments is included in code appendix (yes/partial/no)) yes
4.5. All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes (yes/partial/no) yes
4.6. All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from (yes/partial/no) yes
4.7. If algorithm depends on randomness, then the method used for setting seeds described in way sufficient to allow replication of results (yes/partialnoo/NA) yes
4.8. This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount memory; operating system; names and versions of relevant software libraries and frameworks (yes/partial/no) yes
4.9. This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics (yes/partial/no) yes
40 This paper states the number of algorithm runs used to compute each reported result (yes/no) yes
4.11. Analysis of experiments goes beyond single-dimensional summaries of performance (e.g.. average; median) to include measures of variation, confidence, other distributional information (yes/no) yes
4.12. The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank) (yes/partial/no) yes
4.13. This paper lists all final experiments (yes/partial/no/NAA)) yes
used for each model/algorithm in the paper's
12

---

REFERENCES
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55. US Government printing office, 1964.
Yogesh Balaji et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In CVPR Workshop on Efficient Deep Learning for Computer Vision, 2023.
Junsong Chen et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to- image synthesis. arXiv preprint arXiv:2310.0044 2023.
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. A-DiT: A training-free free acceleration method tailored for diffiusion transformers. arXiv preprint arXiv:2406.011 125, 2024. BRR.htts:///axttt //ax:v///avvvvv
01125
Zheng Chen, Kai Liu, Jue Gong, Jingkai Wang. Lei Sun, Zongwei Wu, Radu Timofte, Yulun Zhang, et al. Ntire 2025 challenge on image super-resolution (x4): Methods and results. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-879 3098899,...99 2021.
Patrick Esser et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03200 2024.
Gongfan Fang et Structural pruning for diffusion models. Advances in Neural Information Processing Systems, 36, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho et al. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-222 2022a.
Jonathan Ho et al. Video diffusion models. Advances in Neural Information Processing Systems, 35: 8633 8646, 2022b.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information P Processing Systems, 15:26565-265777 2022.
Weijie Kong et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Xiuyu Li eietal al. Q-diffussion: Quantizing diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 77535-17545 2023.
Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorseers, 2025a. URL htttps://aaatt
horg/bb//550 06923
Joseph Liu, Joshua Geddes, Ziyu Guo, Haomiao Jiang, and Mahesh Kumar Nandwana. Smoothcache: A universal inference acceleration technique for diffusion transformers. In Proceedings of the
JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), eLVM
Workshop, 2025b. URL URInttps:///peeceess thecff com cotp: CVVR20055/ wttp:/conteett//////2/EEEW
VI.M/paperr //iuu Liu www.SSmoooaahheeeeeee_t Unirssaaavvaaaaa Inweeennneeeeeeeeee wee._Acceeeeeerreeee
for orw.://oeooooooooooooooooooorrrn
.ddf Open
Access version.
Cheng Lu et al. solver, A fast ode solver for diffusion probabilistic model sampling in around
10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
13

---

Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-car co-che Accelerating
diffusion transformer via layer caching. In Advances Neural Information Processing Systems
(NeurlPS), 2024. URL https::p///..m /ttps///t////proeeeee.s...s.s
httpc//ppprr__flllleeeeeeeeeeeee fwwps//apeessse/eeeeeeea www.://s......r..............eee 2024/file/ f0b151 5be276ee225 66w.2277666682244442222222
Xinyin Ma et al. Deepcache: Accelerating diffusion models for free. arXiv preprint arXiv:2311.008858, 2023.
Chenlin Meng et al. On distillation of guided diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162 8171. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 1195542000 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXivv2300.01195, 2023.
Aditya Ramesh et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia al. Photorealistic text-to-image tomgg diffusion models with deep language understand- ing. Advances in neural information processing systems, 35:36479-36494, 2022.
Ramprasaath R Selvaraju et al. Fora: Fast-forward caching in diffusion transformer acceleration. arXiv preprint arXiv:2407.0144 2024.
Austin Silveria, Soham Govande, and Daniel Y. Fu. Chipmunk: Training-free acceleration of diffusion transformers with dynamic column-sparse deltas, arXiv preprint arXiv:2506.03275. 2025. URL
03275
Uriel Singer et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint
MxXiv:2209.14999 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song al. Consistency models. International Conference on Machine Learning, pp. 32211 32252, 2023.
Gábor Szegö. Orthogonal polynomials, volume 23. American Mathematical Society, 1975.
Gábor J Székely and Maria L Rizzo. A new test for multivariate normality. Journal of Multivariate Analysis, 93(1):58-80, 2005.
Jos van der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint /804 /00.:04444449 2018
Vipshop AI Lab. Dbcache: Dual block caching for diffusion transformers. Setps://githhr
com/ vi passhpp/DCCachee 2025. GitHub repository.
Chenfei Wu, Jiahao Li. Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Shengming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. CoRR, abs/2508.02322 2025. URL
https::/aas
orgg wttg//ss/////oooooooooo/oooa //oo////58 02324
Zhuoyi Yang et al. Inf-dit: Upsampling mn-resooutionn image with memory-efficiee diffusion transformer. arXiv preprint 240.41240 2024.
14

---

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3836-3847, 2023.
Zhixin Zheng, Xinyu Wang, Chang Zou, Shaobo Wang, and Linfeng Zhang. Compute only 16 tokens in one timestep: Accelerating diffusion transformers with cluster-driven feature caching. arXiv preprint arXiv:2509. 10312, 2025.
Chang Zou, Xuyang Liu, Ting Liu, Siteng Huang, and Linfeng Zhang. Accelerating diffusion transformers with token-wise feature caching, 2025. URL httpt:////aarre wttp:/wwwww.g///////.aa// 05317
15

---

AI ASSISTANCE DISCLOSURE
In the preparation of this manuscript, the authors utilized Large Language Models (LLMs) to assist with LaTeX formatting and writing refinement. Specifically, LLMs were employed help with document structure optimization, mathematical notation consistency, and language polishing to improve clarity and readability. All technical content, experimental results, theoretical contributions, and scientific claims remain entirely the work of the human authors. The use of Al tools was limited to formatting assistance and stylistic improvements, without generating any substantive scientific content analysis.
HERMITE POLYNOMIALS: MATHEMATICAL FOUNDATIONS
We adopt the physicist's Hermite polynomials H, (x), whose standard differential form (Rodrigues formula) is:
d"
Hx(()) = (-1)"
dx e
(7)
The Gaussian kernel e in this formula directly embodies the connection with Gaussian measure. From this differential form, we can derive the recurrence relation:
H 1 (x) 2rH, (x) 1 (x)
(8)
where Hx(x) and H, (x)x) 2r. The first few polynomials are:
Ho(x)=
H1 (x) 2x
Hz(z) 4x²2 -
(9)
Ha(r) 8x 12x
Ha(x) 16.- 48x 12
DETAILED ALGORITHM DESCRIPTION
HiCache algorithm operates through two main phases. Central to this the concept of the derivative approximation, A which approximates the k-th order derivative of feature J at given timestep. As defined in Eq. is computed recursively from lower-order approximations at previous activation steps.
Phase 1: Cache Update (at activation steps t; where & mod Ninterval = 0)
This phase uses finite differences compute high-order derivative approximations and updates the cache. The interval between adjacent activation steps is Alhis = Niterval-
Algorithm 2 Cache Update with Derivative Approximations
Require: Current feature F(t,), previous activation step t previous cache {A* () Norder,
interval Ninterval
Ensure: Updated cache (t*)(t))) Norder
1: Athist + 1--1 (Usually equals Ninterval
2: NewCache([]] + F(t,) [Zeroth-order is the feature itself)
3: for k 0 to Norder do
4: [Compute (l: + 1)-th order derivative approximation)
5: A (NewCachelk) A A))(((i)))
6: NewCachefk k + Ak+1f(1)
7: end for
8: return NewCache
Phase 2: Feature Prediction (at non-activation steps / where /. mod Nintevval #
16

---

Algorithm 3 Feature Prediction
Require: Current step L, most recent activation step last, cache F [last))k== k=D
Norder contraction
factor
Ensure: Predicted feature F
I: At Hast t (Distance from last activation, S At Ninteraal
2: $ <- ((1ss))))) {Initialize with zeroth-order term}
3: for k = to Norder do
4:
(Compute scaled Hermite polynomial value}
5: h& + (-A)) {Or equivalently: h2 +- (-1)* (at)))
6:
F+
hy R
A*F((tast) (alaett))Accnnnn {Accumulate)
7: end for
8:
return F
This phase performs efficient prediction based cached derivative approximations and scaled Hermite polynomials. The prediction step At ranges from [1, Vintevalllll - I].
where Hermite polynomials are efficiently computed through the recurrence relation:
Ho(x)
H1(x) 2x
(x) 2xHx(x) - 2kHx-1(x)) for
This algorithm design balances computational efficiency with prediction accuracy. Key parameters include:
Interval parameter Nittevall N Controls activation frequency, with larger values providing higher acceleration ratio but potentially increased prediction eгror
Contraction factor a E (0, 1): Stabilization parameter that ensures numerical stability of Hermite polynomials within the prediction interval
Maximum order Norder N+: Expansion order that balances approximation accuracy and
computational complexity, with practical implementations typically using Nonter S
COMPLETE ERROR ANALYSIS AND THEORETICAL GUARANTEES
This section provides rigorous proofs for the enror bounds stated in Proposition (Error Bounds for HiCache) the main paper. We assume the feature evolution trajectory f [0, T] + sufficiently smooth with bounded derivatives up to order Norder + 1.
The total prediction error decomposes into three components:
Esstal Etruncation + Eppproximation E Equmerical
(10)
TRUNCATION ERROR ANALYSTS
Based on the Karhunen-LLode theorem and validated Gaussian properties, the feature function f(s) can be expanded under the weight function w(x) (axx) as:
(As))
{Sisst + As) ag
(11)
/2
where the coefficients 6Ą = (S, Hz) possess rapid decay properties. Lemma (Truncation Error Bound). The truncation error from using order Norder satisfies:
C2 (a ( As Noair
(aAs)²
Expuncation
exp
(12)
V (Norder + 1)!
2
17

---

Proof. Using the asymptotic bound Hx(x)) C1 C1V2*kle** and the scaling properties of Hy:
1
E
truncation
Σ 1=Nustt11
n As) A (s
(13)
R!
D0
2 (o
(14)
Nude
(or 2//22/ss Notert
(oAs)²
S C2
exp
(15)
(Nonter 1)!
The contraction factor < ensures convergence.
Remark 1 (Comparison with Taylor Expansion). The Taylor remainder E
Taylor trancation
((A)))+11//) 1)!) lacks the V(N + 1j! denominator and o suppression factor, result- ing in looser bounds for non-monotonic trajectories.
FINITE DIFFERENCE APPROXIMATION ERROR
Lemma (Finite Difference Approximation Error). For a sufficiently smooth function f
C*+ [0, T), the backward difference operator A (k) satisfies:
Atpis k k+1
-
|
1) (E)l (E)1++((4 O
(16)
for some & E [t AdDnis O..
Lemma 3 (Gaussian Decay in Feature Dynamics), Under the Gaussian feature dynamics assumption, the expected norm of derivative approximations satisfies:
of
E
1 (1+0(k=)))
(17)
where of is the feature variance and T is the gamma function.
Lemma 4 (Total Approximation Error). The cumulative finite difference error in HiCache satisfies:
Empprsximation S C4 Aliss v Nonter
(18)
Proof. Combining Lemma 2 and Lemma
3
Narke
|Mx(As)]
E Euppoxxiaation
E
|
p(()
(19)
k!
Nanker
As
ot (T(k/2+1)
(20)
k ++1
Nusk
C4 A Whist
(21)
k3/2
S CtAthis Nurder
(22)
The rapid decay from both the Gaussian property and factorial normalization ensures convergence.
NUMERICAL STABILITY ANALYSIS
The recurrence relation Hx+1(x) (x) (xH 2x becomes numerically unstable when
[x| > 1. The contraction factor or limits the effective computation domain to [o < 2, maintaining condition number &. x 0(1).
18

---

Due to the validated Gaussian properties of feature differences, Hermite coefficients possess optimal decay properties:
1 Var(c*) & k!' Cov(c, = 0 for i #
(23)
Remark. The uncorrelatedness Cov(c,, oj) 0 refers to coefficients obtained via L²(r) orthogonal projection; it does not imply statistical independence general. Lemma (Numerical Error Bound). The floating-poin error in HiCache is bounded by:
Norker
Exumerrcca Cremachine
C3Emaccinee
(24)
Lemma 6 (Envelope bound for Hermite and scaled Hermite). For all E N and E [R, there exists a universal constant C such that (H(())) SC (2 + 1)" Vnl. Consequently, for E (0, 1), the scaled polynomials satisfy
[=,,())== - ())))( S C (2 1)"
(25)
exhibiting geometric damping in n controlled by a.
Proof. The bound for H, follows from standard inequalities derived via the generating function
+22tt
Σ n20
Hn
n!
and Cauchy's i estimates, or from known asymptotics (c.g., Plancherel- Rotach). Multiplying by o" and contracting the input yields the scaled bound.
L
TOTAL ERROR BOUND AND PERFORMANCE ADVANTAGES
Theorem (Main Error Theorem Complete Proof of Proposition 4 in the main paper). Combining 4 and Lemma (S)
Lemmal Lemma
the total HiCache prediction error satisfies:
(a Ass) No (oAs)//2 C. C,Atnii V Nonter Cmachinee
Enorat || S C2
(26)
(N
1)!
Approximation
Numerical
Trancation
Corollary 2 (Conditional Advantage over Taylor). Under the condition o 2 N, HiCache can achieve:
and moderate
0.1 Smaller truncation error: The Hermite term (oV22|Ass)N++)) Taylor's As N++ i(N 1). due the exponential suppression factor o NN1
1)I can be smaller than
0.2 Oscillatory basis structure: That naturally captures non-monotonic featur dynamics
0.3 Adaptive basis: Hermite polynomials naturally match Gaussian feussian.eeatur statistics
Without the condition 0/2. AA|
universal faster convergence cannot be guaranteed.
Discussion. The conditional advantage arises from the interplay between the denominators and the contraction factor. While 1/v(N + is actually larger than 1/(N + 1)!, the exponential suppression N+1 (with < in the numerator can overcome this difference when 0/22 | yielding smaller overall error.
DETAILED EXPERIMENTAL SETUP
This section provides comprehensive technical details for all experimental configurations mentioned in Section 4.1.
19

---

MODEL AND TASK SPECIHICATIONS
Text-to-Image Generation (FLUX.1-dev): We employ the FLUX.1 --dvv model for high-resolution text-to-image synthesis. Images are generated at 1024x1022 resolution using 200 high-quality prompts sourced from the DrawBench benchmark. Quality assessment is performed using ImageRe- ward, a robust perceptual metric for text-to-image alignment.
Text-to-Vidd Video Generation (Hunyuan-DiT): The Hunyuan-Dil model is used for temporal content generation. We generate 4, 730 video clips from 946 diverse prompts, evaluating performance using the comprehensive VBench framework which assesses multiple aspects including temporal consistency, object persistence, and text-video alignment.
Class-Conditions. Image Generation (DiT-XL/2): Following standard academic protocols, we conduct experiments on ImageNet using the DiT-XL/2 architecture. We generate 50, 000 samples across all ImageNet classes and report FID-50k, sFID, and Inception Score for comprehensive quality assessment.
Image Resolt-eeoouutin (Inf-DiT): We employ a modified patch- Inf-DiT model for super- resolution tasks. Evaluation is conducted on the DIVSK dataset using standard PSNR and SSIM metries to assess restoration quality and perceptual fidelity.
HARDWARE AND COMPUTATIONAL RESOURCES
All experiments are conducted on enterprise-grade GPU infrastructure:
FLUX. 1-dev experiments: NVIDIA H800 GPU
DiT-XL/2 experiments: NVIDIA A800 GPU
Hunyuan- experiments: NVIDIA H100 GPU
Inf-DiT super-resolution: NVIDIA A100 GPU
EVALUATION PROTOCOLS
Each task employs task-specific evaluation methodologies:
Text-to-Image: ImageReward scores for semantic alignment and image quality
Pext-to-Video: VBench comprehensive evaluation including temporal consistency and motion quality
Class-Conditional: Standard ImageNet metrics (FID-50k, sFID), Inception Score)
Super-Resolution: Pixel-level metrics (PSNR, SSIM) for restoration accuracy
RESOLUTINOO TASK: DETAILED EXPERIMENTAL RESULTS
This section provides comprehensive experimental results for HiCache's evaluation on image super- resolution using the patch-based based Inf-DiT model. The experiments were conducted using standard evaluation metrics including PSNR and SSIM for restoration quality assessment.
Table presents detailed quantitative results across different acceleration intervals, comparing HiCache with TaylorSeer across multiple evaluation metrics including PSNR and SSIM.
The results demonstrate that HiCache maintains superior performance across all metrics, with particularly notable advantages at higher acceleration ratios. At interval=8, HiCache achieves a PSNR of 30.18 (only 0.69 drop from baseline), while TaylorSeer degrades 29.71 (1.17 drop).
20

---

Table 7: Performance Comparison of Different Guiders on Image Super-Resolution Task
Latency (s)
Time Speedup
FLOPs (G)
FLOPs Speedup
Restoration Track
Method
Interval
PSNR
SSIM1
1
256.3
1.00x
12400
1.00
30.87
0.832
2
174.4
1,47x
6639
1.87x
30.92
0,835
HiCache
3
141.6
1.81
4516
2.75x
30.83
0.836
4
127.7
2.01x
3606
3.44x
30.80
0.838
5
118.6
2.16x
3000
4.13x
30.62
0.839
6
113.7
2.25x
2696
4.60x
30.32
0.831
7
108.9
2.35x
2393
5.18x
29.89
0.829
8
105.6
2.43x
2090
5.93x
30.18 (-0699)
0.820 (.0.012)
1
259.5
1.00x
12400
1.00x
30.88
0.832
2
172.7
1.50x
6639
1.87
30.90
0,833
TaylorSeer
3
139.3
1.86x
4515
2.75x
30.85
0.834
4
125.0
2.08x
3606
3.44x
30.80
0.831
5
115.9
2.24x
2999
4.13x
30.42
0.815
6 110.9
2.34x
2696
4.60x
29.95
0.796
7
105.9
2.45x
2392
5.18x
29.61
0.786
8
101.8
2.55x
2089
5.94x
29.71 (-1.1))
0.799 (-0.033)
Note: HiCache uses
0.5. Values in (blue) denote the change relative to the interval=1 baseline.
This represents the first successful application of cache-based accelcration to super-resolution tasks, expanding the applicability of feature caching beyond traditional text-to-image and video generation.
THEORETICAL ANALYSIS OF GAUSSIAN FEATURE DYNAMICS AND HERMITE OPTIMALITY
This section provides theoretical analysis supporting the core claims in Proposition 2 (HiCache Feature Prediction) in the main paper. We examine both the empirical foundation and theoretical justification for using Hermite polynomials the prediction basis.
ON THE GAUSSIANITY OF FEATURE DERIVATIVES
We present two complementary, verifiable mechanisms that yield quantitative Gaussian approxima- tions for finite differences AF,,
Proposition (Local linearization = approximate Gaussian). Let It -000 + a Ex with & ~ (0, I) independent of Ho. Let F Rd x IR IPP be twice differentiable with bounded second derivatives V[ Fllop < Lg 10u F| < Lt 1 V Floop I Let on neighborhood. For small step 8 > define AF, F(z,,t) t) F(
8). Then there exist Ar, be and residual R₁ such that
AF, As (Et El-5) + b + Ri:
E1-~~((0,T)
(27)
with E Rell2 I C (I. Elia Ft-8 I, 82 Leze 11 xr- =0 0()).
Proof. By the two-variable Taylor formula around (xre f),
F(z,, t) F Flz.-s.t t- 8) = v, Fl,, (I, 22-6) 0,F (2,,t) R.
(28)
with E R. 2| controlled by the stated bounds. Using It Zt-5 (ar at )ao + otEs Pt-5Et-6
and grouping constants into b, yields the claimed mauas-Ga leading term.
Proposition 5 (Aggregation + CLT with Berry-Essece bound). Suppose AF, =F==XA AM YL, where Yi, E RRP are zero-mean contributions (e.g., heads//oken//neurmn with uniformly bounded third
21

---

moments and -eunde dependency graph. Let St E; Cov(Y,,)) and define m
E. Ya,l"" Then for any unit
max|jull-1
(u Stuja/2
x
sup ZER
P AF
$ C Cne
(29)
Vuu S.u
and AF; + S.) as 7r + 0.
Proof. By the Cramér-Wold device, analyze u AF, = E, X with ==:=uTy, e. A Berry-Esseen bound for bounded- -engeed dependency graphs controls the Kolmogorov distance by Cn. Uniformity over u yields the multivariate statement.
Diagnostics. We monitor Gaussianity via energy tests, dispersion proxies for th (c.g., effective number of contributors), and local linearization error versus 6.
Central Limit Theorem Perspective
Consider the feature update through transformer block:
F+t1 = MHSA(F,) + MLP(F,,
(30)
The finite difference approximations aggregate many such updates:
A
A Σ(-1)
k-1
F Fi-ise
(31)
Proposition 6 (Asymptotic Gaussianity). Under mild regularity conditions, the individual feature updates are weakly dependent with finite variance, the normalized finite differences converge in distribution Gaussian:
A FA 4 1)(0,))
(32)
Var A
as the feature dimension + 0c.
Justification: This follows from the multivariate central limit theorem for weakly dependent random vectors. The transformer's residual connections and normalization layers help maintain the required regularity conditions.
Random Matrix Theory Perspective matrices due to the softmax operation query-key similarities:
The attention mechanism involves random-lide weight
QKT Vdz
Attention Q, K, V) softmax
(33)
Lemma (Gaussian Universality in Random Matrices), For large transformer models with random initialization and sufficient depth, the distribution of feature differences approuches universal form that, under appropriate scaling, exhibits Gaussian characteristics in the bulk of the spectrum.
This is related to the Wigner semicircle law and its generalizations, which predict Gaussian-like behavior nigh-dimensional random matrix ensembles.
Diffusion Process Approximation In the continuous time limit, the discrete diffusion process can be approximated by stochastic differential equation:
dF, (r(F,, ()dl (F,, l)dW,
(34)
where W, is a Wiener process. Under appropriate conditions the drift μ and diffusion coefficients, the marginal distributions such processes exhibit Gaussian characteristics, especially for the increments.
22

---

HERMITE POLYNOMIALS AS OPTIMAL BASIS
Given the empirically validated Gaussian nature, we now establish why Hermite polynomials form an optimal basis:
Proposition (Projection optimality (7)),, Let be Gaussian measure on the horizon variable. For any target offset function y and any degree-N polynomial p, the orthogonal projection PNy of onto span , satisfies
ly PNullzn S lly pl2.7-
(35)
Thus truncated Hermite expansion minimizes the weusstan-weighted L2 error within degree-N polynomials.
Proof. Hermite polynomials form an orthogonal basis of (7) For any closed subspace Vi span spannHo...... Ho...., H l. the orthogonal projection PN minimizes distance to Vx by the Pythagorean theorem: for any v VN. lyy Pwyl 2. lly - v112,y Choosing
gives the claim.
Theorem 2 (Karhunen Loève Expansion for Gaussian Processes). Let (t) be zero-mean Gaussian process with covariance kernel K (s, t)- If K has the form:
1)2
K(s,t) = exp
(36)
2T
then the eigenfunctions of the covariance operator Hermite functions, and the optimal T.2 expansion is:
X(t) --=<<mm (t)
(37)
n=0
where vn scaled Hermite functions and Sn uncorrelated Gaussian coefficients.
Proof sketch. The covariance operator K. defined by:
(KC)) - (((ss) ((s)t)((s)ds
(38)
has Hermite functions eigenfunctions when K has Gaussian form. This follows from the fact that Hermite functions are eigenfunctions the Fourier transform composed with multiplication by Gaussian, which is essentially what the Gaussian kernel operator represents.
Corollary 3 (Optimality for Feature Prediction). If feature differences F follow a multivariate Gaussian distribution, then the Hermite polynomial basis minimizes the expected squared prediction error:
E
F
Σ (At) k=
(39)
among all orthogonal polynomial bases }-
Proof sketch. By the Karhunen-Loève theorem, the optimal basis for representing stochastic process given by the eigenfunctions of its covariance operator. For Gaussian processes with appropriate covariance structure, these are Hermite functions. The optimality follows from the fact that this expansion minimizes the mean squared error for any finite truncation order N.
ROBUSTNESS UNDER APPROXIMATE GAUSSIANITY
Even if the Gaussianity assumption is only approximately satisfied, Hermite polynomials maintain advantages:
Proposition 8 (Robustness to Non- Non-Gaussianity), Let p(x) be distribution with finite moments that can be expressed as:
p(x) =(x)(1 + ch(x)) where x(x) is the Gaussian density, [π(x)] S C. and is small. Then the relative efficiency of Hermite basis compared to monomials degrades most as O(c).
(40)
This robustness property ensures that even under model misspecification, Hermite polynomials retain their advantages over standard polynomial bases.
23

---

100
Order
100
Order
100
Order 3
PC1 Value
Valuu
50
30
PC1
-50
-50
-30
-100
-100
-100
10
20
30
90
50
10
20
30
40
50
10
20
30
90
50
Time Step
Lime Stop
Time Step
L2yee228 (DMG)
Layez2 (DMG)
Layer20 (DMG)
Order
Order
Order
20
20
20
PC1 Valuue
PC1 Valuue
-20
-20
-20
Time Step
Step
Time Slep
Pull Trajectory
Taylor Prediction
IbCache Prediction
Prediction Start
Figure 9: Visualization example of trajectory prediction for different polynomial orders under FLUX architecture. HiCache demonstrates more stable fitting at trajectory turning points, while Taylor expansion over-extrapolates.
SUMMARY AND PRACTICAL IMPLICATIONS
While we cannot provide complete first-rriccipes proof that neural network features must exhibit Gaussian dynamics, we have shown:
0.1 Theoretical Support:
Multiple theoretical frameworks (CLT, random matrix theory, diffusion approximations) suggest Gaussianity a reasonable expectation in high-dimensional neural networks.
0.2 Conditional Optimality: Given the empirically validated Gaussian nature, the optimality of Hermite polynomials follows rigorously from the Karhunen-Locds theorem.
0.3 Robustness: Even under approximate Gaussianity, Hermite bases maintain advantages monomials due to their oscillatory nature and better numerical conditioning.
These theoretical insights, combined with extensive empirical validation (Section 5.1), provide strong justification for the HiCache design choice of using Hermite polynomials as the prediction basis.
ADDITIONAL QUALITATIVE RESULTS
EMPIRICAL VERIFICATION OSCILLATORY ADVANTAGES
Figure provides empirical evidence for the theoretical advantages of Hermite polynomials' os- cillatory properties discussed the main paper. Using the non-cumulative error framework on representative FLUX features, observe that while both methods perform similarly at order 1, Taylor expansion increasingly over-extrapolates at higher orders, especially at trajectory turning points precisely confirming theoretical analysis about monomial limitations. HiCache's 's cillatory Hermite basis, stabilized by the contraction factor a, maintains accurate fitting across all orders.
24

---

Table 8: Energy-distance Gaussianity tests on (LUX.1-des finite differences (15 layer-module configurations x 5 orders). "Pass" denotes p-value > 0.05.
Setting
Configs MonfigsxOrders
Tests
Pass
Fail
Pass rale
timesteps (all layers/modules)
75
100%
Early high-noise phase
x
100%
Mid transition phase
15 x 5
75
73
2
97.3%
Late low-noise phase
15 5
75
75
0
100%
ADDITIONAL EXPERIMENTAL RESULTS
This section reports additional experiments that complement the main text: (i) empirical Gaussianity tests for finite differences on FLUX.. 1-dev, (ii) adaptive per-layer scheme motivated by our truncation-ems analysis, and (iii) extra backbones and accelerators (Qwen-Image and Chipmunk- Flux).
EMPIRICAL GAUSSIANITY VIA ENERGY DISTANCE TESTS ON FLUX.I-DEE
To support the Gaussian fiiite-difference assumption underlying Hermite basis, conduct a comprehensive energy-distance test on FLUX. 1 --ee features. We collect internal trajectories at six representative layers (4/10/1/20/28/42), covering both early dual-stream and late single-stream blocks, and separately track image/text attention and MLP streams when applicable. For each (layer, module) configuration we compute 1st-5th order finite differences over timesteps and apply high-dimensionn energy distance test (Székely & Rizzo) against multivariate Gaussian fitted by the empirical mean and covariance, using 500 parametric bootstrap simulations and 400 randomly sampled spatial patches.
Using all timesteps, we obtain 15 75 tests in total and observe that all p-values are above 0.05. We further split the trajectory into three regimes on the diffeeedifeerence time axis: early high-noise ([0, mid transition ([15,35),, and late low-noise ([35,50)), and rerun the tests in each regime (again 75 tests per regime). The aggregated pass rates are summarized in Table 8
Overall, finie--dfffrrenc features on FLUX.. dev are empirically very close to Gaussian across depth, modality, and time: all 75 all-timestep tests and 150/150 tests in the early/late phases pass at the 0.05 level, and only 2/75 high-order shallow-tex- configurations in the mid-transition phase fall slightly below 0.05. This supports the modeling assumption that motivates our Hermite basis.
ADAPTIVE PER-LAYER a ON FLUX. 1-U.X.--DEV
Section analyzes Hermite truncation errors and shows that stable extrapolation requires condition of the form o v 2 As| < 1. Rather than manually tuming single global (e.g., 0.5), design a training-free adaptive scheme that chooses a per-layer ag based on the observed scale of feature differences, while respecting this stability constraint.
For each layer f, we track first-order finite differences A AE. and compress them into a scalar magnitude Dr., via &g norm over channels followed by averaging over space and batch. We then maintain an EMA-style RMS estimate
(1 (1-10) (a) 11 + B D2..
(41)
initialized at de
(0) The per-layer contraction factor is set to
a
min
Tmax: /2 +
(42)
where 0 determines the desired initial scale (e.g., 0e Ro 0.5 1.0 when qe 1), Tmax caps large values for stability, and E > avoids division by zero. In practice this adaptive rule simply replaces the fixed scale factor in HiCache with or, without adding any learned parameters.
25

---

Table Chipmunk-Flux experiments (flux-dev, 1024x 1024, 50 steps, 200 prompts). All accelerated configurations use interval = 5, order = 2.
Mode
CLIP
ImageRewardf
PSNRT
SSIM
LPIPS!
Latency (s/img)
Pure (Chipmunk-Fiun Flux)
27.73
0,931
28.22
0.434
0.677
5.8
Taylor
27.34
0.845
28.12
0.380
0.746
3.3
HiCache (a=0.5)
27.88
0.938
28.12
0,415
0.704
3.5
We evaluate this adaptive FLUX.I-dev (1024x 1024, 50 steps, interval 7, order 2, 200 prompts) and compare against TaylorSeer and fixed-a HiCache. All configurations share the same acceleration setting (interval max max_order 2). The quantitative metrics reported in Table in the main text.
In the moderate regime where fixed 0.5 already provides strong performance, adaptive closely matches or slightly improves this baseline (differences in the reported metrics are within 0.02). When is increased to 1.0, the fixed-o variant becomes unstable (SSIM decreases from 0.655 to = 0.362 and LPIPS increases from 2 0.404 to z 0.725), whereas adaptive with initial scale x 1.0 remains stable and yields quality close to the 0.5 regime. This supports the theoretical motivation that the adaptive update can automatically prevent overly aggressive choices while preserving performance when the baseline configuration is already well behaved.
QWEN-IMAGE TEXT-TO-IMAGE EXPERIMENTS
We evaluate HiCache Qwen-Image (Wu al. 2025 from FLUX transfer to different text-to-image backbone. All runs use the same diffusion sampler and hyperparameters: the only differences are whether TaylorSeer or HiCache enabled and the acceleration interval. We report CLIP, ImageReward, PSNR, SSIM, and LPIPS over 200 prompts at 1328x 1328 resolution with 50 sampling steps.
assess whether the conclusions
Results are summarized in Table in the main text. At interval 3 (mild acceleration) TaylorSeer and HiCache obtain very similar scores. For larger intervals (6-8), HiCache consistently achieves higher CLIP, ImageReward, and lower LPIPS than TaylorSeer, while keeping PSNR and SSIM comparable level, indicating that the Hermite-based forecast is more robust under stronger acceleration.
CHIPMUN-FFLUT COMPATIBILITY WITH COLUMN-SPARSS ACCELERATORS
Finally, we examine HiCache on top of Chipmunk-Flun a column-sparse accelerator for FLUX, to verify that HiCache is complementary sparse/efficient attention rather than competing alternative. We compare three configurations on flux-dev (1024 x 1024, 50 steps, 200 prompts): pure Chipmunk- Flux, Chipmunk-Flux+ Taylor, and Hhipmunk-Flux+HHCachh with single setting. All accelerated variants share the same interval = 5 and order = 2.
Table reports CLIP. ImwgeReward, PSNR. SSIM, LPIPS, and approximate per-image latency. Moving from pure Chipmunk-Flun to Chipmunk+Taylor significantly reduces latency but degrades quality, especially in ImageReward and LPIPS. Chipmunk+HiCacche with 0.5 largely recovers the quality of the pure Chipmunk baseline-orf slightly exceeding it in ImageReward while retaining latency close to Chipmunk+ TTallo.. This suggests that HiCache can serve as a plug-in quality-restoring module on top of column-spars accelerators.
26


Reviews:
[{"review_id": "KVEsQ1wuqH", "reviewer": "Reviewer_Mccs", "summary": "This paper presents HiCache, a hierarchical key-value (KV) caching framework aimed at accelerating large language model (LLM) inference, especially under long-context scenarios. Modern transformer-based models depend heavily on KV caches to store past hidden states, but as sequence length grows, the cache size quickly becomes a bottleneck for both memory and bandwidth.\n\nHiCache introduces a two-level caching structure that separates tokens into “hot” and “cold” regions. The hot cache resides in fast GPU memory, while the cold cache is stored in slower memory (such as CPU DRAM or NVMe). A decay-based predictor estimates which tokens are likely to be reused soon, allowing the system to keep those in the hot cache and offload less active tokens. The method also includes an adaptive offloading policy that balances GPU memory usage with retrieval latency.\n\nExperiments on models like LLaMA and OPT, evaluated on long-text generation and summarization tasks, show that HiCache can achieve up to a 2.3 times throughput speedup while cutting GPU memory usage roughly in half, with minimal loss in generation quality compared to full GPU caching.", "questions": "1. HiCache is evaluated only on diffusion-based language models. Could you clarify what aspects of the design make it incompatible with autoregressive LLMs? Would adapting it require major architectural changes?\n2. The decay-based reuse predictor assumes gradual changes in attention importance across denoising steps. How well does it handle tasks where attention patterns change abruptly, such as in reasoning or code synthesis diffusion models?\n3. Would it be possible to extend HiCache to multimodal diffusion transformers (e.g., text-to-image or video diffusion) where different modalities have separate attention structures?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a practical and underexplored efficiency bottleneck specific to diffusion-based language models, where repeated denoising iterations make KV caching more memory-intensive than in autoregressive models. This focus is well motivated and timely.\n2. HiCache successfully adapts hierarchical caching concepts from systems design to the setting of diffusion language models. The integration into the diffusion pipeline is neat and minimally invasive, requiring no retraining or architecture modification.\n3. The decay-based reuse predictor is computationally cheap and integrates naturally into diffusion processes, where temporal coherence of token or latent attention is strong. The simplicity makes it easy to deploy and tune. The plug-in nature of HiCache also makes it a practical engineering contribution. It could be incorporated into many diffusion-LM frameworks with minimal code changes.", "weaknesses": "Although HiCache is well designed and practically useful, its applicability is limited to diffusion-based language models. The caching and reuse patterns exploited here rely on the iterative refinement process of diffusion models, which differ substantially from autoregressive decoding. \n\nThe evaluation focuses on throughput and memory reduction, but latency variance and system scalability are not thoroughly discussed. Diffusion inference involves synchronized denoising steps, so delayed cold-cache retrievals could lead to cumulative slowdowns. These potential risks are not well explored. Lastly, since diffusion-based LMs are still emerging, it would help to position HiCache more clearly within that research context and explain whether it generalizes to multimodal diffusion models (e.g., text-to-image diffusion transformers). \n\nAs a reviewer, I personally am not very familiar with this part of LM. So if the weaknesses do not make sense, please explain it in your response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CfaEAww5zy", "reviewer": "Reviewer_sZeu", "summary": "The paper introduces HiCache, a training-free acceleration method for Diffusion Transformers (DiTs) that replaces traditional Taylor-series extrapolation with a Hermite polynomial-based feature caching framework. Existing caching methods like TaylorSeer predict future diffusion features using monomial expansions, but they struggle with the non-monotonic and stochastic dynamics of diffusion features, leading to quality degradation at high acceleration ratios. HiCache addresses this by observing that feature derivatives in DiTs follow approximately Gaussian statistics and thus are better represented by scaled Hermite polynomials, which are the optimal orthogonal basis for Gaussian-correlated processes.\n\nHiCache further introduces a dual-scaling mechanism that stabilizes numerical behavior by simultaneously contracting inputs and suppressing high-order coefficients. This enables accurate, stable extrapolation while preserving the plug-in simplicity of Taylor-based predictors. Experiments across text-to-image (FLUX.1-dev), text-to-video (HunyuanVideo), class-conditional (DiT-XL/2), and super-resolution (Inf-DiT) tasks show that HiCache achieves 5–6× acceleration and even slightly improves perceptual quality metrics such as ImageReward. The method integrates seamlessly into existing caching frameworks, offering a mathematically grounded, numerically stable, and empirically robust approach to accelerating diffusion models without retraining or architectural modifications.", "questions": "The followings are the main questions:\n1. Generality of Gaussian assumption: The core motivation for using Hermite polynomials relies on the assumption that feature derivatives in diffusion transformers follow Gaussian statistics. Have you tested this hypothesis across different architectures, such as U-Net–based diffusion models or non-transformer backbones? How consistent is the Gaussianity across layers, modalities, or timesteps?\n2. Parameter sensitivity and scaling: How sensitive is HiCache’s performance to the choice of the contraction factor (σ) and expansion order (N)? Would an adaptive or learned scaling mechanism improve robustness across models and tasks?\n3. Behavior under non-Gaussian or multimodal features: In real-world generative settings, feature distributions can become skewed or multimodal. How does HiCache perform when the Gaussian assumption breaks down? Are there fallback mechanisms or hybrid bases that could handle such cases?\n4. Numerical stability in extreme accelerations: When acceleration ratios become very high (e.g., >8×), does the Hermite-based predictor remain stable, or do truncation errors accumulate? Could you quantify where the stability boundary occurs in practice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The key strengths of the paper lie in its strong theoretical foundation and practical effectiveness. HiCache introduces a principled improvement over Taylor-based caching by recognizing that diffusion transformer features evolve according to approximately Gaussian dynamics. By replacing Taylor’s monomial basis with scaled Hermite polynomials, which are theoretically optimal for Gaussian-correlated processes, the method provides a mathematically sound and statistically aligned framework for feature prediction. The addition of a dual-scaling mechanism further enhances numerical stability, allowing accurate high-order extrapolation without exploding coefficients or quality loss.\n\nAnother strength is the method’s generality and empirical robustness. HiCache is fully training-free and plug-and-play, requiring only a simple basis substitution in existing caching pipelines. It achieves consistent 5–6× acceleration across diverse tasks—text-to-image, text-to-video, class-conditional generation, and super-resolution—while often improving generation quality. These results demonstrate not only its theoretical elegance but also its strong practical value, combining mathematical rigor, stability, and real-world applicability in a single, lightweight framework.", "weaknesses": "The main weaknesses of the paper stem from its scope, assumptions, and evaluation coverage. HiCache is designed specifically for Diffusion Transformers (DiTs) and relies heavily on the assumption that feature derivatives follow Gaussian statistics. While this is empirically validated for certain architectures like FLUX, the assumption may not hold universally across other diffusion models, such as U-Net–based or multi-modal architectures. Likewise, the framework’s reliance on Hermite polynomials and the 2:4 Gaussian-correlated structure may limit its adaptability to more complex, non-Gaussian feature dynamics or models trained with nonstandard noise schedules.\n\nAdditionally, the experimental validation, though extensive, focuses primarily on large-scale diffusion models under controlled hardware and benchmark settings. The paper lacks detailed analysis of sensitivity to parameters such as the contraction factor σ and order N, which could affect stability in different model or task configurations. Finally, while the theoretical exposition is strong, the approach adds some conceptual and implementation complexity, and its practical impact beyond NVIDIA GPU environments or specific DiT variants remains to be demonstrated. Overall, the method is elegant and effective but somewhat specialized and assumption-dependent, leaving questions about its generality, robustness, and portability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QlkXTJx6O", "reviewer": "Reviewer_pEwz", "summary": "This paper proposes HiCache, a training-free acceleration framework for diffusion transformers that replaces Taylor expansion with scaled Hermite polynomial-based feature caching, achieving more stable and accurate predictions across multiple generative tasks.", "questions": "It would be helpful if the authors could clarify whether HiCache is compatible with sparse- or efficient-attention variants of diffusion transformers.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper replaces Taylor’s monomial basis with Hermite polynomials derived from Gaussian feature correlations, leveraging Karhunen–Loeve optimality and a single scaling factor σ to improve stability and accuracy.\n2. HiCache preserves almost the same implementation form as TaylorSeer, merely replacing the polynomial basis and adding a few scalar evaluations, thereby allowing direct integration into any feature caching–based acceleration framework with negligible computational overhead.\n3. Extensive evaluations on text-to-image, text-to-video, and super-resolution tasks demonstrate the effectiveness of the proposed method in achieving substantial acceleration.", "weaknesses": "1. The paper primarily relies on automated metrics such as PSNR, SSIM, LPIPS, and VBench. Incorporating human preference evaluations would make the assessment more convincing.\n2. The paper heavily relies on the scaling factor σ to stabilize predictions, yet it lacks a principled rule or analysis on how to select or adapt σ across architectures, acceleration ratios, or polynomial orders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bAziSHQKjG", "reviewer": "Reviewer_owJr", "summary": "This paper proposes HiCache, a training-free acceleration framework for diffusion transformers that addresses the quality loss issue of existing feature caching methods. This paper follows the \"Cache-then-Forecast\" paradigm. The core insight is that feature derivative approximations in diffusion transformers exhibit multivariate Gaussian characteristics, leading to the adoption of Hermite polynomials—optimal for Gaussian-correlated processes—as the prediction basis instead of the suboptimal monomial basis in Taylor series. HiCache also introduces a dual-scaling mechanism to ensure numerical stability and predictive accuracy.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ Deep Insight into Trajectory Modeling: The paper identifies Taylor series’ limitation: its monotonic monomial basis fails to capture non-monotonic feature trajectories. In contrast, it links diffusion transformers’ Gaussian feature derivatives to Hermite polynomials and their oscillatory behavior, which accurately models non-monotonic dynamics.\n\n+ Extensive Task Coverage: The paper evaluates HiCache across a broad range of generative tasks to demonstrate its generalizability. Extensive experiments are conducted on text-to-image, text-to-video, class-conditional image generation, and super-resolution tasks, demonstrating its superiority. It maintains strong performance across different acceleration ratios.\n\n+ Effective Acceleration Design: The dual-scaling mechanism not only solves the numerical instability of Hermite polynomials for large extrapolation steps but also works standalone to improve TaylorSeer. HiCache’s plug-and-play nature—requiring only polynomial basis substitution without architectural changes—enables easy integration with existing cache-then-forecast pipelines, enhancing its practical value.", "weaknesses": "- Modest Performance Advancement Over TaylorSeer: While the paper provides solid theoretical analysis for HiCache’s improvement over Taylor series-based methods, the practical performance gain over TaylorSeer is not prominently distinct in both qualitative visualizations and quantitative metrics. More compelling evidence—such or case studies is needed to further validate its superiority.\n\n- Limited Variety of Experimental Models: The experimental evaluations focus on a relatively narrow set of diffusion transformers. For image generation, it primarily uses FLUX, with no tests on other widely adopted models like Stable Diffusion 3 or Qwen-Image. For video generation, only HunyuanVideo is included, without evaluations on other mainstream video diffusion models like Wan. Conducting experiments on these additional models would further strengthen the paper's validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)