Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
Authors: Hanru Bai
Weiyang Ding
Difan Zou
Venue: NeurIPS 2025 poster ()
Keywords: One-step Generation
Diffusion Models
Koopman Operators
Interpretable Image Synthesis
Primary area: deep_learning
TL;DR: We propose explicit and interpretable one-step generation framework that retains the advantages of traditional diffusion models, such as access to intermediate states and fine-grained control, while enabling fast sampling.

Abstract: Diffusion models have achieved impressive success in high-fidelity image generation but suffer from slow sampling due to their inherently iterative denoising process. While recent one-step methods accelerate inference by learning direct noise-to-image mappings, they sacrifice the interpretability and fine-grained control intrinsic to diffusion dynamics, key advantages that enable applications like editable generation. To resolve this dichotomy, we introduce **Hierarchical Koopman Diffusion**, a novel framework that achieves both one-step sampling and interpretable generative trajectories.  Grounded in Koopman operator theory, our method lifts the nonlinear diffusion dynamics into a latent space where evolution is governed by globally linear operators, enabling closed-form trajectory solutions. This formulation not only eliminates iterative sampling but also provides full access to intermediate states, allowing manual intervention during generation. To model the multi-scale nature of images, we design a hierarchical architecture that disentangles generative dynamics across spatial resolutions via scale-specific Koopman subspaces, capturing coarse-to-fine details systematically. We empirically show that the Hierarchical Koopman Diffusion not only achieves competitive one-step generation performance but also provides a principled mechanism for interpreting and manipulating the generative process through spectral analysis. Our framework bridges the gap between fast sampling and interpretability in diffusion models, paving the way for explainable image synthesis in generative modeling.

Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
Hanru Bai Fudan University hrbai230m. fudan. edu.
Weiyang Ding" Fudan University dingæy@fudan edu.
Difan Zou
The University of Hong Kong dzou@cs hku. hk
Abstract
Diffusion models have achieved impressive success in high-fidelity image gen- eration but suffer from slow sampling due their inherently iterative denoising process. While recent one-step methods accelcrate inference by learning direct noise-to-image mappings, they sacrifice the interpretability and fine- line-grained con- trol intrinsic to diffusion dynamics, key advantages that enable applications like editable generation. To resolve this dichotomy, we introduce Hierarchical Koop- man Diffusion, a novel framework that achieves both one-step sampling and interpretable generative trajectories. Grounded in Koopman operator theory, method lifts the nonlinear diffusion dynamics into a latent space where evolution is governed globally linear operators, enabling closed-form trajectory solutions. This formulation not only eliminates iterative sampling but also provides full access intermediate states, allowing manual intervention during generation. To model the multi-scale nature of images, we design a hierarchical architecture that disen- tangles generative dynamics across spatial resolutions via scale- scale-specific Koopman subspaces, capturing coarse-to-fine details systematically. We empirically show that the Hierarchical Koopman Diffusion not only achieves competitive one-step generation performance but also provides principled mechanism for interpreting and manipulating the generative process through spectral analysis. Our framework bridges the gap between fast sampling and interpretability in diffusion models, paving the way for explainable image synthesis in generative modeling.
1 Introduction
Diffusion models [29, 7, 30] have achieved remarkable success in image generative tasks [3]. How- ever, despite producing figh-iidelity samples, the sampling process of diffusion models typically requires an expensive iterative procedure, which limits their applicability in real-world scenarios [18]. Accelerating sampling diffusion models thus remains a critical challenge. To overcome this limitation, recent work has focused on efficient one-step inference for diffusion models to replace the costly iterative sampling process.
Existing approaches toward this goal fall into several distinct paradigms. A widely adopted paradigm distillation-based methods [39, 42], which aim to distill pre-trained diffusion models into efficient one-step generators. Among these, Knowledge Distillation (KD) [17], Progressive Distillation (PD) [27], and (distilled) Rectified Flow [14, 13] considered classical approaches, laying the foundation
"Correspondng author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

for recent advances in one-step generation. Another influential direction involves consisteney models [32. 31], which learn time-consistent mappings from noisy inputs to clean data.
Although these methods achieve strong performance in one-step image generation, they fundamentally rely on learning direct noise-to-image mappings, bypassing the temporally coherent denoising trajectory upon which diffusion models are built. As a result, they lack access to intermediate generative states of the diffusion trajectory that are crucial for interpreting and controlling sample evolution from noise to image, thus leading to limited interpretability of the generation process. This lack of interpretability not only obscures how semantic and structural information gradually emerges during sampling but also limits the ability to intervene at specific stages along the generative trajectory- an ability that enables controllable image synthesis at inference time."
Our goal is to develop an explicitly interpretable one-step generation framework, which is not merely to achieve high generation quality in single step, but provide a principled understanding of the generative process and thus enable fine-grained control along the diffusion trajectory. Motivated by Koopman operator theory [20. 24], which projects the nonlinear dynamics into an observable space where evolution is linear via function space lifting, we propose an explicil one-step generation paradigm that mapping the entire deterministic sampling trajectory of diffusion models into latent space where closed-form ODE solutions exist. This latent space, theoretically grounded in Koopman theory, is referred to as the Koopman space. In this space, the dynamics of the diffusion process are governed by linear operator, thus revealing an explicit, analytically tractable form for the evolution to enable an interpretable mapping from noise to data. We realize this framework by jointly learning the mapping and its associated linear operator. In our explicit framework, all intermediate states along trajectories are analytically accessible, thus naturally allowing for introducing additional supervision on these states to guide the noise-to-iman mapping more precisely and enabling fine fine-grained control of the generation dynamics, a capability not afforded by implicit methods.
While the Koopman-based modeling provides a principled way to enable explicit one-step sampling, its standard formulations operate in single latent space, implicitly assuming uniform dynamics across all spatial and semantic scales. However, generative processes, particularly in visual domains, exhibit inherently multi-scale behaviors: global structures emerge early and evolve slowly, while fine-grained textures form later with more rapid variations [34]. Ignoring this scalc-specific nature limits the model's capacity to represent the full complexity of image generation. To address this limitation, we reformulate hierarchical Koopman modeling framework that explicitly decomposes the generative dynamics across multiple spatial resolutions. Features at different scales are extracted via U-Net encoder and projected into separate Koopman subspaces, each governed by its own linear operator. This design allows the model to track the evolution of visual content from coarse layout to fine detail, aligning with the hierarchical nature of visual perception and offering improved generation fidelity. Our key contributions are summarized as follows:
We propose a novel interpretable one-step generation paradigm, referred to as Hierarchical Koop- man Diffusion (HKD), via hierarchical Koopman dynamics. This paradigm uniquely integrates explicit intermediate generative states into the process, enabling control along the diffusion tra- jectory. Moreover, our framework provides novel analytical aspect for diffusion models using spectral tools in dynamical systems theory to analyze the underlying generative mechanisms.
We provide a theoretical justification that our Koopman explicit formulation is provably more expressive than directly learning black-box mapping from noise to image using standard neural networks for learning the diffusion process.
We conduct experiments on the CIFAR- 10 and FFHQ datasets to demonstrate the competitive one- step generation performance of our proposed framework. Beyond generation quality, we interpret the underlying generative dynamics via principled spectral analysis, revealing a quantitative correspondence between spectral components and semantic image attributes. Moreover, we further justify the interpretability of our framework through an image editing experiment that targets frequency-specific intervention at the intermediate stage of the diffusion trajectory.
"Some distillation biiiaattoo---edddddee fast generation methods have explored multi-step editing for text-to- image,, which is beyond the scope of this work. We focus on trajectory level intervention during one-step generation.

---

2 Background
Diffusion Models. We consider the continuous-time diffusion models [33], where data is generated by reversing a stochastic process that gradually adds Gaussian noise. Let (x) denote the data distribution. The diffusion process is described by the stochastic differential equation (SDE): dæ; fl (t)dd, where t E [0, T. (u(:;-) -) and (-)) are the drift and diffusion coefficients, respectively. (eE.T) represents standard Brownian motion. The distribution of æ, pr pr()) with po(a)) = Pdata (z). Notably, there exists ordinary differential equation (ODE), called the Probability Flow ODE, whose solution trajectories sampled at &. follow Pr)):
log dæ; [p(z,,) f) ((tg"Vloope(aa dt,
where V (x is the score function of (x.))
Koopman Operators. Koopman theory provides a framework for analyzing nonlinear dynamical systems by transforming them into a linear form. The formal definition of the Koopman operator is given below.
Definition 2.1 (Koopman Operator [21]). Consider fiitte-immensoonal state space X C R with state evolution described by T++ ((xx), t E [to, tı], where & : X - X is the state transition operator. The Koopman operator K. is linear operator that acts infiniteddimensione space of observable function I" + R, such that
(Ko () (acta))gg((0 (xx()
Instead of modeling the state x, directly, the Koopman operator captures the evolution of observables 2; g (x) (x), 9m (x)] in a lifted space, where each g, is component of the observable function g. In continuous time, this yields linear system /z//dt Az,, enabling spectral analysis of nonlinear dynamics. Formal details on approximating with finite observables are in App. A.2.
3 Hierarchical Koopman Diffusion for Fast Generation
We propose a novel denoising generative model called Hierarchical Koopman Diffusion (HKD) that enables one-step deterministic sampling for diffusion models via Koopman operator theory. By leveraging the Koopman operator's ability to use tools of linear dynamics such as spectral analysis, our formulation not only enables efficient one-step ODE sampling but also offers a principled dynamical interpretation of generative modeling. This section is organized as follows: we first introduce the theoretical formulation of our framework (Sec. 3.1), then describe the corresponding supervision strategy (Sec. 3.2), and conclude with implementation details (Sec. 3.3). We subsequently present theoretical results that justify the superiority of the proposed method under ideal conditions (Sec. 3.4).
3.1 Framework Formulation
Encoder. Given the trajectory (ateerr) satisfing ODE in Eq. (1), we construct a hierarchical Koopman representation by applying an encoder
Eo : 24 RCXHXW + (z(") Rdxxhi (xxxhhxww) 1/2
Eo adopts a U-Net style downsampling architecture, just as the right part of Fig. 1. Here, de is the number of latent channels and (h₁, wy) is the image resolution at level l. The encoder approximates independent Koopman observable functions (t) Cx Rd xwi at each level by obtaining (1) g (1) (xx), where (!) captures the dynamics corresponding to its spatial scale.
Hierarchical Koopman Subspace. Each latent feature z (1) R44 xhxxxrx is extracted at level I, with (f,().)) E IRd denoting the feature vector at spatial position (i, We model (t, evolves linearly under vaatially-varying linear operator (0) (i.j) e && xd, which is specific to each spatial location (i. just the middle part of Fig. 1:
dz 1 (1,) A (1) (i,g) zt ( dt
(i,5), V(i,)), te [e,
(3)
3

---

Image X
Probability
Noisy Image X₃
32 x
Sporial Hierareby Actoss Multi-Reso essolutions
32 32
Aguzi Reur Paie
Koopman Dynamics
(L.
x.16
Koopman Dynamics
Koopman Dynamics
encoder E
decoder D
Koopman Dynomics
Denoising
Figure The framework of the proposed method. The HKD model first hierarchically extracts lifferent-level features from the given noisy image at any time by encoder E. Secondly, the Koopman dynamics model is applied for each level to the skips and bottleneck. Last, uniform decoder D performs the mapping from the Koopman spaces back to the image space.
The latent evolution at each position thus follows an independent linear dynamical system governed by its local Koopman operator. Allowing A A (i,j) to vary across spatial locations enables the model capture heterogeneous temporal-frequent behaviors at different regions, thus leading to finer-grained and spatially adaptive dynamics modeling.
Without loss of generality, (1) (i, could be modeled in the form of block-diagonalizzbbl i.e.,
A (i.j) diag A (1) (i.j), (i.s). A (i.j), A (1,(i.))))
with di even and each block A (i, E 222 corresponding to a pair of complex conjugate a (6.)) (f.) This eigenvalues 0 - 1 of (1) (i,j), given explicitly by A (i.j) = (2. a (L.))
reduces the number parameters compared to modeling full matrix. The rationality of this modeling can be guaranteed by the Prop. C.I. App. C.I. Due to the linear evolution in the Koopman
space, the mapping between latent states time and for level I and location (i,j) j) can be explicitly expressed as
(1) (i, e (1.5)(e-)) (i.j).
The explicit formulation enables the network to learn sufficient one-step mappings from =t, Vt [e, 7] to z. during the training process. Consequently, it allows direct mapping from wT to w, in the inference time, thus achieving efficient one-step sampling without the need for iterative integration.
Decoder. Finally, at the target time step c, the evolved of evolved latent features (2(") M1=1 is decoded to generate the sample xe, The decoding process is performed by a decoder Díg which implements the mapping
Do (z( | + x. E RCxH
At each decoding level, the decoder upsamples the latent features from the lower-resolution level and integrates them with skip features obtained by evolving the encoder outputs through Koopman dynamics, just as the left part of Fig. 1.
3.2 Intermediate State Supervision: Trajectory Consistency Loss
A key advantage of our explicit framework over implicit mappings is the ability to supervise inter- mediate states during generation. Unlike conventional methods that supervise only the endpoint, approach explicitly constrains the trajectory from noise to image by enforcing consistency with

---

Figure 2: Image reconstruction from noisy inputs time highlights the consistency enforced by framework. The left-most image is obtained from æy and the right-most most from Zy-
expected denoising dynamics along the path. Hence, we introduce a new trajectory consistency loss for the supervision of intermediate states. Fig. 2 presents the reconstructed images from noisy inputs all time steps, showcasing the consistency properties promoted by our proposed loss function. Definition 3.1 (Trajectory Consistency Loss). The trajectory consistency loss is defined as
Lt-comii En (e.T) [al (Dol(els-()A e!!)) (1)))
where the expectation is taken with respect to U [c, T), and æ, is given by Eq. (1). Distance d(.,-) 2 0 is in the image space and equals 0 if and only if the argument variables are the same.
This trajectory consistency loss enforces that any intermediate state Tt, after being encoded and evolved through Koopman dynamics to time yields decoded prediction that matches the ground truth image &, obtained by Eq. (1). However, the most direct supervision would be to enforce alignment between the encoder-dens deriived Koopman representation at time and its analytically evolved counterpart from time within the Koopman space. Although conceptually straightforward, enforcing consistency directly in the latent space may lead to suboptimal results in practice, as it reduces training flexibility and introduces gradients that poorly reflect perceptual discrepancies. Therefore, we instead adopt alternative image-space formulation in Eq. (6), which enables the use of perceptually meaningful distance metrics, such as the Learned Perceptual Image Patch Similarity (LPIIPS), to better capture semantic fidelity. Importantly, we show in App. C.2 that, under structural assumptions, minimizing the trajectory consistency loss in image space is theoretically equivalent to minimizing its latent-space counterpart.
3.3 Implementation
Training. We train the model using total loss L consist Lrecon: where ao enforces consistency along the diffusion trajectory, and Crecan = d D (ws(fe(r-T) e(l)) (x+)))=) su- pervises accurate one-step mapping from noise to clean image. The distance function defined as d(x, y) = AI + X2 CLPPS with X2 = I and êι annealed to shift from coarse alignment to perceptual refinement. The full model, Eg. Da. and (A (A)) M=
trained end-to-end. The training algorithm is presented in Alg. 1. More training details are provided in App. B.1.
One-Step Sampling, xτ by
At inference time, we could obtain the final generated sample &. from noise
De((e(e-T)a") ell) HEn)
Specifically, Given noise sample, project it into the Koopman space, perform one-step evolution via Koopman dynamics, and decode the result back to the data space. The detailed sampling algorithm of sampling is presented in Alg. 2.
Koopman Spectral Analysis. Our framework offers a distinct perspective on diffusion generation through Koopman spectral analysis. Each block A (i.j) in Eq. (4) represents a local dynamical mode, with eigenvalues af (i, + 4355)) (v, encoding the growth/decay and oscillation of trajectory. This explicit spectral interpretability further allows for controllable image generation by selectively modifying certain frequency components. More details of Koopman spectral analysis of framework can be found in App. B.2.
5

---

Algorithm The training algorithm.
1: Inputs: Trajectory (ej.rr from well-trained diffusion model; The number of samples 8
2: Outputs: Network parameters and Φ; Koopman matrices (A()E
3: Initialize 0 and & by a pre-trained U-Net
Initialize A (1) + O, / ,......
5: for i = 0 to num -iiiitrrr11 do
6: Sample S. = {1: 4 10, T U {T}
Uniformly sample the intermediate time Apply the HKD model F is the feature extractor in LPIPS
7: Reconstruct it. ( E0 ((=)) fort
8: Let L = Eies, || mo || + (1=())) F(mo)]]
9: Update 0, Φ and (0) by the gradients
10: end for
11: Return 0, Φ and {A (A 11=1
Algorithm 2 The sampling algorithm.
I: Inputs: Trained HKD including network parameters 0, & and Koopman matrices H
2: Outputs: Predicted image ă. = De((e/<TTA") ell) (zT)))
3.4 Theoretical Results
In order illustrate the feasibility of our proposed method, we theoretically analyze the error bounds caused by the proposed method, namely erTиKD, and by conventional one-step methods erTone-step Intuitively, neural networks of the complex functions. In the proposed formulation, the encoder and decoder are commonly simple due the elementary observable function, hence, the HKD generation, serving composition of three functions, is overall simpler than end-to-end one-step methods. Algebraically speaking, we have еттнкт S e/Tone-step
size are faced with larger errors when estimating more
(1s) where
reasonably small error caused by the Koopman process.
This inequality holds mainly because the proposed framework first transfers the complicated data space to the simpler Koopman space at small cost and then controls the Koopman trajectory error by the explicit formulation the Koopman theory. Thm. 3.1 provides an informal description of the conclusion where k = O(N +) + o(m 8) is small and that substantiates the common knowledge with N and being the size of the dataset and the dimension of the Koopman space respectively. A formal description and proof of the theorem are given in App. C.3.
Theorem 3.1 (Comparison of Error Bounds for One-Step Diffusions). Let ETThKD and erT be the ideal estimation errors of our proposed HKD model and end-to-end one-step model with the same total number of activation functions we have,
eITHKD S CTT stee O(s).
where K = max
3ammm Poop
2//amm Piul
+ o(m 5) with N and m
VN(A -2m 4mm )-Pnnt /3om² dominating being the size of the dataset and the dimension of the Koopman space respectively. The parameter a is the standard deviation of the observables, 6 0 when the network is perfectly trained, Pint and Psup the true spectral range of the Koopman operator, and m-dimensional Koopman process adequately model the Koopman space.
V 2m 5) 3omm
when the
To formally prove the above result, we first proposed evaluate the network size by the number of activation functions it uses, as they are the ones causing non-linearity of the network. Secondly, we introduced the simplicial error as the deviation in the simplicial complex estimation. Fig. empirically shows that the metric of same-density triangulations reveals the complexity of 2D surfaces, demonstrating our idea. Consequently, the simplicial error quanti- tatively evaluates the complexity of the data and noise spaces and the mappings between them. Finally, using the ideas above, we analytically prove Thm. 3.1.
Figure 3: More complex sur- faces need denser triangula- tion for the same error.

---

Experiments
We conducted experiments to present the promising one-step generation capability our framework compared with other one-step baseline methods (Sec. 4.1). Notably, beyond generating high-quality images, we further provide empirical insights toward understanding the underlying dynamics of diffusion models through Koopman spectral analysis (Sec. 4.2). This analysis demonstrates the dynamical interpretabiliity of our approach and further enables controllable image editing (Sec. 4.3). An ablation study was finally performed show the key contributions of framework (Sec. 4.4). Extensive results highlight both the generation quality and interpretability ad advantages of our framework, distinguishing it from distillation- and consistency-motd soaecey-oool-bbbseee paradigms.
4.1 Compare with Prior One-step Generation Methods
In this section, we evaluated our proposed framework on the CIFAR-IO [12] and FFHQ datasets [9], focusing exclusively on one-step generation methods within the diffusion model family. We used the standard Fréchet Inception Distance (FID) [6] metric and report FID-50k scores, following prior work [42]. Our model employs the U-Net [26] architecture from EDM [8] as the backbone for both the encoder and decoder, initialized with pretrained weights. All models were trained using Adam optimizer [11] with a constant learning rate of x
10 and a weight decay of 0.95. App. D.4 provides additional implementation details of the experiments.
Comparison Prior Work. We evaluated our framework against two dominant one-step generation approaches: distillation-based meth- ods and consistency models, with comparisons to multi-step diffusion baselines for context. The multi-step diffusion baselines include: (I) DDPM [7]: The foundational denoising diffu- sion model; (2) Score SDE [33]: Continuous- time diffusion via reverse-time SDEs; (3) DDIM
Table I: Sample quality on CIFAR-10 dataset.
Methods
NFE(()
FID(L)
Multi-Step Diffusion Models
DDPM [7]
1000
3.17
Score SDE [33]
2000
2,38
DDIM [30]
100
4.16
DDIM [30]
10
13.36
[30]: Efficient deterministic sampler from diffu-
EDM [8]
35
1.97
sion ODEs; (4) EDM [8]: State-of-the-art diffu- sion backbone (used in our framework).
EDM [8]
5.62
Diffusion Distillation
For the distillation the comparison extends to traditional leading methods: (1) KD [17]: Traditional knowledge distillation method with direct teacher-studes mimicry; (2) PD [27]: Progressive distillation with step reduction through sequential student- teacher training: (3) CD (LPIPS) [32]: Consis- tency distillation with LPIPS loss; (4) DMD [39]: Distribution matching distillation with the distribution-matching loss; (5) and (dis- tilled) ReFlow series: including I-Rectified flow, 2-Rectified flow, 3-Rectified flow [14] and 2-Rectified flow++ [13] with reflow and distilla- tion. For the wawiistenccymoee-bbaseeeee method, we included: (1) CT [32]: Consistency training including CT (LPIPS)) iCT [31], and iCT-deep [31], which provides a comparison to the end- to-end traditional consistency model, and its im- proved versions. (2) ECM [4]: Easy Consistency Models, which initializes the network weights with the ones from pretrained score model. More details of comparison methods are App. D.5.
one-step paradigm,
KD [17]
9.36
PD [27]
8.34
CD (LPIPS) [32]
3.55
DMD [39]
3.77
1 Rectifiie flow [14]
6.18
Rectified flow [14]
4.85
3 Rectified flow [14]
5.21
2-Rectified flow++ [13]
3.38
Consistency Model
CT (LPIPS) [32]
8.70
CD (LPIPS) [32]
3.55
iCT [31]
2.83
iCT-deep [31]
2.51
ECM [4]
3.60
HKD
3.30
While enhanced methods such as iCT-deep [31] have pushed performance boundaries, they suffer from high sensitivity to hyperparameters. Moreover, state-of-the-art consistency training typically requires nearly a week of training on GPUs [4], and remains unstable in practice [16]. Retraining with consistency distillation (CD) even results in degraded FID of 10.53, as reported
Results.
7

---

(1) noise
Input Noise
Spectrum
(b) mid
(3)
high
Generated
Image
all
Ground
Truth
Figure 4: Visualization of Spectral Contributions in Generated Images Across Koopman Modes. (a) visualizes the contribution of Koopman spectrum with the smallest real-part magnitudes to the reconstructed state over time. It corresponds to the noisy part of the image. (b), (c) and (d) illustrate progressively larger spectral components and their reconstructed image components.
in [10]. In contrast, our method achieved comparable performance within just 2-3 days on 8xV100 GPUs.
Moreover, we significantly improved the training stability thanks to design choices: (1) the exponential formu- lation in the Koopman space ensures sufficiently large gradients for spectra with magnitudes near mitigating the high-variance gradient issues commonly seen in con- sistency models [28]: and (2) supervising the Koopman trajectory at multiple time points enables more stable and averaged spectrum estimation, leading to more reliable mode evolution modeling. Experimental evidence of the improved training stability on the CIFAR-10 10 dataset is provided in App. D.I.
Table 2: Sample quality FFHQ.
Methods
NFE(()
FID(J)
DDIM [30]
10
18.30
EDM [8]
79
2.47
EDM [8]
15
9.85
ECM [4]
5.99
HKD
1
5.70
Additional experiments on FFHQ 64x64 (Tab. 2) validated the effectiveness of our method on the high-resolution and structurally complex dataset. See App. D.1 for additional visualizations, per-image wall-clock timings for various generation methods, and conditional generation results.
4.2 Koopman Spectral Analysis for Generative Process
We conducted an analysis of A to further investigate how individual spectral components influence the generative trajectory. We first tracked the contribution of each Koopman spectrum (cumulative effect, namely CE) to the reconstruction over time. Then, at each resolution level I, we sorted the eigenvalue pairs across spatial locations by the real parts and applied spectral masking by retaining only those within a target range (smallest, intermediate, or largest), zeroing out the rest. The masked representations are decoded to visualize the contribution of selected spectral modes.
Fig. 4 summarizes the results of the two spectral analyses; the left shows the contribution of grouped Koopman modes reconstruction over time, while the right links spectral ranges to corresponding image structures. By selectively activating spectral bands and decoding the latent representations, we observe clear semantic hierarchy: low-range modes capture global structure, mid-range modes recover overall shape and pose, and high-range modes refine local details. These results demonstrate the semantic interpretability of Koopman spectra and their potential for controllable image synthesis. More Koopman spectral analysis results are provided in App. D.2.
4.3 One-Step Image Editing: A Case of Model Interpretability
We evaluated the interpretability of our framework through an image editing experiment that targets frequency-specine interventions at the intermediate state of the diffusion trajectory. We performed editing by injecting high-frequency features from a reference image into the generated image at mixing ratios of 10%, 20%, 50%, 80%, and 90%, applied at the middle time step to demonstrate

---

injection ratios from reforence image Teatures
low 10%
20%
90%
WX
hisa
Original Imsagg
Refirence Image
mixing high freq.
Figure 5: One-step image editing via frequency-awarc interventions along the diffusion trajectory. We controlled the image generation by the high-frequence features from reference image through injecting them into the lower-left half of the generating image at different mixing ratios (10%. 20%, 50%, 80%, 90%).. The modifications Columns 2-6 showcase the frequency-aware editing, where only high-frequence components were mixed, preserving the low-frequee low-frequency structure of the original image. Column 7 is for frequency- agnostic editing, where all-frequency features of reference images are mixed with all frequency bands of the original image a mixing ratio of 90%. We exhibit the results from both datasets.
performed the midpoint of the Koopman trajectory.
temporal editability. For comparison, frequency-agnen editing mixes the full-spectrum content of the reference and generated images at the same step.
As shown in Fig. 5, increasing the injection ratio of reference image features from 10% to 90% gradually reveals more facial details from the reference, demonstrating that our frequency-aware editing establishes meaningful correspondences through interpretable frequency decomposition. In contrast, eequeeccyagnnoste editing disrupts global structures, indicating a lack disentangled control. These results further validate the interpretability of our framework.
We further perform an image editing experiment of image recuperation the impainting and coloring tasks on CIFAR-10 10 dataset following Algorithm 4 from [32], which iteratively mixes reference image with the generated image along the generative trajectory by adding and removing noise at each time step t. The results are presented in Fig. 6.
4.4 Ablation Study
We present an ablation study on the CIFAR- 10 dataset to evaluate the contributions of key components in frame-
Table 3: Ablation study on CIFAR-10.
work: Koopman evolution (Koop.), the trajectory consis- tency loss (onsconsis), and hierarchical design (hierar.). Tab. 3 shows that the one-step baseline model removing the Koop- man dynamies results in FID of 5.72. Upon it, additional Koopman evolutions at the skips and bottleneck paths, and introducing trajectory consistency loss further improve the results. In addition, a hicrarchical Koopman dynamics de- sign also contributes to better generation quality.
Settings
FID (L)
Koop.
Le Consist
hierar.
5.72
/
5.57
/
X
4.78
3.30
Related Work
One-step Generation for Diffusion Models. Prior works have distilled pretrained diffusion models into efficient one-step generators. Knowledge distillation [17] trains a student model to replicate the denoising behavior of the teacher, rectified flows [14] reformulate diffusion as ODEs for distillation, all considered as classical approaches. Recent works further improve distillation-basee generation, such as DMD [39], which aligns one-step generators with diffusion models via KI. divergence, and SlimFlow, which [42] enhances rectified flows through compression of model size. Consistency

---

Original Image
Masked Image
Impainting
Gray-Scale Image
two sumples
Coloring
Figure 6: The visualization of image recuperation in the impainting and coloring tasks. The exper- iment is performed by Algorithm 4 from [32], which iteratively mixes the image with a reference image created by adding and removing noise at time with decreasing from T to O.
models [32] take different route, suffering from instability and inefficiency, Improvements include fine-tuning from diffusion models [4] and adversarial enhancements [5]. However, these methods prioritize performance at the cost of losing the inherent interpretability and controllability offered by diffusion models.
Koopman Operators. Koopman operator theory has enabled advances in system analysis [19], control [23], and optimization [25] by extracting globally linear structures from nonlinear dynamics. Recent work extends its use time series modeling [22, 40, 2]. In addition, many other works have proposed approximating the Koopman operators using neural networks. For instance, KNF [35] leverages DNNs learn the linear Koopman space and the coefficients of chosen measurement functions. To our knowledge, we are the first to introduce Koopman theory into the field of image generation. See App. A.1 for more related work.
6
Discussion
HKD is an interpretable one-step generation framework that retains access to intermediate states and fine-grained control-while enabling one-step sampling. It is the first introduce interpretability and intermediate-state intervention into one-step generation. By bridging fast sampling and interpretability. HKD opens new possibilities for explainable image synthesis. While demonstrates competitive one-step generation performance and enables spectral interventions, several directions remain open for future exploration. First, although we adopt standard training paradigm without relying on adversarial optimization or heavy tuning, integrating advanced training techniques (e.g., adversarial learning) may further improve generation quality. In addition, while HKD performs well on standard-resolution datasets, its potential for high-resolution generation, afforded by its hierarchical design, remains underexplored. Moreover, the explicit spectral decomposition in our framework naturally supports interpretable interventions, making it well-suited for a range of semantic editing tasks. Beyond the frequency-aware manipulations we demonstrate, HKD opens up possibilities for text-guided editing, sptrifute-specific control. While these directions are currently unexplored, they underscore the broader applicability of our framework beyond pure generation. Our framework provides a solid foundation for these future advancements.
current formulation already
Acknowledgments
We would like thank the anonymous reviewers and area chairs for their helpful comments. This work was partially supported by the National Natural Science Foundation of China (No. 12471481, U24A2001), the Science and Technology Comnission of Shanghai Municipality (No. 23ZR 1403000), and the Open Foundation of Key Laboratory Advanced Manufacturing for Optical Systems, CAS (No. KLMMKF2022033, We also acknowledge the support (to Zou) from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A15150 and the central fund from HKU IDS.
10

---

References
[1] Julius Aka, Johannes Brunnemann, Jörg Eiden, Arne Speerforck, and Lars Mikelsons. Balanced neural odes: nonlinear model order reduction and koopman operator approximations. arXiv preprint arXiv:2410.10172 2024.
[2] Hanru Bai and Weiyang Ding. KoNODE: Koopman-driven neural ordinary differential equations with evolving parameters for time series analysis. In Forty-second International Conference on Machine Learning, 2025.
[3] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. Advances in neural information processing systems, 14:8780-8794 2021.
[4] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. arXiv preprint 2tt.:2400.115445 2024.
[5] Shelly Golan, Roy Ganz, and Michael Elad. Enhancing consistency-based image genera- tion via adversarialy-trainn classification and energy-based discrimination. arXiv preprint arXiv:2405.16260, 2024.
[6] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[7] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[8] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffus generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
[9] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[10] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[11] Diederik Kinga, Jimmy Ba Adam, et al. A method for stochastic optimization. International conference on learning representations (ICLR), volume 5. California, 2015.
[12] Alex Krizhevsky, Vinod Nair, Geoffrey Hinton, et al. The CIFAR-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar html, 55(5):2, 2014.
[13] Sangyun Lee, Zinan Lin, and Giulia Fanti. Improving the training of rectified flows. Advances in Neural Information Processing Systems, 37:63082-63 6319,, 2024.
[14] Qiang Liu, Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint 22X1v:2209.14777 2022.
[15] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long, Koopa: Learning non-stationary time series dynamics with Koopman predictors. Advances in neural information processing systems, 36: 16.12227-12290, 2023.
[16] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.1108/ 2024.
[17] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 2021.
[18] Weijian Luo, Zemin Huang, Zhengyang Geng, J Zico Kolter, and Guo-jun Qi. One-step diffusion distillation through score implicit matching. Advances Neural Information Processing 3 17:115777-115400 2024.

---

[19] Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950, 2018.
[20] Alexandre Mauroy, Y Susuki, and Igor Mezic. Koopman operator in systems and control, volume 484. Springer, 2020.
[21] Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41:309-325, 2005.
[22] Ilan Naiman, N Benjamin Erichson, Pu Ren, Michael W Mahoney, and Omri Azencot. Gen- erative modeling of regular and irregular time series data via Koopman VAEs. arXiv preprini arXiv:2310.02619, 2023.
[23] Abhinav Narasingam, Sang Hwan Son, and Joseph Sang-ll Kwon. Data-driven feedback stabilisation of nonlinear systems: Koopman-base based model predictive control. International Journal of Control, 96(3):770-78) 2023.
[24] Samuel E Otto and Clarence W Rowley. Koopman operators for estimation and control of dynamical systems, Annual Review of Control, Robotics, and Autonomous Systems, 4(1):59-87, 2021.
[25] William T Redman, Maria Fonoberova, Ryan Mohr, Ioannis G Kevrekidis, and Igor Mezic. An operator theoretic view on pruning deep neural networks. arXiv preprint arXiv:2110.14855 2021.
[26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted Intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234-241. Springer, 2015.
[27] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprini 2rXiv:2202.00512. 2022.
[28] Gianluigi Silvestri, Luca Ambrogioni, Chieh-Hsin Lai, Yuhta Takida, and Yuki Mitsufuji. Training consistency models with variational noise coupling. arXiv preprint arXiv:2502. 18197, 2025.
[29] Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[30] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025002, 2020.
[31] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 14189, 2023.
[32] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages
PMLR, 2023.
[33] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[34] Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-plas diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[35] Rui Wang. Yihe Dong, Sercan O Arik, and Rose Yu. Koopman neural operator forecaster for time-series with temporal distributional shifts. In The Eleventh International Conference on Learning Representations, 2023.
[36] Yuanchao Xu, Jing Liu, Zhongwei Shen, and Isao Ishikawa. Reinforced data-driven estimation for spectral properties of Koopman semigroup stochastic dynamical systems. arXiv preprint arXiv:2509.044 04265, 2025.
12

---

[37] Yuanchao Xu, Kaidi Shao, Isao Ishikawa, Yuka Hashimoto, Nikos Logothetis, and Zhongwei Shen. A data-driven framework for Koopman semigroup estimation in stochastic dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 35(10)100312) 10 2025.
[38] Yuanchao Xu, Kaidi Shao, Nikos Logothetis, and Zhongwei Shen. ResKoopNet: Learn- ing Koopman representations for complex dynamics with spectral residuals. arXiv preprint arXiv:2501.00701, 2025.
[39] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pages 6613-6623, 2024.
[40] Ronghua Zheng, Hanru Bai, and Weiyang Ding. KooNPro: A variance-aware Koopman probabilliticc model enhanced by neural process for time series forecasting. In The Thirteenth International Conference on Learning Representations, 2025.
[41] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
[42] Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. Slimflow: Training smaller one-step diffusion models with rectified flow. In European Conference on Computer Vision, pages 342-359. Springer, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
In Section 6, we discuss the limitations and possible future works for this paper.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification:
We provide proofs for every theory and proposition in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental settings and training details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: After organizing the code, we will release code to support full reproducibil- ity. Details for reproducing results are described in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ ctttpp..cc/ubbbceec guggguddsssse CoodeSubmis ssi assionolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental settings and training details in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance of the spectrum is analyzed and reported in the Koopman spectral analysis section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide experimental settings and training details in Appendix D. also give the information computer resources in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://neett /ttpar/////eupp//eeee cc httpp////c.//////ccccceceeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The models and datasets used are standard and pose no specific high-risk misuse scenario. No new models with high dual-use potential are released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httpaa:://erpp....eee http://p.....ooofeeeeee///////// for what should or should not be described.
20

---

Appendix Overview
The following lists the structure of the appendix, with links the respective sections.
A Background Knowledge Supplement:
A.1 Related Work
A.2 Koopman Operator: From Seanite-Dimensionee to Finite-Dimennn Dimensional Approximation
Framework Supplement:
B.1 Training Details
B.2 Koopman Spectral Analysis
Theoretical Supplement:
C.1 Proposition: Rationality of the Block Diagonalization of A
C.2 Proposition: Image-Space Trajectory Consistency Loss and its Latent-Space Counter- part
C,3 Formal Statement and Proof of Theorem 3.1
D Experimental Supplement:
D.1 Additional Experimental Results for Sec. 4.1
D.1 Conditional Generation on CIFAR-10
D.I Visual Results
D.I Training Stability Results
D.1 The Wall-clock Time Per Image for Generation Methods
D.2 Additional Experimental Results for Sec. 4.2
D.3 Additional Experimental Results for Sec. 4.3
D.4 More Implementation Details about Experiments
D.5 More Details about Comparison Methods
E Broader Impact
21

---

A Background Knowledge Supplement
A.1 Related Work
One-step Generation for Diffusion Models. Several prior works have explored distilling pretrained diffusion models into efficient one-step generators. Knowledge distillation [17] trains student model to mimic the full sampling trajectory of a pretrained diffusion model. Rectified flows [14] reformulate diffusion sampling as continuous normalizing flow, allowing distillation via ODE simulation. Progressive distillation [27] gradually reduces the number of inference steps during training, enabling faster sampling with minimal performance loss. Recent works further improve distillation-basee generation, such as SiD [41], which reformulates forward diffusion processes as semi-implicit distributions, DMD [39], which enforces the one-step image generator to match the diffusion model at the distribution level by minimizing an approximate KL divergence, and SIM [18], which computes the gradients for a wide class of score-based divergences between a diffusion model and a generator. In parallel, SlimFlow [42] builds on rectified flows by exploring joint compression of inference steps and model size to enhance efficiency. Consistency models [32] offer an alternative approach, learning to map noisy inputs directly to clean outputs. These models can either be trained end-to-end or used as distillation students. Due to their training instability and inefficiency, various improvements have been proposed, such as [4], which fine-tunes consistency model starting from a pretrained diffusion model, [28], which trains consistency models with variational noise coupling. and [5] by enhancing consistency models via adversarialy-tre trained classification and energy- basedd discrimination. However, most of these works ignore the interpretability of the generation process.
Koopman Operators. Over the past two decades, Koopman operator theory has attracted growing interest, enabling progress in dynamical system analysis [36, 38, 37]. control [23], optimization [25], and forecasting [15]. These methods utilize Koopman-based representations to extract globally linearizable structures from nonlinear dynamics, improving interpretability and control. More recently, Koopman operators have been increasingly applied to time series modeling [22, 40J. In addition, several works have leveraged neural networks to approximate Koopman operators. For example, [1] proposes balanced neural ODEs to approximate Koopman operators without predefined dimensionality, while [? ] uses DNNs to learn both the Koopman space and coefficients of selected observables. In our work, similarly employ neural networks to learn observable functions for end-to-end training. To the best of our knowledge, we are the first to introduce Koopman theory into image generation.
A.2 Koopman Operator: From
to Finite-Dimensional Approximation
In theory, the Koopman operator K. is an infiniteddimensional linear operator that governs the evolution of observable functions under nonlinear dynamical system. That is, for dynamical map $ X + X and any observable function g : X - TR., the Koopman operator acts as
(Kg)(z) = gx)())).
(7)
However, this operator acts on an infinite-dimensional function space, which is not directly tractable in practice. To enable computation, the Koopman operator is commonly approximated in a finite-dimensional subspace. This is achieved by selecting a finite set of basis functions [ui, .2...., Um] to represent the space of observables. We then approximate the action of KC on this basis as
Ku(Ø)= Ku(æ),
where K E mR is a finite-dimensional matrix known as the Koopman matrix. Given an observable 9 projected onto the subspace as & é Kg. ui). : (g. u2).. ... (g. un.)T, its evolution under can be approximated by (())) Re E K u(x).. Furthermore, if we consider vector of observables g Pu that is linearly related the basis via some invertible matrix P. the Koopman dynamics in this new coordinate system are given by the similarity transform:
g((())) x PKP-1 gz().
Several numerical algorithms have been developed to compute the Koopman matrix K from data:
(1) Dynamic Mode Decomposition (DMD): DMD is a widely used data-driven technique that approximates the Koopman operator using lincar regression on snapshots of system states. It assumes
22

---

linear relationship between time-shifted observables and solves for the best-fit linear operator K such that - K in the observable space. Variants include Extended DMD (EDMD) and Hankel-DMDD for richer function spaces.
(2) Extended Dynamic Mode Decomposition (HDMDD): EDMD generalizes DMD by applying the method in a lifted feature space defined by nonlinear basis functions (e.g., polynomials, radial basis functions). This corresponds choosing specific u and solving for the best Koopman matrix in this subspace.
(3) Neural Approximations: More recently, neural networks have been used to learn either the Koop- man embedding (i.e., the basis functions u) or the Neural Approximations use trainable architectures to discover a latent space where linear evolution via a Koopman operator holds approximately. They enable scalable and adaptive modeling of complex, high-dimensional dynamics. In our work, we adopt this method approximate the Koopman operators.
B Framework Supplement
B.1
Training Details
We optimize model using a total loss that combines the trajectory consistency loss Licconsiat that supervises intermediate states along the diffusion trajectory with a reconstruction loss
Lrecon dD (( ())) el (oT)) at.)
from the noise ør to enforce the accurate one-step mapping from noise to clean image, i.c.,
-= Licconsiit Lrezon-
Define that x. & Dus (fele-tt)
Eg (x:) then Liaal EL d(z xx), where the elements of T are sampled from U[e,, T) and d(x; y) is a metric composed of 12-distance, i.c., MSE loss, and L.IPS loss in our model:
d(x, y) X CM,E y) CLLLPIS y)
To balance perceptual fidelity and optimization stability, we fix de = and apply an annealing schedule Yy. allowing the model to initially focus on coarse alignment before gradually emphasizing perceptual refinement.
For network architecture, adopt the mature U-Net encoder and decoder backbone used in diffusion models for both the encoder and decoder modules in our formulation to leverage existing diffusion models' design efficacy. To improve training efficiency and reduce mode collapse, further initialize the encoder and decoder with pre-trained weights from diffusion models, which provide structured latent-to-output mappings, and retain rich hierarchical features acquired during large-scale training.
The entire model, including Eg. Dæ. and (A [ M= trained end-to-end using the loss Lusaa-
B.2 Koopman Spectral Analysis: New Lens on Interpreting Diffuusion Dynamics
Our framework offers fundamentally different viewpoint on the analysis of the diffusion generation process via Koopman spectral analysis. Beyond enabling one-step sampling, the proposed formulation naturally provides novel dynamical perspective of diffusion- based generative models through the lens of Koopman spectral analysis. By explicitly modeling the latent evolution system, our approach allows for the application of spectral tools from dynamical systems theory to reveal the underlying spectral structure of generative dynamics and provide new theoretical tools for analyzing and interpreting dynamical modes during sampling.
linear dynamical
Specifically, each block A (1) (i, directly characterizes local dynamical mode, with cigenvalues Ok (i. + ( (i, encoding the growth/decay rate and oscillatory frequency of the diffusion trajectories. The eigenvalues of Koopman matrix K (i. associated with each mode can be computed as
A (1.)),
o ( coos (1.))A))) isn (1.)(())))))
(10)
23

---

providing a principled spectral description of the generative dynamics. The magnitude (*((K-)) (1,5) e (tJ) determines the stability of each mode, while the imaginary part of ( (1) (i, captures the frequency modes of oscillations. Benefiting from the block-diagonal structure of (i,)), our framework enables direct analysis of its spectral components to interpret the generative behavior of the model by directly observing the values of aj (i, and B (i. j). (1) (5) (i.j) and imaginary parts $ (i,j) and low a (i. and 2 (i. j)
Specifically, the high real parts O' of each block correspond to low- and high-frequency modes in the generation process, respectively.
This explicit spectral interpretability further allows for controllable image editing by selectively modifying certain frequency components, enabling targeted modification of structural or fine-grained details in the generated images.
C Theoretical Supplement
C.1 Proposition: Rationality of the Block Diagonalization A
We present the following proposition ensure the rationality of the block diagonalization of the linear operator A.
Proposition C.1. We define the set for the l-layer encoders Sg - ( (8= & 0..... 0 f
((0)++b( and the set for the decoders So - {D Dio..... .......D220D) Di i w s+bl b/) Then for & E Sp E SD and the Koopman matrix K, there exists another set of Sg, D E Sp
and K Rmxm that K- I diag {A1, AAx and A,
0g -
Bi
such that,
03
De De- = De- 1K-
(11)
Proof. We assume that the eigenvalue decomposition of matrix K in the complex field is K PAP-1 where A A diagfA1. Å1 ... , År,, Ar+1 Ar+2, Am-r3 with A X1, 1 S i the conjugated imaginary eigenvalues and A,, + S S m r the real eigenvalues.
Re X 1
Im X
Let
A
,1
I
i
and
A,
Im A
Re A;
A
0
+
1
we use them to define
K.
Fur-
A
-
thermore,
let
P
Re pl, Im .I. Pr. Im Pr:
pr+1 Pr+1
Pm-r
Pm-r
2
2
2
where
P
[11 PI, .rr Pr. Pr+1: Pr+2. Pm-rl
and
Q
V2
Re pl. Im pl,
Re pl. Im pl. pl+s Peti Pm-r pm-r
where p-1
2
pl.pl. ... .Pi D/. p P/42. necessarily identity.
Pm_.JT. Note that pt's 's are row vectors and PQ is NOT
Last but not least, let W (8) A (o) and ( Qb(0) in and w
w P) P in D then
De i6 1KK ---(k-)((()) 1((x),
=D, o ..... D2 D2 (D) -(K 1 (w/3))
b
(6) + +bb (D) + ((0))
-D.o... D2 (2 (D) Pe -ik -ik- (8) ( (())))
(D))
=D Pe (k--(kk(@@ [(())
De 1(K - ((()))())
where = holds due
P
Pi
-tA LA,
- Si m and D./V2
1
v
24

---

2Re !
[V2Re
22l pcl
-A;
(12)
21m p.
2 |Re pi
c (-((a-11
cos(
(t,,,)
-taas- sin(-t:83
Re p! Im p
Im pi]
(13)
---1(au- sin( -1,3)
e e-1a--1 cos( [1Bb)]
[o0((133)
(in()
Re p! -Im p!_
=2c (la, 1))
[Re Ps
Im p:]
(14)
sin(t,3.)
(t,(),
= 1(a--1)
Re pt
Im pi]
1
c
t/
0
[
-il
Re p! Im p!
(15)
i
eBi
- -(o
= 1a, [ps
prl
c
13,i
e
0
p !!
=
lps
p
(16)
for Ssisr.
C.2 Proposition: Image-Space Trajectory Consistency Loss and its Latent-Space Counterpart
In this part, we first propose the informal version of the equivalence between image-space trajectory consistency loss and its latent-space counterpart under specific assumptions. Then we present the formal version.
Proposition C.2 (Equivalence of Trajectory Consistency Losses under Structural Assumptions, Informal). Assume that the decoder H. = F o Da, VF is differentiable with respect each Koopman state w and satisfies isotropy and level disentanglement properties with respect to the Koopman decomposition. Then, the image-space trajectory consistency loss:
[d [d (Do(fele- c/2 (1.))
Li-consist Be- TI
(17)
is approximately equivalent to the following latent-space loss:
L
Σ
0F OFo2
D Ow!!)
-4 || 11
( (mx), (1-T))
1 22
Cr-consist RS (E
WT
(18)
where .(...) denotes suitable distance metric such as squared Log-oor or perceptual distances like LPIPS with a corresponding feature extractor F, i.e. d(x, y) & - The approximation holds up small residual terms governed by the assumptions detailed in the derivation.
Proof. Our proof relies three key assumptions. The assumptions are (1) the level-based Koopman states w are disentangled across levels labeled by I, (2) the mapping H isotropic w.r.f. the Koopman states, and (3) the spectrums of the Koopman observables are close to 1.
Intuitively, it requires (1) Koopman states in different levels depict different features, (2) each Koopman state has the same unit and contributes equally to the loss function, and (3) the Koopman space sufficiently models the latent evolution. detailed version of the derivation, with the divergence from the assumptions included, is
Ex (T,,),,T)) T), ram
[o D
le -e
800) (x:)]+]]
FoD [m1
[ 1= 11
(19)
E
H(w)) - H(w[-1)
1/3 +o(p),
2
(20)
[=]
0H
| 2) Ed (ax)) - w/ 1| + olp + E),
11)
(21)
(t)(
1=1
(*0.)
8H
((l) c (x,) w 1
E
+ +oppe+0))
(22)
2
25

---

where parameters wi - (c-tta E/ (+:)) U ( correspond the assumptions above respectively. The error term consists of three small-value terms where P is the extent of entanglement among levels, is the amount of anisotropy of H. and ó is the range of Koopman spectrum.
(1) 1 Jk=t+1 1 and the equations (-+) and (m=))
Proposition C.3 (Equivalence of Trajectory Consistency Losses under Structural Assumptions, For-
mal). Under the assumptions that (1) the level-based Koopman states w. are strongly disentangled across levels with an upper bound of correlations p. (2) the mapping H is isotropic w.r.1. the Koopman states, with a lower bound of (7 with is non-zero weights for sufficient models, and (3) the spectrums of the Koopman observables are close to I, which leads to the fact that the maximal spectrum a of matrix A has small positive value. Following the notations w H. F, and d in Proposition C.2., we have
[al (Do(fele--)a al1) (2))) 22.
L Li-consist -Bn-Z(/e.T
(23)
1 (1-
(1-P2))
p²L 4 e-aa.
Σn En..EN (xx)) (t--T) (1) WT
2
(24)
which showcases that the minimization of Lt-consist enforces the minimization of estimation error in the Koopman space.
Proof. Under the given assumptions, we have
2
Be- Te::: FFo D(wL) F FD D/woll12 =E E Σ [H(wi) HH(m)-1) (25)
2
Bl]H(um) - H(wr-1112 +
B(H(w))) - H(wm) (1{((u))))((( - H(wxk-1))
(26)
1= l7
p²L.
4
1
[l(H(ww)) - H(W
(27)
4
(xx)
L²
( Eelee ( ol1) (axt) cx)( u0!' 100'
1
(28)
(1x)
p²
L Σ
1)) B11E8" (xx) - clt A
(4)
e
w
(29)
4
where parameters wi 2 (e & (x:) (I:)) u (w) @ 3
- In addition, the inequality holds under the assumption (1) where the maximal entanglement of Koopman observables at different levels
max LA=E,...
Corr [H((w)) - H(w;-1), H(w)) - H(wx-1)] is small. On the other hand,
L
(***)
the inequality and > holds when r7(0) A An (1 ( + Aw) - H(w ||Aw 12 is the minimal spectrum of H. and & (A) is the spectral radius of matrix A (1)
Under the assumptions, there holds p + 0+ and a -> 0+. Consequently, minimizing the left-hand side (as in the proposed algorithm) is equivalent to the minimization of Koopman observables to the right-hand side.
C.3 Formal Statement and Proof of Theorem 3.1
In this section, we provide the formal statement and proof of Theorem 3.1. Before that, we first give Definition C. Proposition C.3, and Lemma C. to introduce Theorem 3.1.
In order to illustrate the feasibility of our proposed method, we propose to measure the complexity of functions by the estimation accuracy using the simplicial complex created by Delaunay triangulation.
26

---

Def. C.1 first introduces the simplicial enror, which reflects the complexity of the subset S2 as EN (S2) if and only if dim f2 m and 9 is convex.
Definition C.1 (The Simplicial Error). For a compact subset of an m-dimensional manifold S C Mm C CR,, the simplicial error of it is (N2) - infsca, [Sm(E)]= (=( where E is a set of N generic point samples in 22 { _ supzen min d(x, s) is the enrrr of the m-dimensional Delaunay triangulation Sm (=) of the point set E, and d(x, s) & infyes || - yll is the minimal distance from simplex The norm || is commonly Lg-norm.
We then propose measure the theoretical error bounds for network estimation by the error between its graph and a closest simplicial complex. Firstly, Prop. C.4 defines the function complexity. which intuitively influenced by the complexity of its domain and range sets. The conclusion in Lem. C.1 further shows bound for the estimation error of the function by a network using ReLU activations. The error is closely related to the function complexity we defined, which is coherent with the common knowledge, as the network creates simplicial complex with each activation creating a surface between simplicials.
Proposition C.4. The function simplicial error is (f) e ex(i(a, f(z)))a Q)) for f: -. We then have max(En (I), (T)) I (f( & infsco, SB) (T())
Proof. We first prove the left side.
ex(f)=
inf
sup
min d(0,s)
>
inf ECG 1Sm N
sup min d(PnO, DEE smm
ECB 1s.. EE)l
OCE (=)
inf P SS.. [Pa
min O Pa (P)E)
sup
d(Pa0, P(os) = =
where Pa is the projector to Ω. Similarly,
eN(s) = inf
sup min d(0,s) O0E sE.,..) (E) 2
inf BCO (E)
sup min d(PrO, Prs)
sce |s.. (B)=
inf PrECO [P;
sup
min
d[Pr0,
inf
$ 5) (T) 2 (I).
BCR
(BI=N
On the other hand, the right-hand side holds due to,
EN(F)
inf BCe 1S.. (E)|=N
sup min inf Leee (E)m.E) PEs
[||Pa(0 (((lP(00=))1l111 1/1
inf sce [3)| N
sup min inf Lee (.. (E) PE#
[==
inf
(()
Eco
18m. (B)=
where = holds as the norm inside inf is a constant w.r.t. 6, y and S.
Lemma C.1. The estimation of function f 22 + I by network F with n. ReLU activations (12n_((m+2)(())) 2 1 1-6 8 where indicates the amount undertrained.
satisfies P (I) - [())] <
Proof. With reference the universal approximation theorem, each (m - 1)- dimensional face of the simplex matches a cuspidal point of a ReLU activation. For a network with n., activations, the maximal number of faces is n. which means there are at most [2n_ simplices in the simplicial m
complex, as each simplex has m 1 faces and all faces clamped between (exactly) two simplices.
For activations other than ReL.U, one may use intrinsic Delaunay triangulation to replace the Delaunay triangulation.
27

---

Theorem C.1 (Formal Statement of Theorem 3.1). If the noise space ær E E C R and the Koopman space V C I. D are compact, the ideal estimation error of our proposed HKD model with encoder & and decoder 'D networks of na/2 activation functions is smaller than that of an end-to-end one-step model F with na activation functions, i.e, erTuRD S eer stepme-step + O(k) holds for any one-step diffusion where
сттнкр = =[m_/(2+1]](EE) + n ((ma/(D+1))()) and ETT eTTone-step = +12ma/(22+11]
for the one-step mapping F, the encoder & and decoder in the framework... The additional Koopman error K. is
K L max
2v/3am Poup
2//3mmm Pint
+ o(m=5),
NNC 2m V3/3mm
22mm
V [N -2m 8) Paat 33mm2
which is small when the Koopman dimension m is sufficient and the dataset size N is large enough. Here, parameter a is the standard deviation of the observables, ô is O when the network is perfectly trained, (Dint, psup) the true spectral range of the Koopman operator and r + -00 when the m-dimensional Koopman process adequately model the Koopman space.
Proof. Under the assumption of AE-based data compression methods, we assume that the data are approximately on an m-dimensional manifold MM C Ra** defined by a bijection 3 : M <> R and that the projection of the dataset on the manifold c M. Let v 3(12) be compact subset of R. that encodes the dataset.
We consider (p)(p) as function w.r.t spatial position p and define operators F. ax(()) + (10) G. ax, + g² (1) V log p. (mx hence we have,
(Fxw,) = [ - 89, öt 3(F - G (fxx))tt
which satisfies the definition of Koopman process where the operator is Kr2 = I + 3 9,)
Letting & (Ka. K),,, for the observables. We have the bound that
P (IIKK, K.) S Ne 1 2 1 8,
where = -2/30m22(k /
N66 2m 8)
- v3 om?]
+ o(m =mm
Here, the bound factor K₁ is related to time t while it has uniform bound factor,
K (8) A sup Ke max tE(o.T]
2v/amm Poup
2v pint
+ o(m omm=5))
V ( 2m 8) 3)-m3mm?
v (8-222m 2m p
33rm²
where Pint A inf, r(C,) and Psup sup,r r(Ka).
The estimated Koopman model with an encoder & estimating mapping 3 and decoder D estimating 8-1 1 (with each of them having na/2 activations) has probabilistic error bound,
PP Dz xoll S {),, where 2, =zT + [ (K. 1)2,, ds, zт E(xT);
and vp 3 Yo: y. = yr + 1' (Ki -T)y, ds, yr BOTT
2P( - 3 'zoll + 1 -2-
20 - 3 2 "yol (
SP(IDD2 - 3 3-'zol < | (8 (8-1))
IP[[ -yoll S (6/3)) ((o)))((oT___ - -1)/p)
P[[ yr S ببب (3)),
2( 6/3)3 2 1-8.
Here, 2 holds due to Lemma and ブ holds when
(8 (3-1) + r 1811 (66/3) (( 11/p + r(8-1)) (0031)))0()) oo . no (8)
E = = no
28

---

where r(3 1) is the spectrum radius of F-1 and the bound for || -Voll is given by
|| -vll =| yrll + |,, (K, I) y.)ds
+ ff II(Ks ko))=olds,
2| Vrl + p / ||2, - yolldo + kε(T - 1),
12.--yllds
where p - max Pint, Psup }} and Ç is a bound of ||z|||- Let Dr satisfies = - yr and dit. pD; Kς which leads to de || 2,11 S D, as they are positive trajectories. The
solution to Dj D. KS [e - -1] Dye
For ideal K2 and 8, e dominated by et # (8 + (8) as p is small and r(B = 1.
On the other hand, the one-step methods aim at estimating to = F( or f. wt by a network with na activation functions. The error satisfies, according to Lemma 1,
IP (IIF(xr) æoll S ef2ma/ (F)) 2 1 6.
Note that ideally, 2T E = C R zo E Ω C Mm and 8zp V C R", where Egr and A compact, hence
S (F),
(30)
4²41
2 max
200
(=),
g
2no n2+1
(92)
< Proposition 1
(31)
- (92)
+ Ω is compact
(32)
=I] ( ([), (V)]l
E and v compact
(33)
where A satisfies FΛ ( E (12)
[ (V)
(34)
29 Ba (8-1)
ET ,,
Proposition
(35)
(36)
Using Lem. C.I, we may reach the conclusion in Thm. C.I, which shows that if the neural network models are trained properly, the theoretical error bound of the proposed method is smaller than that of any other end-to-end one-step method.
D Experimental Supplement
D.1 Additional Experimental Results for Sec. 4.1
Conditional Generation on CIFAR-10. In this part, provide the conditional generation results on CIFAR-10 to show the generalization ability of our framework on class-conditional tasks. The results are provided in Tab. 4.
Table 4: Class-conditional sample quality on CIFAR-10 dataset.
Methods
NFE(J)
FID(L)
Score SDE
2000
2.20
EDM
35
1.79
Visual Results. In this section, we provide ad-
DMD (w/o REG)
1
5.58
ditional qualitative results on CIFAR-10 (uncon- ditional), FFHQ (unconditional), and CIFAR- 10 (conditional) to further demonstrate the vi- sual quality and effectiveness of our proposed method. The results are presented Fig. 7, Fig. 8, Fig. 9, respectively.
DMD (w/o KL)
1
3.82
DMD
1
2.66
HKD
1
2.77
Training Stability Results. We include the results training HKD five times independently on the CIFAR-10 dataset in Tab. 5. The FID scores and their standard deviation are reported in the table below. We summarize the FID results over every 10 epochs during training across the five runs, demonstrating stable convergence behavior.
29

---

Figure 7: The additional visualization of image generations from HKD trained by the CIFARIO dataset.
Figure 8: The additional visualization of image generations from HKD trained by the FFHQ dataset.
The Wall-clock Time Per Image for Generation Methods. We provide the wall-clock time per image for different generation methods in Tab. 6, measured on an NVIDIA V100 GPU. These methods were trained on the CIFAR-10 dataset.
D.2 Additional Experimental Results for Sec. 4.2
We provide additional visualization results of image models on the CIFAR-10 dataset in Fig. 10, and on the FFHQ dataset in Fig. 11. As shown in Fig. 11, the reconstructed images using low-frequency components primarily capture the semantic structure of the face. The mid-frequency components contribute the overall contour and shape details of the face, while the high-frequency components responsible for fine-grained details such as facial hair.
30

---

Figure 9: The additional visualization of conditional image generations from HKD trained by the CIFARIO dataset.
Figure 10: The additional visualization of image modes for the CIFARIO dataset. The rows correspond to those the manuscript.
D.3 Additional Experimental Results for Sec. 4.3
Additional Results One-step Image Editing via Frequency-aware Interventions. We provide the additional results on the one-step image editing experiment via frequeney-awand interventions along the diffusion trajectory. The results provided in Fig. 12, 13.
D.4 More Implementation Details about Experiments for Sec. 4.1
Network Architecture. In our experiment, both the encoder and decoder designed to follow the architecture of the EDM [8] encoder and decoder, leveraging the proven effectiveness of existing diffusion model designs, similar to consistency models [32]. Inspired by ECM [4], we utilize EDM's pre-trained weights for our encoder and decoder modules to provide structured latent-to-output mappings and prevent mode collapse, retaining rich hierarchical features acquired during large-scale training. For each linear operator A (1) (i, which adopts a block-diagonal structure, we perform vectorization by concatenating the real and imaginary parts parts-placing the real components in the first half of the vector and the imaginary components the second half. The optimization is then conducted over the elements of this vectorized form. We train our model end-to-end.
31

---

Table FID scores (mean standard deviation) during training across epochs.
Epoch 0
Epoch 10
Epoch 20
FID (Mean Std)
11.82+0.29
5.70+0.21
4.71
Epoch 30
Epoch 40
Epoch 50
FID (Mean + Std)
4.09+0. 18
3.52+0.11
3.31=0.03
Epoch 52
Epoch 53
Before Training
FID (Mean Std)
3.31
3.30#0.02
447.13
Table 6: NFE and per-image latency across different generation methods.
DDPM
Score SDE
DDIM
DDIM
NFE Latency
1000
2000
100
15.32s
45.89s
0.51s
2.35s
EDM
EDM
NFE
35
15
Latency
0.69s
0.30s
0.26s
0.28s
CD (LPIPS)
DMD
RF-1
RF-2
NFE
Latency
0.02s
0.03s
0.03s
0.03s
RF-3
RF-2++
CT (LPIPS)
iCT
NFE
Latency
0.03s
0.03s
0.02s
0.02s
iCT-deep
ECM
HKD
NFE
Latency
0.02s
0.02s
0.03s
Compared conventional diffusion models, our framework introduces only one additional parameter component, the linear operator A. However, due to the structured block-diagonal assumption imposed on A, the overall increase in parameter count minimal.
In addition, the EDM U-Net architecture supports augment labelllbell_e as an input, which we set to None. Additionally, in EDM, the U-Net receives the noise scale a as an input. Although the observable functions in our model are inherently time- iene-nndeenddet, we follow the EDM convention and feed the time-dependent (t) into the encoder to maintain architectural compatibility and enhance representational capacity. In contrast, our decoder only operates on the final timestep; therefore, consistently receives (1 1) as input. The EDM architecture also allows for class labels as input. For unconditional generation tasks on CIFAR-10 and FFHQ, set class labels None. For conditional generation on CIFAR-10, we adopt the same label configuration as EDM.
class_labels
Hyperparameter Setting. We conduct experiments using 8 NVIDIA V100 GPUs, with batch size of 256 for CIFAR-10 dataset and 64 for the FFHQ dataset. The loss function weights are
set A2 = and \ 10 (omma spech/ spcc) where A1 decays exponentially over the course of training. For the implementation of the trajectory consistency loss, we employ a Monte Carlo sampling strategy: in each training iteration, four intermediate timesteps are randomly sampled uniformly, and the loss is computed as the average over these sampled time points.
D.5 More Details about Comparison Methods
To provide comprehensive comparison, we categorize the baseline methods into three groups: multi- step diffusion models, distillation-bast one-step models, and consistency- based models. Below, we summarize the key characteristics and mechanisms of each method.
Multi-step Diffusion Baselines. DDPM (Denoising Diffusion Probabilistic Models) [7]
foundational diffusion model that learns to reverse a fixed Markovian noising process via iterative
32

---

Figure 11: The visualization of image modes for the FFHQ dataset. The rows correspond to those in the manuscript. The third row, indicating the low-frequency semantics, underwent a downsampling to avoid the uncanny valley effect.
Figure 12:
The visualization of frequency-awar interventions of images from CIFARIO. The rows correspond to the columns in the manuscript, meaning the original image for row I, frequency-awann editing at factors of 10%, 20%, 50%, 80%, and 90%, frequency-agnostic editing at factor of 90%, and the reference image, respectively.
denoising. The model is trained using mean squared error (MSE) loss between predicted and actual noise. Sampling typically requires hundreds to thousands of steps.
2. Score SDE (Score-based Generative Modeling via Stochastic Differential Equations) [33] A Generalization towards continuous time via stochastic differential equations (SDEs). The model trained via score matching and enables sampling using reverse-time SDE ODE solvers, improving sample quality and flexibility.
3. DDIM (Denoising Diffusion Implicit Models) [30] A non-Markovian and deterministic variant of DDPM that introduces sampling ODE, enabling fast sampling with fewer steps while preserving high visual fidelity. It is compatible with pre-trained DDPM models without requiring retraining.
33

---

Figure 13: The visualization of frequency-aware interventions of images from FFHQ. The rows correspond to the columns in the manuscript.
4. EDM (Elucidating the Design Space of Diffusion Models) [8] A state-of-the-art diffusion frame- work with improved noise scheduling (log-normal distribution) and score network preconditioning. achieves superior performance across datasets and serves as the backbone in our proposed framework.
Distillation-basee One-step Models. 1. KD (Knowledge Distillation) [17] A straightforward distillation approach where student model learns to replicate the output of a pre-trained diffusion teacher via MSE loss. While efficient, performance
degrade with large step-size reductions.
PD (Progressive Distillation) [27] Gradually distills a high-step teacher into low-step student via intermediate models, reducing the number of steps incrementally. This approach improves stability and fidelity over direct distillation.
3. CD (LPIPS) (Consistency Distillation with Perceptual Loss) [32] Extends distillation by employ- ing perceptual similarity metries (LPIPS) the training objective. It improves visual quality by preserving perceptual features across time steps.
4. DMD (Distribution Matching Distillation) [39] Aims to directly match the student's output distribution to that the teacher using divergence- based objectives (c.g.. KL divergence). This results in more accurate distribution alignment and improved sample diversity.
ReFlow Series (Rectified Flow and Distilled Variants) [14, 13] Utilizes rectified flow framework to optimize ODE trajectories for efficient sampling. Variants such as 1-, 2-, and 3-Rectified Flow, and 2- Rectified Flow++ offer different trade-off's between speed and quality. Distilled versions further reduce sampling cost to step with minimal performance loss.
Consistency-based Models. 1. CT / iCT / iCT-deep (Consistency Training) [32, 31] CT models are trained to produce consistent outputs when given different noise levels. Improved variants (iCT and iCT-deep) enforce deeper consistency across time steps, using perceptual and cosine similarity losses to enhance training stability and sample quality.
2. ECM (Fasy Consistency Models) [4] Simplifies consistency training by initializing model parame- ters with those from pretrained score-based model. This reduces training complexity and improves convergence without requiring complex schedule tuning.
34

---

E
Broader Impact
The proposed Hierarchical Koopman Diffusion (HKD) framework introduces a novel approach to balancing sampling efficiency and interpretability in generative modeling, which may have significant and multifaceted societal impact. By enabling one-step image synthesis with transparent generative dynamics, HKD has the potential to make image generation systems more accessible, controllable, and explainable attributes that are essential for responsible deployment in sensitive domains such as design, and scientific simulation.
The explicit modeling of generative trajectories and access intermediate states could enhance human-AI collaboration, enabling users to guide or edit synthetic content with semantic intent, thus reducing risks of undesired generation and improving user trust. Furthermore, the spectral inter- pretability introduced by our framework can offer insights into the internal mechanics of generative models, fostering research in model debugging, safety validation, and faimess auditing.
In the broader landscape, our work contributes to the growing field of explainable generative AI, reinforcing the possibility of building models that are not only performant but also transparent and controllable. We hope this inspires further research toward interpretable, reliable, and socially responsible generative systems.
35


Reviews:
[{"review_id": "8qawZg7y0N", "reviewer": "Reviewer_Vr9m", "summary": "This paper introduces Hierarchical Koopman Diffusion (HKD) to bridge the gap between fast, one-step generative models and the interpretable, controllable trajectories of traditional diffusion models. It is based on the Koopman operator theory to linearize the nonlinear diffusion ODE dynamics. \nThe authors propose a U-Net-like architecture to encode a noisy image x_t into a hierarchical set of latent observables, which are then evolved in a single step via a learned linear Koopman operator. A corresponding decoder then maps these evolved latents back to the image space. This paper shows that this framework not only enables one-step generation but also provides interpretability through spectral analysis of the learned Koopman operators. The paper conducts experiments on CIFAR-10 and FFHQ, demonstrating competitive FID scores.", "questions": "- How critical is the initialization from a pre-trained EDM model (U-Net)?\n- The authors argue against supervising in the latent space because it \"reduces training flexibility and introduces gradients that poorly reflect perceptual discrepancies\" (line 158). Can you please provide more evidence?\n- Could one not achieve similar frequency-based editing by performing a Fourier transform on the feature maps at different levels of a standard U-Net-based one-step model and manipulating those coefficients?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- The application of Koopman operator theory to accelerate diffusion model sampling is a novel direction. \n- One of the strengths is the successful demonstration of interpretability. The spectral analysis in Figure 4, showing a correspondence between eigenvalues and semantic image structures, is interesting.\n- HKD achieves competitive FID scores for a one-step generator on both CIFAR-10 (3.30) and FFHQ (5.70).", "was_revised": true, "final_justification": "The response addressed my concerns. I'd maintain my score and recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the positive feedback and for recognizing the novelty, interpretability, and performance of our approach.\n\n***\n\n**[Q1. Importance of the initialization from a pre-trained EDM model]**\n\nThe initialization from a pre-trained EDM model is important for improving training efficiency of our framework, as it provides well-structured feature representations in the early stages, which facilitates faster convergence. However, this initialization is not essential to the framework’s design: our model can be trained from scratch and the pre-trained model is used purely to reduce training time. Specifically, on the CIFAR-10 dataset, training with a pretrained model takes only 2–3 days on 8×V100 GPUs, whereas training from scratch requires approximately one week.\n\n------\n\n**[Q2. Justification against supervision in the latent space]**\n\nWe apologize for the confusion. We will clarify this point more explicitly in the revised version. The LPIPS study [1] provides evidence that accurately capturing perceptual discrepancies requires a carefully designed and fixed feature extractor. Supervising in the Koopman space with fixed feature extractor $f$ (e.g., VGG) is equivalent to applying a perceptual loss under the extractor $f\\circ\\mathcal E$, where $\\mathcal E$ is trainable encoder. This training of $\\mathcal E$ lacks the perceptual alignment of pretrained perceptual encoders, leading to gradients that may poorly reflect actual perceptual differences in image space.\n\nFurthermore, enforcing supervision in the latent space restricts the encoder's ability to adapt freely to downstream tasks. It may hinder the model’s capacity to learn more expressive or task-specific features. For these reasons, we opt for supervision at the image level, which allows the encoder to evolve flexibly and ensures better alignment with perceptual quality.\n\n[1] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" *Proceedings of the IEEE conference on computer vision and pattern recognition*. 2018.\n\n***\n\n**[Q3. Achieving similar frequency-based editing via a Fourier transform on the feature maps]**\n\nYes, this is a very interesting idea. In principle, we could achieve frequency-based editing by applying Fourier transforms to feature maps at different levels of a standard U-Net-based one-step model and manipulating the corresponding frequency coefficients. \n\nHowever, directly injecting high-frequency components without an intermediate evolution process may lead to distortions, because even small spatial misalignments between the source and target images can disrupt the high-frequency details. In contrast, the Koopman-based spectral decomposition provides a structured intermediate evolution, which helps preserve fine details and ensures more stable frequency-level control."}, {"review_id": "BlDe82qWrl", "reviewer": "Reviewer_q79C", "summary": "This paper proposes Hierarchical Koopman Diffusion (HKD), a framework that leverages Koopman operator theory to enable one-step sampling for diffusion models while preserving interpretable generative trajectories. It also achieves competitive results for one-step generation.", "questions": "1.\tDo you have results on ImageNet at 64×64 or 256×256 resolutions? It would be helpful to see whether the method scales effectively to larger and more diverse datasets.\n2.\tHow do the extracted features impact the effectiveness of the Koopman operator in this context? It would be valuable to include more discussion on what constitutes good feature extraction for achieving better one-step results.\n3.\tIs it necessary to extract multi-scale features from images to apply the Koopman operator theory effectively? For better clarity, I suggest including a simple toy example to illustrate how the Koopman framework works in practice — for example, mapping a standard Gaussian to another simple distribution.\n4. Does the Koopman framework require pretrained models, or can it be trained entirely from scratch?", "limitations": "See weakness and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-structured, with a solid theoretical foundation and clear technical novelty. The application of Koopman operator theory to diffusion distillation is particularly interesting. The authors also provide extensive experimental validation, detailed theoretical derivations, and ablation studies.\n\n\nWeaknesses:\nThe paper does not include results on larger or more challenging datasets such as ImageNet, leaving the method’s scalability and generalization to more complex data unproven. While the reported results are competitive, they do not reach state-of-the-art performance, raising questions about the practical benefit of introducing the Koopman operator framework compared to existing simpler or more established approaches.", "was_revised": true, "final_justification": "The response addressed most of my questions and concerns. I prefer to maintain my score.", "rebuttal": "Thank you sincerely for your thoughtful feedback. We deeply appreciate the recognition of our work’s theoretical foundation, methodological novelty, and comprehensive experimental validation.\n\n***\n\n**[W1. Results on larger or more challenging datasets]**\n\nTo demonstrate the generalization ability of our framework to larger and more complex datasets, we conducted experiments on the more complex FFHQ dataset at a resolution of 64×64. The results, presented in Table 2, highlight the potential of our framework on higher-resolution and more challenging data. \n\nWhile we have not yet included results on even larger datasets such as ImageNet, we believe our approach is inherently scalable due to its modular Koopman-based formulation, which allows effective training across tasks of different scales and compatibility with various feature extractors, such as DiT and other network architectures, which enables it to adapt to more complex image generation tasks. Extending our method to larger-scale datasets remains an important and promising direction for future work.\n\n***\n\n**[W2. The practical benefit of introducing the Koopman operator framework]**\n\nWe appreciate the reviewer’s thoughtful comment on the practical utility and comparative advantage of introducing the Koopman operator framework. We would like to clarify that, beyond achieving competitive one-step generation performance, our framework offers additional benefits in terms of interpretability and controllability by preserving the diffusion trajectory, which is generally unavailable in existing one-step generation approaches. This property allows the use of training-free sampling-path control techniques, as shown in Figure 5.\n\nMoreover, our method achieves competitive performance without extensive hyperparameter tuning, highlighting its practical applicability. Compared to existing one-step methods, our framework only introduces an additional Koopman module, which requires learning the matrix $\\boldsymbol{A}$. However, the design of this matrix is tridiagonal, meaning it does not significantly increase the number of parameters. From an implementation perspective, once the Koopman evolution function is written, it can be easily integrated into the existing encoder-decoder pipeline with just one additional line of code. Thus, our framework remains both lightweight and effective, building upon established methodologies while introducing minimal additional complexity.\n\n***\n\n**[Q1. Results on ImageNet at 64×64 or 256×256 resolutions]**\n\nWe apologize for not having results on ImageNet at 64×64 or 256×256 resolutions. Due to the computational cost, training on ImageNet 64x64 with 8×A100 GPUs would require more than one week. \n\nIn addition to experiments on the CIFAR-10 dataset, we demonstrate the generalization ability of our method on the more complex and higher-resolution FFHQ dataset (64×64) in the manuscript. These results highlight the potential of our framework on more challenging data. \n\n***\n\n**[Q2 (I). Impact of feature quality on Koopman modeling]**\n\nThank you for your insightful question. In our framework, the features extracted by the encoder help identify representative and expressive modes (as shown in Figure 4 of the manuscript, where different semantic features are associated with different modes). The Koopman operator focuses on assigning spectral components to these features (or modes) to learn how they evolve over time. \n\nTherefore, the extracted features impact the Koopman framework by influencing the tractability of the identification of the spectrum. **It can be concluded that features with more simplex trends of evolution ease the training of the Koopman operator.** For instance, a good estimation of the diffusion noise would be excellent as its trend is as simple as dropping throughout the evolution. The requirement is mainly caused by the simplification we applied in Appendix C.1 where we formulated the Koopman operator into a tridiagonal matrix. The formulation complicates the depiction of features with multiple dynamics, such as border features containing both semantic evidence and sharp details, which correspond to the middle and high frequency features in Fig. 4(c) and (d), respectively. Having these kinds of features would result in the best spectra to model their evolution, but also a loss in performance. \n\nHowever, more redundant dimensions in the Koopman space can solve the problem by selecting better features for reconstruction. Therefore, given a sufficient number of Koopman dimensions, the overall result mainly depends on the representativeness of the extracted features, just as any one-step model would do, which leads to the second requirement of good features.\n\n**[Q2 (II). What constitutes good feature extraction for achieving better one-step results]**\n\nRegarding the impact on the overall one-step results, good features require two properties, i.e., **(1) each feature has a simplex evolution trend as in one of the modes visualized in Fig. 4, and (2) features combined are sufficiently representative.** The necessity of the second property can be seen in the rows (a)-(d) in Fig. 4 as missing representative features degrade the image quality. These properties require the encoder to disentangle multiple dynamics and preserve as many faithful image characteristics as possible. The first requirement, as discussed above, improves the effectiveness of the Koopman operator but is harder to achieve as the pre-trained extractor does not initially satisfy the condition. Fortunately, the impact can be tackled by a sufficient number of Koopman dimensions. The second requirement, however, dominates the quality of one-step image generation and requires a better pre-trained model and fine-tuning of hyperparameters. \n\n***\n\n**[Q3. Necessity of multi-scale features and a simple toy example]**\n\nExtracting multi-scale features is not of great necessity to applying the Koopman theory. As shown in the ablation study, hierarchical modeling improves generation quality by helping organize dynamical modes across levels. However, even without hierarchical modeling, the model can still learn a sufficiently expressive latent representation at a single resolution. This representation captures the full set of dynamical modes required for accurate one-step prediction under Koopman evolution.\n\nRegarding the visualization of the toy example, we will include it in the Appendix to demonstrate the distribution evolution of one pixel during the revision to showcase the evolution estimated by the Koopman framework.\n\nThe Koopman-space trajectory of each pixel, evolving from a standard Gaussian to its true data distribution, closely mirrors the local frequency spectral evolution patterns shown in Figure 4(e).\n\nIn this figure, we use a tinted background to illustrate the distribution evolution from time $T$ to $0$ where the color band at time $t$ represents the momentary probability density of a Koopman state. The density function is estimated by the histogram of a certain element in the state from 1000 randomly sampled images. In the foreground, we plot the trajectories corresponding to the values of this element using the Koopman framework.\n\n***\n\n**[Q4. Pretraining dependency and trainability from scratch]**\n\nThe Koopman framework can be trained entirely from scratch. Initializing with a pretrained model is solely intended to improve training efficiency and accelerate the convergence of our model. Specifically, on the CIFAR-10 dataset, training with a pretrained model takes only 2–3 days on 8×V100 GPUs, whereas training from scratch requires approximately one week. This efficiency gain arises because the pretrained model provides the Koopman module with reasonably good features at the early stage of training, which speeds up the learning process. However, if given sufficient training time, the encoder can still learn to extract representative features without any pretraining."}, {"review_id": "hiW6NN7zAv", "reviewer": "Reviewer_MPjE", "summary": "The paper introduces a novel one-step diffusion model based on Koopman operators. Unlike existing few/one-step diffusion models, the proposed Hierachical Koopman Diffusion (HKD) offers several advantages, including improved interpretability during generation and the elimination of iterative sampling for ODE solving.\n\nThe HKD framework adopts a hierarchical architecture that projects a noisy input $x_t$ into a set of latents $z_t$ at different spatial resolutions, referred to as Koopman subspaces. The original nonlinear diffusion dynamics of $x_t$ is modeled as a linear evolution within these subspaces, enabling closed-form trajectory solutions and effectively eliminating the need for iterative sampling.\n\nThe authors provide a theoretical justification showing that by explicitly modeling diffusion dynamics in a simpler linear form via Koopman operators introduces less error compared to existing one-step diffusion models, which may struggle with learning complex end-to-end one-step mappings.", "questions": "1. I wonder whether the hierarchical modeling with varying spatial resolutions from the U-Net architecture is essential to the proposed framework. Can the framework also be applied to DiT-based architectures, given that many recent image foundation models are built on DiT?\n\n2. In the main paper, the encoder is not conditioned on the timestep t, but I saw a description in the appendix that the noise level is fed into the EDM U-Net. I wonder if you've observed any performance changes when not conditioning the encoder on the noise level.\n\n3. Can HKD perform multi-step generation by recursively computing $x_{t-1}$ from $x_t$ through the encoder-decoder pipeline, rather than predicting $x_0$ in a single forward?", "limitations": "The limitations are written in Strengths and Weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is well written and easy to follow. \n\nIntroducing Koopman theory into the diffusion model literature is novel and theoretically beneficial, opening up a promising new direction for accelerating diffusion models. \n\nCompared to prior work, HKD slightly lags behind iCT but outperforms the other baselines. However, I believe the current results may not reflect optimal hyperparameter settings. The method's novelty outweighs these limitations, and I see strong potential in the proposed approach. \n\nIn terms of weaknesses, the paper lacks some key experiments and includes a few claims that appear overstated and would benefit from clarification.\n\nThe authors claim that HKD offers better training stability than iCT, but no experimental evidence is provided to show the training stability of HKD. Additionally, the comparison with recent few-step diffusion models is somewhat limited—these baselines are only included on CIFAR-10, and are omitted for the other datasets. Furthermore, the highlighting in the tables does not reflect the best results, which may be misleading to readers.", "was_revised": true, "final_justification": "I believe this submission presents sufficient novelty and makes a meaningful contribution to the generative modeling literature. Thus, I vote for Accept.", "rebuttal": "We sincerely thank the reviewer for the encouraging feedback and constructive suggestions, which have been valuable in improving the clarity and quality of our paper. We are especially grateful for the recognition of the novelty, the promising new direction, and the strong potential of our approach.\n\n***\n\n**[W1. Training stability of HKD]**\n\nThank you sincerely for pointing this out, and apologize for not including experimental evidence on training stability. We will add such results in the revised version to better support our claim.\n\nFrom a theoretical perspective, the training stability of HKD is supported by two key aspects of our formulation: (1) The exponential formulation in the Koopman space ensures sufficiently large gradients for spectra with magnitude around 1, which shows superiority in the effectiveness of switching underlying ODEs and helps mitigate the instability issues often encountered in consistency models. Consistency models may suffer from high-variance gradient estimators due to sharp decision boundaries in the ODE flow, making them difficult to train [1]. (2) The supervision of the Koopman trajectory with inputs at multiple time points ensures that the proposed method captures the ideal spectrum for the evolution of modes in a more stable and averaged manner, which enhances the stability.\n\nAs experimental evidence, we include the results of training HKD five times independently on the CIFAR-10 dataset. The FID scores and their standard deviation are reported in the table below. We summarize the FID results over every 10 epochs during training across the five runs, demonstrating stable convergence behavior.\n\n|                   | Before Training | Epoch 0    | Epoch 10  | Epoch 20  | Epoch 30  | Epoch 40  | Epoch 50  | Epoch 52  | Epoch 53  |\n| ----------------- | --------------- | ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| FID (Mean ± Std ) | 447.13          | 11.82±0.29 | 5.70±0.21 | 4.71±0.17 | 4.09±0.18 | 3.52±0.11 | 3.31±0.03 | 3.31±0.02 | 3.30±0.02 |\n\n[1] Silvestri, Gianluigi, et al. \"Training consistency models with variational noise coupling.\" *arXiv preprint arXiv:2502.18197* (2025).\n\n***\n\n**[W2. The comparison with few-step methods on FFHQ]**\n\nWe apologize that some recent few-step diffusion models were not included in our evaluation on the FFHQ dataset. This is because these methods were not originally trained or evaluated on FFHQ, leading to the unavailability of official pretrained models and standardized evaluation protocols for this dataset.\n\nAmong the main comparison baselines we selected, one is the few-step EDM model, which is a state-of-the-art diffusion backbone. While EDM performs very well with a large number of steps, its performance drops significantly in the few-step regime.\n\nAnother key baseline is ECM, a representative one-step consistency model that, like our method, initializes the network weights from a pretrained score model. We chose to compare against ECM under the same configuration to fairly assess our method’s effectiveness on more complex datasets such as FFHQ.\n\n***\n\n**[W3. The highlighting in the tables]**\n\nWe sincerely apologize for the confusion caused by the highlighting in the tables. Our original intention was to draw attention to the results of our proposed method, not to imply that they represent the best performance overall. We will remove the highlighting in the revised version to ensure clarity. Thank you for pointing this out.\n\n***\n\n**[Q1 (I). The hierarchical modeling with varying spatial resolutions]**\n\nWe appreciate the reviewer’s insightful question. While the hierarchical architecture enhances the representational capacity, it is not essential for the theoretical validity or functioning of the proposed Koopman-based framework.  The hierarchical design, inspired by the U-Net architecture, encourages a structured decomposition of modes across different spatial resolutions. This facilitates a frequency-aware partitioning of the feature space and enriches the model's representational capacity by allowing different resolution bands to capture different aspects of the dynamics. Thus, as shown in the ablation study, incorporating hierarchical modeling can improve generation quality. \n\nHowever, we would like to emphasize that the core mechanism of our framework, i.e., learning a latent representation amenable to Koopman-based evolution, is not strictly dependent on the hierarchical structure. In principle, even without hierarchical modeling, the network has the capacity to learn the necessary information at the final resolution layer. That is, the model will naturally be incentivized to encode the full dynamical modes into the final latent features to enable accurate one-step prediction through the Koopman operator.\n\n**[Q1 (II). Applied to DiT-based architectures]**\n\nYes, our framework can be applied to DiT-based architectures. Specifically, the input tokens of each DiT block can be used for Koopman evolution. Our Koopman framework only requires a feature representation with sufficient expressiveness to model temporal dynamics and enable one-step generation. As long as the feature extractor, whether a U-Net or a DiT, provides such features, our method remains applicable.\n\n***\n\n**[Q2. Condition on the time *t*]**\n\nThank you sincerely for your careful reading and thoughtful observation. We have observed a minor drop in performance when removing noise-level conditioning, particularly in convergence speed. For example, without conditioning, the FID at epoch 50 reaches 3.83, which is slightly higher than the 3.30 achieved when the noise level is explicitly provided. This degradation is expected, as images at different noise levels lie in different feature spaces. Removing the noise level input reduces the encoder’s ability, especially when pretrained, to map noisy inputs into a unified latent space, thereby slightly compromising training efficiency and performance.\n\n***\n\n**[Q3. Multi-step generation capability]**\n\nOur framework can, in principle, support recursively computing $x_{t-1}$ from $x_t$ through the encoder–Koopman–decoder pipeline. However, this would require modifications to the current training setup. In our work, we focus on one-step generation, where the Koopman model is always trained to evolve features toward a fixed target time $t=0$, and decoding is only performed at this final time step to recover $x_0$. To enable multi-step generation by recursively computing $x_{t-1}$ from $x_{t}$, we would need to adjust both the Koopman evolution targets to perform one-step evolution from $z_{t}$ to $z_{t-1}$ and the decoder to support feature decoding at arbitrary intermediate time steps, not just at $t=0$."}, {"review_id": "NSxHRO22fX", "reviewer": "Reviewer_xGGM", "summary": "This paper try to remove the iterative process in the diffusion models by utilizing koopman dynamics in the hierarchical latent space of the diffusion models. In specific, they introduced a encoder that encode the latent from UNet to koopman subspace and then utilize the Koopman dynamic model to skip the intermediate timesteps to get the latent for the decoder. This enabled lower latency image editing. The experiment results shows close to best performance while the proposed method does not need to tune the hyperparameters.", "questions": "1. It is beneficial for the audience that the authors include the latency analysis (i.e. a table with different methods and wall clock time for each image to generate/edited)\n2. Also, I suggest the authors to include experiments on PIE-Bench[1], which is a benchmark dataset for evaluating image editing performance. This helps the audience to understand the effectiveness of the proposed method for image editing task.\n3. I would like the author to further clarify the proof laid out in appendix C.2, especially from line 14 to line 15 where what kind of assumption you use to establish the equivalence. Also line 16 and 18, how to understand the equivalence between these two. This question is pivotal for me since I am willing to change my score to strong accept if I the authors fully address my concerns and the proof is verified.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n1. The paper shows an alternative way of doing one-step generation for diffusion model. The idea is novel and could lead to broader impact.\n2. The paper is easy to follow and clear.\n\nWeakness:\n1. It is beneficial for the audience that the authors include the latency analysis (i.e. a table with different methods and wall clock time for each image to generate/edited)\n2. Also, I suggest the authors to include experiments on PIE-Bench[1], which is a benchmark dataset for evaluating image editing performance. This helps the audience to understand the effectiveness of the proposed method for image editing task.\n3. I would like the author to further clarify the proof laid out in appendix C.2, especially from line 14 to line 15 where what kind of assumption you use to establish the equivalence. Also line 16 and 18, how to understand the equivalence between these two.\n\n\n[1]: Ju, Xuan, et al. \"Direct inversion: Boosting diffusion-based editing with 3 lines of code.\" arXiv preprint arXiv:2310.01506 (2023).", "was_revised": true, "final_justification": "The authors has addressed my concerns and I like this paper. The paper is technically sound and has the potential to have broader impact to the community.", "rebuttal": "Thank you sincerely for your thoughtful review. Your constructive suggestions have been truly valuable in helping us improve the clarity and overall quality of the paper.\n\n***\n\n**[W1&Q1. The latency analysis]**\n\nThank you for the helpful suggestion. We provide the wall-clock time per image for different generation methods in the table below, measured on an NVIDIA V100 GPU. These methods were trained on the CIFAR-10 dataset. We will include this latency analysis in the revised version of the paper.\n| **Method**  | **DDPM**         | **Score SDE**    | **DDIM**         | **DDIM**           | **EDM**    | **EDM** | **KD**     | **PD**   | **CD (LPIPS)** | **DMD** |\n|-------------|------------------|------------------|------------------|--------------------|------------|---------|------------|---------|----------------|--------|\n| NFE    | 1000             | 2000             | 10               | 100                | 35         | 15      | 1          | 1       | 1              | 1      |\n| Latency | 15.32s            | 45.89s           | 0.51s            | 2.35s              | 0.69s      | 0.30s   | 0.26s      | 0.28s   | 0.02s          | 0.03s  |\n| **Method**  | **1-Rectified flow** | **2-Rectified flow** | **3-Rectified flow** | **2-Rectified flow++** | **CT (LPIPS)** | **iCT** | **iCT-deep** | **ECM** | **HKD** |        |\n| NFE     | 1                | 1                | 1                | 1                  | 1          | 1       | 1          | 1       | 1              |        |\n| Latency | 0.03s            | 0.03s            | 0.03s            | 0.03s              | 0.02s      | 0.02s   | 0.02s      | 0.02s   | 0.03s          |        |\n***\n**[W2&Q2. Experiments on PIE-Bench]**\n\nThank you for the valuable suggestion. We would like to clarify that our work does not focus on the text-based image editing task, which PIE-Bench primarily evaluates. As noted in Footnote 1 (line 41) of the manuscript, apart from one-step generation, our study centers on diffusion trajectory control rather than text-guided image editing. Adapting our framework to PIE-Bench would require integrating a text-conditioning branch and retraining the model, which is beyond the current scope. Nevertheless, we agree that this is an interesting and valuable direction and consider it a promising avenue for future work.\n\n\n***\n\n**[W3&Q3. The proof laid out in Appendix C.2]**\n\nThank you for your careful reading and for pointing out the unclear parts in Appendix C.2, particularly (14)–(15) and (16)–(18). We apologize for the lack of clarity and will clarify the overall proof in the revised version.\n\nThe derivation presented in (14)–(15) of Appendix C.2 relies on two key assumptions to hold. **The assumptions are: (1) the level-based Koopman states $\\boldsymbol w_t^{(l)}$ are disentangled across levels labeled by $l$, and (2) the mapping $\\mathcal H$ is isotropic w.r.t. the Koopman states**.  Intuitively, this requires that (1) Koopman states at different levels depict different features, and (2) each Koopman state contributes equally to the loss function. A detailed version of the derivation, explicitly accounting for the $o(\\cdot)$ terms arising from the assumptions, is\n\n$\\mathbb E\\_{t\\sim\\mathcal U[\\varepsilon , T), \\boldsymbol x\\_t\\sim p\\_\\text{data}}\\left\\Vert \\mathcal F\\circ\\mathcal D\\left[\\left\\\\{\\textrm e^{-t\\boldsymbol A^{(l)}}\\mathcal E\\_{\\boldsymbol \\theta}^{(l)}(\\boldsymbol x\\_t)\\right\\\\}\\_{l=1}^L\\right]-\\mathcal F\\circ\\mathcal D\\left[\\\\{\\boldsymbol w\\_0^{(l)}\\\\}\\_{l=1}^L\\right]\\right\\Vert\\_2^2$\n\n$\\overset\\star=\\sum\\_{l=1}^L\\mathbb E\\Big\\Vert \\mathcal H\\left[\\\\{ \\textrm e^{-t\\boldsymbol A^{(l)}}\\mathcal E\\_{\\boldsymbol \\theta}^{(l)}(\\boldsymbol x\\_t)\\text{ at element }l\\\\}\\right]-\\mathcal H\\left[\\\\{\\boldsymbol w\\_0^{(l)}\\text{ at element }l\\\\}\\right]\\Big\\Vert\\_2^2+o(\\rho )$\n\n$=\\sum\\_{l=1}^L\\mathbb E\\ \\left\\Vert\\frac{\\partial \\mathcal H}{\\partial \\boldsymbol w^{(l)}}\\right\\Vert\\cdot \\Big\\Vert \\textrm e^{-t\\boldsymbol A^{(l)}}\\mathcal E\\_{\\boldsymbol \\theta}^{(l)}(\\boldsymbol x\\_t) - \\boldsymbol w\\_0^{(l)}\\Big\\Vert\\_2^2+o(\\rho +\\varepsilon )$\n\n$\\overset*=\\sum\\_{l=1}^L\\mathbb E\\ \\left\\Vert\\frac{\\partial \\mathcal H}{\\partial \\boldsymbol w^{(l)}}\\right\\Vert\\cdot \\left\\Vert\\textrm e^{-t\\boldsymbol A^{(l)}}\\right\\Vert\\cdot \\Big\\Vert \\mathcal E\\_{\\boldsymbol \\theta}^{(l)}(\\boldsymbol x\\_t) - \\boldsymbol w\\_0^{(l)}\\Big\\Vert\\_2^2 +o(\\rho +\\varepsilon +\\delta )$\n\nwhere the equations $\\overset\\star=$ and $\\overset*=$ correspond to the two assumptions above, respectively. The error term $o(\\rho +\\varepsilon +\\delta )$ consists of three small-value terms: $\\rho$, which measures the extent of entanglement among levels; $\\varepsilon$, which quantifies the anisotropy of $\\mathcal H$; and $\\delta$, which is the range of Koopman spectrum. These error terms reflect the degree to which the assumptions are violated; since our assumptions are generally easy to satisfy in practice, the resulting error is expected to be small.\n\nRegarding Eqs. (16) and (18), we sincerely apologize for the confusion. Eq. (16) already implies the conclusion stated in Proposition C.2, while Eq. (18) provides a corollary that the proposed loss function at $\\boldsymbol x_0$ not only provides supervision on the latent-space trajectory, but also infers supervision on $\\boldsymbol x_t$, the image-space trajectory. We will remove Eqs. (17)–(18) in the revised version to clarify the proof."}]

Author rebuttal:

Meta-review: This work proposes a fast inference algorithm for diffusion model based on the Koopman theory. The Koopman theory is a framework that lifts a nonlinear dynamics to a linear dynamics. In this paper, Koopman theory is leveraged to approximate the nonlinear backward diffusion with a linear system in a feature space. Once this feature space is learned, fast inference can be achieved by solving this linear system in closed form. In addition to efficiency, the authors claim interoperability is another benefit of the proposed method. All reviewers agree that applying Koopman theory to diffusion model sampling is a novel and interesting idea. Several weaknesses are pointed out including the scalability of the algorithm to high resolution images. Most of comments have been properly addressed in the rebuttal as indicated in final justification. Overall, this is an interesting contribution to diffusion generative models.
Decision: Accept (poster)