Title: Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
Venue: NeurIPS 2025
Accepted: yes
Keywords: One-step Generation

Diffusion Models

Koopman Operators

Interpretable Image Synthesis
Abstract: Diffusion models have achieved impressive success in high-fidelity image generation but suffer from slow sampling due to their inherently iterative denoising process. While recent one-step methods accelerate inference by learning direct noise-to-image mappings, they sacrifice the interpretability and fine-grained control intrinsic to diffusion dynamics, key advantages that enable applications like editable generation. To resolve this dichotomy, we introduce **Hierarchical Koopman Diffusion**, a novel framework that achieves both one-step sampling and interpretable generative trajectories.  Grounded in Koopman operator theory, our method lifts the nonlinear diffusion dynamics into a latent space where evolution is governed by globally linear operators, enabling closed-form trajectory solutions. This formulation not only eliminates iterative sampling but also provides full access to intermediate states, allowing manual intervention during generation. To model the multi-scale nature of images, we design a hierarchical architecture that disentangles generative dynamics across spatial resolutions via scale-specific Koopman subspaces, capturing coarse-to-fine details systematically. We empirically show that the Hierarchical Koopman Diffusion not only achieves competitive one-step generation performance but also provides a principled mechanism for interpreting and manipulating the generative process through spectral analysis. Our framework bridges the gap between fast sampling and interpretability in diffusion models, paving the way for explainable image synthesis in generative modeling.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper try to remove the iterative process in the diffusion models by utilizing koopman dynamics in the hierarchical latent space of the diffusion models. In specific, they introduced a encoder that encode the latent from UNet to koopman subspace and then utilize the Koopman dynamic model to skip the intermediate timesteps to get the latent for the decoder. This enabled lower latency image editing. The experiment results shows close to best performance while the proposed method does not need to tune the hyperparameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The paper shows an alternative way of doing one-step generation for diffusion model. The idea is novel and could lead to broader impact.\n2. The paper is easy to follow and clear.\n\nWeakness:\n1. It is beneficial for the audience that the authors include the latency analysis (i.e. a table with different methods and wall clock time for each image to generate/edited)\n2. Also, I suggest the authors to include experiments on PIE-Bench[1], which is a benchmark dataset for evaluating image editing performance. This helps the audience to understand the effectiveness of the proposed method for image editing task.\n3. I would like the author to further clarify the proof laid out in appendix C.2, especially from line 14 to line 15 where what kind of assumption you use to establish the equivalence. Also line 16 and 18, how to understand the equivalence between these two.\n\n\n[1]: Ju, Xuan, et al. \"Direct inversion: Boosting diffusion-based editing with 3 lines of code.\" arXiv preprint arXiv:2310.01506 (2023).\n\nQUESTIONS:\n1. It is beneficial for the audience that the authors include the latency analysis (i.e. a table with different methods and wall clock time for each image to generate/edited)\n2. Also, I suggest the authors to include experiments on PIE-Bench[1], which is a benchmark dataset for evaluating image editing performance. This helps the audience to understand the effectiveness of the proposed method for image editing task.\n3. I would like the author to further clarify the proof laid out in appendix C.2, especially from line 14 to line 15 where what kind of assumption you use to establish the equivalence. Also line 16 and 18, how to understand the equivalence between these two. This question is pivotal for me since I am willing to change my score to strong accept if I the authors fully address my concerns and the proof is verified.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel one-step diffusion model based on Koopman operators. Unlike existing few/one-step diffusion models, the proposed Hierachical Koopman Diffusion (HKD) offers several advantages, including improved interpretability during generation and the elimination of iterative sampling for ODE solving.\n\nThe HKD framework adopts a hierarchical architecture that projects a noisy input $x_t$ into a set of latents $z_t$ at different spatial resolutions, referred to as Koopman subspaces. The original nonlinear diffusion dynamics of $x_t$ is modeled as a linear evolution within these subspaces, enabling closed-form trajectory solutions and effectively eliminating the need for iterative sampling.\n\nThe authors provide a theoretical justification showing that by explicitly modeling diffusion dynamics in a simpler linear form via Koopman operators introduces less error compared to existing one-step diffusion models, which may struggle with learning complex end-to-end one-step mappings.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and easy to follow. \n\nIntroducing Koopman theory into the diffusion model literature is novel and theoretically beneficial, opening up a promising new direction for accelerating diffusion models. \n\nCompared to prior work, HKD slightly lags behind iCT but outperforms the other baselines. However, I believe the current results may not reflect optimal hyperparameter settings. The method's novelty outweighs these limitations, and I see strong potential in the proposed approach. \n\nIn terms of weaknesses, the paper lacks some key experiments and includes a few claims that appear overstated and would benefit from clarification.\n\nThe authors claim that HKD offers better training stability than iCT, but no experimental evidence is provided to show the training stability of HKD. Additionally, the comparison with recent few-step diffusion models is somewhat limited—these baselines are only included on CIFAR-10, and are omitted for the other datasets. Furthermore, the highlighting in the tables does not reflect the best results, which may be misleading to readers.\n\nQUESTIONS:\n1. I wonder whether the hierarchical modeling with varying spatial resolutions from the U-Net architecture is essential to the proposed framework. Can the framework also be applied to DiT-based architectures, given that many recent image foundation models are built on DiT?\n\n2. In the main paper, the encoder is not conditioned on the timestep t, but I saw a description in the appendix that the noise level is fed into the EDM U-Net. I wonder if you've observed any performance changes when not conditioning the encoder on the noise level.\n\n3. Can HKD perform multi-step generation by recursively computing $x_{t-1}$ from $x_t$ through the encoder-decoder pipeline, rather than predicting $x_0$ in a single forward?\n\nLIMITATIONS:\nThe limitations are written in Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Hierarchical Koopman Diffusion (HKD), a framework that leverages Koopman operator theory to enable one-step sampling for diffusion models while preserving interpretable generative trajectories. It also achieves competitive results for one-step generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well-structured, with a solid theoretical foundation and clear technical novelty. The application of Koopman operator theory to diffusion distillation is particularly interesting. The authors also provide extensive experimental validation, detailed theoretical derivations, and ablation studies.\n\n\nWeaknesses:\nThe paper does not include results on larger or more challenging datasets such as ImageNet, leaving the method’s scalability and generalization to more complex data unproven. While the reported results are competitive, they do not reach state-of-the-art performance, raising questions about the practical benefit of introducing the Koopman operator framework compared to existing simpler or more established approaches.\n\nQUESTIONS:\n1.\tDo you have results on ImageNet at 64×64 or 256×256 resolutions? It would be helpful to see whether the method scales effectively to larger and more diverse datasets.\n2.\tHow do the extracted features impact the effectiveness of the Koopman operator in this context? It would be valuable to include more discussion on what constitutes good feature extraction for achieving better one-step results.\n3.\tIs it necessary to extract multi-scale features from images to apply the Koopman operator theory effectively? For better clarity, I suggest including a simple toy example to illustrate how the Koopman framework works in practice — for example, mapping a standard Gaussian to another simple distribution.\n4. Does the Koopman framework require pretrained models, or can it be trained entirely from scratch?\n\nLIMITATIONS:\nSee weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Hierarchical Koopman Diffusion (HKD) to bridge the gap between fast, one-step generative models and the interpretable, controllable trajectories of traditional diffusion models. It is based on the Koopman operator theory to linearize the nonlinear diffusion ODE dynamics. \nThe authors propose a U-Net-like architecture to encode a noisy image x_t into a hierarchical set of latent observables, which are then evolved in a single step via a learned linear Koopman operator. A corresponding decoder then maps these evolved latents back to the image space. This paper shows that this framework not only enables one-step generation but also provides interpretability through spectral analysis of the learned Koopman operators. The paper conducts experiments on CIFAR-10 and FFHQ, demonstrating competitive FID scores.\n\nSTRENGTHS AND WEAKNESSES:\n- The application of Koopman operator theory to accelerate diffusion model sampling is a novel direction. \n- One of the strengths is the successful demonstration of interpretability. The spectral analysis in Figure 4, showing a correspondence between eigenvalues and semantic image structures, is interesting.\n- HKD achieves competitive FID scores for a one-step generator on both CIFAR-10 (3.30) and FFHQ (5.70).\n\nQUESTIONS:\n- How critical is the initialization from a pre-trained EDM model (U-Net)?\n- The authors argue against supervising in the latent space because it \"reduces training flexibility and introduces gradients that poorly reflect perceptual discrepancies\" (line 158). Can you please provide more evidence?\n- Could one not achieve similar frequency-based editing by performing a Fourier transform on the feature maps at different levels of a standard U-Net-based one-step model and manipulating those coefficients?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}