Title: One-Step Offline Distillation of Diffusion-based Models  via Koopman Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Koopman theory

diffusion models

offline distillation

one-step generation

dynamical systems
Abstract: Diffusion-based generative models have demonstrated exceptional performance, yet their iterative sampling procedures remain computationally expensive. A prominent strategy to mitigate this cost is *distillation*, with *offline distillation* offering particular advantages in terms of efficiency, modularity, and flexibility. In this work, we identify two key observations that motivate a principled distillation framework: (1) while diffusion models have been viewed through the lens of dynamical systems theory, powerful and underexplored tools can be further leveraged; and (2) diffusion models inherently impose structured, semantically coherent trajectories in latent space. Building on these observations, we introduce the *Koopman Distillation Model* (KDM), a novel offline distillation approach grounded in Koopman theory - a classical framework for representing nonlinear dynamics linearly in a transformed space. KDM encodes noisy inputs into an embedded space where a learned linear operator propagates them forward, followed by a decoder that reconstructs clean samples. This enables single-step generation while preserving semantic fidelity. We provide theoretical justification for our approach: (1) under mild assumptions, the learned diffusion dynamics admit a finite-dimensional Koopman representation; and (2) proximity in the Koopman latent space correlates with semantic similarity in the generated outputs, allowing for effective trajectory alignment. Empirically, KDM achieves state-of-the-art performance across standard *offline distillation* benchmarks - improving FID scores by up to 40% in a single generation step.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Koopman Distillation, an approach rooted in Koopman theory for nonlinear dynamics, in which a pretrained diffusion model is distilled by learning an autoencoder on the latent representation of the model and a linear operator inside that space in order to transfer data to noise.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The overall math an intuition behind the approach is well thought out, and exceedingly interesting and insightful. Unlike many modern distillation methods, this approach is novel and is rooted in both empirical observations and theoretical insights that make for a compelling distillation method.\n- The presented analysis in 6.1 is very interesting, and constitutes a thoughtful empirical analysis of the proposed KDMs.\n\n**Weaknesses**\n- I think the overall empirical results are a bit weak. The authors make clear emphasis the delineation between online and offline methods, but I think this emphasis is overstated and somewhat attempts to cover their weaker results.\n- Especially on unconditional CIFAR-10, KDMs barely beat *any* other distillation method, besides their chosen baseline of GET. The authors also incorrectly portray 2-rf++ as just a continuous flow method, when in fact **it is also a form of offline distillation** if I'm not mistaken.\n- Additionally, the relevant offline distillation methods that the authors *should* be comparing against, ADD and LADD, are generally done on much large Text2Image models rather than these small tasks (as GET is not particularly adopted if I'm not mistaken), which makes it hard to assess the overall future performance of KDMs, and whether such a framework could be competitive with these primarily adversarial methods, which are the most common forms of offline distillation. \n\n\n**Final Decision**: Overall, I do not think this paper is ready for publication yet. While I thoroughly enjoyed the methodological contribution and found these insights exciting and potentially useful for the wider acceleration community, it is hard to ignore the overall poor performance of the approach, and thus in all the paper seems like an interesting, but generally less performant distillation method. I would recommend the authors attempt what they can to either scale up their experiments, continue to explore the empirical design space of the method to improve performance, or recreate existing popular distillation methods at the scale feasible for their compute budget to demonstrate the strengths of KDMs.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe authors do recognize the limitations of their performance, though I do think the overall poor performance is a bit understated and downplayed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Koopman Distillation Model (KDM), a novel method for offline distillation of diffusion models. The main innovation is its use of Koopman operator theory to represent the non-linear process of generating an image from noise in diffusion models as a simple linear transformation in a learned embedding space. This provides  a principled way to achieve single-step generation.\n\nThe authors provide theoretical proofs showing that the diffusion process can be accurately represented by a finite-dimensional Koopman operator and that this representation preserves the semantic structure of the generated images. Empirically, KDM outperforms previous offline distillation methods on standard benchmarks like CIFAR-10, FFHQ, and AFHQv2, improving FID scores by up to 40% in a single generation step. The method is not only fast, achieving a 4x speedup in training time compared to prior work, but also scales effectively with model size.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n### **Quality**:\nOverall, the paper is of high quality and supports its claims with extensive empirical validation, which includes experiments on standard benchmark datasets (CIFAR-10, FFHQ, AFHQv2) using standard evaluation metrics (FID, IS) as well as a human evaluation study.\n\n### **Clarity**:\nThe paper is well-written and clear. The paper is logically structured and the method is described in detail, including the specific loss function and model architecture.\n\n### **Significance**:\nBy developing a method for single-step offline distillation that is not only fast but also achieves state-of-the-art results, the paper provides\nprovides a practical tool that both researchers and practitioners are likely to use and build upon. The efficiency gains are significant, with a reported 4x speedup in training and over 8x faster sampling compared to the previous offline SOTA.\n\n### **Originality**:\nThis paper presents a original idea: it uses Koopman operator theory to linearize the complex generation dynamics of diffusion models, leading to a novel approach for offline distillation. Moreover, the insight that the semantic structure in the noise-to-image mapping of diffusion models is preserved in the learned embedding of the Koopman operator is an original and valuable contribution in its own right.\n\n---\n\n# Weaknesses:\n\n### **Contribution of the  $L_{\\text{Koopman}}$ loss**:\n\n- It's not clear to what extent the $L_{\\text{Koopman}}$ loss contributes to the success of the approach. The authors include an adversarial loss \"to further refine the distillation process\", however without the adversarial component (see Table 5) their method actually performs worse than their main point of comparison (GET).\n- Moreover, GET itself is not adversarially trained so the comparison between an adversarially trained KDM and a non-adversarially trained GET is not totally fair. The authors mention in Appendix D.3 that they struggled to train GET with an adversarial loss - it would be helpful to also see the results/loss curves for these attempts.\n\n### **Weighting of the $L_{\\text{Koopman}}, L_{\\text{adv}}$ losses**:\n\n- Following up from the previous comments, I wonder if the authors tried different weightings for the $L_{\\text{Koopman}}$ and  $L_{\\text{adv}}$ losses? Currently, the weighting on the different components of $L_{\\text{Koopman}}$ is uniform, even though they may potentially lie on different scales during training. Moreover, there is no ablation for different values of $\\lambda_{\\text{adv}}$. It would be nice to see some ablations for this to better understand the effect of the adversarial loss and also the different components of $L_{\\text{Koopman}}$.\n\n### **Practical relevance of the bound in Theorem 5.2**:\n\n- Thm. 5.2 establishes that semantic proximity is preserved, showing that the distance between two generated images is bounded by the distance between their latent representations. However, clearly the quality of this bound depends on the norm $||C_\\eta||$ - can the authors show what kind of norms $||C_\\eta||$ actually achieves in practise?\n\nQUESTIONS:\nI would appreciate it if the authors can answer the below questions from the weaknesses section above:\n\n- Point 2. from \"Contribution of the $L_{\\text{Koopman}}$ loss\".\n- Point 1. from \"Weighting of the $L_{\\text{Koopman}}, L_{\\text{adv}}$ losses\".\n- Point 1. from \"Practical relevance of the bound in Theorem 5.2\".\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the Koopman distillation model for offline distillation, in order to compress diffusion models. The authors observed that there is lack of theoretical adoption of dynamic systems theory to diffusion models. It is also empirically found that there is semantic coherence in latent spaces induced by diffusion models. From these two observations, Koopman distillation model (KDM) is proposed and theoretically analyzed. It distills diffusion models via linear evolution in a learned embedding space which is generated by an encoder given a noisy input. From the embedding, a decoder then reconstructs the clean samples. The authors also theoretically proved the existence of finite-dimensional Koopman representations and semantic structure preservation under mild assumptions. Experimental results also demonstrate its performance compared with baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. *Originality*: This work novelly proposed the one-step Koopman distillation model (KDM) as an offline distillation of diffusion models. It is of originality to consider the Koopman operator in distillation of diffusion models, which did not appear in past research.\n\n2. *Clarity*: The presentation is clear and logically linear. The whole work is easy to follow. The motivations and methods are clearly articulated in terms of the current limitations of diffusion models and the mechanism and pipeline of KDM. \n\n3. *Significance*: The excessive computational cost is a current pressing issue of diffusion models as a powerful visual content generator. The proposed KDM is important to solve this problem which is also vital.\n\n**Weaknesses**\n\n1. *Quality*: The performance of the proposed KDM does not of the state-of-the-art among all the presented compression methods for diffusion models. Although KDM is an offline distillation method, the experimental results did not demonstrate the advantages of it over CTM as diffusion distillation methods.\n\nQUESTIONS:\n1. *Training loss*: The design of the loss function contains the adversarial loss in order to enhance the quality of the generated images. However, it may introduce the instability of the training process and the difficulty to converge. Would the authors demonstrate the training loss convergence curve in their experiments and compare it with GET and CTM?\n\n2. *Memory cost*: In experiments, the quality of generated images is compared on the same scale of parameters across KDM and GET. However, as claimed in the introduction that KDM as an offline distillation method is more efficient than diffusion distillation methods such as CTM. Hence could the comparison include CTM on the parameter efficiency or other metrics to demonstrate the efficiency of KDM over CTM?\n\n3. *Higher resolution dataset*: The experiments are conducted on CIFAR-10, FFHQ 64×64 and AFHQv2 64×64 datasets. Would the authors present the experiments on ImageNet-1k to evaluate the performance of KDM on higher resolution datasets?\n\n4. *Clarification*: A minor question is on the notations. Could the authors clarify and explain further on the differences between $C_\\mu$ and $C_\\eta$?\n\n5. *Quality*: The quality and motivations should be stronger. The current prominent compression method for diffusion models are consistency models and their variants. Although the privacy is mentioned as one of the advantages, experiments do not demonstrate and support it. It would improve the quality to include them and articulate them in experiments briefly.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Koopman Distillation Model (KDM), a novel offline distillation framework designed for one-step generation. The proposed method is grounded in Koopman operator theory, a classical mathematical framework that seeks to represent complex nonlinear dynamics as a globally linear operator acting on a transformed space of \"observable\" functions. In practice, KDM employs an encoder-linear-decoder architecture. A neural network encoder maps a noisy input into a high-dimensional latent space; a learned, finite-dimensional Koopman matrix then propagates the latent state forward in a single, linear step; finally, a decoder network reconstructs the clean, final sample from the evolved latent state. The authors provide theoretical justification for this architecture, proving that the learned diffusion dynamics admit a finite-dimensional Koopman representation under mild assumptions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. Novel application of Koopman operator theory to the problem of offline diffusion model distillation.\n2. The paper provides a solid theoretical foundation for its method. Key contributions include a proof that the learned diffusion dynamics can be represented by a finite-dimensional Koopman operator under mild assumptions, which justifies the core architectural choice. \n3. The writing is clear and easy to follow. \n\n**Weakness**\n1. The proposed method cannot really utilize more inference-time compute for better generation, unlike Consistency Models.\n2. I do not think that the community can agree over the statement on offline distillation methods are more scalable and modular. Generally, the community has moved away from this family of approaches.\n3. The empirical results are very weak. The paper only conducts experiments on toyish datasets like CIFAR and FFHQ. The results are lackluster. Without adversarial training, it is worse than the only offline distillation baseline (7.83 vs 6.91). It is not at all comparable to the likes of DMD and CM.\n\nQUESTIONS:\n1. Forgive me if I have missed it, but I would like to see the relationship between the number of pre-constructed noise-data pairs and the distillation performance. One big problem with offline distillation is the possibility of prior holes.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-03"}