Title: Learning to Integrate Diffusion ODEs by Averaging the Derivatives
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative models

diffusion models
Abstract: To accelerate diffusion model inference, numerical solvers perform poorly at extremely small steps, while distillation techniques often introduce complexity and instability. This work presents an intermediate strategy, balancing performance and cost, by learning ODE integration using loss functions derived from the derivative-integral relationship, inspired by Monte Carlo integration and Picard iteration. From a geometric perspective, the losses operate by gradually extending the tangent to the secant, thus are named as secant losses. The target of secant losses is the same as that of diffusion models, or the diffusion model itself, leading to great training stability. By fine-tuning or distillation, the secant version of EDM achieves a $10$-step FID of $2.14$ on CIFAR-10, while the secant version of SiT-XL/2 attains a $4$-step FID of $2.27$ and an $8$-step FID of $1.96$ on ImageNet-$256\times256$. Code is available at \url{https://github.com/poppuppy/secant-expectation}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for few-step diffusion. Existing fast samplers suffer from significant performance degradation when using a small number of time steps, while previous diffusion distillation methods are prone to training instability and complexity.\nIn the proposed method, a neural network learns to approximate the secant function, representing the average rate of change along the solution paths of the PF-ODE. This training is performed using a loss function designed from the perspective of Monte Carlo integration of the tangent function, inspired by Picard iteration.\nThis paper introduces four loss variants, corresponding to combinations of two time-step sampling schemes during training and whether the training is a distillation process or performed from scratch. Theoretical justification for the proposed losses is also provided.\nExperiments on CIFAR-10 and ImageNet 256 x 256 demonstrate that the proposed method exhibits consistent stability across various hyperparameter settings and achieves relatively good performance with a small number of time steps.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper is well organized.\n* The paper provides theoretical guarantees for the validity of the proposed losses. The theories show an interesting connection between training iterations with stop gradient and Picard iteration.\n* The proposed method demonstrates performance comparable to or better than most existing methods, as shown in the tables.\n* Extensive evaluations were conducted across various settings to examine the behavior of the proposed method.\n\n### Weaknesses\n1. The comparison to and discussion of previous consistency models on ImageNet-256 x 256 appears to be insufficient. The paper only mentions training instability issues encountered by previous consistency models on ImageNet-256 x 256. It remains unclear whether appropriate hyperparameters and settings were used for training these models. A more detailed discussion of the failure cases of consistency models would enhance the clarity and quality of the paper and highlight the stability of the proposed method.\n2. The condition for $|s - t|$ in Theorems 2 and 3 appears to be too restrictive for practical few-step diffusions. The magnitude should be $O(L^{-1})$ for Theorem 2 and $O(L^{-1/2})$ for Theorem 3, where $L$ denotes the Lipschitz constant of $v(x_t, t)$ and $f_\\theta (x_t, t, s)$, respectively. It is generally not guaranteed that the neural network or the flow is Lipschitz in practical cases. This could be a limitation of the proposed method.\n3. There is subtle missing information and errors in the formulas.\n    * The definition of the prime symbol ', around Eq. (1) and Proposition 1, is unclear, while I presume it denotes the derivative with respect to time t.\n    * The variables a and b are not used in the statement of Theorem 3 in the main part of the paper. Please clarify the relationship between $[a, b]$ and $[t, s]$ in Theorem 3.\n    * In the proof of Th.3, there are confusing duplications of the variables, $dr$, in the nested integrations. Please use different variables for the nested integrals, e.g., $dr$ and $d\\tilde{r}$.\n    * $=0$ in the middle of the equation around L554 is possibly a typo.\n\nQUESTIONS:\n* Could you please provide more detailed discussion and information regarding the failure cases of consistency models? Alternatively, performance comparison with consistency models on another benchmark would be also helpful.\n* Could you please address the issues raised in Weakness (3)?\n\nLIMITATIONS:\nOne potential limitation is that $|s-t|$ should be very small to ensure the convergence to the desirable solution in practical cases.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach to accelerate diffusion model inference by learning to integrate ODEs through \"secant losses,\" which approximate the average derivative (secant) between two time points instead of relying on traditional numerical solvers or complex distillation techniques. The authors introduce several variants of secant losses (SDEI, STEI, SDEE, STEE) and demonstrate their effectiveness on CIFAR-10 and ImageNet-256×256, achieving competitive FID scores with fewer steps (e.g., 10-step FID of 2.14 on CIFAR-10 and 8-step FID of 1.96 on ImageNet).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The idea of learning secant approximations instead of relying on traditional ODE solvers or distillation is innovative and well-motivated from a geometric perspective.\n\n2. The paper provides theoretical guarantees (Theorems 2 and 3) that justify the proposed losses under certain conditions.\n\n3. The method achieves competitive results on standard benchmarks, outperforming some fast samplers and distillation techniques.\n\n4. The approach can be applied via fine-tuning or distillation and supports classifier-free guidance.\n\nWeaknesses\n1. Table 2 shows underwhelming results: At comparable or larger NFEs, the method underperforms \"Joint Training,\" \"Few-step Distillation,\" and \"Few-step Training/Tuning\" approaches (e.g., CTM, IMM, rectified flows). This raises concerns about whether the theory overlooks critical practical factors (e.g., error accumulation in large steps).\n\n2. Theoretical assumptions may be too restrictive: The Lipschitz condition is not empirically validated, and the paper does not discuss how violations affect performance.\n\n3. Performance varies drastically across loss types (e.g., SDEI vs. STEE), suggesting the current formulation may not be SOTA. The authors should justify why certain losses (e.g., STEE) perform better and whether the theoretical framework aligns with empirical trends.\n\n4. Theorems 2 and 3 only prove unbiasedness for SDEI and STEE. Missing proofs for STEI and SDEE are a significant gap, as these losses are empirically evaluated.\n\n5. Importance sampling (Eq. 18) is unmotivated: The paper does not explain why uniform sampling is insufficient, especially since training iterations typically compensate for variance.\n\n6. The nested Monte Carlo estimation (e.g., sampling $r$ and $t$) likely requires large sample sizes for unbiasedness, but the paper omits analysis of convergence rates or computational trade-offs.\n\n7. Presentation Issues: (1) Typos: Theorem 3 mixes bold/non-bold variables ($x_1$ vs. $\\mathbf{x}_2$).  (2) Proposition 1 lacks attribution: This is a known result (e.g., from diffusion/flow matching literature) but is presented without citation.\n\nQUESTIONS:\n1. Please quantify the sample complexity introduced by nested Monte Carlo. How much does the inner-loop Monte Carlo increase sample complexity? Does it negate the claimed efficiency gains?\n\n2. Why does the method underperform alternatives (Table 2) despite elegant theory? Are the assumptions (e.g., Lipschitz tangents) unrealistic in practice?\n\n3. For Table 7, why do some losses (STEE) work better? Is the theory missing key practical considerations?\n\n4. Provide proofs for STEI and SDEE unbiasedness. Currently, only SDEI/STEE are covered.\n\n5. Justify the importance sampling in Eq. 18. Why not uniform sampling given enough iterations?\n\n6. Compare wall-clock time (not just NFE) against distillation methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel acceleration and distillation method aimed at significantly speeding up the inference process of diffusion models. By leveraging the framework of ordinary differential equation (ODE) integration, their approach learns an efficient solver that approximates the continuous reverse diffusion dynamics. This enables a dramatic reduction in the number of sampling steps required during generation, thereby balancing the trade-off between computational cost and model performance. The proposed method not only preserves the high quality of generated samples but also reduces inference latency, making diffusion models more practical for real-world applications. Through extensive experiments, the authors demonstrate that their approach achieves competitive performance compared to standard diffusion solvers while requiring substantially fewer function evaluations\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper introduces a geometrically motivated “secant loss” that bridges the gap between diffusion models (tangent) and numerical integration (secant), offering a fresh perspective on few-step diffusion inference.\n\n2.The use of Monte Carlo integration and Picard iteration to derive the loss is theoretically sound and practically simple.\n\nWeakness:\n\n1.The proposed method shares a similar starting point with MeanFlow, as both optimize the average velocity field based on the integral of the instantaneous velocity field. Could the authors clarify the key differences between their approach and MeanFlow?\n\n2.MeanFlow directly learns the average velocity field, making its objective function simpler, with complexity concentrated in differentiation. In contrast, the proposed method eliminates reliance on differentiation but focuses more on learning a more consistent (straighter) sampling trajectory, which seems more suitable for distillation rather than training a model from scratch. This suggests that MeanFlow might have broader applicability. Could the authors further discuss the scalability of their method?\n\n3.The one-step generation performance appears relatively weak. Could this be due to the limitations of the proposed loss function, which does not directly learn the average velocity field itself? More results on one-step generation would be helpful.\n\n4.The paper introduces many similarly named variants of the method, making the methodology and experiments somewhat difficult to follow. A clearer distinction between these variants would improve readability.\n\nQUESTIONS:\nSee details in above weakness. Authors should answer these questions corresponding to above comments: 1. Key Differences with MeanFlow; 2. Scalability and Applicability; 3. One-step Generation Performance; 4. Clarity on Method Variants\n\nLIMITATIONS:\nYes.  The authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel family of training objectives, termed \"secant losses\", to train few-step diffusion/flow neural solvers. This is framed as learning the \"secant\" of the ODE trajectory by matching it to the average of the \"tangents\" along that path. The final, tractable loss functions are derived using principles from Monte Carlo integration and a Picard-iteration-inspired estimation scheme. The framework is flexible, offering four distinct variants for both distilling a pre-trained teacher model and training a few-step generator from scratch. Empirically, the method demonstrates competitive performance and superior training stability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. A primary strength of this work is its originality and the clarity of its core concept. The essential proposal of the paper, that the average velocity is the average of (instantaneous) velocities, is very intuitive.\n2. Writing is clear and easy to follow, and the figures are well-made, e.g. Fig.1.\n3. A good collection of ablation studies that validate the design choices and provide practical guidance to users.\n\n**Weakness**\n1. Consistency models, or any model that does any-size jumps (like MeanFlow, shortcut models), can be understood as a gradual transition from tangents to secants. The contribution of the work is not really a framework or the contrast between tangents and secants, but rather a new way of training these any-step predictors.\n2. Since the contribution of the paper is really a new way of training an existing family of models, we need to compare their empirical performances. It seems like in almost all cases, it lags behind IMM without the help of guidance intervals. The scores on CIFAR are not really convincing either, given the competition. And if we include the comparisons with MeanFlow, the presented results do not inspire confidence in the proposed method. I understand MeanFlow is concurrent work (so I do not really consider this in my evaluation), but it still will greatly hinder the impact of the paper, i.e. why use the proposed losses over others?\n\nQUESTIONS:\n1. I do not quite understand the significance of the sampling of r. The theory demands a uniform weighted average right? So even if one utilizes a different sampling distribution, they need to rebalance via importance sampling.\n2. The end-point version of the training requires the model to be able to invert (from data to noise), which could be a waste of model capacity.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}