Learning to Integrate Diffusion ODEs by Averaging the Derivatives
Authors: Wenze Liu
Xiangyu Yue
Venue: NeurIPS 2025 poster ()
Keywords: generative models
diffusion models
Primary area: deep_learning

Abstract: To accelerate diffusion model inference, numerical solvers perform poorly at extremely small steps, while distillation techniques often introduce complexity and instability. This work presents an intermediate strategy, balancing performance and cost, by learning ODE integration using loss functions derived from the derivative-integral relationship, inspired by Monte Carlo integration and Picard iteration. From a geometric perspective, the losses operate by gradually extending the tangent to the secant, thus are named as secant losses. The target of secant losses is the same as that of diffusion models, or the diffusion model itself, leading to great training stability. By fine-tuning or distillation, the secant version of EDM achieves a $10$-step FID of $2.14$ on CIFAR-10, while the secant version of SiT-XL/2 attains a $4$-step FID of $2.27$ and an $8$-step FID of $1.96$ on ImageNet-$256\times256$. Code is available at \url{https://github.com/poppuppy/secant-expectation}.

Learning to Integrate Diffusion ODEs by Averaging the Derivatives
Wenze Liu
Xiangyu Yue"
MMLab, The Chinese University of Hong Kong
Abstract
To accelerate diffusion model inference, numerical solvers perform poorly at extremely small steps, while distillation techniques often introduce complexity and instability. This work presents an intermediate strategy, balancing performance and cost, by learning ODE integration using loss functions derived from the derivative- integral relationship, inspired by Monte Carlo integration and Picard iteration. From geometric perspective, the losses operate by gradually extending the tangent the secant, thus are named as secant losses. The target of secant losses the same as that of diffusion models, the diffusion model itself, leading to great training stability. By fine-tuning or distillation, the secant version of EDM achieves 10-step FID of 2.14 CIFAR-10 10, while the secant version of SiT-XL/2 attains a 4-step FID 2.27 and an 8-step FID of 1.96 on ImageNet-256 x 256. Code is available /tsttp::/gginae com htwp//opppppysscccnttttto exppeetttttteeeeetttiiio..o
1 Introduction
Diffusion models denoising their noised versions, which be formulated by stochastic differential equations (SDEs) and the corresponding probability flow ordinary differential equations (PF-ODEs) [4]
4 generate images by reversely
E(0)
(R,..))
1)f (x..r)
Over recent years, diffusion models have transformed the land- scape of generative modeling, across multiple modalities such as image [5] 6 [7] 8), video 99 TO I1] and audio 12 13 14).
Xx.t
fe (x t,
Despite their remarkable performance, a significant drawback of diffusion models is their slow inference speed: they typically require hundreds to thousands of number of function evalua- tions (NFEs) to generate a single image. Considerable research has focused on reducing the number of required steps, with common approaches falling into categories including faster 18 for diffusion SDEs PF-ODEs, and
(a) Estimating interior point
(x,,,)
(t r)re (x,,1.
fa (R,,,, t,
samplers [15 [6]
17
(b) Estimating end point &;
diffusion distillation
19
20
21
22
23 24
25
26
27
28 29
30
Figure 1: Formulation of secant losses. We employ an Eg loss be- tween the learned secant and tan- gent at random interior time point. By estimating either the interior the end point, we derive two vari- ants shown in (a) and (b). times face risks of model collapse and over-smoothing in the generated outputs.
35 methods that distill pretrained diffusion models to few-step generators. However, faster samplers ex- perience significant performance degradation when operating with small number of function evaluations (typically fewer than NFEs). Meanwhile, distillation approaches frequently introduce substantial computational overhead, complex training procedures, or training instabilities. Additionally, they some-
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 2: Selected 8-step samples on ImageNet 256 x 256.
In this work, we propose a simple intermediate solution to reduce the inference steps of diffusion models through easy distillation fine-tuning. A diffusion model constructs the PF-ODE by fitting the derivative of the noised image with respect to time, as indicated by the green line in Fig. From geometric perspective, we refer to the diffusion model the langent since describes the instantaneous rate of change of the noised image as the time difference approaches zero. Correspondingly, we define the rate of change of the noised image between two time points as the secant, marked in orange in the figure. Unlike diffusion models, we use neural networks to model the secant function instead. Based on the observation that the secant is the average of all tangents between two time points, we establish an integral equation as our loss function, termed secant losses. This average is implemented by taking the expectation with respect to random time variables in Monte Carlo fashion. Since the secant or the tangent at each training iteration, we let the model estimate the other, inspired by Picard iteration. Two scenarios for estimating the interior the end point are shown in Fig. ! (a) and (b), respectively. This results in a straightforward loss formulation: the distance between the learned secant with respect two given time points and tangent at random intermediate time between them, as presented at the top of Fig. Intuitively, secant losses work by gradually extending from the tangent to the secant. Unlike consistency models [33 discretization errors or rely on explicit differentiation loss calculation, our approach avoids explicit differentiation while preserving accurate solutions at sufficiently small time intervals under mild conditions. More importantly, the target of our secant losses is identical to that of diffusion models or the diffusion model itself. This parallel to diffusion models provides significantly greater training stability compared with consistency models.
only sample one noised image for evaluating either
35 34] 36), which either introduce
We evaluate our method on CIFAR-10 (37) and ImageNet-2566 256 38 datasets, using EDM 6 and SiT 33] as teacher diffusion models, respectively. Our experiments demonstrate that diffusion models can be efficiently converted their secant version, with significantly slower accuracy degradation compared to conventional numerical solvers as the step number decreases. On CiFAR-10, our approach achieves FID scores of 2.14 14 with 10 steps. For ImageNet-256 x 256 in latent space, obtain a 4-step FID of 2.78 and an 8-step FID of 2.33. With the guidance interval technique [4), the performance is further improved to 2.27 with 4 steps and 1.96 with steps.
2 Related Work
Few-step diffusion distillation and training. Reducing inference steps in diffusion models is commonly achieved through distillation. As an application of knowledge distillation [41), direct distillation methods [9] [00 generate noise-image pairs by sampling from pretrained diffusion model and train a one-step model on this synthetic dataset. Similarly, progressive distillation [21 22 iteratively trains the models to merge adjacent steps toward one-step model. These methods often introduce significant computational overhead due extensive sampling or training costs. Adversarial distillation approaches 23 24 25 apply GAN-style losses [2] to supervise the one-step distribution,

---

potentially increasing training instability, Variational score distillation 226 27 28 29 30
31 32
and score identity distillation 44 44 also employ distribution-lever distillation, while introduce additional complexity due to the simultaneous optimization of two online models, and may potentially result in over-smoothing artifacts in the generated outputs. Consistency distillation [33 35
leverages the consistency property of PF-ODEs to train models to solve these equations directly, but faces stability challenges [33 35| and may require model customization [35]. While direct
training of consistency models [33 45 35 is feasible, they typically underperform consistency
distillation under high data variance [35| Similarly, Shorteut Models [46] utilize the consistency
property regularization in the loss function, which could be considered as simultaneously training and distillation. Rectified flow 47 48] adopts a multi-time training strategy aimed at gradually straightening the trajectory of the PF-ODE. Similar to consistency models, our work also introduces losses for both distillation and training. However, our loss function emphasizes local accuracy, which enhances stability, though the accuracy decreases more at larger time intervals.
Fast diffusion samplers. Common numerical solvers, such as the Euler solver for PF-ODEs and the Boller-Mauuyama solver for SDEs, typically require hundreds to thousands of NFEs to sample an image from diffusion models. Since SDEs involve greater randomness, samplers based on them generally require much more NFEs to converge 2 49
6
50) [1]] 52 53]. Consequently, existing fast
samplers are usually based on PF-ODEs, including both training- free
freenng-feeeeeeee and learnable methods. Training- free methods leverage mathematical tools to analyze and formulate the solving process. Examples include high-order samplers like the Heun sampler (6], exponential samplers [16] 17 [54] 55 and parallel samplers (7). Despite their convenience, these methods struggle in low NFE scenarios. Data-driven, learnable samplers enhance performance by training lightweight modules to learn hyperparameters in the sampling process, such as coefficients (58), high-order derivatives (59], and time schedules [18 60 62] 63 64). However, both training-fres and learned faster samplers still
experience significant performance degradation when the step count falls below ten. Our method is also based on solving the diffusion ODE, and it degrades more gradually than conventional diffusion samplers as the step count decreases. Our approach can be seen an effective compromise between fast samplers and diffusion distillation methods.
3 Diffusion Models
For simplicity, we review diffusion models within the flow matching framework [65 47 66 Let Pa denote the data distribution, and æo ~ Pa represent a data sample. Let z ~ NN I) be a standard Gaussian sample. The noised image with respect to and at time ( is defined as 2t 0,400 + 0, where E [o, 1] represents the intensity added noise. For a given neural network vg the training objective is formulated as:
(D) = Ezo.,, (0 t) - (a/20 + o(2))1
where of2 and of are time derivatives. An established result from the literature on diffusion and flow matching 65 66 is as follows:
Proposition 1. When (ng(() reaches the minimum, the optimal solution oz t) is
v( t) Bm. (a/20 + z(z)
(2)
And the associated PF-ODE
dx: = v(x,,t) t) dt guaranteed to generate my ~ Pa at starting from NO, I) ttt= =1
(3)
4 Learning Integral with Secant Expectation
In this section, we first introduce how to parametrize the model as the secant function. We then explain our derivation of loss functions based on Monte Carlo integration and Picard iteration.
4.1 Secant Parametrization
Given the PF-ODE Eq. 3
solve x. at time s starting from æ, at time t, one calculates
x. = 24 + J, v(x,,, r)dr.
(4)
3

---

Algorithm 1 Secant Distillation by Estimating the Interior Point (SDEI)
Algorithm 2 Secant Training by Estimating the End Point (STHE)
Input: dataset D, neural network fa- teacher diffusion model v, learning rate
Input: dataset D, neural network f o. learning
rate repeat
repeat
0
++0
0 +
Sample x ~ D, = 10
Sample ~ D, 2 ~ (0,
Sample 1 and
Sample and
Sample r ~ 21 [0, 11. t r(s
Sample ~ 44 11. t r(s
Dt to (1 t)
3, + r + (1 r)z
c, - 2t + (r t)f 10((x,, r)
ã f 2, + (t 1 1-rr)fo (xr.
0, f rl r),
u, x -
(()) R...t.t.r
( ( (1 -1,
[()) Bez.ts f L, (,
urrl1
< V)
0 < V)C(())
until convergence
until convergence
We define the secant function from 2; at time to time as
2 v( t),
= 5,
f(x,,
8 t [ r)drr
(5)
if =
Since
J. v(,,, v(x t),
f(x,,tt)) lim f (xr,, e-28
s.
lim 8-8
(6)
8
f (1t,t, s) is continuous at And, we parametrize the neural network f (2x,, t, represent this secant function. If fa t, s) is trained accurately fitting f(x,, t, s), we can directly jump from 2t to ws using
x. Dc + [. v( r)dr a. + (s (0
We choose this parameterization for its simplicity and clearer geometric interpretation, though other similar formulations [3 36 46) would also viable in practice.
4.2 Secant Expectation
Examining Eq. (5) from probabilistic perspective, we have
1 // u(zr, r)dr = ((
f t, -))
s-t
(8)
which indicates that can calculate the integral ب Je o(æ,,r)dr by uniformly sampling random values of r ~ (t, s and computing their average. However, during training, we cannot obtain a large number of calculate this integral accurately. We address this challenge by formulating the integral as the optimal solution to a simple objective that involves only one w, at time:
LNave( (z.....-U((..)) (offo(z, t, v(x,,, r)112
This approach is inspired by the diffusion objective that leads from Eq. 1 to Eq. (2) Inspecting Eq. (9). we observe that we can only access either 2; æ, at each training step, but not both simultaneously. To address this issue, we draw inspiration from Picard iteration by estimating one of the solutions between and v, given the other. We refer to the family of obtained loss functions in this way as secant losses. Specifically, one way is to sample 2;, = agao + a. and estimate æ, using
- cz (r t)f (2,,t,))
(10)
We do not assume specific order relation between and s, and we sightly abuse the notion r u(t,s) by meaning U(min(t, s), max(t, (}) for simplicity.

---

where 0 denotes the stop-gradient version of 0. This transforms the loss function in Eq. (9) into
LSDEI(0) = -E.........rr(((t.s))) IIf (z,,, v( + (r - -f (2 i, r),,))))) r))
(11)
where ESDEI stands for secant distillation by estimating the interior point, meaning we sample the end point and estimate the interior solution at r, According to Eq. (6). we can also directly train the few-step model by incorporating the diffusion loss termed as
Lsna (0) Ex (.t. Ilf (Tt,
t) - fo (It + (r t)f (Iz, t, r.)
(12)
+XEmo II (z T, 7) (a. o22)
where À is a constant to balance diffusion loss and secant loss, and T is a time step indicating the time sampling in the two parts independent. Here STEI denotes secant training by estimating the interior point.
Alternatvely,
sample x, = D. + 0, and estimate the solution at t from as it = 3r + - r) (2r, r,t),
(13)
which we term secant distillation by estimating the end point (SDEE). The loss from Eq. (9) then becomes
ESDEE (0) (. (ff (ar + (f (z,,,,,), (, va r)1 (14)
In the above equation, since we directly use the sampled x, evaluate vi (2,,, r). we can alternatiely
use To + 0f to estimate v(zr, r) like diffusion models do in Eq. 4 to Eq. (2). This leads to the training version termed as secant training by estimating the end point (STEE):
LSTEE (0) F..., [ + (1. (1- -f (Irr, v, l), 8) (a + o(=)
(15)
As theoretical justification for the above losses, we present the following theorems, which corresponds to CSDEI (0) and CSTEE(F), The other two can be derived from these.
Theorem 2 (SDEI). Let fo tt, 8) be neural network, and v( t) E(a + (21). Assume
oxz, t) is L-Lipschitz continuous in its first argument, i.e., [| 1, t) - n Ø2. t)lla S || - æ2l12
for all 1 22 R", L [0, 1]. Then, for each fixed I, in sufficient small neighborhood for some h > 0, if LSDEr(0) (0) reaches its minimum, we have Jo( 1,8) (2t,,
4
Theorem 3 (STEE). Let folær, t, s) be neural network, and v( t) 18 + Assume fo fo t, s) is L-Lipschitz continuous in its first argument, i.e., [z)
fo ,))l2 S
L| - Iz l12 for all æı, Ty R", t, [0, 1]. Then, for each fixed ba C [0, with b sufficiently small, if (6T reaches its minimum, we have f fo t, s) = ff f 8) for any [t,s| S (a.
Remark 4. There are notable differences between variants that estimate the interior or end point. As illustrated in Fig. D (a), the estimation direction of &, aligns with the direction from æ; to Es. Consequently, Theorem states that we can fix t and progressively move s further away. In contrast, for variant (b), the direction between é; estimation and secant learning must be opposite. This requires sampling time pairs where and s can appear in any order for accurate estimation. As a result, Theorem 3 shows that increasing the distance
t| is accompanied by expanding the interval where and s are randomly sampled rather than simply moving. s. In short, t and s play symmetric roles in scenario (b), whereas their roles can be asymmetric in scenario (a).
The proofs of the above theorems rely on properties of conditional expectation and techniques related to the -icard-Lindelff theorem, specifically by constructing a convergent sequence and applying the Banach fixed-poine theorem. Detailed proofs of the total four losses are provided in Appendix [A] We note that while the proofs only guarantee local accuracy, the expansion to larger time intervals is achieved through bootstrapping process. We provide illustrations of using LSDEI (Eq. (IT) for distillation and LSTER (Eq. (5)) for training in Algorithm [ and Algorithm respectively. And we illustrate the other two in Appendix D
Comparison with consistency models. To achieve the same objective of fitting the average velocity (the secant), consistency models (33] (MeanFlow (67) uses differentiation to construct its loss, whereas our method uses integration. With identical model parameterization, the two methods can be viewed as differential and integral counterparts. The above fundamental difference leads to the following distinctive features of secant losses: i) Local accuracy. Consistency models rely on either iiteereneebbasel approximations of derivatives or explicit derivative terms in their loss functions, which can lead to training instability. In contrast, our loss functions do not involve explicit derivatives. and the solution is accurate when 8 is near to t.
5

---

Table Cost comparison among dif- fusion loss and secant losses.
sompling ongeots
samplingagge ets
Loss
Teacher
#Forward
#Backward
Cni
(a) Discrete sempling of stop sampling only
(b) Continuous sampling
Csnm
4. more stop sampling support
CaTEI
X
Camm
/
3
Figure Discrete vs. continuous sampling of when estimat- ing the interior point.
Cam
X
2
1
ii) Target stability. We refer to the stability of the prediction target in the loss function as target stability. Under the flow matching interpolant and model parametrization Eq. (5), compare diffusion loss Eq. I secant losses and consistency
140
Crigl Mutt
Cango foutcn
sunter Desteran
sotttnes
loss 33 34 67
(cr(0) = (f., (of ao (,))
(16)
and
d dt (a t, s))
CCD(0) fl ())
Torring Beration
(17)
Figure 3: Standard deviation of in- spected terms tions. The label format follows loss type (inspected term). The target stability of secant losses is signif- icantly better than that of consis- tency losses.
where CT and CD stand for consistency training and consis- tency distillation, respectively. One tency losses, the item 28 fa (x,, s) 5) (or its discrete version (2- (2. is model-dependent and suscepti- ble numerical issues, which contributes to training instability. In stark contrast, the target of secant losses is either identical to the target diffusion losses (a/æs + o(z), or the diffusion model itself (v(2,, We illustrate standard deviation of different losses across training iterations in Fig. This intrinsic target stability provides a compelling explanation the robust training performance observed with secant losses.
training itera-
that in consis-
4.3 Practical Choices
Our proposed method is designed for simplicity and robustness, mirroring the design of standard diffusion models. In general it uses: i) A simple time sampling strategy (following diffusion models), ii) a standard loss weighting (following diffusion models), iii) a simple Mean Squared Error (MSE) loss (following diffusion models), iv) stable loss target discussed in Section 4.2 The strong parallels between our method and standard diffusion models provide powerful cue: if standard diffusion
model loss works well on a given dataset, is highly likely our secant losses will too. Here we will discuss more practical designs in application.
Diffusion initialization. As explained in Remark 4 initialization such that ( f, t) v(2,, t), which largely accelerates the learning process.
pretrained diffusion model the
Application of classifier-free guidance (CFG) [68]. For all the secant losses except LSTEE. can embed CFG as an additional input into the model (22]. Specifically, in the loss functions we substitute v( with
o (x,, t) (x,, + ( (x,,))
(18)
where is the guidance scale, and the superscripts g. and stand for the model with guidance, the unconditional model and the conditional model, respectively. In contrast, the treatment to LYTEE is similar to training diffusion models. We can train unconditional and conditional models by randomly dropping the class label, and apply CFG at inference via
f 4 8) f (m,, 1,8) 8) + w (a, (, (20,, L, 8))-
(19)
Step intervals at each sampling step. Adjusting the step intervals may lead to better performance 6 18] While for simplicity, always use uniform sampling steps. Specifically, if the number of steps

---

is N, then the steps (t, are (A - = 0, ..... N 1. We provide details of sampling in Appendix[]
The sampling of and 8 when estimating the interior point. Theorem 2 states that we can fix and randomly sample This enables few inference fixed step count N by setting N-1,.. t E {1, NN .... :.... N: 0} in training. When f = N. we randomly sample s E [ J However,
this approach constrains us using exactly N steps during inference. Figure 4 illustrates this with an example where N = 4. If we attempl to use more NFEs, such as 2N,, we cannot determine the value of f0 2N-1 2N-2) at the second step, as the model was not trained with 2N-1 2N
2N
2N
X because the case where t= and 2-2 was not included in training. To enable a flexible step-accuracy trade-off in LSDEI we should continuously sample i, as shown in Figure E (b). This approach allows inference with any number of NFEs greater than N. Furthermore, if the model is required to perform inversion during training. The sampling strategy of t and can chosen according to specific application requirements, due to the limited model capacity. However, for variants that estimate the end point, the sampling of t and must be both continuous and bidirectional.
Similarly, with fewer NFEs, such as 7. we cannot compute fo
N
from images back to Gaussian noise, should incorporate cases of
The sampling of r. In Eq. 8 we can alter the sampling distribution of r by
J. o( (r)tt.
f( 1,
= S. o( r)
(r) ) q())dr
(20)
u(r)
= ( (r( g(r)
where (r)) is another probability density function that has positive density value within the limits of integration. This means we can unevenly sample based on the importance, i.e., how we allocate the training effort across different time intervals. We note that while importance sampling can reduce variance, it is not the focus of our paper. The method used in this paper to reduce variance is to employ stable target.
Resource costs. As the loss formulations suggest, compared to the diffusion loss LDir, our proposed loss functions require additional computational resources for forward evaluation and/or backward propagation. We summarize the computational costs in Table ! and test the practical usage in Appendix G.2
5
Experiments
In this section, we first compare our method with other related approaches, Subsequently, we conduct ablation studies validate the design choices of the proposed loss functions.
5.1
Implementation Details
conduct experiments on common used image generation datasets including CIFAR-10 [37 and ImageNet-256 256 x 256 [38 For quantitative evaluation, employ the Fréchet Inception Distance (FID) [87] score for both datasets, with additional Inception Score (IS) metric for ImageNet-2555 x 256. There UNet-based EDM [6] and transfommrrbased DiT [ models involved in the experiments, and we load the pretrained weights of EDM and SiT 39 for the two models respectively. More details can be found in Appendix
5.2 Main Results
Overall Comparison. We conduct comprehensive comparison among various few-step diffusion approaches, including fast samplers, few-step distillation, and few-step training/fine-tuiing methods. The results are summarized in Tables Sitees[[aa and 3 On CIFAR-10, our SDEI variant achieves interme- diate performance between faster samplers and few-step distillation/rraiingg methods. Although
7

---

Table 3: Class-conditional results on ImageNet-255 256.
Table 2: Unconditional image gen-
eration on CIFAR-10.
Method
FID4
IS
Steps
#Params
Diffusion
Method
FID Steps
Diffusion
ADM
10.94
100.98
250
554M
CDM 75
4.88
158.71
8100
DDPM
3.17
1000
SimDiff 76
2.77
211.8
512
2B
Score SDE (deep) 4
2.20
2000
LDM-4 8
3.60
247.67
250
400M
EDM [ (Teacher)
1.97
35
U-DiT-L
3.37
246.03
250
916M
Flow Matching 66
6.35
142
U-ViT-H 78
2.29
263.88
50
501M
Rectified Flow
2.58
127
DIT-XL/2
2.27
278.24
250
675M
SiT-XL/2
30
(Teacher)
2.15
258.09
250
675M
Fast Samplers
Fast Samplers
DPM-Solver [15]]
4.70
10
DPM-Solver++ 16
2.91
10
Flow-DPM-Solie
79
16
3.76
241.18
8
675M
DPM-Solver-v3 17
2.51
10
UniPC
55
7.51
10
DEIS [54]
4.17
10
LD3
64
4.32
7
UniPC 55
3.87
10
GAN
LD3 66
2.38
10
Joint Training
BigGAN NO
6.95
171,4
1
112M
GigaGAN 81
3.45
225.52
1
590M
Diff-Instrue
29
4.53
1
StyleGAN-XL XL
2.30
265.12
166M
DMD
3.77
1
Masked and AR
CTM 34
1.87
2
SiD 45
1.92
1
VQGAN 83
15.78
78.3
1024
227M
SiD A 69
1.5
1
MaskGIT
6.18
182.1
8
227M
SiM 77
MAR-I 85
1.55
303.7
256
943M
VAR-d30 86
1.97
334.7
10
2B
Few-step Distillation
Few-step Training/Tuning
KD 19
9.36
1
4.51
2
iCT 73
20.3
2
675M
DFNO 20
3.78
1
Shortcut Models 46
7.80
4
675M
2- R-eetifiee Flow +7
4.85
1
IMM (XL/2) 74
7.77
1
675M
TRACT 71
3.32
2
3.99
2
675M
PID
3.92
1
2.51
4
675M
CD 38
2.93
2
1.99
8
675M
sCD
35
2.52
2
STEI (Ours)
7.12
241.75
1
675M
Few-step Training/Tumi
4.41
242.00
2
675M
iCT
73
2.46
2
2.78
269.87
4
675M
+guidance interval 40
2.27
273.76
675M
ECT
45
2.11
2
2.36
247.72
8
675M
sCT
35
2.06
2
+guidance interval
40
1.96
276.12
8
675M
IMM
74
1.98
2
STEE (Ours)
3.02
274.00
4
675M
SDEI (Ours)
3.23
4
+guidance interval
40
2.55
275.83
675M
2.14
10
2.33
274.47
8
675M
+guidance interval
40
1.96
275.81
8
675M
consistency models [33 45 34 35) demonstrate superior performance this dataset, observe training instability issues. More notably, when scaling to ImageNet-256 x 256 with greater data variation, these models diverge in experiments In contrast, proposed loss functions exhibit consistent stability and rapid convergence, enabling efficient model distillation or fine-tuning that requires merely 1% of the original SiT training duration (50K 100K versus 7M iterations). On the larger-scale ImageNet-256 x 256 dataset, our approach consistently outperforms fast samplers and ranks second only to the recent IMM 774) in both 4-step and 8-step settings. Notably, for one-step generation, STEI even surpasses both IMM and 4-step Shorteut Models [46] By incorporating interval CFG [40), our method further achieves 4-step FID of 2.27 and 8-step FID of 1.96, where we simply ignore the guidance for the first step in 4-step generation and the first two steps in 8-step generation. Fig. 2 showcases generated samples using steps on ImageNet-256 256 with our
best-performing model. While both effective, our method differs from IMM: ours is grounded in the
Due training divergence of consistency models, cite the iCT 75 result reported in 74

---

Table 4: The effect of the weighting factor À in LSTE on ImageNet-256 256 with 4 sampling steps.
0365
00700
0.02
0350
0.003
A
FID4
1St
0340
0.009
0.00
0.1
3.96
20
0325
0.5
3.15
232.15
Taining teeration (0
Trining flerations Rk
1.0
2.84
249.57
2.0
3.96
213.56
Figure 5: The effect of X in LSTHI on training dynamics.
exact solution of the PF-ODE (therefore the performance of our models is bounded by the teacher model SiT (9]), whereas IMM relies on distribution-leve losses [88]. For classifier-free guidance, IMM follows Eq. 19). while STEI applies the integral- foogral-form CFG following Eq. (18). This difference may explain our superior performance in one-step generation.
Comparison of secant losses. We evaluate the four types of secant losses proposed in Section 4 on CIFAR-10 and ImageNet-256 x 256 datasets. The result can he found at Table in Appendix G In the absence of classifier-free guidance, similar trends emerge across both datasets. The distillation variants generally demonstrate superior performance compared to the training counterparts, with the exception of STEI on ImageNet-256 256. Furthermore, variants with interior-point estimation perform better and degrade more slowly as the step number decreases, which can be explained by three aspects: i) Input correctness. For inner-point variants, the input to the training network the clean data æt, while in end-point variants, the input is the estimated ω,. Fitting fixed, clean input distribution, is more efficient than fitting input dependent of the model historical output. ii) Error accumulation path, i.e., the path from clean data to the prediction destination. In inner- inner-point variants, the path is æt > S. while in end-point variants is x., >> Zt -> 2.g. Generally. the path in end-point variants is longer than that in inner-point variants. iii) Model capacity. As per Fig, [ and Algorithm 2 the and-point variants additionally require inversion, which costs part the model capacity. With classifier guidance applied, the performance gap narrows ImageNet-256 x 256, with the STEE variant achieving the best 8-step FID 2.33. Based on the computational costs presented Table] [ we recommend using LSDEI for distillation and LSTEE for training. These variants offer optimal performance while minimizing both computational time and GPU memory requirements compared to other methods.
5.3 Ablations
The loss weighting in LSTEL- We investigate the balancing factor À in LSTHL which comprises the diffusion loss and the secant loss. 5 the training dynamics of both losses (excluding the À factor in the secant loss). When À increases from 0.1 to 1.0. observe reduction in the secant loss while maintaining relatively stable diffusion loss levels. However, at À = 2, the model exhibits significant deterioration in diffusion loss, accompanied by increased instability in the secant loss. The quantitative results presented in Table that À = achieves the optimal balance, yielding the best overall performance.
iillustrates
The sampling of and s in interior-point estimation. As shown in Fig. [ LSDEI and LsTa offer flexibility in sampling the time point and s during training. Given the fixed model capacity, different sampling strategies lead to varying perfor- mance characteristics. For fixed N-step generation, discrete sampling suffices; enable step-performance trade-off, con- tinuous sampling of becomes necessary; to incorporate the capability of inversion, bidirectional sampling < and t > s) is required. The results in Table 5 demonstrate that the best performance achieved with the discrete sampling and generation-only configuration (first row), and each additional functionality comes at the cost of decreased performance.
Table 5: The effect of (, sampling on CIFAR-10 with steps.
Gen.
Inv. I. sampling FID
X
discrete
3.23
/
discrete
3.63
x
continuous
4.29
continuous
5.47

---

The sampling of r. Following Eq. 20), we know that we can use alternative distributions q(r) sample r. Here we investigate sampling strategies for The cases include standard uniform sampling and biased sampling schemes that favor positions closer to or the midpoint st2 2 These biased sampling strategies are implemented using truncated normal distributions, denoted as (J, a, b) where μu represents the mean, a the standard deviation, and [a, b] the truncated interval. Details provided in Ap- pendix E As the results in Table 6 indicates, all sampling strategies works comparable, and the uniform strategy works slightly better than others.
Table 6: 4-step results of different dis- tributions on ImageNet-256 x 256 with SDEE We first sample and obtain r r= + (s- t)r.
i distribution
FID4
1St
u(0.1) 1)
3.57
222.83
TN 0,1,
3.84
215.57
(.,0
3.59
221.44
(1,1005,0,,)
3.71
216.93
Training from scratch. We evaluate the scalability and from- scratch training capabilities of the secant loss using LSTE as the representative loss function. We conduct experiments with varying model capacities, including DiT-S/2, DiT-B/2, DiT-L/2 and DiT- XL/2. Figure 6 presents the evolution of FID scores during the first 1000K training iterations, with a guidance scale of 1.5 used sampling phase. The results clearly show correlation between model capacity and generation quality, with larger architectures consistently achieving superior performance. After extending the training to 3000K iterations, our DiT-XL/22 model achieves an 8-step FID of 2.68, which is slightly higher than the fine-tuning FID. This is expected, given that the training period of the teacher model SiT 39 (7000K iterations) significantly longer than our training duration.
40-
20
400 500 Training Iterattion (k0
6:00 7022 8000 500 2002
Figure 6: Results of training from scratch on ImageNet-256x 256. The guidance scale is 1.5.
6
Limitations
While our method can stably and rapidly convert a diffusion model into a few-step generator, there is still significant performance gap between 1-step and 8-step generation. Additionally, the performance on ImageNet 256 256 relies heavily on classifier-free guidance, and the theoretical relationship between CFG and secant losses may be explored in future work. Lastly, method requires training data, which may present constraints data-line data-limited limited scenarios.
Conclusion
In this paper, we introduce novel family of loss functions, termed secant losses, that efficiently learn integrate diffusion ODEs through Monte Carlo integration and Picard iteration. Our proposed losses operate by estimating and averaging tangents learn the corresponding secants. We present both theoretical and intuitive interpretations of these secant losses, and empirically demonstrate their robustness and efficiency on CIFAR-10 and ImageNet 256 x 256 datasets.
Acknowledgments
This work is partially supported by the National Natural Science Foundation of China (Grant No. 62306261), and The Shun Hing Institute of Advanced Engineering (SHIAE) Grant (No. 8115074). This study was supported in part by the Centre for Perceptual and Interactive Intelligence, a CUHK- led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Govermment. This work also partially supported by Hong Kong RGC Strategic Topics Grant 4001/4-403/24-N, and CUHK-CUHK CUHK-CUHK(S2)-DDS)) Joint Collaboration Fund YSP26-4760999.
References
[1] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. pmlr, 2015.
10

---

[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[3] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[4] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.133550 2020.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[6] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of basfe generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
[7] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195 4205, 2023.
[8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[9] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22563-22575, 2023.
[10] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633 8646, 2022.
[11] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[12] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009 19766,, 2020.
[13] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916-13932. PMLR, 2023.
[14] Haohe Liu, Zehua Chen, Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audh generation with latent diffusion models. arXiv preprint arXiv:2301. 2023.
[15] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[16] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[17] Kaiwen Zheng, Cheng Lu. Jianfei Chen, and Jun Zhu. Dpm-solver-v3; Improved diffusion ode solver with empirical model statistics. Advances Neural Information Processing Systems, 11555002-5552,, 2023.
[18] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2021.

---

[19] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02.888 2021.
[20] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International conference on machine learning, pages 42390 42402.. PMLR, 2023.
[21] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005114 2022.
[22] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[23] Axel Sauer, Dominík Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87-103. Springer, 2024.
[24] Zhendong Wang, Huangjic Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan:: Training gans with diffusion. arXiv preprint arXiv:2206.02262, 2022.
[25] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pages 1-11, 2024.
[26] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neurul Information Processing Systems, 36:8406-8441, 2023.
[27] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613-6623, 2024.
[28] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in Neural Information Processing Systems, 37:47455 47487, 2025.
[29] Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36: 36:76525-76546, 2023.
[30] Weijian Luo, Colin Zhang, Debing Zhang, and Zhengyang Geng. Diff-instruct Towards human- preferree one-step text-to-image generative models. arXiv preprint 1rX.v.2410.20898, 2024.
[31] Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. Em distillation for one-step diffusion models. arXiv preprini arXiv:2405. 16852, 2024.
[32] Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching, Advances in Neural Information Processing Systems, 37:36046-36070, 2025.
[33] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01466) 2023.
[34] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[35] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410. 1001, 2024.
[36] Nicholas M Boffi, Michael S Albergo, and Eric Vanden- Vanden-Fijnden. Flow map matching. arXiv preprint arXiv:2406.07507 2024.
12

---

[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[38] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 225. Ieee, 2009.
[39] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pages 23-40. Springer, 2024.
[40] Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.0777 07724, 2024.
[41] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.0251,,
[42] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[43] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
[44] Zemin Huang, Zhengyang Geng, Weijian Luo, and Guo-jun Qi. Flow generator matching. arXiv preprint arXiv:2410. 10, 2024.
[45] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406. 02024406.4444,, 2024.
[46] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint arXiv:2410. 12557, 2024.
[47] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 2tt:::200000000:33 2022.
[48] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. The Twelfih International Conference on Learning Representations, 2023.
[49] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[50] Alexia Jolicoeur-Martiee Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. /4080, 2021.
[51] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2 106.022006.0013, 2021.
[52] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.06550 2022.
[53] Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05555 2022.
[54] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXhv:2204.13902 2022.
[55] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances Neural Information Processing Systems, 36: 49869, 2023.
13

---

[56] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36:4263-4276, 2023.
[57] Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, and Tsung-Hui Chang. Accelerating parallel sampling of diffusion models. In Forty-first International Conference on Machine Learning, 2024.
[58] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covari- ance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022.
[59] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150-30166, 2022.
[60] Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7777-7786, 2024.
[61] Amirmojtaha Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. arXiv preprint arXiv:2404. 14507, 2024.
[62] Defang Chen, Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ode-based diffusion sampling. arXiv preprint arXiv:2405. 11326, 2024.
[63] Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo
Li. Accelerating diffusion sampling with optimized time steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8292-8301, 2024.
[64] Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, and Mathias Niepert. Learning to discretize denoising diffusion odes. arXiv preprint arXivv2405. 15506, 2024.
[65] Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[66] Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
[67] Zhengyang Geng, Mingyang Deng. Xingjian Bai, Zico Kolter, and Kaiming Hc. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.1344 2025.
[68] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12555 12598, 2022.
[69] Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410. 14919, 2024.
[70] Weijian Luo, Zemin Huang, Zhengyang Geng, J Zico Kolter, and Guo-jun Qi. One-step diffusion distillation through score implicit matching. Advances in Neural Information Processing Systems, 37:115377-115400 2024.
[71] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. Xiv preprint 1.X1:23033.04248. 2023.
[72] Joshua Tian Jin Tee, Kang Zhang, Hee Suk Yoon, Dhananjaya Nagaraja Gowda, Chanwoo Kim, and Chang D Yoo. Physics informed distillation for diffusion models. arXiv preprint arXiv:241 1.08788, 2024.
[73] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.1411 2023.
[74] Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. arXiv preprint 2003v2250..05555 2025.
14

---

[75] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[76] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213-
13232. PMLR, 2023.
[77] Yuchuan Tian, Zhijun Tu, Hanting Chen, Jie Hu, Chao Xu, and Yunhe Wang. U-dits: Down- sample tokens in u-shaped diffusion transformers. arXiv preprint urXiv:2405.027300 2024.
[78] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669-22679, 2023.
[79] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410. 10629, 2024.
[80] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 180iv:180. 2018.
[81] Minguk Kang, Jun-Yan Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 10124-10134, 2023.
[82] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[83] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[84] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11325, 2022.
[85] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424-56445. 2024.
[86] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 7778889-84865, 2024.
[87] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[88] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In Interna- tional conference on machine learning, pages 1718-1727. PMLR, 2015.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly present the contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

For Theorem 2 and Theorem 3 we provide the assumptions and proofs in Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide experimental details in Appendix
H
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification: We only provide the detailed settings and algorithm. We will make the code public in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl wwwppbblccgguiicuceccccccoeooopp wet.ggddddeeoeeebbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Due to limited computing resources, we do not report error bars in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of computing resources in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrrppcccpppu/.cccccccaa
Answer: [Yes]
Justification:
The research conducted conform the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
This work is not yet ready for real applications, it has little societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not have data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets are properly credited, and the license and terms of use can be found in the references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will release new assets in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not include experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not include participant study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Appendices
A Proofs
Proposition 1. When
LDI = = (1),
(a/w0 + o12)11
reaches minimum, the optimal solution ( 1) is v( t) = Ex (2/20 + 0z2)))).
Proof. Since
(DIH(0) = , (,,,, 1) - (a/20 + o/2)
Ex, E. Do (x, t) - (0/20 + x(2)
Em [Bao | (.,, ()1 - 2I (1 (x, 1), 0, + [22)
+ 1 - x|
Ex.. (110g ( 112 zo ((10 (), a420 d(2) [cog]
+ RRm.= + x
= Em (1110 a.,, 1)12 2( (ee 1), Emo (2/20 + o(21(zz])
+ Emm.=1lac/ + 0
Ex,, (1|v0 (x,, 1 2vv : t), v t)) + (1 6)12
-1|v(201: t) 112 Ix |||a/200 + o/212/00.)
Ex..t (11eg t) va t) [| 1) E (ll + dz2111221))
have the optimal solution v% (2x, () ().,,))-
The above is a basic result literature diffusion models [66]. We prove here since it has close relation the following theorems.
Theorem 2 (SDEI), Let f ( s) be neural network, and v(x,, t) Eia + Assume v(z, t) L-Lipschitz continuous in its first argument, i.e., [| t) a æz2 t)ll2 S L æ x2lle for all 1 22 E R", t E [0. 1]. Then, for each fixed t, in sufficient small neighborhood is t S h for some > 0, if (0) reaches its minimum, we have J fo( = f(x,, 1,
Proof. We prove it in two steps.
First, we show that the following loss function
[ v(x, + (r 1)f r)dr12 (21)
(0) = Em s.t [f (o 4,
has the same derivative with respect to (O). Let a, 2t + (r t)f (aee, t, r). it follows since
VoLSDE(()) VgEau.ztt.
|ffo t. r)
Vo (1ffolr,, s) ( o() + ( r)1
VoEmo,z,t. (s- ( ((f( J ((kt,r))dr
4,
+ I. || 111rrrr
s J v(t,,, r)dr
= Vo - (21,, -
23

---

Second, we show that (0) = O implies f,) = f t) in each sufficient small neigh-
borhood of Using Picard iteration, we construct a sequence f, satisfying fn 0n g-t
t, r), r)dr. Without loss generality, we may assume h > We have - Se o(x, + (r - (
sup *E(t.t+Ai
|5 1sn+1 (a,, (, ((11,6,))15
(s 1)2 I. v( + - (x),
1
sup *E((,++b]
1. v(x; + (r 1)f((())))))))) (,),))
$ L sup sE(t,Ht]] (s - t)2 J. (r-t)(fnx (x),tt) f(a,, r))dr112
xE(t,tth] (s 1)2 {. (r (r- sup re]t,th]]
L sup
[Ifn(,,,t,r) t(xr,t
Lh sup (2 f ( (,),
3 sE(t,t+h)
When s = f, we also have
1 f (2 t, s)1 - L f(x,, s)1B.
1
Therefore, sup n+I (z,, s) f s)1 converges to 0 when j. < i.e. h < 1 This sE[t,(+]]
means we find a neighborhood [t, + it] for each f, such that when (SDEr(0) reaches its minimum, f f s).
we have
Theorem 3 (STEE). Let fo s) be a neural network, and v(2,, t) =
Assume fo fx) is L- L-Lipschitz continuous in its first argument, i.e., llf ,t, s) fo,,,, s)ll s
T. a|(a- - æ22/22 for all æ1, x2 E RR,, 1,8 E [0, Then, for each fixed (a, b) C [0, 1] with a
sufficiently small, if (f) reaches its minimum, we have f fo( s) = f f t, 8) for any [t, C bj.
Proof. First, show that LSTEE (0) have the same derivative with respect to 0 as the following loss function
(.) = En.. Ar. + (t r)f r,),,, - v(x,, r)1..
(22)
follows since
VoLSTEE(O) = V VeEee [. + (1-
- (0 + 222)1l2dr
Vo t. Exois ( IIl
+ / [olloo(r, (t
- a x0 + 2 xgJdr
VoBurr,t [: (Beo. + r) ( [2]
- -2Ex..> lifo (t r) (t-r)f (2,,
+ [z,]
+ Emgg |1 + 0,221110=111
VoBar. 1. llfo + (t r) o- (Z,, r,,, t), 1
-2 J. (Folar + (t- r)5o (ær, r, 1), 1, s), 20 + of.22ar.)))d
+ / Eu0o=2lla,,00 + of221/aa,,d
24

---

VoBmm,t. J. Ilf (1fo (t- r2,, v(0r, 1)
Then, notice that when L. (0) achieves its minimum, we have
(0) I. fo + (t - r) (o,,, T, t), s)dr J. v( r)drl =
(23)
r, s) 5) = fa + (t r)f 0- (ar, r, t), s). and assume (0) reaches the If not, denote 9o r,, minimum without Jt g0 (,, t, s.r r J o r )dr.
Let (er 4, 9o (r. (, + - de v(ar, r)dr - 5-t Se go, v, l s)dr, then we have
1
110 (2,, v(σ,, r1)l
L (1 r,t,s) v(x,,1) r) 3 L v(a,,r)dr s s --1 J. Io 1,
1
1
(1r, v(ær, r))
+ 1 1 r(drr 10------ s-t L. go r, (d)
1
+ 2 J. (9, v( r),
( o(x
8 8- - I go s)drr)d
= I (1go( r;1,8) v(ar,))112
+ (s t)ll 5 s- [ v(x,,r)dr
1
5 s L go r,
1
2( J. (go(zn),ttss) - v( rr, f v(z,,r)rr -
f. g0(2,,,,t,s)dd 1 J. v(x,, r)dr 8 s-t - S. r, (s - t)ll - t (1..( v(rr,r)lldd 1)2drr
L (,,( 1112dr
which is contradictory.
Next, we prove that minimizing (0) implies fo(a,, t, = f(x,, s) for all [t S [a, b] S [0, 1], where 6 is sufficiently small.
Using Picard iteration, we construct a sequence fn satisfying fo = 0, f fn + (t (z,,,, t,, tt)
When # we have
sup t,sElab)
If f t,s) f(e,, t, s)
1
[ 1l 1lf (xt, f(x,, s)
sup (,sE(a.)]
1 - I:¹ |l 1Ifmii (2,,, (n + (t r) (2,, f), 1)
< sup
(,a
8 1" 1x0 - Er xr- (t-r) (2r, 1)
L sup
(,5
f" |2 + - (t, x, - fx 12
= sup
t,)E(a.0)
25

---

J rf sup neE(a,b]
L sup
n (2rr 1)
(, 8
(s (s-
L sup
sup
(n T,) f(ar,
$ Lb a)²
sup b.Eaa,b
- far,,t,)
Also when t, the inequality satisfies.
One can see with sufficiently small a, sup IIf (xt,,t) f( s),, converges to [a)
0. Hence, there exists small interval, such that when LSTEE(!) reaches the minimum, we have fo (w),,, f s).
Corollary (STEI). Let f be neural network, and let ((rr, be the velocity field. Assume væv,,t f) is L-Lipschitz continuous in its first argument. If Cster(0) reaches its minimum, then for any interval where t and h < T. we have f folxi,t, f(z,,, s).
Proof. The loss (STE(()) is defined as a sum of a secant term and a diffusion term
(0) = / (z,, (x,, where ã, Tt + (r t)fe (xx, r). For (oT to reach the minimum, both two terms must reach the minimum. By Proposition]] | the diffusion term achieves the minimum and only if
So (ãc, r,,,,, r) + fo
(a + of x [2
f (, B(a + (1,) v(v,
Substituting this result into the secant term, it becomes the one studied proof of Theorem The remainder of the proof follows identically.
Corollary (SDEE). Let f oZ t, s) be a neural network that is L-Lipschitz continuous its first argument. For any fixed interval [a, C [0, 1] with sufficiently small, if (SDEE(()) reaches its minimum, then fo( 8) (ax, t, s).
Proof. The SDEE loss is defined as
SSEE (0) = [Ell f + (t (t-r)fo-(ae) (x,, r),,,,,)) This loss function is identical to the intermediate objective L., analyzed in the proof of Theorem The proof, therefore, follows that of Theorem directly.
B
Derivations
B.1 Applying to EDM
The training objective of EDM 6 is
LEDM (0) E. Fo (Cm(())) (y + n), Smvis(())) (o (g) Copp(r) - (y) + n)) (24)
where Fg is the neural network, w(or the loss weighting factor, Cil(o) Vo=+o?"
Coul (a)
Co)
Cnoise In((o), and 0.5 is the variance of the image data Pa- gF+07 NO,0000 2T), we can denote y by æo, n by az in our notation. Then Eq. 24 can
o3
Vo'to?
Since n be simplified to
w(O) o3
σ
LEDM (8) Eg
FoFFo
Ty +
Ouz
Croise (0)
Od
o02
00 +
o²
(25)
xn +
Oaz
Vo2
00 o
26

---

The Trigflow framework [3] demonstrates that if letting t = arctan and 2t cos(t)ax +
oi where E [0, 211 training with the simplified objective
w(l) 02
St Croise (Od tan(t)) ) (-sin(())oo + cos(t)ouz) /
LEDM (0) = 2.
(26)
leads to the diffusion ODE
dæ
Et odFo
(oa tan tan(t))
(27)
dt
0d
As result, to adapt the notions in Section
3 to the EDM loss, we can modify the range of time
to [0, 5) substitute with oa and let v( -oaFo 0t = sin(t)).
(oo Croise (an()))) ar, cos((),
[
For practical choices in transferring EDM the secant version, simply set the loss weight =(()
du Besides, since -> or > 2 makes Cooise (aa tan(t)) 4 II(aa tan(t)) co, we constrain and s within the range of [0.001, - 0.00625].
B.2 Applying to SiT
The training objective of SiT 39 is the flow matching loss
(0) = Ex. (2ce t) - (x1 z0)1
(28)
To adapt our notation to loss Eq.
28 we can substitute mo with æ,, and with æo, and set oz = t
and og =
C Model Parametrization
C.1 The Secant Version of EDM
The parametrization is similar to that in CTM (34) Specifically, we apply the same time embedder for the extra s input as that employed for I. In each UNet block, we add affine layers the embedding, which projects the embedding into the AdaGN [5] parameters, akin to that for embedding. The resulting AdaGN parameters from both and f are then added up. The weights of all the added layers randomly initialized.
C.2 The Secant Version of DiT
We clone (including the pretrained parameters) the time embedder of as the embedder. The original time embedding is replaced with half of the summed embeddings from t and s. This design ensures fa (,,,t, t) u uo(xx, (z,, t) when loading the pretrained SiT weights. For classifier-free free guidance, we add another time embedder for the CFG scale whose weights are randomly initialized.
We also tried the parametrization in Section C.T for DiT, and find that the performances compara- ble. However, this implementation introduces significantly more parameters (~ 226M), since the module for producing AdaLN parameters is very large.
D Training Algorithms for STEI and SDEE
Similar to Algorithm for LSDEI and Algorithm 2 for LSTEE we provide the training algorithms with LSTEI and LSDEF in Algorithm and Algorithm respectively.
'The loss configuration follows linear path and velocity prediction. Details can be found at https:// gi titb.b. www.womm....... com//illlsma/S111
27

---

Algorithm 3 Secant Training by Estimating the Interior
Point (STEI)
Algorithm 4 Secant Distillation by Estimat- ing the End Point (SDEE)
Input: dataset D, neural network f learning rate
Input: dataset D, neural network Se teacher diffusion model v, learning rate
repeat
++0
Sample x ~ D, 10
repeat
Sample 1 and
8 0
Sample r ~ (0 11. t+ r(s-t)
Sample x ~ D, ~ N(O,I)
Dt to (1 t)
Sample l and 8 [
2, + 2, + (r t)f r.)
Sample r ~ 10, + -
t- + r-r
v, - r(
Sample E (0,
3r
ē; t x, + (t- (t-r))fe(xx.r11 (x,, [
2r + Ta (1 -)2
U, +-++- -
0a r)
(( = Exsst.r 1f fo s) fo
0 + - V)) until convergence
+ x ff f ur11
0 - 8 11 v)((()
until convergence
Algorithm Sampling i, in SDEI and STEI for EDM
Algorithm 6 Sampling of 4, in SDEE and STEE for EDM
Input: Gaussian distribution parameter Pimea and Pstd, number of steps N, boundary con- stants E₁ and 62
Input: Gaussian distribution parameter Piman and PAu. number of steps N, boundary con- stants 6 and 62
Output: sampled time point and
Output: sampled time point i and
Sample a ~ ~N Psa)
Sample o ~N([meaa Pss)
arctan( 2.)
arctan( _))
.T
Clip into [e1. $ - E2]
Clip l into (e1, c2)
Round to be discrete
Sample (0, 1
Sample d ~ (0, + + d Clip s into (61, czl
d + d with probability of 0.5 s t
Clip into [e1, E2)
Algorithm 7 Sampling of 1, 8 in SDEI and STEI for DiT
Algorithm 8 Sampling of L, s in SDEE and STEE
for DiT
Input: number of steps N
Input: number of steps N
Output: sampled time point i and s
Output: sampled time point i and
Sample d ~ 2/(0, 4)
Sample d ~ (1 +
Sample f ~ 74(0, d-
Sample t (10,1-d)
Round be discrete (SDEI only)
+- + d
+ 1
Swap and s with probability 0.5
E Sampling Time Points During Training
Sampling: and for EDM. We sample from the Gaussian proposal distribution with Pmean = - 1.0 and Puu = 1.4 (35), which we find sightly better than the default configuration of Pinaan 11. and Pau 1.2 in EDM (6) and derive by 1 = arctan( (-) Then, we sample the distance d = |s || by ~ (0, )), where is the pre-set number of steps. the loss estimates the interior point, we round the sampled / to be discrete; while for losses that estimate the end point, we randomly multiply -1 with probability 0.5. Finally, we get s by illustrated in Algorithm 5 and Algorithm 6
d. The sampling processes
28

---

Samples
Samples FundkernttssA1
Sample Tuschemdoo.00
08
0.6
06
0.6
0.
0.4
0.4
0.4
0.4
0.
02
02
02
0.0
02
02
04
0.6
0.
1.0
0.0
22
04
0.6
04
1.0
0.0
02
0.1
2.6
08
1.0
0.0
0.0
02
0.1
16
08
1.
Figure 7: Uniform distribution and truncated normal distribution under different values.
Sampling and for DiT. We first sample the distance d = by d u1(0, k). Then,
sample t by t ~ 14(0, -d) d). If estimating the interior point, we round t to be discrete and derive by = t d: if estimating the end point, get s by t and randomly swap t and with probability of 0.5. The sampling strategies are illustrated in Algorithm and Algorithm X
Sampling r. The default sampling strategy for r follows uniform distribution between the endpoints and 8. Specifically, we first draw (00, 1) from a standard uniform distribution, then obtain i(s t).
through the linear transformation
In our ablation studies, we explore alternative sampling strategies using the truncated normal distribu- tion. Specifically, we sample F from a truncated normal distribution F TN ar, 0, 1), where the distribution is confined to the interval [0. 1]. The value of is then obtained through the same linear transformation
f(s t). Let ((F) denote the probability density function of TN (p. a, O, 1).
Assuming s * without loss of generality, we can derive
r g~)(r) = 1s - 9( 1
(29)
Then Eq. (20) becomes
f( 1, = 10:0=Errghhn Erng(r) (20,, Purir.s) (r)
((r)
= EE
(30)
((F)
The probability distributions of under different values are illustrated in Fig.
Sampling Algorithms Inference
For simplicity, we always sample adjacent time steps with uniform spacing. The sampling process differs between STEE-trained models and those trained with SDEI, STEI, SDEE, due to their distinct handling of classifier-free guidance. The detailed sampling processes are presented in Algorithm 9 and Algorithm I0
Algorithm 9 Sampling Using Models Trained with SDEI, STEI and SDEE
Input: model fg- number of steps N, guidance scale Output: sampled image En Sample σ ~ (0,
for i = t do
N/ fola x-i, N-i N-i-1
2 P N
end for
29

---

Table 7: The performance comparison of secant losses on CIFAR-10 and ImageNet-256 x 256.
CIFAR-10 FID
ImageNet-256 x 256
Type
Steps
FID
ISt
FID (w/ CFG)
IS (w/ CFG)1
SDEI
22.67
43.49
54.40
8.97
253.25
SDEI
5.88
20.83
93.17
4.81
257.56
SDEI
3.23
14.21
116.03
3.11
258.81
SDEI
2.27
9.14
139.69
2.46
248.36
STEI
36.87
38.44
56.60
7.12
241.75
STEI
9.21
21.10
95.77
4,41
241.99
STEI
4.04
10.91
135.03
2.78
269.87
STEI
2.59
7.64
157.03
2.36
274.72
SDEE
10.19
19.99
96.87
3,96
247.20
SDEE
3.18
9.46
136.97
2.46
258.94
STEE
4 10.55
22.82
83.87
3.02
274.00
STEE
8
3.78
12.98
110.03
2.33
274.47
Algorithm 10 Sampling Using Models Trained with STEE
Input: model fa. number of steps N, guidance scale Output: sampled image Sample 31 I)
for i = to N - do
if w then
fø N N
N-i N N + f0 No N N + w w(f6(a No
N=
N-1
N
N-i-l)
fo fo Nai, N N
N-1
N N-i-1)
end if
T N-i-1 t- x xZ fo(
N-i N-i-1)
N
end for
G Additional Quantitative Results
G.1 Additional Performance
For detailed comparison among secant losses, we provide more results CIFAR-10 and ImageNet-
256 x 256 in Table 7
G.2 Training Efficiency
In practical application, methods like continuous-time CMs analytical Jacobian-vector product (JVP) operations compute the loss. This imposes significant computational burden, especially in Py Torch. To provide a direct comparison, we benchmark the training speed and memory usage under the same EDM setup (the batch size is 512 8 A.00 GPUs with 64 per GPU) on CIFAR-10. The results are shown Table
33 35
and MeanFFlw (67 require
H Experimental Settings
The detailed experimental configurations are presented in Table 9 Basically, we follow the settings of EDM and SiT. Except for STEE on ImageNet-256 x 256, we multiply the learning rate by factor of 0.1. For CIFAR-10 dataset, we use the DDPM+ architecture, and adopts the Trigflow framework. For ImageNet-256 x 256, we cache the latent codes on disk, and for simplicity we disable the horizontal flip data augmentation. For SDEI, STEI and SDEE, we embed CFG scale as conditional input to the model, with the CFG range [1, 2] for 2-step, 4-step and 8-step models and [1, 2.5] for 1-step ones. In the experiment concerning training from scratch ImageNet-256 x 256,
30

---

Table 8: Comparison on training efficiency among different approaches.
Method
Training Speed (sec/KIMG)
Memory Usage (G, Per GPU)
EDM
(teacher diffusion model)
0.57
8.69
MeanFlow 67
1.04
38.73
SDEI
0.91
9.64
STEI
1.42
16.95
SDEE
0.91
9.64
STEI
0.72
9.34
Table 9; Experimental settings of four loss functions on different models and datasets.
CIFAR-10
ImageNet-256 x 256
SDEI
STEI
SDEE
STEE
SDEI
STEI
SDEE
STEE
Model Setting
Architecture
DDPM+
DDPM+
DDPM++
DDPM
DLT
DIT-XL2 675
DIT-XL22
DIT-XL22
Params (M)
$5
$5
675
675
675
0d
0.5
0.5
Guose (t)
In(a, tan f)
In(era tan f)
In(or tan é)
In(a,, tan t)
Boundary f1
Boundary
0.00625
0,00625
0.00625
0.00625
Initialization
EDM
EDM
EDM
EDM
STF-L/2
STF-L/2
SIT-XL/2
SIT-XL22
Truining Selting
Precision
Ip32
Ips2
Ip32
1p32
Ep16
Ip16
Ip16
Ipl6
Dropout
0.2
0
Optimizer
RAdam
RAdam
RAdam
RAdam
Adam
AdamW
AdamW
Adam W
Optimizer E
lc-8
lc-8
le-8
lc-8
le-8
le-8
lc-8
le-8
31
0.9
0.9
0.9
S2
0.99
0.99
0.99
0.99
0.999
0.999
0.999
0.999
Leariigg Rate
le-4
1c-4
16-4
lc-4
le-5
Ie-5
1o-5
lc-4
Weight Decay
U
U
0
Batch Size
512
512
512
512
256
256
256
Training iters
100K
100k
100k
100k
100K
100k
100k
100k
t, sampling
discrete
discrete
continuous Uniform EDM's
continuous Uniform EDM's
discrete
continuous
continuous Uniform 0.9999
continuous
sampling
Uniform
Uniform EDM's
Uniform
Uniform 0.9999 0.1
Uniform
EMA Rate
EDM's
0.9999
0.9999
Label Dropout
0.1
Embed CPG
V
ar flip
X
x
X
x
X
maintain identical settings those specified in the STEE column, while only disable the pretrained weights and alter the model size. To calculate the FID and IS score at evaluation, we use the codebase provided in MAR (85) for simplicity. The result of Flow-DPM-Solver [79, [66 in Table is with CFG scale of 1.5. All the experiments can be done on a sever with NVIDIA A100 GPUs.
More Visualizations
Additional visualization results are presented for both datasets. For CIFAR-10 dataset, provide 4-step and 8-step results using LSDEI in Fig. 8 and Fig. 9 respectively. For ImageNet-256 x 256, extended visualizations of 8-step generation using LSTER displayed in Fig. 10 and Fig. 11
31

---

o
P
o
Figure 8: Uncurated 4-step samples unconditional CIFAR-10.
32

---

o
m
Figure 9: Uncurated 8-step samples unconditional CIFAR-10.
33

---

Figure 10: Uncurated 8-step samples on ImageNet-2566 x 256. Guidance scale is 2.1, and the guidance of first two steps is ignored.
34

---

Figure 11: Uncurated 8-step samples on ImageNet-256- x 256. Guidance scale is 2.1, and the guidance of first two steps is ignored.
35


Reviews:
[{"review_id": "Er3lxa97ZP", "reviewer": "Reviewer_v8s1", "summary": "The paper introduces a novel family of training objectives, termed \"secant losses\", to train few-step diffusion/flow neural solvers. This is framed as learning the \"secant\" of the ODE trajectory by matching it to the average of the \"tangents\" along that path. The final, tractable loss functions are derived using principles from Monte Carlo integration and a Picard-iteration-inspired estimation scheme. The framework is flexible, offering four distinct variants for both distilling a pre-trained teacher model and training a few-step generator from scratch. Empirically, the method demonstrates competitive performance and superior training stability.", "questions": "1. I do not quite understand the significance of the sampling of r. The theory demands a uniform weighted average right? So even if one utilizes a different sampling distribution, they need to rebalance via importance sampling.\n2. The end-point version of the training requires the model to be able to invert (from data to noise), which could be a waste of model capacity.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1. A primary strength of this work is its originality and the clarity of its core concept. The essential proposal of the paper, that the average velocity is the average of (instantaneous) velocities, is very intuitive.\n2. Writing is clear and easy to follow, and the figures are well-made, e.g. Fig.1.\n3. A good collection of ablation studies that validate the design choices and provide practical guidance to users.\n\n**Weakness**\n1. Consistency models, or any model that does any-size jumps (like MeanFlow, shortcut models), can be understood as a gradual transition from tangents to secants. The contribution of the work is not really a framework or the contrast between tangents and secants, but rather a new way of training these any-step predictors.\n2. Since the contribution of the paper is really a new way of training an existing family of models, we need to compare their empirical performances. It seems like in almost all cases, it lags behind IMM without the help of guidance intervals. The scores on CIFAR are not really convincing either, given the competition. And if we include the comparisons with MeanFlow, the presented results do not inspire confidence in the proposed method. I understand MeanFlow is concurrent work (so I do not really consider this in my evaluation), but it still will greatly hinder the impact of the paper, i.e. why use the proposed losses over others?", "was_revised": true, "final_justification": "Overall, I like the originality of the proposed method, but I still have reservations regarding its practical usefulness and impact. At its current stage, it seems like just another way of doing things rather than a convincingly better way.", "rebuttal": "We sincerely thank the reviewer for the valuable and constructive feedback. We are also grateful for the recognition of our work's originality and clarity. We will address the reviewer's concerns below.\n\n### **Regarding Weakness 1**\n\n> **Reviewer:** Consistency models, or any model that does any-size jumps (like MeanFlow, shortcut models), can be understood as a gradual transition from tangents to secants. The contribution of the work is not really a framework or the contrast between tangents and secants, but rather a new way of training these any-step predictors.\n\nWe agree with the reviewer's assessment. Our core contribution is indeed the development of the **secant expectation algorithms** for training few-step, ODE-based generative models. The discussion of the relationship between tangents and secants is intended to provide a  clearer geometric intuition (like that in MeanFlow paper) and a foundational basis for our model parametrization and loss construction, rather than being the central contribution of this paper.\n\n### **Regarding Weakness 2**\n\n> **Reviewer:** Since the contribution of the paper is really a new way of training an existing family of models, we need to compare their empirical performances. It seems like in almost all cases, it lags behind IMM without the help of guidance intervals. The scores on CIFAR are not really convincing either, given the competition. And if we include the comparisons with MeanFlow, the presented results do not inspire confidence in the proposed method. [...] why use the proposed losses over others?\n\nWe thank the reviewer for raising this important point. We compare the mentioned methods as follows.\n\n**Comparison with IMM:** We wish to highlight a fundamental conceptual distinction between the two methods, which places them in different categories of few-step generative models. IMM adopts distribution-level loss—the Maximum Mean Discrepancy (MMD) loss.  In contrast, our approach is particle-based and grounded in finding the exact solution of the ODE. And our contribution lies in developing a stable and efficient new method for particle-based few-step diffusions. In this context, our work has its unique advantages compared to other related methods as follows.\n\n**Comparison with MeanFlow, Consistency Models (CM), Consistency Trajectory Models (CTM), etc.:** We wish to highlight three key advantages of our approach: local accuracy in theory, target stability in principle, efficiency in application.\n\n1.  **Theoretical Soundness:** From a theoretical standpoint, a key feature that distinguishes our method is its ability to achieve local accuracy without requiring explicit differentiation in the loss function, as shown in Theorem 2 and Theorem 3.\n\n2. **Target Stability:**\n    Another crucial point for stable training is the target stability, i.e., the stability of the prediction target in the loss function. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):\n\n      * **Diffusion Models (Flow Matching):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{v}\\_\\theta(\\\\boldsymbol{x}\\_t,t)-(-\\\\boldsymbol{x}_0+\\\\boldsymbol{z}))$. The target is the velocity estimation $-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z}$.\n      * **MeanFlow / CM / CTM (similar):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{f}\\_\\theta(\\\\boldsymbol{x}\\_t,t,s)-((-\\\\boldsymbol{x}_0+\\\\boldsymbol{z})+(s-t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$. The target is $-\\\\boldsymbol{x}_0+\\\\boldsymbol{z} + (s - t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$. The latter item $\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$ (or the discrete version $\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) is highly unstable, which has been substantially studied in [1].\n      * **Our Method:** The targets are either the velocity estimation $(-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z})$ or the velocity $\\\\boldsymbol{v}(\\\\boldsymbol{x}_t,t)$ itself.\n\n    Our method is the **only** one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10, where we track the specific items in the loss functions, and record the **standard deviation** in the following table. The statistics shown in the table clearly support the stability of our method.\n\n    | Loss (Inspected item)/Training iteration | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500 |\n    | :--- | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :--- | :---: | :---: |\n    | CM ($\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$) | 0.2627 | 15.4339 | 30.9154 | 40.7293 | 79.1750 | 101.8844 | 97.2164 | 134.4441 | 72.3149 | 97.4782 |\n    | CM ($\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) | 0.2810 | 10.1618 | 34.4472 | 88.6847 | 39.1018 | 59.5919 | 143.4175 | 62.3149 | 69.5990 | 137.7932 |\n    | SDEI ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 3.1713 | 3.0427 | 4.1342 | 3.4504 | 3.3060 | 2.6381 | 3.2294 | 3.5089 | 3.4080 | 3.2673 |\n    | STEI ($\\\\boldsymbol{f}_{\\theta^-}(\\\\boldsymbol{x}_r,r,r)$) | 4.4030 | 4.2680 | 4.1115 | 3.8282 | 4.3783 | 4.5072 | 4.3712 | 4.0905 | 3.9820 | 4.0967 |\n    | SDEE ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 2.8809 | 3.2181 | 3.5562 | 3.0221 | 3.7394 | 3.0247 | 3.8801 | 3.2247 | 4.0112 | 3.5830 |\n    | STEE ($-\\\\text{sin}(t)\\\\boldsymbol{x}_0+\\\\text{cos}(t)\\\\boldsymbol{z}$) | 4.2049 | 4.1783 | 4.6112 | 4.0290 | 4.6454 | 4.5360 | 4.9838 | 4.9387 | 5.0872 | 4.3094 |\n\n3.  **Training Efficiency (Speed and Memory):** In practical application, methods like continuous-time CM and MeanFlow require Jacobian-vector product (JVP) operations to compute the loss. This imposes a significant computational burden, especially in PyTorch (while the MeanFlow authors use JAX, PyTorch remains the more widely adopted framework). To provide a direct comparison, we benchmarked training speed and memory usage under the same EDM setup (the batch size is 512 on 8 A100 GPUs with 64 per GPU) on CIFAR-10 as in our main text.\n\n    | Method | Training Speed (sec/kimg) | Memory Usage (Per GPU, GB) |\n    | :--- | :---: | :---: |\n    | Diffusion Model (Teacher) | 0.57 | 8.69 |\n    | MeanFlow | 1.04 | 38.73 |\n    | SDEI | 0.91 | 9.64 |\n    | STEI | 1.42 | 16.95 |\n    | SDEE | 0.91 | 9.64 |\n    | STEI | 0.72 | 9.34 |\n\n    As the table shows, our method's training overhead is modest. For example, STEI adds only **26%** to the teacher model's training time, whereas MeanFlow adds over **82%**. The difference in memory consumption is even more stark: STEI increases usage by only **7%**, while MeanFlow requires a **346%** increase. And the total training duration required of our method is much less. For example, both fine-tuning from existing teacher models, our method converges with only 50M images on CIFAR-10, whereas sCM requires 200M.\n\nIn summary, while our method may not achieve state-of-the-art FID scores, it presents a compelling trade-off. For scenarios where training speed, memory efficiency, and stability are critical—especially when working with large models or limited computational resources, and modest step numbers like 4 or 8 are acceptable, our approach provides a highly practical and efficient solution. \n\n### **Regarding Question 1**\n\n> **Reviewer:** I do not quite understand the significance of the sampling of $r$. The theory demands a uniform weighted average right? So even if one utilizes a different sampling distribution, they need to rebalance via importance sampling.\n\nMathematically, a uniform weighted average and a non-uniform distribution rebalanced with importance sampling are equivalent. However, in the context of deep learning, the choice of sampling distribution corresponds to how we allocate the **training effort** across different time intervals. We apologize for not clarifying this motivation in the main text. Inspired by recent findings in the diffusion model literature (e.g., EDM[2], SD3[3], FasterDiT[4]), where focusing training efforts on specific time intervals was shown to accelerate convergence, we decide to explore if a similar strategy would benefit our method. We experiment with different distributions for sampling the time points $r$. However, our empirical results indicate that allocating training effort uniformly across all intervals works best.\n\n### **Regarding Question 2**\n\n> **Reviewer:** The end-point version of the training requires the model to be able to invert (from data to noise), which could be a waste of model capacity.\n\nThe end-point variant of our training algorithm indeed requires the model to learn the inversion from data back to noise, which could command additional model capacity compared to the intermediate-point variant. And here we would like to emphasize that there is a trade-off between model capacity and training computation. As shown in Table 1 in the main text, the benefit of this approach is a more efficient training step: while STEE may use more model capacity, it saves **two forward pass evaluations and one backpropagation step** per iteration compared to STEI.\n\n[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025\n\n[2] Elucidating the design space of diffusion-based generative models. Tero Karras et al. NeurIPS 2022\n\n[3] Scaling rectified flow transformers for high-resolution image synthesis. Patrick Esser et al. ICML 2024\n\n[4] FasterDiT: Towards faster diffusion transformers training without architecture modification. Jingfeng Yao et al. NeurIPS 2024"}, {"review_id": "IWxaR2gpWn", "reviewer": "Reviewer_JYzA", "summary": "This paper introduces a novel acceleration and distillation method aimed at significantly speeding up the inference process of diffusion models. By leveraging the framework of ordinary differential equation (ODE) integration, their approach learns an efficient solver that approximates the continuous reverse diffusion dynamics. This enables a dramatic reduction in the number of sampling steps required during generation, thereby balancing the trade-off between computational cost and model performance. The proposed method not only preserves the high quality of generated samples but also reduces inference latency, making diffusion models more practical for real-world applications. Through extensive experiments, the authors demonstrate that their approach achieves competitive performance compared to standard diffusion solvers while requiring substantially fewer function evaluations", "questions": "See details in above weakness. Authors should answer these questions corresponding to above comments: 1. Key Differences with MeanFlow; 2. Scalability and Applicability; 3. One-step Generation Performance; 4. Clarity on Method Variants", "limitations": "Yes.  The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.The paper introduces a geometrically motivated “secant loss” that bridges the gap between diffusion models (tangent) and numerical integration (secant), offering a fresh perspective on few-step diffusion inference.\n\n2.The use of Monte Carlo integration and Picard iteration to derive the loss is theoretically sound and practically simple.\n\nWeakness:\n\n1.The proposed method shares a similar starting point with MeanFlow, as both optimize the average velocity field based on the integral of the instantaneous velocity field. Could the authors clarify the key differences between their approach and MeanFlow?\n\n2.MeanFlow directly learns the average velocity field, making its objective function simpler, with complexity concentrated in differentiation. In contrast, the proposed method eliminates reliance on differentiation but focuses more on learning a more consistent (straighter) sampling trajectory, which seems more suitable for distillation rather than training a model from scratch. This suggests that MeanFlow might have broader applicability. Could the authors further discuss the scalability of their method?\n\n3.The one-step generation performance appears relatively weak. Could this be due to the limitations of the proposed loss function, which does not directly learn the average velocity field itself? More results on one-step generation would be helpful.\n\n4.The paper introduces many similarly named variants of the method, making the methodology and experiments somewhat difficult to follow. A clearer distinction between these variants would improve readability.", "was_revised": true, "final_justification": "The author has addressed most of my concerns, I am satisified with the rebuttal, I would like to keep my positive rating", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and for recognizing the theoretical soundness and simplicity of our method. We will address the concerns below.\n\n### **Question 1. Key Differences with MeanFlow**\n\nThe core difference can be summarized succinctly: **To achieve the same objective of fitting the average velocity (the secant), MeanFlow uses differentiation to construct its loss, whereas our method uses integration.** With identical model parameterization, the two methods can be viewed as differential and integral counterparts.\n\nThe authors of MeanFlow notes in Section 4.1 that, “However, directly using the average velocity defined by Eq. (3) as ground truth for training a network is intractable, as it requires evaluating an integral during training.” The essential difficulty is, evaluating the integral requires a large amount of samples (i.e., a full inference-like trajectory). But in deep learning, using such many samples to evaluate the integration in one iteration is impractical. MeanFlow circumvents this by using differentiation, while we use Monte Carlo integration, where the key is to convert the many-sample evaluation to one-sample evaluation of the integral.\n\nThe above fundamental difference leads to our method's distinctive features: local accuracy in theory, target stability in principle, efficiency in application.\n\n1.  **Theoretical Soundness:** From a theoretical standpoint, a key feature that distinguishes our method is its ability to achieve local accuracy without requiring explicit differentiation in the loss function, as shown in Theorem 2 and Theorem 3.\n\n2. **Target Stability:**\n    Another crucial point for stable training is the target stability, i.e., the stability of the prediction target in the loss function. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):\n\n      * **Diffusion Models (Flow Matching):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{v}\\_\\theta(\\\\boldsymbol{x}\\_t,t)-(-\\\\boldsymbol{x}_0+\\\\boldsymbol{z}))$. The target is the velocity estimation $-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z}$.\n      * **MeanFlow / CM / CTM (similar):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{f}\\_\\theta(\\\\boldsymbol{x}\\_t,t,s)-((-\\\\boldsymbol{x}_0+\\\\boldsymbol{z})+(s-t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$. The target is $-\\\\boldsymbol{x}_0+\\\\boldsymbol{z} + (s - t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$. The latter item $\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$ (or the discrete version $\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) is highly unstable, which has been substantially studied in [1].\n      * **Our Method:** The targets are either the velocity estimation $(-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z})$ or the velocity $\\\\boldsymbol{v}(\\\\boldsymbol{x}_t,t)$ itself.\n\n    Our method is the **only** one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10, where we track the specific items in the loss functions, and record the **standard deviation** in the following table. The statistics shown in the table clearly support the stability of our method.\n\n    | Loss (Inspected item)/Training iteration | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500 |\n    | :--- | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :--- | :---: | :---: |\n    | CM ($\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$) | 0.2627 | 15.4339 | 30.9154 | 40.7293 | 79.1750 | 101.8844 | 97.2164 | 134.4441 | 72.3149 | 97.4782 |\n    | CM ($\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) | 0.2810 | 10.1618 | 34.4472 | 88.6847 | 39.1018 | 59.5919 | 143.4175 | 62.3149 | 69.5990 | 137.7932 |\n    | SDEI ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 3.1713 | 3.0427 | 4.1342 | 3.4504 | 3.3060 | 2.6381 | 3.2294 | 3.5089 | 3.4080 | 3.2673 |\n    | STEI ($\\\\boldsymbol{f}_{\\theta^-}(\\\\boldsymbol{x}_r,r,r)$) | 4.4030 | 4.2680 | 4.1115 | 3.8282 | 4.3783 | 4.5072 | 4.3712 | 4.0905 | 3.9820 | 4.0967 |\n    | SDEE ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 2.8809 | 3.2181 | 3.5562 | 3.0221 | 3.7394 | 3.0247 | 3.8801 | 3.2247 | 4.0112 | 3.5830 |\n    | STEE ($-\\\\text{sin}(t)\\\\boldsymbol{x}_0+\\\\text{cos}(t)\\\\boldsymbol{z}$) | 4.2049 | 4.1783 | 4.6112 | 4.0290 | 4.6454 | 4.5360 | 4.9838 | 4.9387 | 5.0872 | 4.3094 |\n\n3.  **Training Efficiency (Speed and Memory):** In practical application, methods like continuous-time CM and MeanFlow require Jacobian-vector product (JVP) operations to compute the loss. This imposes a significant computational burden, especially in PyTorch (while the MeanFlow authors use JAX, PyTorch remains the more widely adopted framework). To provide a direct comparison, we benchmarked training speed and memory usage under the same EDM setup (the batch size is 512 on 8 A100 GPUs with 64 per GPU) on CIFAR-10 as in our main text.\n\n    | Method | Training Speed (sec/kimg) | Memory Usage (Per GPU, GB) |\n    | :--- | :---: | :---: |\n    | Diffusion Model (Teacher) | 0.57 | 8.69 |\n    | MeanFlow | 1.04 | 38.73 |\n    | SDEI | 0.91 | 9.64 |\n    | STEI | 1.42 | 16.95 |\n    | SDEE | 0.91 | 9.64 |\n    | STEI | 0.72 | 9.34 |\n\n    As the table shows, our method's training overhead is modest. For example, STEI adds only **26%** to the teacher model's training time, whereas MeanFlow adds over **82%**. The difference in memory consumption is even more stark: STEI increases usage by only **7%**, while MeanFlow requires a **346%** increase. And the total training duration required of our method is much less. For example, both fine-tuning from existing teacher models, our method converges with only 50M images on CIFAR-10, whereas sCM requires 200M.\n\n### **Question 2. Scalability and Applicability**\n\nWe would like to clarify that our method, like MeanFlow and Consistency Models, is also based on the exact solution of the diffusion ODE and **learns the same average velocity field**. Our approach does not straighten or alter the ODE trajectory itself, which is a technique used by other methods like 2-Rectified Flow [2]. Therefore, there is no inherent difference in applicability between our method and MeanFlow. Our method is designed for, and has been demonstrated to work effectively in both distillation and training settings. Regarding scalability, we refer to Section 5.3 in the main text, where we scale the model size. The results show that performance consistently improves with a larger model, which can be a supporting evidence for scalability on model size. \n\n### **Question 3. One-step Generation Performance**\n\nWe provide more 1- and 2-step results in the table below.\n\n| Type | Steps | CIFAR-10 FID | ImgNet-256 FID | ImgNet-256 IS |\n| :--- | :---: | :---: | :---: | :---: |\n| SDEI | 1 | 22.67 | 8.97 | 253.25 |\n| SDEI | 2 | 5.88 | 4.81 | 257.56 |\n| STEI | 1 | 36.87 | 7.12 | 241.75 |\n| STEI | 2 | 9.21 | 4.41 | 241.99 |\n\nAs explained in Question 2, akin to MeanFlow, our method also learns the average velocity. For explanation of the weaker one-step generation, we can see in Theorem 2 and Theorem 3 in the main text, our loss functions emphasize on **local accuracy** without explicit differentiation. Our loss function is most accurate over shorter integration intervals, and its performance naturally degrades as the step interval becomes very large. We would like to highlight that it is a trade-off. Methods like CM, MeanFlow optimize for global accuracy but do so by introducing either discretization errors or explicit derivative terms in their loss functions, which leads to potential training instability. Our method's \"sweet spot\" is in the modest few-step regime (e.g., 4 or 8 steps). For scenarios where training speed, memory efficiency, and stability are critical—especially with large models or limited resources—and a modest number of steps is acceptable, our approach provides a highly practical and efficient solution. We acknowledge this limitation and note the performance gap between 1-step and 8-step generation in the paper.\n\n### **Question 4. Clarity on Method Variants**\n\nThe naming convention is based on two key design choices, estimation strategy and training mode. According to Figure 1, the main goal is to learn the secant function $\\\\boldsymbol{f}(\\\\boldsymbol{x}_t,t,s)$ such that we can jump from $\\\\boldsymbol{x}_t$ to $\\\\boldsymbol{x}_s$ at inference. First, our loss target is set as a random tangent (the green line in the figure). To apply Monte Carlo integration and Picard iteration, we should derive an end point $\\\\boldsymbol{x}_t$ and a uniformly selected interior point $\\\\boldsymbol{x}_r$. Considering that we can only access to one of these two at each training iteration, we propose to use the model itself to estimate either the interior point from the true end point, or the end point from the true interior point. We denote the former as “EI”, i.e., “estimating the interior point”, while name the latter as “EE”, i.e., “estimating the end point”. Furthermore, since the target can either be a pretrained diffusion model, or be estimated during training, both “EI” and “EE” include a distillation (D) version and a training (T) version. At the front, we add an “S” denoting “secant losses”, leading to our total four variants: SDEI, STEI, SDEE and STEE.\n\n[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025\n\n[2] Flow straight and fast: Learning to generate and transfer data with rectified flow. Xingchao Liu et al. ICLR 2023"}, {"review_id": "BRV1Vjd0Kg", "reviewer": "Reviewer_NLNA", "summary": "The paper proposes a novel approach to accelerate diffusion model inference by learning to integrate ODEs through \"secant losses,\" which approximate the average derivative (secant) between two time points instead of relying on traditional numerical solvers or complex distillation techniques. The authors introduce several variants of secant losses (SDEI, STEI, SDEE, STEE) and demonstrate their effectiveness on CIFAR-10 and ImageNet-256×256, achieving competitive FID scores with fewer steps (e.g., 10-step FID of 2.14 on CIFAR-10 and 8-step FID of 1.96 on ImageNet).", "questions": "1. Please quantify the sample complexity introduced by nested Monte Carlo. How much does the inner-loop Monte Carlo increase sample complexity? Does it negate the claimed efficiency gains?\n\n2. Why does the method underperform alternatives (Table 2) despite elegant theory? Are the assumptions (e.g., Lipschitz tangents) unrealistic in practice?\n\n3. For Table 7, why do some losses (STEE) work better? Is the theory missing key practical considerations?\n\n4. Provide proofs for STEI and SDEE unbiasedness. Currently, only SDEI/STEE are covered.\n\n5. Justify the importance sampling in Eq. 18. Why not uniform sampling given enough iterations?\n\n6. Compare wall-clock time (not just NFE) against distillation methods.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The idea of learning secant approximations instead of relying on traditional ODE solvers or distillation is innovative and well-motivated from a geometric perspective.\n\n2. The paper provides theoretical guarantees (Theorems 2 and 3) that justify the proposed losses under certain conditions.\n\n3. The method achieves competitive results on standard benchmarks, outperforming some fast samplers and distillation techniques.\n\n4. The approach can be applied via fine-tuning or distillation and supports classifier-free guidance.\n\nWeaknesses\n1. Table 2 shows underwhelming results: At comparable or larger NFEs, the method underperforms \"Joint Training,\" \"Few-step Distillation,\" and \"Few-step Training/Tuning\" approaches (e.g., CTM, IMM, rectified flows). This raises concerns about whether the theory overlooks critical practical factors (e.g., error accumulation in large steps).\n\n2. Theoretical assumptions may be too restrictive: The Lipschitz condition is not empirically validated, and the paper does not discuss how violations affect performance.\n\n3. Performance varies drastically across loss types (e.g., SDEI vs. STEE), suggesting the current formulation may not be SOTA. The authors should justify why certain losses (e.g., STEE) perform better and whether the theoretical framework aligns with empirical trends.\n\n4. Theorems 2 and 3 only prove unbiasedness for SDEI and STEE. Missing proofs for STEI and SDEE are a significant gap, as these losses are empirically evaluated.\n\n5. Importance sampling (Eq. 18) is unmotivated: The paper does not explain why uniform sampling is insufficient, especially since training iterations typically compensate for variance.\n\n6. The nested Monte Carlo estimation (e.g., sampling $r$ and $t$) likely requires large sample sizes for unbiasedness, but the paper omits analysis of convergence rates or computational trade-offs.\n\n7. Presentation Issues: (1) Typos: Theorem 3 mixes bold/non-bold variables ($x_1$ vs. $\\mathbf{x}_2$).  (2) Proposition 1 lacks attribution: This is a known result (e.g., from diffusion/flow matching literature) but is presented without citation.", "was_revised": true, "final_justification": "The paper proposes a simple and stable method for integrating diffusion ODEs using a derivative-free loss. I find the core idea — learning secants — to be both effective and well-motivated. While the method section could be more clearly presented, especially regarding the loss formulation and the rationale behind the observed stability, the authors have addressed my concerns and clarified key points. I have also independently validated the method's effectiveness, and thus I believe the paper makes a meaningful contribution. I have accordingly raised my score.", "rebuttal": "We sincerely thank the reviewer for the valuable and constructive feedback. We are also grateful for the recognition of our method as innovative and well-motivated. We will address each of the reviewer's concerns below.\n### **Weaknesses 1 (Question 2)**\n\nOur theoretical framework grounded in Theorems 2 and 3 is designed to emphasize **local accuracy** in the absence of explicit differentiation, which guarantees that our method can work well at short intervals, empirically like 4 or 8 steps. As a result, we position our method between faster samplers and few-step diffusion distillation/training methods, as our method provides good trade-off between performance and efficiency. We provide additional experiments to show the stability and efficiency against other methods. Please also see the response to Reviewer v8s1 regrading Weakness 2.\n\n### **Weaknesses 2**\n\nWe agree with the reviewer that Lipschitz continuity is not generally guaranteed for a neural network in practice. However, we work in the context of diffusion models, where it is a necessary condition to apply foundational results like the Picard-Lindelöf theorem, which guarantees the existence and uniqueness of the ODE solution. This provides a rigorous framework for analyzing our algorithm's convergence, stability, and correctness, common to many works in this field. For example, the seminal work, consistency models [1] and consistency trajectory models [2], both include the Lipchitz condition in the Theorems. And the theoretical guarantee is under the worst case; in practice, the strong empirical stability of our method suggests that our training procedure implicitly guides the network to a well-behaved state, thus closely approximating the conditions required by the theory.\n\nFor violations, i) if the function is not Lipschitz, the convergence is not guaranteed. However, according to the experiments, our method consistently trains models with great stability and performance, suggesting the function appears to be locally smooth in the relevant regions of the state space. ii) The Lipchitz constant is not small enough. In this situation, the small numerical error from one step could be amplified in subsequent steps. Considering this, we position our method between fast samplers and diffusion distillation/training methods, as a good trade-off between performance and efficiency.\n\n### **Weaknesses 3 (Question 3)**\n\nFirst, we would like to correct an error in Table 7 of the appendix. The 4- and 8-step FIDs for STEI on CIFAR-10 should be **4.04** and **2.59**. We discovered that dropout was not correctly enabled in our previous experiments, contrary to the setting of 0.2 specified in Table 8.\n\nIn Table 7, the difference happens mainly between estimating inner or end point (training/distillation achieves similar performance), which can be explained by three aspects.\ni) **Input correctness**. For inner-point variants, the input to the training network is the clean data $\\boldsymbol{x}_t$. While in end-point variants, the input is the estimated $\\\\hat{\\boldsymbol{x}}\\_t$. Fitting a fixed, clean input distribution, is more efficient than fitting an input dependent of the model historical output. ii) **Error accumulation path**, i.e., the path from clean data to the prediction destination. In inner-point variants, the path is $\\boldsymbol{x}_t\\to \\boldsymbol{x}_s$, while in end-point variants it is $\\boldsymbol{x}_r\\to \\boldsymbol{x}_t \\to \\boldsymbol{x}_s$. Generally, the path in end-point variants is longer than that in inner-point variants. iii) **Model capacity**. As per Figure 1 and Algorithm 2, the end-point variants additionally require inversion, which costs part of the model capacity.\n\n### **Weaknesses 4 (Question 4)**\n\nWe thank the reviewer for pointing out this omission. We include the proofs for STEI and SDEE below. We will add these to the final manuscript. (\\boldsymbol is omitted to save the character count.)\n\nCorollary 4. (STEI) Let $f_\\theta(x_t,t,s)$ be a neural network, and let $v(x,t)$ be the true velocity field. Assume $v(x,t)$ is L-Lipschitz continuous in its first argument. If $\\mathcal{L}\\_{STEI}(\\theta)$ reaches its minimum, then for any interval where $|s-t| \\le h$ and $h < 3/L$, we have $f_{\\theta}(x_{t},t,s)=f(x_{t},t,s)$.\n\nProof. The loss $\\mathcal{L}\\_{STEI}(\\theta)$ is defined as a sum of a secant term and a diffusion term: $$ \\mathcal{L}\\_{STEI}(\\theta) = \\mathbb{E}[||f_\\theta(x_t,t,s)-f_{\\theta^-}(\\hat{x}\\_r,r,r)||\\_2^2] + \\lambda\\mathbb{E}[||f_\\theta(x_\\tau,\\tau,\\tau)-(\\alpha_\\tau'x_0+\\sigma_\\tau'z)||\\_2^2],$$ where $\\hat{x}\\_r = x_t+(r-t)f_{\\theta^-}(x_t,t,r)$. For $\\mathcal{L}\\_{STEI}(\\theta)$ to reach the minimum, both two terms must reach the minimum. By Proposition 1 of our paper, the diffusion term ahieves the minimum if and only if $$ f_\\theta(x_t, t, t) = \\mathbb{E}(\\alpha_t'x_0+\\sigma_t'z|x_t) = v(x_t,t).$$ Subsitituting this result into the secant term, it becomes the one studied in the proof of Theorem 2. The remainder of the proof follows identically.\n\nCorollary 5. (SDEE) Let $f_\\theta(x_t,t,s)$ be a neural network that is L-Lipschitz continuous in its first argument. For any fixed interval $[a, b]\\subseteq[0,1]$ with $b-a$ sufficiently small, if $\\mathcal{L}\\_{SDEE}(\\theta)$ reaches its minimum, then $f_\\theta(x_t,t,s)=f(x_t,t,s)$.\n\nProof. The SDEE loss is defined as: $$ \\mathcal{L}\\_{SDEE}(\\theta) = \\mathbb{E}\\_{x_0,z,t,s,r\\sim\\mathcal{U}(t,s)}[||f_\\theta(x_r+(t-r)f_{\\theta^-}(x_r,r,t),t,s)-v(x_r,r)||\\_2^2].$$ This loss function is identical to the intermediate objective $\\mathcal{L}\\_{*}(\\theta)$ analyzed in the proof of Theorem 3. The proof, therefore, follows that of Theorem 3 directly.\n\n### **Weakness 5 (Question 5)**\n\nMathematically, uniform sampling and a non-uniform distribution rebalanced with importance sampling are equivalent. However, in the context of deep learning, the choice of sampling distribution corresponds to how we allocate the **training effort** across different time points. We apologize for not clarifying this motivation in the main text. Inspired by recent findings in the diffusion model literature (e.g., EDM [3], SD3 [4], FasterDiT [5]), where focusing training efforts on specific time intervals was shown to accelerate convergence, we decided to explore if a similar strategy would benefit our model. However, empirical results indicate that allocating training effort uniformly works best for our method.\n\n### **Weakness 6**\n\nAs stated in the main text below Eq. (9), “This approach is inspired by the diffusion objective that leads from Eq. (1) to Eq. (2).”, our use of Monte Carlo estimation for sampling data and time points **within one batch** is analogous to the standard practice in training diffusion models. We follow this manner and choose the same batch size (e.g., 256 for ImageNet, 512 for CIFAR-10) as these baseline models. The experimental results validate that it is sufficient for stable convergence. Regarding the analysis of convergence rates, to our knowledge, it is still an open research problem in the field of deep generative modeling. To directly address the reviewer's point on sample sizes and computational trade-offs, we conduct ablation study on the effect of batch size on performance. We train models with SDEI and EDM on CIFAR-10 with varying batch sizes and report the 4-step FID. The table below suggests that our method is robust to the batch size. We will add these results to the appendix of our paper.\n| Batch size | FID (same iteration) | FID (same epoch) |\n| :--- | :---: | :---: |\n| 64 | 3.62 | 3.24 |\n| 128 | 3.48 | 3.23 |\n| 256 | 3.33 | 3.22 |\n| 512 | 3.23 | 3.23 |\n\n### **Weakness 7**\n\nWe thank the reviewer for indicating the presentation issues. We will correct the typos. For Proposition 1, as stated in the appendix after the proof of Proposition 1, “The above is a basic result in literature of diffusion models [66].” We will also add the attribution and references to the main text as well.\n\n### **Question 1**\n\nThe additional complexity is only introduced to the **training phase**, i.e., the extra forward and backward passes as summarized in Table 1. Other minor computations, such as sampling time steps $s$ and $r$, are negligible. Practical running speed and memory cost can be referred to the response to Reviewer v8s1, regarding Weakness 2. Our method has almost no impact on the inference/sampling phase, as the only architectural modification is the injection of an additional time condition, which has a negligible computational cost. Therefore, it does not affect the claimed efficiency gains.\n\n### **Question 6**\n\nSince few-step diffusion distillation/training methods typically do not modify the model architecture, the running time per NFE is the same as diffusion models. We list some results in the table below (the first is on CIFAR-10, and the second is on ImageNet-256), where we use one A100 GPU with a batchsize of 8.\n\n| Method | FID | NFE | Run time (ms) |\n| :--- | :---: | :---: | :---: |\n| EDM (Teacher) | 1.97 | 35 | 677.43 |\n| CD | 2.93 | 2 | 53.71 |\n| IMM | 1.98 | 2 | 54.82 |\n| SDEI (Ours) | 3.23 | 4 | 88.42 |\n| SDEI (Ours) | 2.14 | 10 | 216.76 |\n\n| Method | FID | NFE | Run time (ms) |\n| :--- | :---: | :---: | :---: |\n| SiT-XL/2 (Teacher) | 2.15 | 250($\\times$2) | 16041.28 |\n| Shortcut Models | 7.80 | 4 | 238.27 |\n| IMM | 2.51 | 4($\\times$2) | 359.98 |\n| IMM | 1.99 | 8($\\times$2) | 624.48 |\n| STEI (Ours) | 2.27 | 4 | 240.05 |\n| STEI (Ours) | 1.96 | 8 | 369.51 |\n\n[1] Consistency models. Yang Song et al. ICML 2023\n\n[2] Consistency trajectory models: Learning probability flow ode trajectory of diffusion. Dongjun Kim et al. ICLR 2024\n\n[3] Elucidating the design space of diffusion-based generative models. Tero Karras et al. NeurIPS 2022\n\n[4] Scaling rectified flow transformers for high-resolution image synthesis. Patrick Esser et al. ICML 2024\n\n[5] FasterDiT: Towards faster diffusion transformers training without architecture modification. Jingfeng Yao et al. NeurIPS 2024"}, {"review_id": "MGeVZmeWtD", "reviewer": "Reviewer_RFUp", "summary": "This paper proposes a method for few-step diffusion. Existing fast samplers suffer from significant performance degradation when using a small number of time steps, while previous diffusion distillation methods are prone to training instability and complexity.\nIn the proposed method, a neural network learns to approximate the secant function, representing the average rate of change along the solution paths of the PF-ODE. This training is performed using a loss function designed from the perspective of Monte Carlo integration of the tangent function, inspired by Picard iteration.\nThis paper introduces four loss variants, corresponding to combinations of two time-step sampling schemes during training and whether the training is a distillation process or performed from scratch. Theoretical justification for the proposed losses is also provided.\nExperiments on CIFAR-10 and ImageNet 256 x 256 demonstrate that the proposed method exhibits consistent stability across various hyperparameter settings and achieves relatively good performance with a small number of time steps.", "questions": "* Could you please provide more detailed discussion and information regarding the failure cases of consistency models? Alternatively, performance comparison with consistency models on another benchmark would be also helpful.\n* Could you please address the issues raised in Weakness (3)?", "limitations": "One potential limitation is that $|s-t|$ should be very small to ensure the convergence to the desirable solution in practical cases.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* The paper is well organized.\n* The paper provides theoretical guarantees for the validity of the proposed losses. The theories show an interesting connection between training iterations with stop gradient and Picard iteration.\n* The proposed method demonstrates performance comparable to or better than most existing methods, as shown in the tables.\n* Extensive evaluations were conducted across various settings to examine the behavior of the proposed method.\n\n### Weaknesses\n1. The comparison to and discussion of previous consistency models on ImageNet-256 x 256 appears to be insufficient. The paper only mentions training instability issues encountered by previous consistency models on ImageNet-256 x 256. It remains unclear whether appropriate hyperparameters and settings were used for training these models. A more detailed discussion of the failure cases of consistency models would enhance the clarity and quality of the paper and highlight the stability of the proposed method.\n2. The condition for $|s - t|$ in Theorems 2 and 3 appears to be too restrictive for practical few-step diffusions. The magnitude should be $O(L^{-1})$ for Theorem 2 and $O(L^{-1/2})$ for Theorem 3, where $L$ denotes the Lipschitz constant of $v(x_t, t)$ and $f_\\theta (x_t, t, s)$, respectively. It is generally not guaranteed that the neural network or the flow is Lipschitz in practical cases. This could be a limitation of the proposed method.\n3. There is subtle missing information and errors in the formulas.\n    * The definition of the prime symbol ', around Eq. (1) and Proposition 1, is unclear, while I presume it denotes the derivative with respect to time t.\n    * The variables a and b are not used in the statement of Theorem 3 in the main part of the paper. Please clarify the relationship between $[a, b]$ and $[t, s]$ in Theorem 3.\n    * In the proof of Th.3, there are confusing duplications of the variables, $dr$, in the nested integrations. Please use different variables for the nested integrals, e.g., $dr$ and $d\\tilde{r}$.\n    * $=0$ in the middle of the equation around L554 is possibly a typo.", "was_revised": true, "final_justification": "The additional discussion and tables in the rebuttal support the advantage of the proposed method with respect to training stability.\nAlthough the issues regarding the Lipschitz condition would remain on theoretical side, the practical experiments verify the effectiveness of the proposed method as a trade-off between performance and efficiency to a certain degree.", "rebuttal": "We sincerely thank the reviewer for the valuable feedback. We are grateful for the recognition of our paper's clear organization and the interesting connections provided by our theory. We address the reviewer's concerns below.\n\n### **Regarding Weakness 1**\n> **Reviewer:** The comparison to and discussion of previous consistency models on ImageNet-256$\\times$256 appears to be insufficient. The paper only mentions training instability issues encountered by previous consistency models on ImageNet-256$\\times$256. It remains unclear whether appropriate hyperparameters and settings were used for training these models. A more detailed discussion of the failure cases of consistency models would enhance the clarity and quality of the paper and highlight the stability of the proposed method.\n\nWe thank the reviewer for indicating this important aspect. For more detailed discussion, we focus on the analysis of the **target stability**. Target stability is a rather crucial factor to train few-step diffusion models, which means the stability of the prediction target in the loss function [1]. The difference in target stability between CM and ours is essential in principle, which cannot be influenced by specific hyperparameters or settings. We apologize for not elaborating on this in the main text. For simplicity, let's compare the targets under the OT-FM interpolant ($d$ denotes some distance metric):\n\n* **Diffusion Models (Flow Matching):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{v}\\_\\theta(\\\\boldsymbol{x}\\_t,t)-(-\\\\boldsymbol{x}_0+\\\\boldsymbol{z}))$. The target is the velocity estimation $-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z}$.\n* **MeanFlow / CM / CTM (similar):** $\\mathcal{L}(\\theta)=d(\\\\boldsymbol{f}\\_\\theta(\\\\boldsymbol{x}\\_t,t,s)-((-\\\\boldsymbol{x}_0+\\\\boldsymbol{z})+(s-t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$. The target is $-\\\\boldsymbol{x}_0+\\\\boldsymbol{z} + (s - t)\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$. The latter item $\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)$ (or the discrete version $\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) is highly unstable, which has been substantially studied in [1].\n* **Our Method:** The targets are either the velocity estimation $(-\\\\boldsymbol{x}\\_0+\\\\boldsymbol{z})$ or the velocity $\\\\boldsymbol{v}(\\\\boldsymbol{x}_t,t)$ itself.\n\nOur method is the **only** one among these to feature a target with stability comparable to that of standard diffusion models. This inherent stability of the prediction target helps explain the robust training behavior we observe. To see the practical target stability, we also conduct an experiment with EDM on CIFAR-10 (the first table below) and also SiT on ImageNet-256 (the second table below), where we track the specific items in the loss functions, and record the **standard deviation** in the following table.\n\n| Loss (Inspected item)/Training iteration | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500 |\n| :--- | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :--- | :---: | :---: |\n| CM ($\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$) | 0.2627 | 15.4339 | 30.9154 | 40.7293 | 79.1750 | 101.8844 | 97.2164 | 134.4441 | 72.3149 | 97.4782 |\n| CM ($\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) | 0.2810 | 10.1618 | 34.4472 | 88.6847 | 39.1018 | 59.5919 | 143.4175 | 62.3149 | 69.5990 | 137.7932 |\n| SDEI ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 3.1713 | 3.0427 | 4.1342 | 3.4504 | 3.3060 | 2.6381 | 3.2294 | 3.5089 | 3.4080 | 3.2673 |\n| STEI ($\\\\boldsymbol{f}_{\\theta^-}(\\\\boldsymbol{x}_r,r,r)$) | 4.4030 | 4.2680 | 4.1115 | 3.8282 | 4.3783 | 4.5072 | 4.3712 | 4.0905 | 3.9820 | 4.0967 |\n| SDEE ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 2.8809 | 3.2181 | 3.5562 | 3.0221 | 3.7394 | 3.0247 | 3.8801 | 3.2247 | 4.0112 | 3.5830 |\n| STEE ($-\\\\text{sin}(t)\\\\boldsymbol{x}_0+\\\\text{cos}(t)\\\\boldsymbol{z}$) | 4.2049 | 4.1783 | 4.6112 | 4.0290 | 4.6454 | 4.5360 | 4.9838 | 4.9387 | 5.0872 | 4.3094 |\n\n| Loss (Inspected item)/Training iteration | 10 | 20 | 30 | 40 | 50 | 100 | 200 | 300 | 400 | 500 |\n| :--- | :---: | :---: | :---: | :---: | :--- | :---: | :---: | :--- | :---: | :---: |\n| CM ($\\\\frac{d}{dt}\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s))$) | 32.7539 | 60.0200 | 62.2378 | 98.2755 | 139.4081 | 299.9317 | 307.6916 | 843.4188 | 1444.6832 | 1512.0031 |\n| CM ($\\\\frac{\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_t,t,s)-\\\\boldsymbol{f}\\_{\\theta^-}(\\\\boldsymbol{x}\\_{t-\\Delta t},t-\\Delta t,s)}{\\Delta t}$) | 44.9663 | 75.9018 | 145.2046 | 158.4578 | 279.9238 | 373.5405 | 316.0669 | 740.3381 | 1397.3980 | 1622.9102 |\n| SDEI ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 3.9272 | 4.2610 | 5.5458 | 4.7783 | 5.0217 | 4.3936 | 4.2743 | 4.8920 | 5.6929 | 5.7791 |\n| STEI ($\\\\boldsymbol{f}_{\\theta^-}(\\\\boldsymbol{x}_r,r,r)$) | 5.4390 | 5.9166 | 4.8327 | 6.4482 | 6.0106 | 4.8009 | 6.2592 | 6.3983 | 4.0272 | 5.8081 |\n| SDEE ($\\\\boldsymbol{v}(\\\\boldsymbol{x}_r,r)$) | 4.3985 | 6.5008 | 4.4519 | 3.6164 | 5.2973 | 4.7842 | 5.3170 | 5.3725 | 5.2417 | 6.0224 |\n| STEE ($-\\\\boldsymbol{x}_0+\\\\boldsymbol{z}$) | 7.4445 | 6.8171 | 10.1249 | 9.7260 | 10.0295 | 10.1284 | 8.5793 | 9.6876 | 9.2053 | 9.9978 |\n\nThe statistics shown in the tables clearly support the stability of our method. While, for CMs, the target is unstable, especially when the data has large variance (e.g., the latent space of ImageNet-256). The instability largely affects the training dynamics, potentially leading to worse performance, or even training divergence.\n\n### **Regarding Weakness 2**\n> **Reviewer:** The condition for $|s−t|$ in Theorems 2 and 3 appears to be too restrictive for practical few-step diffusions. The magnitude should be $O(L^{−1})$ for Theorem 2 and $O(L^{−1/2})$ for Theorem 3, where $L$ denotes the Lipschitz constant of $v(x_t,t)$ and $f_\\theta(x_t,t,s)$, respectively. It is generally not guaranteed that the neural network or the flow is Lipschitz in practical cases. This could be a limitation of the proposed method.\n\nWe agree with the reviewer that Lipschitz continuity is not generally guaranteed for a standard neural network in practice. However, we work in the context of diffusion models, where this condition is a necessary condition to apply foundational results like the Picard-Lindelöf theorem, which guarantees the existence and uniqueness of the ODE solution. This provides a rigorous framework for analyzing our algorithm's convergence, stability, and correctness, a practice common to many works in this field. For example, the seminal work, and perhaps the most related to our work, consistency models [2], and consistency trajectory models [3], both include the Lipchitz condition in the Theorems. And the theoretical guarantee is under the worst case; in practice, the strong empirical stability of our method suggests that our training procedure implicitly guides the network to a well-behaved state. The learned function appears to be locally smooth in the relevant regions of the state space, thus closely approximating the conditions required by the theory.\n\nAnd we also agree with the reviewer that, the requirement on $|s-t|$ affects the performance in extremely low-NFE settings. In this situation, the small numerical error from one step could be amplified in subsequent steps. Here we would like to highlight that it is a key trade-off compared to consistency models in the theoretical design. CMs emphasize global accuracy, but introduce either discretization error or explicit differentiation part in the loss function, which significantly affects the training stability. Our method achieves local accuracy without explicit differentiation, and has a very stable target as diffusion models. We believe this presents a valuable and complementary approach to CMs. And we position our method between fast samplers and diffusion distillation/training methods, as a good trade-off between performance and efficiency.\n\n### **Regarding Weakness 3**\n> **Reviewer:** There is subtle missing information and errors in the formulas.\n* The definition of the prime symbol ', around Eq. (1) and Proposition 1, is unclear, while I presume it denotes the derivative with respect to time $t$.\n* The variables $a$ and $b$ are not used in the statement of Theorem 3 in the main part of the paper. Please clarify the relationship between $[a,b]$ and $[t,s]$ in Theorem 3.\n* In the proof of Th.3, there are confusing duplications of the variables, $dr$, in the nested integrations. Please use different variables for the nested integrals, e.g., $dr$ and $d\\tilde{r}$.\n* $=0$ in the middle of the equation around L554 is possibly a typo.\n\ni) You are correct; the prime symbol denotes the derivative with respect to time $t$. We will explicitly state this definition in the revised paper.\n\nii) In Theorem 2, we fix a $t$ and move $s$ in a small neighborhood. While in Theorem 3, it requires sampling time pairs where $t$ and $s$ can appear in any order (also see Figure 1). Therefore, we use a interval $[a,b]$ to depict that increasing the distance $|s-t|$ is accompanied by expanding the interval where $t$ and $s$ are randomly sampled rather than simply moving $s$, as discussed in Remark 4.\n\niii) Thank you for catching this typo. We will use different variables for the nested integrals in our revision to avoid confusion.\n\niv) Thank you for pointing out this error. We will correct it in the revised manuscript.\n\n[1] Simplifying, Stabilizing and Scaling Continuous-time Consistency Models. Cheng Lu and Yang Song. ICLR 2025\n\n[2] Consistency models. Yang Song et al. ICML 2023\n\n[3] Consistency trajectory models: Learning probability flow ode trajectory of diffusion. Dongjun Kim et al. ICLR 2024"}]

Author rebuttal:

Meta-review: While Reviewer v8s1 raised concerns regarding the stability v.s. performance claim of the proposed loss, the reviewers overall appreciated the novelty and interest of the method for learning ODE-based generative models. Given the proliferation of similar one-step ODE-based approaches with related parameterizations and objectives, the authors are encouraged to provide more detailed comparisons to better highlight their contribution. Overall, I recommend **acceptance**.
Decision: Accept (poster)