Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
Authors: Kaiwen Zheng
Yuji Wang
Qianli Ma
Huayu Chen
Jintao Zhang
Yogesh Balaji
Jianfei Chen
Ming-Yu Liu
Jun Zhu
Qinsheng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Distillation
Consistency Models
Few-Step Generation
MeanFlow
Primary area: generative models
TL;DR: We propose techniques to address issues of continuous-time consistency distillation and scale up to 14B video diffusion models for the first time.

Abstract: Although continuous-time consistency models (e.g., sCM, MeanFlow) are theoretically principled and empirically powerful for fast academic-scale diffusion, its applicability to large-scale text-to-image and video tasks remains unclear due to infrastructure challenges in Jacobian-vector product (JVP) computation and the limitations of evaluation benchmarks like FID. This work represents the first effort to scale up continuous-time consistency to general application-level image and video diffusion models, and to make JVP-based distillation effective at large scale. We first develop a parallelism-compatible FlashAttention-2 JVP kernel, enabling sCM training on models with over 10 billion parameters and high-dimensional video tasks. Our investigation reveals fundamental quality limitations of sCM in fine-detail generation, which we attribute to error accumulation and the “mode-covering” nature of its forward-divergence objective. To remedy this, we propose the score-regularized continuous-time consistency model (rCM), which incorporates score distillation as a long-skip regularizer. This integration complements sCM with the “mode-seeking” reverse divergence, effectively improving visual quality while maintaining high generation diversity. Validated on large-scale models (Cosmos-Predict2, Wan2.1) up to 14B parameters and 5-second videos, rCM generally matches the state-of-the-art distillation method DMD2 on quality metrics while mitigating mode collapse and offering notable advantages in diversity, all without GAN tuning or extensive hyperparameter searches. The distilled models generate high-fidelity samples in only $1\sim4$ steps, accelerating diffusion sampling by $15\times\sim50\times$. These results position rCM as a practical and theoretically grounded framework for advancing large-scale diffusion distillation. Code is available at https://github.com/NVlabs/rcm.

Published as a conference paper at ICLR 2026
LARGE SCALE DIFFUSION DISTILLATION VIA SCORE- REGULARIZED CONTINUOUS-TIM CONSISTENCY
Kaiwen Zheng! :2
Yuji Wang'
Qianli Ma?
Huayu Chen 1.2
Jintao Zhang'
Yogesh Balaji?
Jianfei Chen'
Ming- Yu Liu?
Jun Zhu'
Qinsheng Zhang?
Dept. of Comp. Sci. & Tech., BNRist Center, THU-Bosch MI Center, Al Institute, Tsinghua
NVIDIA
CCorresponding Author
https://research.. wwwiiiiiiiii..m..........ii /aaos//labs///oa dir/rcm
ABSTRACT
Although continuous-time consistency models (e.g.. sCM, MeanFlow) are the- oretically principled and empirically powerful for fast academic-scale diffusion, its applicability to large-scale text-to-image and video tasks remains unclear due to infrastructure challenges in Jacobian- vector product (JVP) computation and the limitations of evaluation benchmarks like FID. This work represents the first effort to scale up continuous-time consistency to general application-Ievel image and video diffusion models, and make JVP-based distillation effective at large scale. We first develop a paraleeissm-compaibbl FlashAttention-2 JVP kernel, enabling sCM training on models with over 10 billion parameters and high-dimensional video tasks. Our investigation reveals fundamental quality limitations of sCM in fine-detail generation, which we attribute error accumulation and the "mode- covering" nature of its forward-divergen the score-regularized continuous-time consistency model (rCM), which incorpo- rates score distillation as long-skip regularizer. This integration complements sCM with the "mode-sceking" reverse divergence, effectively improving visual quality while maintaining high generation diversity. Validated on large-scale mod- els (Cosmos-Prees Preddct,, Wan2.1) up to 14B parameters and 5-second videos, rCM generally matches the state-of-the-an art distillation method DMD2 quality met- rics while mitigating mode collapse and offering notable advantages in diversity, all without GAN tuning extensive hyperparameter searches. The distilled mod- els generate high-fidelity samples in only ~ 4 steps, accelerating diffusion sam- pling by 15x 50x. These results position rCM as a practical and theoretically grounded framework for advancing acrge diffusion distillation.
objective. To remedy this, we propose
teacher
DMD2
IOM (Ours)
SID
consisency distillaionn
scare disiilation
Figure 1: 5 random video samples from 4tsp sCM, DMD2, SiD, and rCM on Wan2.1 1.3B. rCM resolves the quality issues of sCM while showing clear superiority to DMD2/SiD in generation diversity, exhibiting highly similar object posticon/orientaaioonnnttoooo to teacher and sCM.

---

Published as a conference paper at ICLR 2026
Modal Comvistency Model
Score Distillation
CAN
JCM (Ours)
Divergence Type
Forward (Mode- Coverin))))ng
High Diversity
Low Quality (bluridistortion)
Reverse (Mode- Seeking)
High Quality
x
Low Diversity (mode collapss)
High Quality & Diversity
x
x
Easy Tune
Figure 2: High-level comparison of diffusion distillation methods. Despite the theoretical existence of forward divergence, GANs in practice still suffer from limited diversity and model collapse.
INTRODUCTION
Diffusion models have been the cornerstone of generative AI, driving remarkable progress in visual domains such as image and video synthesis (Dhariwal & Nichol 2021
Karras et al. [2022] Ho et al.,
2022
Rombach et al. 2022; Esser et al.) 2024| Brooks et al., 2024 Bao et al., 2024 Wan et al.
2025
Gao et al. 2025 They excel in generation quality, diversity, training stability and scalabil-
ity compared to generative counterparts like generative adversarial networks (GANs) (Goodfellow) 2014), albeit suffering from slow inference. Training free acceleration via specialized sam- plers (Song et al.] 2021a Zhang & Chen 2022 Lu et al. 2022b) still requires over 10 steps to produce satisfactory samples due to the inherent discretization errors of numerical solvers, whereas training-based distillation enables few-step or even single-step generation.
al.
Representative diffusion distillation methods include knowledge distillation (Luhman & Luhman
2021), progressive distillation (Salimans & Ho 2022] Meng et al. (002)). consistency distilla-
tion (Sng al.,
2023 Song & Dhariwal [2023) score distillation (Wang al. 2023 Luo al.
2023b Yin et al. 2024b a Salimans et al. [2024] Zhou et 2024) and adversarial distillation (Sauer
al. 2024bb a Lin
2024][2025g Among these, consistency models (CMs) (Song et al. 2023)
are particularly appealing, as they circumvent the complexities associated with synthetic data gen- eration or GAN training, maintain generation diversity, and achieve competitive performance on image benchmarks. More recently, continuous-time CM (sCM) (Lu & Song 2024) has emerged as a theoretically principled and elegant extension that, compared to its discrete-time predecessors, eliminates inherent discretization errors, decouples training from specific samplers, and dispenses with heuristic annealing schedules. When combined with consistency trajectory models (Kim et al., Heek et al. (024). sCM further gives rise to the popular MeanFlow (Geng et al. 2025).
2023
However, the applicability of sCM to real-world, large-scale diffusion models remains unclear. Although sCM demonstrates scalability by distilling models up 1.5B parameters on ImageNet 512x512, practical application scenarios pose substantially different challenges. Modern large- model training typically relies on infrastructures such BF16 precision, FlashAttention and context parallelism (CP), which complicate and incur numerical errors in sCM's Jacobian-vector product (IVP) computation. Moreover, prior evaluations are limited weakly conditioned ImageNet bench- marks measured by FID, while text-to-ine text-to-image (T2I) and text-to-video (T2V) tasks are strongly con- ditioned and emphasize fine- -reinerried attributes such as text rendering, which FID does not capture. Currently, score- and iiweessrialldistlllaal methods, such as DMD2 (Yin et al. 2024a), remain the state of the art for large-scale diffusion distillation.
Our work represents the first effort to scale up continuous-tmme consistency and JVP to general application-Ievel image and video diffusion models. To this end, we design dedicated infrastructure by developing a FlashAttention-2 JVP kernel and enabling compatibility with parallelisms including FSDP and CP. This allows us to explore sCM's scaling behavior by applying it to 10B+ models and nigh-dimensional video data. Through this investigation, reveal the quality issues of sCM in finc-detail generation and identify the error accumulation characteristic of CMs.
At conceptual level, we argue that the property of diffusion distillation methods is governed by their underlying divergence (Figure - forward (e.g.. CMs), whose objectives are defined on real or teacher-generee generated data, and reverse (e.g., score distillation), where the student is supervised self-generated samples. Forward divergence, commonly used in pre-training is known to encour-
For example, McanFlow aims train few-step model from scratch.
2

---

Published as a conference paper at ICLR 2026
age "mode-coveringe by penalizing underestimation of any training sample likelihoods, which often results in spread-out densities and low sample quality. In contrast, reverse divergence, commonly used in post-training, is inherently "mode-seeking" and excels in generation quality, despite suffer- ing from mode collapse and low diversity.
Motivated by this complementarity, distillation as a long-skip regularizer. This design naturally pairs with sCM: the two supervision signals operate on the forward (external) and reverse (self-generated) data paths, respectively. The broader philosophy of jointly leveraging forward and reverse divergences echoes recent work such training to achieve state-of state-of-the-arg art FID on ImageNet, and DDRL (Ye et al. (025). which incorporates the supervised fine-tuning stage into large-scale diffusion reinforcement learning. We term the resulting distillation framework, together with our other techniques like stable time-derivative computation, the score-regularized continuous- time consistency model (rCM).
address the quality limitations of sCM by integrating score
as DDO (Zheng et al. 2025a). which incorporates reverse KL into maxxiuu--lkkelhoon
rCM requires no engineering complexities such as multi-stage training, GAN tuning or extensive ar- search. We validate its scalability on unprecedentedly large-scale models (Cosmos-PP Preooctt2 (Ali et al. 2025), Wan2.1 I (Wan et al. 2025)), covering T21 and T2V tasks up to 5 seconds and 14B parameters. Empirically, rCM matches or even surpasses DMD2 on quality met- rics, while mitigating mode collapse and offering notable advantages in generation diversity. These results establish rCM as a promising and practical direction for large-scale diffusion distillation.
Extension to Autoregressive Video Diffusion The paradigm of rCM is also promising to autore- gressive video diffission (Yin et al. 2025) for interactive world models. In particular, the currently dominant approach. Self-Forcing Huang et al. 2025), can be viewed as a well-instantiated reverse- KL-style DMD tailored to a bidirectional teacher and a causal student. rCM suggests that forward- divergence- based distillation by causal teacher with teacher forcing could potentially complement self-forcing and enhance diversity and motion dynamics, paving the way for future exploration.
BACKGROUND
2.1 DIFFUSION MODELS
Diffusion models (DMs) (Ho al., 2020; Song et al., 2020) learn continuous data distributions by gradually perturbing clean data æo Paitaa with Gaussian noise, which generates trajectory { along with associated marginals {q: {==) and then learning to reverse this process. The for- ward process follows a closed-form transition kernel (n)((>)/ = ((,,ao of I) with predefined
t=
noise schedule at, ot, enabling reparameterization as ay o,c ~ (O I). The sampling process of diffusion models can follow the probability flow ordinary differential equation (PF-ODE) dæ, [f()), (t)V i log (xx)) dt, where f(t) d llog (²)(()
do? 2 2 at oi.
and Va, log (=x))) is the score function (Song et al. 2020). A key property of diffusion mod- els is the theoretical equivalence of different parameterizations: the network may predict the score (V log gt )). the noise (e), the clean data (.Dg), or the velocity (v), with optimal predictors be- ing analytically interconvertible (Zheng et al., 2023b). With velocity parameterization Vg. diffusion models are trained by minimizing the mean square error (MSE) Eno-puanee. (w)(t (l (2 t) v v//2),
where the regression target is = ågæ0 ō,e (denote f. == /df//dt), and the PF- simplified dæ. (2 t), commonly known as flow matching (Lipman al. 2022). A notable special case, rectified flow (Liu et al. 2022), employs the schedule o₂ 1,0
de
2.2 CONSISTENCY MODELS
Consistency models (CMs) (Song et al. 2023) aim to learn consistency function fo (æt, f) + wo which maps the point D; at arbitrary time f on the teacher PF-ODE trajectory to the initial point æn- The consistency function must satisfy the boundary condition fo 0) = œ. To ensure unrestricted form and expressiveness of the student neural network F8 (x,, t). fo is parameterized as fo t) = (tip(t) + Cou(()) Fo (Cmn (t)æ, Croise(t) (t)) with Cakip 1, Cour (0) 0. This parameterization aligns with practices in diffusion models (Karras et al.) (022). fo is the direct counterpart of the clean data predictor (denoiser) in diffusion models, and typically initialized from the teacher denoiser f freache-
3

---

Published as a conference paper at ICLR 2026
CM's objective is to ensure consistent outputs at adjacent timesteps t At and t on the teacher trajectory. Discrete-time CMs minimize the objective with At 0:
Exo-Panne [wo(t) (2t, f), So (0c- -
(I)
where w(.)) is positive weighting function, d(::.) is distance metric, 0 is the stop-gradient version of 0, and at is obtained by solving the teacher PF-ODE from (æ,, 1) to - At with nu- merical solvers. Discrete-time CMs suffer from discretization enrors and require manually designed annealing schedules for At (Song & Dhariwal 2023] Geng et al.
2024).
Continuous-time CMs, represented by sCM (Lu & Song 2024),
offer a clean upgrade by the CM loss simplifies to fo.fo-(x:: dæ: + (x
taking the limit At 7 0. When d(x y) = Euo fo(t) (t) Tf (t) where ( y ylz.
x
de
de
di
dæ: = theacher (xI, 1). sCM em- is the tangent of fo at (æt,,) 1) along the teacher ODE trajectory de ploys the TrigFlow noise schedule at cos(t), oz sin(t) and precondítioning Casp(!)
cos(t), (t)
sin(t) such that Fe is exactly the velocity predictor Vg. The loss fur-
ther reduces
to
Ex Exo ExorrPonneett
Fo t) Fo- (x1,t) w(t) df. (m..))
where dfg (m)))) di
de
cos(t)(FF) (xe, t) VR, Fo- (me,, 1) day, + OF, (m).t) can be computed using the forward-mode automatic differentiation, St Jacobian- vector product (JVP). This objectíve is a simple MSE which enforces the instantancous self-consistency at (Zt, t). Recent works McanFlow (Geng et al. 2025) and AYF (Sabour et al.
de,)
(in + )F and the full derivative (x
dr
de
2025)
are essentially a combination of sCM and consistency trajectory models (CTM) (Kim et al, under the rectified flow schedule (see Appendix
2023
2.3
SCORE DISTILLATION
Score distillation methods aim match the student distribution po with the teacher distribution Pheacher. where samples æ ~ Pa are generated via æ = Ga(z), p(z) from a noise prior p(z). Directly matching clean, high-dimensional data distributions notoriously difficult (Song & Ermon 2019). A standard remedy is to introduce a "diffused" version by perturbing x through forward diffusion process, yielding æ, with marginal p', and to minimize certain reverse divergences:
min EE, DD ll )l Dy(pb || Plenoher) = (xx.) [r (T (at+)
(2)
0
where Tit PP P5 (xx)) = Procee (2))
is the likelihood ratio. For instance, variational score distillation
2023b) considers the reverse KL divergence (f(r) log r), also known as distribution matching distillation (DMD) (Yin et al.) (024b); the more recent score identity distillation (SiD) (Zhou al. 2024) considers the Fisher divergence f(r) = V log r1
(VSD) Wang et al. 2023 Luo et al.
The gradient VoE,, || Preacter'l typically involves the generator gradient dG, and the score functions V.,, log p (xx),, V.., log (e which are available from diffusion models.
de
As the student score V,, log ra (xx.) is intractable for the few-step generator Go, an auxiliary fake score network introduced. It learns a diffusion model over æo ~ Po by minimizing Euo-xpo.e. e ftae (x, f) To [ and serves a proxy for the student score. Like the critic/dis- criminator in GANs, the fake score is optimized jointly with the student 0 via adversarial interplay. Both the student and the fake score commonly initialized from the teacher diffusion model.
SCALING UP SCM
We hegin by scaling up sCM to T2I and T2V tasks and investigating its performance under different prompt types (see Table for image and video text prompts used in this paper).
There is data std parameter da in original sCM formulation, inherited from EDM Karras al. 2022
For simplicity, we absorb into æo itself, i.c., define xo oa For simplicity, we absorb Cosise into Fg itself.
for original data æ
4

---

Published as a conference paper at ICLR 2026
3.1 ALGORITHM DETAILS
The original sCM relies on multiple implementation tricks for training stability, often requiring fine- tuning or even retraining the teacher model, which is impractical in most distillation scenarios. We first simplify the sCM implementation without compromising stability.
Adapting to Any Noise Schedule. sCM employs the TrigFlow noise schedule æ; = cos + (tn(,)e, while the teacher model is typically trained under other schedules such as rectified flow. Due to the equivalence between different noise schedules and paramelerizations in diffusion mod- els (Kingma et al. 2021] Zheng et al. 2023b), TngFllow-consister
wrapped teacher can be con- structed without retraining. Specifically, let the teacher time be tnl with noise schedule ages dgn. A reverse mapping & (often analytic) from TrigFlow time / to waa can be derived by matching the signal-to-noise ratio, i.e., by solving 2, (an(t). Denote teacher (aiton, (=am) as the original teacher denoiser (can be transformed from other parameterizations) The wrapped teacher is
Freachelaat. t) == Teeucher (Va2re) + (())), Fwacher (I),
cos(f)æ; fieacher (zz, sin
(3)
All wrapping conversions are cheap and performed under FP64 ensure precision. We also wrap the student in the same way so that the raw student aligns with the raw teacher.
Simplification. As our concerned models do not involve the unstable Fourier time embedding or AdaGN layers mentioned in sCM, but instead adopt positional time embedding, AdaLN, and QK
normalization, we keep the network structure. Following sCM's tangent normalization
El the loss is
(c) = Exro
P Pams Atupa
Fo(xx. t) Fo (xt,, t)
(4)
where PG is a time distribution, c 0.1, and g = w(t) dfe (w..t)
de Although BF16 avoids the
overflow issues in sCM's FP16, we still follow the JVP rearrangement by setting w(t) cos(t) and absorbing it into the JVP computation sCM's adaptive weighting, as also noted in AYF (Sabour
lig| Iblll
ct al. 2025 is unnecessary since
Fg Fo 99
remains nearly constant.
3.2
INFRASTRUCTURE
While JVP can be computed with PyTorch's built-in forward-mode operator torch. func. jvp, is not natively compatible with large-scale training setups, necessitating custom implementations. We detail our infrastructure design in Appendix Candd summarize below.
Flash Attention. FlashAttention-22 (Dao) 2023) widely used in large-scale training to reduce memory cost and improve throughput. To enable efficient JVP computation at scale, we develop a Triton (Tillet et al. 2019) kernel that integrates JVP into the FlastAttention-2 forward pass with similar block-wise tiling, supporting both self- and cross- attention.
FSDP. Fully Sharded Data Parallel (FSDP) (Zhao et al. 2023) reduces the memory footprint by par- titioning models across GPUs, but current torch. func. ivp implementation does not support FSDP modules. We therefore refactor networks to perform JVP within each layer: layers expose standard forward functions while additionally accepting tangent inputs and producing tangent out- puts. As long as FSDP granularity matches the layer boundaries, models remain fully compatible.
CP. Context (or sequence) parallelism partitions the input tensor of shape [B, H, L, C] (batch size B. number of heads H, sequence length L, head dimension C) across P GPUs along the sequence dimension L, enabling training with long inputs. In the Ulysses (Jacobs al. (023) strategy, each GPU first holds a slice of size [B, H, L/P, CJ for QKV. An all-to-all operation then redis- tributes QKV to [B, H/P, L, C] for local attention, followed by another all-to-all to restore the sequence partition. This scheme naturally extends to JVP by distributing the tangents of QKV V in the same way and replacing local attention with our FlashAttention-2 JVP kernel.
"MeanFlow's 's adaptive weighting w = 1/([|]] + c) under its best-performing p = the same as tangent normalization.
We find that the weighting cos(t) and JVP rearrangement are also helpful under BF16.
5

---

Published as a conference paper at ICLR 2026
3.3 PITFALLS OF SCALED- UP SCM
3.3.1 EMPIRICAL OBSERVATION: QUALITY ISSUES
Cosmos Prodict2
Cosmas- Prodict2 148
shacke
SCM
IL44
SCM
11
JENT
SCM
Easy
Wan2.1 1.38 sCM
Prame
Prame
Figure 3: 4-step generation results with pure sCM distillation.
We observe that sCM alleviates the blurriness of discrete-time CMs (Luo et al., 2023a) and are capable of generating sharp images. However, in scenarios requiring high accuracy or temporal consistency, distortions are pronounced. As shown in Figure 3 distillation with pure sCM leads to critical quality issues in both and T2V tasks. (1) For T2I, the outputs are close to the teacher under typical prompts, but quality degradation becomes evident in challenging details, such as small text rendering. Morcover, the issues cannot be solved simply by scaling up model size. (2) For T2V, the high sensitivity of human perception temporal consistency makes artifacts notable across prompts. The results exhibit blurry textures and unstable object geometry across frames (e.g., object interpenetration), producing significant and distracting visual distortions.
requiring fine
3.3.2 THEORETICAL ANALYSIS: ERROR ACCUMULATION
The distortions can be interpreted from the perspective of error accumulation. Intuitively, CMs aim to solve the teacher ODE in step, essentially learning the integral S Fieacher r)dr, where the errors accumulate as t increases. Specifically, in sCM, the learning target is
d fff- (are cooo(t)) (at, t) Fiecheer (x,, t)) - (in(t) + d d
(5)
dt
df
self-feedhackk (TVP)
dr weighted by (t), introduces a first-order self-feedback signal via JVP, which is numerically fragile compared to the zeroth-order signal Fe-, particularly under the limited precision of BF16 (Appendix F..2 Near = 0, the student closely resembles the teacher. As training progresses, errors propagate from small to large i and are amplified by self-feedback. When cos(t) sin(r) - 0 at large the teacher supervision from Frencher vanishes and the learning dynamics dominated by JVP.
dF UFg-
4
SCORE-REGULARIZER CONTINUOUS-TH CONSISTENCY MODELS
4.1 QUALITY REPAIR WITH SCORE REGULARIZATION
As shown in Figure we mitigate quality limitations sCM by introducing score-based regulariza- tion long-skip consistency, which complements sCM with reverse divergence. While SiD (Zhou ct al. 2024) achieves impressive results on academic benchmarks, we observe no clear advantage in T21 and T2V tasks (Figure and instead adopt the more memory-efficient DMD (Yin et al. 2024b):
frakee (w,, freacher ( (xt, 1) mean (abs (Co freahhe (Et, t)))
LOMD (0) P.
sg a0n
20
(6)
where frake is the denoiser of the fake score network, pp is time distribution and sg is the stop- gradient operator. The final rCM objective is (o = (o) + XLDMD (0). where À a balancing weight. Empirically, find À 0.01 generalizes across used models and tasks.

---

Published as a conference paper at ICLR 2026
Plata
Pnotse
Pstudent
Puucher
Forward Consistency!
Loss1
Lass2
tmor
d.y.
dt
Reverse Divergencel
Learning Signal Propagation Accumulation
Teacher:
Student:
Forwara Diftsiion Process
PF-ODE Trajectory
Je (Ao, (Consistency Functton w/o Gradllnn)
instenteneeuu Valociy
fa 0) (Consisteccy Fanntion Grudiant)
Illustration of rCM. Left: the forward consistency objective of sCM propagates error from small large times; Right: reverse-divergence minimization serves long-skip regularizer.
Figure 4:
Rollout Strategy. Student generation Co Pe is required for DMD loss and fake score training. As CM, the student supports arbitrary-step sampling by alternating reverse denoising and forward
3 +Ex- iN o O. We randomly choose the number of simulation steps N from [1, N max and only backpropagate the DMD loss through 2024a), which uses fixed t..... INN, CM should explore the entire time range. We thus adopt stochastic scheme by iteratively drawing f, PD and setting tn min(fn, 11) 1) to ensure a monotonically decreasing timestep sequence.
noising from pure noise: 1 3 - - l/
the final step tN + O. Unlike DMD2 (Yin et al.
4.2 STABLE TIME DERIVATIVE CALCULATION
We propose plug-in techniques to stabilize the JVP dF training. preventing sudden collapse after long training. As first noted in DPM-Solver-v3 (Zeng et al. 2023a) and verified in sCM, instability arises from the partial time derivative F, (It, t), due to the oscillatory nature of trigonometric time embeddings. We find two strategies effective.
dt (V Fieaceer + 3 during rCM
Semi-Continuous Time. We compute (Va; Fg-) Fieacher exactly via JVP, while approximating the time derivative with finite difference: O Fg- (aze; () R3 coa(Ar) (m..t)- F, (f,, with Al (in(
10-4. This method is stable for 2B-scale T2I models and requires no architectural changes.
High-Precision Time. Finite-difference approximation. however, sensitive At and becomes unstable for 10B+ models and video tasks. In these regimes, we revert to the native continuous- time derivative computation via full JVP, but enforce FP32 precision for all time embedding layers using the torch... amp autocast context (as done in Wan). Although this introduces an initial mismatch with pretrained Cosmos networks, ensures stable rCM training.
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Models, Tasks and Datasets. To demonstrate scalability and performance of rCM, we distill Cosmos-Predict2 (Ali et al. 2025) T2I models (0.6B, 2B, 14B) and Wan2.1 Wan et al. 2025) T2V models (1.3B, 14B). We leverage curated data from Ali et al. (2025), supplemented with syn- thetic data generated by Wan2. T2V 14B for Wan distillation. In principle, the training could also rely solely on teacher-generated synthetic data, as Yin et al. (2024b) 2025): Huang et al. (2025).
Implementation. Our implementation builds on the Cosmos-Predde support from FSDP2, Ulysses CP, and selective activation checkpointing (SAC). Training alternates between optimizing the student with the rCM loss and updating the fake score via the flow-matching Frke Tate(a (w, w 0]] The teacher denoiser employs classifier
codebase, with infrastructure
loss L(@ E Emaype cttopo
guidance (CFG) (Ho & Salimans
2022), which is simultaneously distilled into the student. Both

---

Published as a conference paper at ICLR 2026
the student and the fake score networks are initialized from the teacher parameters. We perform full-parameter tuning without LoRA, highlighting the stability and performance of rCM
Evaluation Metrics. We use GenEval (Ghosh al. 2023) evaluate T2I models complex com- positional prompts, such as object counting, spatial relations, and attribute binding. For video gen- eration, adopt VBench (Huang et al. 2024) to systematically assess motion quality and semantic alignment. We report the number of function evaluations (NFE) as a quantification of inference ef- ficiency. For video models, we also report throughput in frames per second (FPS), tested with batch size on single H100, covering both diffusion sampling and VAE decoding stages.
The training algorithm and additional experiment details are provided in Appendix B and D
5.2
RESULTS
Table 1: GenEval Results.
Model
*Purams
Resolution
NFR Overall
Single Ohject
Two Object
Counting
Colors
Position
Attribution
Color
Pretriieed Models
SD-XL
Podell at.l. 2023
2,68
1024
1024
50x2
0.55
0.98
0.74
0.39
0.85
0.15
0.23
SD3 S.NT
[Poer a
3154
2,5B
1024
1072
40x
0.66
0.98
0.78
0.50
0.81
0.24
0:52
SD3
Esser
1024
1024
28x2
0.71
0.98
0.73
0.83
0.34
0.47
1021
SANA-1.5 1.5
Xie e al. 2025
4.88
1021
1021
20=2
0.81
0.99
0.93
0.86
0.81
0.59
0.65
0.68
1950 x 768
35:=2
0.81
1.00
0.97
0.74
0.86
0.39
0.70
Cremos-Predie22 Ali 2025
28
1991 708
35x2
01.83
1.00
0.99
0.73
0.89
0.65
0.73
Models
SDXI-LCM Jo al. 30233
2.68
1024 x 10124
0.50
0.99
0.55
0.38
0.85
0.007
0.14
SDXL-Tumbo Pokkl
2,68
512 512
0.56
1,00
0.72
0.49
0.82
0.11
0.21
SDXL
10724
0.84
Hyper-SDXI
1.00
0.89
SDXL-DMD2
Vin al. 2izaa
2,68
1024 x 1024
0.58
1.00
0.76
0.52
0.88
0.11
0.24
SD3 Lumbo
Esser ot 2024
SB
1024 1024
0.68
0.99
0.89
0.68
0.78
0.23
0.54
FLUX. s-chnell
12B
1024
1021
0.69
0.99
0.88
0.61
0,78
0.30
0.52
SANA-Sprint Chen al.
0.89
1.68
1024
1024
0.75
1,00
0.92
0.59
0.91
0.54
0.55
PredisPr DMD2
068
1,350 x 768
0.77
1.00
0.98
0.76
0.85
11:46
0.66
28
13990
768
0.80
0.99
0.98
0.70
0.87
0.57
0.72
0.74
0.88
0.66
Cosmos-Predist
1.00
0.84
0.58
14B
1360 x 768
0.83
1.00
0.98
0.80
0.86
0.59
0.73
0.6B
1300 x 768
0.78
0.99
0.98
0.74
0.86
0.48
0.66
Cosmos: Prennss-teeit2-- rCM
2B
1300
768
0.82
1,00
0.99
0.76
0.85
0.59
0.74
0.6B
1360 x 19900 x 768
0.78
1.00
0.98
0.72
0.86
0.19
0.66
Cosmus-Prodice rCM
28
0.81
0.99
0.97
0.77
0.85
0.57
0.71
14B
13590
768
0.82
1.00
0.98
0.84
0.89
0344
0.72
Dioock
SOXL LOM 18 stepe]
SOXL Turbo stepst
SOXL- -U ightning ateps)
Hyper SOWL steps)
SEX, DMD2 stepa)
SANA-Spri
steps)
S0SS--LTurba
Hype--LLXX
Caammos-Predie222 staps)
Figure 5: Few-step T2I samples compared open-sourced models. rCM can render fine-grained text details such as "Casio G-Shock", "11:44 AM", , and "Thursday, March 22nd" from the prompt.
We evaluate the proposed rCM both qualitatively and quantitatively, comparing it with pretrained models as well as existing distillation baselines. We use 4-step generation by default, which strikes balance between high sample quality and substantial acceleration over the teacher model.
8

---

Published as a conference paper at ICLR 2026
Table 2: VBench Results for Wan (480p), Retested with Diffusers and our augmented prompts.
Throughput (FPS)
Total Score
Quality Score
Semantic Score
Model
#Params
Resolution
NFE
Pretrained Models
1,3B
832 x 480 x 81
50 x 2
0.72
83.02 83.58
83.95
79.26
Wan2.1 T2V Wan et 2025
14B
832 x 480 x 81
50 x 2
0.18
84.26
80.92
Distilled Models
Wan2.1 T2V + DMD2
1.3B
832 x 480 x 81
14.6
84.56
85.58
80.50
Wan2.1 T2V rCM
1.3B
832 x 480 x 81
14.6
84.43
85.38
80.63
14B
832 x x
4.5
84.92
85,43
82.88
1.3B
832 x 480 x
23.0
84.09
84,90
80.86
Wan2.1 T2V +CM
14B
832
85.05
85.57
82.95
1,38
832
32.3
82.65
83.60
78.82
Wan2.1 T2V + rCM
14B
832 x 480 x
14.4
83.02
83.57
80.81
Table 3: VBench Results for Cosmos (720p).
Throughput (FPS)
T2V
12V
Model
#Params
Resolution
NFE
Score
Score
Cosmos- Cosmos-Predict2 TI2V (Ali et al. 2025
1280
704
0.32
83.03
88.6
Cosmos-Predict2 TI2V FCM
1280
704
4.6
84.40
88.2
1-step
2-step
4-step
Cosmos-Predicns Predict22 2B +OM
11:44
Wan2.1 1.38 + -CM
Figure 6: Comparison between different numbers of sampling steps.
Performance. For T2I, we report GenEval scores in Table and provide qualitative comparisons with open-source models in Figure 5 On Desmos-Predictto rCM closely approaches the teacher's performance and benefits from scaling, with the 14B model achieving state-of-the-aar overall score of 0.83 in just 4 steps. Under challenging prompts such as small text rendering, rCM also matches the SOTA few-step model FLUX.I-schnell in visual quality. For T2V. rCM even surpasses the 480p Wan teacher on VBench (T) reaching a total score of 85 when distilling Wan2.1 14B. We also apply rCM to Dosmos-Predictt with higher resolution of 720p and the additional image-to-video (12V) task (Table 3). where similar phenomena are observed. This does not imply that the distilled model strictly superior to the teacher, particularly in terms of diversity and physical consistency, but highlights rCM's ability to preserve quality under few-step generation.
Comparison with DMD2. We implement the DMD2 (Yin et al. 2024a) baseline by additionally parameterizing a discriminator as a branch of the fake score network and incorporating the non- saturating GAN loss supplement DMD training. This branch takes intermediate features from the fake score network and queries them with single Iearnable token to produce a discrimination logit, akin APT (Lin et al. (025a).. As reported in Tables I and rCM matches or even surpasses DMD2 in generation quality, measured by GenEval and VBench. Moreover, we observe rCM's clear diversity advantage, particularly video generation. As highlighted in Figure [ rCM re- tains the diversity of sCM, while simultaneously resolving sCM's visual quality issues. In contrast, DMD2 tends to produce collapsed generations, where objects converge to similar positions and ori- entations, leading reduced diversity. These findings suggest that jointly leveraging forward- and diverse-ivvergence- based methods forms a promising distillation paradigm, yielding models that simultaneously achieve high quality, strong diversity, and substantial speedups.

---

Published as a conference paper at ICLR 2026
Generation with Fewer Steps. We additionally report rCM's 1-step and 2-step results in Tables and [2] and further compare few-step generation quality in Figure 6 For T2I, rCM produces rea- sonable samples across 1-4 steps. with GenEval scores degrading only slightly under 1- or 2-step settings. For simple prompts, 1-step generations are nearly indistinguishable from 4-step, whereas for more challenging prompts they show clear deficiencies in detailed text rendering. For T2V, the task is more demanding: 1-step outputs appear blurry across prompts and exhibit a marked drop in VBench scores. In contrast, 2-step generations already reach scores close the teacher, though with minor shortcomings in quality and background fidelity. At 4 steps, rCM further refines fine details and even succeeds at rendering sharp text in complex backgrounds, such as street signs. Overall, these results highlight rCM's robustness under extremely few steps, enabling competitive T21 generation with only step and T2V generation with only 2 steps.
== (04:22)
Divesityy
2 01 (8457)
= 0.01 (81.4)
Sweet Spot
0.001
Quality
(82.88)
Figure 7: Video samples from 4-step Wan2.1 1.3B rCM models under different X. For each prompt, we use 5 different random seeds to demonstrate diversity. VBench scores are in the parentheses.
Ablation Study on A. In principle, the balancing weighting À between sCM and DMD losses should control the trade-off between diversity (mode-covering) and quality (mode-secking). To validate this, we perform grid search over {1, 0.1, 0.01, 0.001} on W.n..1 training each model with batch size of 64 for 10k iterations. As shown in Figure 7 larger À (i.c., stronger DMD weighting) results better quality but less diversity, while smaller values exhibit the opposite trend. At granularity of one order of magnitude, we find that À 0.01, as the smallest scale to preserve good quality, offers a "sweet spot" balancing both quality and diversity.
CONCLUSION
In this work, we present rCM, score-regularized continuous-ting time consistency model that scales diffusion distillation to large image and video models. By integrating forward-divergence-baa score distillation, rCM remedies the quality
consistency distillation with reverse-divergence--aee limitations sCM while showing superior diversity advantages compared to DMD2. Our distilled models achieve competitive text-to-image results in single step and text-to-video results in only 2 steps, delivering up to 50x acceleration over teacher models. Looking forward, we believe that combining forward- and reverse-divergene principles provides unifying paradigm that may inspire new research in generative modeling.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models (LLMs) solely as writing assistant for language polishing and improving clarity of presentation. The LLMs not involved in research ideation, methodolog- ical design, experimental execution, or result analysis. All scientific contributions and substantive writing were carried out by the authors.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM101, NSF of China Projects (Nos. 62550004, U25B6003, 92370124, 92248303); Beijing Natural Science Foundation L247011: the High Perfor- mance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.
We thank Guande He, Cheng Lu, and Weili Nie for valuable discussions.
REFERENCES
Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tian- shi Cao, Elizabeth Cha, Yu-Wei Chao, et al. World simulation with video foundation models for physical ai. arXiv preprint 1xXiv:25/1.00060 2025.
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng. Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: highly consistent, dynamic and skilled text-to- video generator with diffusion models. arXiv preprint arXiv:2405..m 04233, 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. 2024. 3, 2024.
URL http://openai..
Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Song Han, and Enze Xie. Sana-sprint: One-step diffusion with continuous-time consistency distillation. arXiv preprint arXiv:2503.0960 2025.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and invertible generative modeling. Advances in neural information processing systems, 32, 2019.
Jacobsen. Residual flows for
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint urXiv:2307.086// 2023.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint 2rXiv:2506.09111 13, 2025.
Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406. /4548, 2024.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint ar0i /3447, 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances Neural Information Processing Systems, 36: 52132-52152, 2023.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pp. 2672-2680, 2014.
Guande He, Kaiwen Zheng, Jianfei Chen, Fan Bao, and Jun Zhu. Consisteney diffusion bridge models. Advances in Neural Information Processing Systems, 37:23516- 2024.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv
preprint (.iv:2403.06880)) 2024.
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho and Tim Salimans. MrXiv:2207.12995 2022.
Classifier-free
diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 60000-6811111 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22/0.02300: 2022.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint Saki::2506.0000000 2025.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Champaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 21807-21818, 2024.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 14509, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 01226555-265777 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:2199--277777, 2021.
Black Forest Labs. Flux. http::/://ggttab..oo//////eeteee
flux 2024.
Sangyun Lee, Yilun Xu, Tomas Geffner, Giulia Fanti, Karsten Kreis, Arash Vahdat, and Weili Nie. Truncated consistency models. arXiv preprint arXiv:2410. 14895, 2024.
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2y-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. Advances in neural information processing systems, 37:75692-75726, 2024a.
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design. arXiv preprint arXiv:4410006677, 2024b.
Shanchuan Lin, Anran Wang, and Xiao Yang. Saxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.1 13929, 2024.
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang. Xuefeng Xiao, and Lu Jiang. Diffusion adversar- ial post-training for one-step video generation. arXiv preprint arXiv:2501.08.. 008316, 2025a.
Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time interactive video gen- eration. arXiv preprint arXiv:2506.09350 2025b.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 17X:0:2210.02747, 2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint wrXiv:2209.03003 2022.
12

---

Published as a conference paper at ICLR 2026
Cheng Lu and Yang Song. Simplifying. stabilizing and scaling continuous-timg consistency models. arXiv preprint arXiv:2410.1108/ 2024.
Cheng Lu, Kaiwen Zheng, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for ccore-based diffusion odes by high order denoising score matching. In International conference on machine learning, pp. 14429-14460) PMLR, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems, 35:5775-5787, 2022b.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint 0rXiv:2201.023860 2021.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04377 2023a.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36.76525-76546, 2023b.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14297-14306, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXivv2307.01952, 2023.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. Advances in Neural Information Processing Systems, 37:1 117362, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your flow: Scaling continuous-time flow map distillation. arXiv preprint arXiv:2506. 14603, 2025.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 22Xiv:2202.005//1 2022.
Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. Advances in Neural Information Processing Systems, 37: 36046-36070, 2024.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIG- GRAPH Asia 2024 Conference Papers, pp. 2024a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis-
tillation. In European Conference Computer Vision, pp. 87-103. Springer, 2024b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. Interna-
tional Conference on Learning Representations, 202
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310. 141199 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2026
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 32211-32252 PMLR, 2023.
Philippe Tillet, Hsiang-Tsiin Tsung Kung, and David Cox. Triton: intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop Machine Learning and Programming Langnages, pp. 10-19, 2019.
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20. 14, 2025.
Fu- Yun Wang. Zhaoyang Huang, Alexander Bergman, Dazhong Shen, Peng Gao, Michael Lingel- bach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency models. 2024.
Advances in neural information processing systems,
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in neural information processing systems, 36:8406-68 118406-8441, 2023.
Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer, 2025. URL htttt:://aaxiiii
abss/2501.8827
Haotian Ye, Kaiwen Zheng, Jiashu Xu, Puheng Li, Huayu Chen, Jiaqi Han, Sheng Liu, Qinsheng Zhang, Hanzi Mao, Zekun Hao, et al. Dega-regul reinforcement learning for diffusion mod- els at scale. arXiv preprint arXiv:2512.04333 2025.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 37:47455-47487, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang. Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 6613-6623, 2024b.
Tianwei Yin, Qiang Zhang, Richard Zhang, William Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 22963-22974, 2025.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. 1277, 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36: 55502-55542, 2023a.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp. 42363- 42389. PMLR, 2023b.
14

---

Published as a conference paper at ICLR 2026
Kaiwen Zheng, Yongxin Chen, Huayu Chen, Guande He, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Direct discriminative optimization: Your likelihood-baase visual generative model is secretly gan discriminator. arXiv preprint arXi::2503.01 01103, 2025a.
Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, and Jun Zhu. Elucidating the preconditioning in consistency distillation. arXiv preprint arXiv:2502.02922, 2025b.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang, Score identity distillation: Exponentially fast distillation pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Consistency Models Consistency models (CMs) (Song et al. (023)
accelerate diffusion sampling
by taking shortcuts along the teacher ODE trajectory and directly predicting the starting point. Con- sistency trajectory models (CTMs) (Kim et al. 2023) and multi-step CMs (Heek et al. 2024 gen- eralize the approach to predict trajectory jumps to arbitrary intermediate points. CDBMs (He et al, 2024) adapt CMs to diffusion bridges models. However, CMs suffer from training instabilities and quality issues such as blur. Subsequent efforts address these limitations by introducing dedicated
annealing schedules (Song & Dhariwal 2023 Geng al.] [024), preconditioning strategies (Zheng
et al.
2025b), or segmented consistency schemes (Wang et al. [0244] [Ren et al. 2024 Lee et al,
2024
Yet these approaches often come with added complexity, such as multi-stage training or ex- lensive hyperparameter tuning. The recent sCM (Lu & Song 2024) represents the most advanced
CM solution, being theoretically principled, practically simple, and empirically effective on aca- demic image benchmarks. MeanFlow (Geng et al.] (025) and AYF (Sabour et al., (025)) which directly combine sCM with CTM, have also drawn significant attention. Nonetheless, the applica- bility of sCM to large-scale, application-Ievel image and video diffusion models remains unclear. SANA-Sprint (Chen et al. 2025) applies sCM to a modest 1.6B text-to-image model, while delib- erately sidestepping the key challenge of JVP computation by relying a base model with linear attention rather than the widely adopted FlashAttention, limiting the application scenarios.
Video Diffusion Distillation Existing practices distill video diffusion models with CMs, score distillation GANs. T2V-Turbo (Li et al. 2024a b) employs CMs but relies additional reward models to enhance quality. By contrast, we conduct pure distillation while still delivering remarkable video quality. APT (Lin et al. (025a) applies an adversarial GAN loss for one-step video generation. Another line of work distills a bidirectional teacher into an autoregressive student to enable real- time streaming video generation. Within this direction, CiusVid (Yin al., 2025) leverages DMD loss with diffusion forcing, while Self- Self-Forcing (Huang et al.] (025) and APT2 (Lin et al. 2025b) introduce student foreing to address the exposure bias inherent in diffusion foreing.
JVPs Generative Modeling Jacobian-vector products (JVPs) are a fundamental computational primitive in generative modeling, as they enable efficient handling of high-dimensional Jacobian in- formation without explicitly materializing the full matrix. They are widely employed in normalizing
flows (Chen et al. 2019) and diffusion models (Song et al. 202 Lu et al. 2022al Zheng al. 20236 a). for example to estimate matrix traces via Hutchinson's 's trick or to derive exact coefficients for the optimal sampler. To the best of our knowledge, this work is the first to integrate JVP sig- nals into large-scale generative model training, with modern FlashAttention architectures, diverse parallelism strategies, 10B+ parameter networks, and high-dimensional video data.
B ALGORITHM
We provide the detailed algorithm of rCM in Algorithm where we adopt slightly different tangent warmup strategy compared sCM. We find the tangent warmup not essential for rCM.
C INFRASTRUCTURE
C.I FLASHATTENTION JVP
FlashAttention-22 2 Dao 2023 is an optimized attention algorithm that reduces memory usage and
improves throughput by tiling the sequence into blocks and streaming intermediate results without materializing the full attention matrix. Given query, key, and value sequences Q R.Nidd K, V e N2xd, where N₁ and N₂ denote sequence lengths and d is the head dimension, the attention output O E NiNixx is computed as
S =
QK T NIXNo
P = softmax(S) E R NixNo
O = PV E R Nixd
where softmax is applied wise. In multi-head attention (MHA), this computation is carried out in parallel across heads well across the batch dimension (number of input sequences).
16

---

Published as a conference paper at ICLR 2026
Algorithm Score-Regularized Continuous-Tine Time Consistency Model (rCM)
Require: dataset D, teacher diffusion model Oreacher with TrigFlow-wrapped consistency function fwacher and v-predictor Funchers student model @ with wrapped fo. Fo. fake score model fjkk with wrapped flake: Fuke. time distributions pG: PD. student update frequency maximal number of simulation steps Naxx number of tangent warmup iterations H, number of total iterations I. Initialize: 0 Oteacher, Ofake f- Owencer
1: for = 1 to I do
2: S H or i mod F 0 then Zo ~ D,E (O, ~ PG, + cos(t) + sin(t)e
3:
// Generator Step
cos(t) sin(t) dF.--
4:
min(1, (111,,,, i JVP(PV (x., t), (cos(t) sin(t) Fionher (xe, sin(t)FFussaa cos(t) sin(t)))
5:
6:
9
cos(1)) sin (() (Fg- (xre (areche( 1)) (cos(t) + cos(t) sin(t) )=)
7:
((0) - Fo Fø- (xi,,) lall
8:
ifi H then
9:
N 7~ ((11 Nmax) Starting from L1 5. iteratively sample timesteps Li,..., IN by in PD, ta = min((n, In-1)
10:
Perform hackward simulation fi S t 2 top +ex tx 0 to obtain zb En (0, I), pn;
11:
12:
vip cos(tn) )ar + sin(tn)en
fa - frcttor ( (D)
13:
(( (()) x sg
mean(anc(a Senee ( (b))))
end
15:
Update the student with loss L(0)
16:
else
N U(1 Nmax)
Critic Step
18:
Starting from
iteratively sample timesteps f fi,..., AN by fn PD.tn min(fn, tn-1) Perform backward simulation 25 t L2 " tzp +-g-1 ix " to obtain w.""
19:
20:
PD, cos(t) + sin(t)e, cos(t)e sin(() Update the fake score Onae with flow- now-matching loss [(Orae) Flaee (x,, 1) w/13
f), (O,
22:
end if'
end for
17

---

Published as a conference paper at ICLR 2026
For the Jacobian- vector product (JVP), seek the tangent to tQ E R Ni xd and tK, tV E R2aa defined as to d0 aQ
R Ni xd given input tangents
do ak
tK d0
dV d By the chain rule,
this can be expressed in matrix form as
tS tQK' T + QtK
tP P tS P o (PP o 1,, (,)
to tPV PtV
where denotes the element- product. Aggregating terms, we obtain
to PtV+
HV
diaagrowsunn H O, where P
As noted Lu Song 20224) both O and to can be computed within single streaming loop, analogous the FlashAttention-2 forward pass. We make this procedure explicit in Algorithm 2
Algorithm 2
FlashAttention-2 Forward Pass with JVP Computation
Require: Matrices Q, K. V, their tangents tQ. tK,tV, block sizes Bey Br. Split Q, tQ into T, blocks Q1...., Qr. and tQ .Q...... tQr, of size B, d. 2: Split K, tK, V, vV into T. blocks K,,..,, KT., tK,,... tKT. Vi,..., VT. WV....,tV; of size B. x d. 3: Split output O into T, blocks O....., Or,, and L into T, blocks Lı,.... LT..
4: Split output tangent tO into T, blocks t........ tO
5: for i = 1 to T, do
6: Load Q2 tQ; from HBM to SRAM.
7:
Initialize m, < (- B. OB, O, + B ri t- OB,, A₁ 08.xd B, -.d,
8:
for j- 1 To do Load K,, tK, V,, tV, from HBM to SRAM.
9:
10:
Compute Sij Q.K tS = tQ Q.tK..
11:
Compute Mmew max (m, rowmaxxX (),))
Compute P Poj exp(S.j M.ow).
13:
Compute knew e mc Marr le + rowsum (...) P.y).
14:
Compute O = diag(e mma 00 + P.j V
15:
Compute A new = diag( (e mman maee A + P tt
16:
Compute H. ijj Pij ② tS.
Compute Fnow m= mire
rowsum H
18:
Compute B Bnew = diag(e (m m4 Meee
B: H,, V j.
19:
Update m. Maew, li Loows O O new, A; A Avows P3s f- Tnew, B; t- B Boew
20:
end for
21:
Compute O diag(Ence ) ' new
22:
Compute L. = Mnew + log ).
23:
Compute C, diag(rnew) JO;
24:
Compute tO, = diag(fus (f
(A; + B, B.).
Write O L,, tO, HBM.
26:
end for
return O, L, tO;
C.2 NETWORK RESTRUCTURING
To make JVP computation compatible with Fully Sharded Data Parallel (FSDP), restructure the forward functions of network layers. Specifically, we define a base class JVP (Listing I that extends torch. nn. Module and supports both standard forward execution and JVP-mode execu- tion. When witherTrue, the forward pass receives and returns both the primals and their tangents, with each primal and the correpsonding tangent wrapped in the TensorMis th tuple type.
18

---

Published as a conference paper at ICLR 2026
For each layer, the original forward logic is moved into -forward, while JVP computation is delegated to _forward.jve using torch. func. jvp. Other components (e.g.. parameter ini- tialization) remain unchanged. Figure shows example restructuring of the RMSNorm layer.
The attention block an exception since the native FlashAttentionn22 does not support JVP compu- tation with torch. func. - vp. When implementing Jode forward of the attention block, replace the self-attention and cross-attention components with our implemented FlashAttention-22 JVP kernel, while the remaining modules still rely torch. func. jvp.
Listing Base class JVP that supports both standard forward execution (- (-forward) and JVP- forward execution (.fowarddjvv))
TensorWithT Tuple Mettorc..Tenor,, Tensor, torch. Tonsor]
class JVP (torch. nn. Module)
def (self
super ()per()_nintee init ()
def forward(self, *args, **xwargs):
withT kwargs prrisssppp""iihh" False)
("withT",
if with!:
return sel for
**karggs)
else:
return self farf.foowwarddooo (Hargs, kkwargs)
def jvp *args, **kwargs):
raise NotImplleennedEzror
def (sel, argg,, **wwargs)
raise NotImplementedBrror
clase aetioca Stoch Kodala)
class iositoca (a00)
(Ipec 1715
Rspec (1 1tit
Nedghe aecamtteriocodt (cir))
Panatetor cocchooooeeeryyy
terchein, ove,, f welght)
texch.en in oved, welghh)
dal Lan5s Mi1
caget gow cenintel, kecpiie: Trie) lepo)
capet gow cenintlly kerpiii Trme) repot
forvardranie, bezeh, Dacsary bopeh Consoc: reburn output self, weiget
__tovaaL_ivptaais xt Tenapelishh] Demaortithhc
Y. K.vicht
ofpceel.._.foraaed 1877 (2601 secuen cos6, T,00s, detachi)
det _torvarddion to
Teracei sorch, Teracer
octpac hooee flost (13-tyaaula
Figure 8: Restructuring example for the RMSNorm layer: (left) original implementation, (right) JVP-enabled implementation.
D EXPERIMENT DETAILS
Training Details. The rCM training configurations for different models and tasks are summarized in Table 4 We maintain a smoothed version of the student parameters using the power EMA (Karras) al. 2024), and use the EMA model for evaluation. We use the AdamW optimizer with B1 32 0.999 and weight decay of 0.01 for both student and fake score optimizers, while disabling gradient clipping, which we find crucial for maintaining training stability of rCM.
Evaluation Details. For GenEval, we repeat the 553 test prompts four times to reduce vari- ance. For VBench, we follow standard practice and use [TT-4o-aagmenterr prompts. We observe that Fmax governs the trade-off between quality and diversity. We adopt timesteps (estann(mmax))) 1.3, 1.0, 0.6] for 4-step sampling and take the first entries when sampling with fewer than 4 steps. We set dmax 80 for high-diversity visualizations, and in some cases increase
19

---

dF,- under BF16 incurs Published as a conference paper at ICLR 2026
it when computing metrics that emphasize high quality. For the 8-step result in Figure |aretan(Onaan (omax), 1.3, 1.0, 1.0, 0.6, 0.6, 0.3, 0.3).
Table Training and evaluation configurations.
T21
EMA Length
0.05
0.05
0.05
0.05
0:05
Butch Size
1024
512
256
256
64
Contest Parllll Size
1
1
10
Leuming Rate (staden))
le-6
le-6
lc-6
3e-6
1c-6
score)
2x-7
40-7
CPG
4.5
5.D
Stadent Update Froquency
5
Maximul Simulation Steps
Tingent Wirmup lierations
0
1000
200
Total lierations
80k
30k
25%
10k
10k
Pooe
log - (-008. 1.6%)
log (-88, 1.f")
log A (-0.8, 1.6%)
log
8 1.65)
logs (/
[ arctan(2)
= arctam(s)
arctam(:)
areeaaatnyy aretanlyTTa TTa)
axetion Pa)
(4
(/000, 1.6))
N7000 1.))
(0.0, L
PD
arctan(s)
arctamf))
arctan(s)
oretam
areton
E MORE DISCUSSIONS
E. CONTINUOUS-TMN CONSISTENCY TRAJECTORY MODELS
sCM can be easily combined with consistency trajectory models (CTM) (Kim et al. 2023 Heek al. 2024), which adds additional time condition < to CMs and consider more fine-grained transitions æ, > s. on the PF-ODE, forming an interpolation between diffusion models and con- sistency models. Specifically, can define consistency trajectory function fa (Iz, + æ, from t to with preconditioning coefficients derived from the DDIM (Song et al. 2021a) step:
fo 1, 8) = cos(t - s)æ; sin(t - 8) Fo t,),,,
Continuous-time CTMs (denoted sCTM) be trained via similar instantaneous objective of sCM by simply changing the coefficients, as s is independent of t and remains uninvolved in the JVP computation w.r.t. t:
|| Fa (2)e, Fe- (2), - -w(l,)) dfo- (x t, s)
27
Ex,,,,
(8)
dt
where
dff (xi,,, -cosscooss--- Fg- (0r,,1,))
dx: sin(( -)) 0x: d.F (I), t, 8)
cos(1- s) 8e
(9)
dt
di
dt
The objective naturally recovers flow matching under s t: when w(t,t) cos(t - s)), it is exactly the same as flow matching; other arbitrary w(t, > 0 gives equivalent objective whose gradient is proportional to that flow matching. Recent methods such Mean- Flow (Geng et al. 2025) and AYF (Sabour et al. (2255) are the same as sCTM under the rectified flow schedule, which simply changes the preconditioning to fo (2r,,,)) xt (I (a L) and adjusts the JVP coefficients accordingly.
(e.g., wt,
For distillation, we also implemented sCTM without extensive hyperparameter tuning, but observed that it underperforms sCM in both quality and diversity on basic T2I tasks (Figure 9.. This suggests that SCTM (or MeanFlow) encounters greater optimization challenges than sCM for diffusion distillation, as learning arbitrary mappings along the ODE trajectory is inherently more demanding than learning the mapping solely to the initial point.
E.2 ANALYSIS OF JVP ERRORS
To avoid overflow issues in FP16, BF16 precision is required for neural network computation in large model training. However, we find that computing the JVP term dt substantially larger numerical emors compared the zeroth-order signal Fg-.
20

---

Published as a conference paper at ICLR 2026
12
4
8
5
7
6
5
7
(a) SCM
(b) sCTM (MeanFlow)
Figure 9: Comparison between sCM and sCTM for distillation. We implement sCTM by adding an additional time condition to the network, which goes through separate embedding layer and is added to the embedding of t before normalization. We adopt the sCTM training objective in Eq. (8). along with sCM tricks such as tangent normalization.
To quantify these errors, compute Fo- using both BF16 and FP32 precision, and measure the relative L2 error with
PEP
2
where BBF16 and FPP32 denote outputs under BF16 and FP32, 6-
respectively. We repeat the procedure for the rearranged JVP term (ts sin(t) d.F. the network precision is altered, while all wrapping conversions remain in FP64, consistent with the main algorithm. Figure 10 reports the relative L2 errors between BF16 and FP32 computations across 100 uniformly sampled timesteps from t 0 to 2 using Nosmos-Predict T2I models of 0.6B and 2B parameters. The results indicate that JVP computation is considerably more sensitive to limited BF16 precision than the network output.
de Note that only
Fa-
6
Fe- cos(t)sin(t))*
2.5
cos(t)sin(t)
5
2.0
Relative L Error
4-
Relative La
1.5
3.
1.0
2
0.5
m
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
0.0
0.2
0.4
0.6
0.8
1.2
1.4
1.6
(a) Cosmos-Pree Predict2-O..e 0.66
(b) Cosmos-Predict22 Predict2-2B
Figure 10: Relative Le errors of the network output and JVP under BF16 precision. Empirically, JVP computation leads to substantially larger numerical errors compared to the network output.
F
PROMPTS
21

---

Published as a conference paper at ICLR 2026
Table 5: Used prompts in this paper.
Prompt
References
Image
Red squirrel drumming on tiny twig and acomn drums in autumn woods
Figure 6
A Casio G- O-Shock digital watch with metallic silver hezel and a black face. The watch displays the time as 11:44 AM on Thursday, March 22nd, with additional features like Bluetooth connectivity, water resistance up to 20 bur, and multi-band radio wave reception. watch strup appears to be made stainless steel, and the overall design emphasizes durability and functionality.
Figure 3 5
alarm clock
Figure 9
Video
A stylish woman walks down a Tokyo street filled with warm glowing neon and unimated city signage. She wears black leather jacket, a long red dress, and black boots, and carries black purse. She wears sunglasses and rod lipstick. She walks confidently and casually. The street is damp and reflective, creating mirror effect of the colorful lights. Many pedestrians walk about.
Figure 16
Animated scene features a closc-up of a short fluffy monster knceling beside a melting red candle. The art style is 3D and realistic, with a focus on lighting and texture. The mood of the painting is one wonder and curiosity, as monster gazes at flame with wide eyes and open mouth. Its pose and expression convey a sense innocence and playfulness. if it is exploring the world around for the first time. The use of warm colors and dramatic lighting further enhances the cozy atmosphere of the image.
Figure 17
The camera follows hehind a white vintage SUV with a black roof rack as it speeds up steep dirt road surrounded by pine trees on a steep mountain slope, dust kicks up from it's tires, the sunlight shines on the SUV il speeds along the dirt road, casting warm glow over the scene. The dirt road curves gently into the distance, with no other cars vehicles in sight. The trees on either side of the road are redwoods, with patches of greenery scattered throughout. The car is soen from the rear following the curve with case. making it seem as steep hills and mountains, with clear blue sky above with wispy clouds.
Figure
on a rugged drive through the rugged terrain. The dirt road itself is surrounded by
A close up view of a glass sphere that has zen garden within it. There is a small dwarf in the sphere who is raking the zen garden and creating patterms in the sand.
Figure
playful raccoon seen playing electronic guitar, strumming strings with its front paws. The raccoon has distinctive black facial markings and bushy tail. It sits comfortably on small stool, its body slightly tilted as it focuses intently on the instrument. The sctting is cozy, dimly lit room with vintage posters on the walls, adding retro vibe. The raccoon's 's expressive eyes convey a sense of joy and concentration. Medium close-up shot, focusing on the raccoon's face and hands interacting with the guitar.
Figure
In an urban outdoor setting, a man dressed in black hoodie and black track pants with white stripes walks toward a wooden bench situated near a modern building with large glass windows. He carries a black backpack slung over one shoulder and holds a stack papers his hand. he approaches the bench, he bends down, places the papers with yellow accents and black pants joins him. She stands beside the bench, facing him, and appears to engage in a conversation. The man continues to review the papers while the woman listens attentively. In the backgruud, other individuals can he seen walking by, some carrying bags, adding to the bustling yet casual atmousphre of the scene. The overall mood suggests a moment of focused discussion or preparation amidst busy environment.
and then sits down. Shortly after,
wearing red jacket
Figure
22


Reviews:
[{"review_id": "zrRsU8cvXB", "reviewer": "Reviewer_stKY", "summary": "The authors propose rCM which scales continuous-time consistency models (sCM) to 10B+ parameter image and video models. For efficient computation of JVP in sCM training, they develop FlashAttention-2 JVP kernels compatible with FSDP and context parallelism. To fix sCM's quality issues in fine details, it also adds score distillation (DMD) as regularizer. This loss results in combining forward-divergence (high diversity) with reverse-divergence (high quality). Experiments on Cosmos-Predict2 and Wan2.1 show 4-step generation matching DMD2 quality with better diversity and speedup over teachers.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper shows good experimental results across comprehensive benchmarks that achieve SOTA from the benefits of large-scale models.\nThis is the first attempt to scale continuous-time consistency distillation to application-level models with over 10 billion parameters and high-resolution video generation, demonstrating high practicality. \nThe contributions on infrastructure, such as FlashAttention-2 JVP kernel, are valuable, allowing large-scale training to be feasible.", "weaknesses": "1. The methodological novelty is limited. rCM is essentially a trivial weighted sum of existing methods (sCM + $\\lambda$DMD). \n2. Section 3.1's \"Algorithm Details\" appears to directly use techniques already proposed in the original sCM paper (Lu & Song 2024), with no clear additional contributions.\n3. It remains unclear whether rCM actually outperforms DMD on smaller-scale models where infrastructure is not a bottleneck in Table 2.\n4. There's no ablation study on the loss.\n5. Could the authors provide a detailed training cost comparison between rCM and DMD2?\n6. I wonder if the authors have attempted combining sCM with GAN training instead of DMD, as it also has a mode-seeking property.\n7. It is unclear what exactly the 'long-skip' refers to in a long-skip regularizer.\n\nOverall, the paper's main contribution appears to be its infrastructure part. However, this component is only briefly mentioned in the main paper, and even the appendix does not seem to reveal any particularly novel methodology or contribution compared to prior work.\nWhile the work has practical value, the authors do not convincingly argue that sCM is essential for large-model distillation. It is conceivable that other approaches might not suffer from the training efficiency issue to the same extent, or that this issue is not as critical as implied.\nThe lack of experiments ablating this specific efficiency claim, or comparing it against alternatives that might not have this issue, makes it difficult to see a significant merit in the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "majXrspjGf", "reviewer": "Reviewer_Lry4", "summary": "The paper introduces score-regularized continuous-time consistency models (rCM), a framework for few-step distillation of large text-to-image (T2I) and text-to-video (T2V) diffusion models. The authors first scale continuous-time consistency models (sCM) to application-level models by implementing a FlashAttention-2–compatible JVP kernel and making sCM work with FSDP and context parallelism. They then diagnose a core limitation of sCM—fine-detail degradation and temporal artifacts—and attribute it to error accumulation and the mode-covering nature of its forward-divergence objective. To fix this, they propose rCM, which adds score-based distillation (DMD) as a long-skip, reverse-divergence regularizer, yielding higher visual quality without sacrificing diversity. On large teachers (Cosmos-Predict2 up to 14B params; Wan 2.1 up to 14B), rCM achieves 1–4 step generation (15×–50× speedups) with GenEval/VBench scores matching or exceeding DMD2, and better diversity, with additional stabilization via semi-continuous/high-precision time-derivative handling.", "questions": "+ Please ablate the sensitivity of $\\alpha$, rollout schedule $p_D$, and the number of rollout steps $N$.\n\n+ Will the FlashAttention-2 JVP kernel, training code, and/or distilled weights be released?\n\n+ What are the total training flops, GPU hours, and memory profiles for 2B/14B and for video? This would help others assess feasibility of rCM at scale.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Point out that sCM’s forward-divergence and JVP self-feedback cause error accumulation and mode-covering behavior; reverse-divergence regularization addresses fine-detail quality.\n\n+ The FlashAttention-2 JVP kernel and compatibility with FSDP/CP are substantial contributions for training at 10B+ and for video; these infra details are often the bottleneck in practice.\n\n+ On T2I GenEval, rCM matches/approaches large teachers in 1–4 steps; on T2V VBench, rCM even surpasses the 480p Wan teacher in total score at 2–4 steps while being much faster.\n\n+ The paper emphasizes that rCM retains sCM’s diversity while repairing quality, contrasting with DMD2’s tendency toward collapse (Fig. 5).", "weaknesses": "+ The claim that a single $\\lambda=0.01$ works “across all models and tasks” is strong; sensitivity to $\\alpha$, rollout schedule $p_D$, and the number of rollout steps $N$ should be ablated. Likewise, the contribution of each stabilization component (semi-continuous vs FP32 time embedding) deserves a clean ablation.\n\n+ While DMD2 is a relevant and strong baseline, additional comparisons would strengthen the case: SiD (score-identity distillation) and MeanFlow/AYF-style CTM variants at similar budgets, as well as recent adversarial post-training for video. The paper mentions these families but does not show comprehensive side-by-side results.\n\n+ Results focus on Cosmos-Predict2 and Wan2.1; evidence on qualitatively different teachers (e.g., SDXL/FLUX/SANA or smaller open models) would underscore generality, especially for non-Transformer or non-FlashAttention architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POe9a07ssc", "reviewer": "Reviewer_J8HH", "summary": "This paper proposes rCM, a score-regularized continuous-time consistency model, to scale diffusion distillation to large-scale T2I and T2V models. It addresses quality issues in pure continuous-time consistency models by integrating reverse divergence-bsaed score distillation as a regularizer. Besides, the authors introduce a lot of infrastructure to help distill the model.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel formulation. The core idea of combining forward and reverse divergence within a consistency model framework is novel, and the authors provide some theoretical supports.\n\n2. Significant scaling. From the experiments, the author seems to be the first work to successfully apply and analyze continuous-time consistency distillation at 10B scale for both image and video generation.", "weaknesses": "1. Limited benchmark. Through all experiments in the paper, the authors only conduct the experiments on GenEval (for T2I), and VBench (for T2V). Honestly speaking, both of them are too old, and the prompts in both benchmarks are too easy. I highly recommend the authors to include more diverse datasets or prompt types.\n\n2. The comparison baseline only contains DMD2. It could be expanded to include other recent methods, especially for the image generation.\n\n3. The substantial engineering efforts, while necessary for training large scale models, might partially overshadow the novelty of the method for some readers, creating a perception of heavy engineering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lioTjvaqkp", "reviewer": "Reviewer_pJkK", "summary": "This paper investigates the key limitations of continuous-time consistency models (sCM) and presents solutions to improve their performance and scalability. The authors first identify two fundamental issues in sCM: error accumulation during integration and a “mode-covering” tendency caused by its forward-divergence objective. To mitigate these problems, they propose a score-regularized variant, rCM, which integrates score distillation as a long-skip regularizer. This modification complements sCM with the “mode-seeking” property of reverse divergence, leading to enhanced sample quality and diversity.\nIn addition, the paper scales sCM and rCM to large-scale text-to-image and text-to-video diffusion tasks, made possible by custom infrastructure designs leveraging FlashAttention-2, Fully Sharded Data Parallel (FSDP), and Context Parallelism (CP). These optimizations enable efficient training of models with over 10 billion parameters. Extensive experiments on large-scale settings demonstrate that rCM achieves visual fidelity and diversity on par with or surpassing existing distillation methods, while maintaining the efficiency of few-step generation.", "questions": "1.\tCould the authors provide stronger theoretical justification or intuition for why combining forward and reverse divergence objectives improves both quality and diversity? How does this differ from formulations such as Jensen–Shannon divergence or distribution matching distillation (DMD)?\n\n2.\tIn Section 3.3.2, how exactly does the inclusion of reverse divergence alleviate error accumulation when the self-feedback JVP term dominates? Are there ablation or analytical results that support this explanation?\n\n3.\tThe paper claims that a fixed λ = 0.01 generalizes across all models and tasks. Was this empirically validated? If so, could the authors provide supporting evidence or sensitivity analysis?\n\n4.\tHow challenging is hyperparameter tuning in practice for rCM compared to baseline sCM and DMD? The paper presents conflicting statements—could the authors clarify this discrepancy?\n\n5.\tWill the authors release training scripts or infrastructure details for reproducing large-scale runs (e.g., 14B models), given the reliance on system-level optimizations like FSDP and Context Parallelism?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Summary: The paper tackles an important problem in the current diffusion distillation landscape by scaling continuous-time consistency models (sCM) to practical, large-scale visual generation tasks, including text-to-image and text-to-video synthesis.\n\n2. Experimental Evaluation: The experiments are extensive and well-executed, covering both GenEval and VBench, with comprehensive comparisons against strong recent baselines for image and video generation.\n\n3. Results: Both qualitative and quantitative results show that the proposed rCM effectively distills large diffusion models, achieving performance comparable to teacher models while being substantially faster. The method also demonstrates strong scalability, successfully applied to models with up to 14B parameters and producing competitive few-step generations.", "weaknesses": "1. Limited theoretical novelty: The paper reads more like a comprehensive technical report than a scientific contribution offering new theoretical insights. While the engineering efforts to scale continuous-time consistency models—such as infrastructure designs leveraging FlashAttention, FSDP, and Context Parallelism—are impressive, these are primarily system-level and implementation-oriented rather than conceptual innovations.\n\n2. Unclear theoretical justification for score regularization: The explanation of how incorporating reverse divergence via the DMD objective mitigates accumulated error in sCM remains insufficient. Without stronger theoretical grounding or ablation evidence, the justification for the proposed regularization appears weak.\n\n3. Inconsistent claims about tuning difficulty: The statement that sCM requires “tedious and impractical tuning” (lines 182–183) contradicts the claim in Figure 1 that CMs are “easy to tune.” These conflicting points reduce clarity.\n\n4. Presentation and visualization issues: In Figure 2, the teacher–student comparison is ambiguous. Annotating which samples correspond to teacher and student outputs, and including teacher references for text-to-video examples, would improve clarity.\n\t\n5. Limited evaluation scope: The generality of rCM is not well established, as experiments are limited to Cosmos-Predict2 and Wan2.1. A broader evaluation across other architectures (e.g., SANA, Flux) would strengthen the claims of generality.\n\n6. Clarity of algorithmic presentation: The main paper should include the key algorithms, clearly indicating which parts are inherited from sCM and DMD, and which represent new contributions. This would make the methodological novelty more transparent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)