CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow-Map Models
Authors: Zheyuan Hu
Chieh-Hsin Lai
Yuki Mitsufuji
Stefano Ermon
Venue: ICLR 2026 Poster (2026)
Keywords: Flow Map Models
Consistency Models
Mean Flow
Mid-Training
Diffusion Model
Generative Models
Primary area: generative models
TL;DR: We introduce Consistency Mid-Training (CMT), a lightweight prior stage that stabilizes flow-map training, reduces training cost by up to 98%, and achieves state-of-the-art FIDs in few-step generation.

Abstract: Flow map models such as Consistency Models (CM) and Mean Flow (MF) enable few-step generation by learning the long jump of the ODE solution of diffusion models, yet training remains unstable, sensitive to hyperparameters, and costly. Initializing from a pre-trained diffusion model helps, but still requires converting infinitesimal steps into a long-jump map, leaving instability unresolved. We introduce *mid-training*, the first concept and practical method that inserts a lightweight intermediate stage between the (diffusion) pre-training and the final flow map training (i.e., post-training) for vision generation. Concretely, *Consistency Mid-Training* (CMT) is a compact and principled stage that trains a model to map points along a solver trajectory from a pre-trained model, starting from a prior sample, directly to the solver-generated clean sample. It yields a trajectory-consistent and stable initialization. This initializer outperforms random and diffusion-based baselines and enables fast, robust convergence without heuristics. Initializing post-training with CMT weights further simplifies flow map learning.  Empirically, CMT achieves state-of-the-art two-step FIDs of 1.97 (CIFAR-10), 1.32 (ImageNet $64\times64$), and 1.84 (ImageNet $512\times512$), using up to $98$\% less training data and GPU time than CMs. On ImageNet $256\times256$, it attains 1-step FID 3.34 with $\sim50$\% less training than MF from scratch (FID 3.43). On MSCOCO T2I, CMT reaches the best FID with $\sim47$\% less training. This establishes CMT as a principled, efficient, and general framework for training flow map models. Code and models are available at https://github.com/sony/cmt.

Published as a conference paper at ICLR 2026
CMT: MID-TRAINING FOR EFFICIENT LEARNING OF CONSISTENCY, MEAN FLOW, AND FLOW MAP MODELS
Zheyuan Hu
Chich-Hsin Lai
Yuki Mitsufuji'a
Stefano Ermon
'Sony AI Sony Group Corporation
Stanford University
*Equal Contribution
zyhu2001 gmmil..amommmmmmmmmmmmmmmm com
chieh-hsin. l @sony com
ABSTRACT
Flow map models such Consistency Models (CM) and Mean Flow (MF) enable few-step generation by learning the long jump of the ODE solution of diffusion models, yet training remains unstable, sensitive hyperparameters, and costly. Initializing from a pre-trained diffusion model helps, but still requires converting infinitesimal steps into long-jump map, leaving instability unresolved. We intro- duce mid-training, the first concept and practical method that inserts a lightweight intermediate stage between the (diffusion) pre-training and the final flow map train- ing (i.e., post-training) for vision generation. Concretely, Consistency Mid- Training (CMT) is compact and principled stage that trains a model map points along a solver trajectory from pre-trained model, starting from prior sample, directly to the solver-generated clean sample. It yields a trajectory-consist
and stable ini- tialization. This initializer outperforms random and diffusion beasdddoo baselines and enables fast, robust convergence without hcuristics. Initializing post-training with CMT weights further simplifies flow map learning. Empirically, CMT achieves state-of-the-an two-step FIDs of 1.97 (CIFAR-10), 1.32 (ImageNet 64 x 64), and 1.84 (emageNet 512 512), using up 98% less training data and GPU time than CMs. On ImageNet 256 x 256, it attains 1-step FID 3.34 with ~500 less training than MF from scratch (FID 3.43). On MSCOCO T2I, CMT reaches the best FID with ~47% less training. This establishes CMT as a principled. efficient, and general framework for training flow map models. Code and models are available https://githuh.com//smm
FID versus Training GPU Hours
INTRODUCTION
Vanilla ECD
100
CMT w/ ECD (Mid)
CMT ECD (Post)
Diffusion models (Ho et al., 2020;
Song & Ermon, 2019) have become a cornerstone of modern generative modeling, yet their practical applica- tion is often hindered by significant computational burden during infer- ence. This latency arises because sam- pling is equivalent to solving a prob- ability flow ordinary differential equa- tion (PF-ODE) (Song et al., 2021), process that requires many iterative steps. To cireumvent this limitation, promising direction focuses di- rectly learning the solution (integra- tion) map of the PF-ODE, which is also referred to as a flow map model.
2-Step FID
31.98
10
1.84
0
100
200
300
400
H100 GPU Hours
Figure 1: FID training time for vanilla ECD (Geng et al., 2025b) and CMT (ours) on ImageNet 512x512. With the pro- posed mid-training. our CMT w/ ECD (as post-trained flow map) achieves SOTA two step FID of 1.84 using only 400 H100 GPU hours (mid- and post-training combined). Under the same budget, vanilla ECD still produces unrecognizable images, and even to reach a reasonable two step FID of 3.38 it requires 4643,90 99 hours. Overall, CMT reduces the total training cost of flow map models by 91.4% while achieving SOTA performance.
Because the PF-ODE flow map lacks a closed form, recent methods learn surrogate maps by enforcing proper-

---

Published as a conference paper at ICLR 2026
ties that any exact flow must satisfy: e.g., Consistency Models (CM) (Song et al., 2023) impose cross sooss-noise-level celf-cooni and Mean Flow (MF) (Geng et al., 2025a) matches time averages along trajectories. However, these objectives (Song et al., 2023; Song & Dhariwal, 2024; Kim et al., 2024; Geng et al., 2025a; Lu & Song, 2025; Sabour et al., 2025) supervise against stop- -raddgddi,nn,, network-dependent pseudo-targets that drift with training dynamics. The lack of true, time-invariant regression target injects bias, yields unstable optimization signals, and slows convergence. While re- cent works observed that initializing from pre-trained diffusion weights can mitigate instability (Geng et al. 2025b; Lu & Song, 2025), this does not address the root cause. Fundamentally, flow map must learn large integrated jumps of the trajectory, whereas diffusion models capture only the infinitesimal movements. This mismatch renders diffusion- depends on brittle heuristies (e.g., time weightings and sampling schedules) yet still suffers from flow map learning's instability and converges slowly (Geng et al., 2025b). In particular, recent studies (Zhu, 2025) have observed that post-training MF, even when initialized from a well-trained large-scale diffusion model (Ma et al., 2024), is prone to divergence and requires careful configuration tuning.
initialization fragile: flow map training then
We address the instability and high cost of training few-step flow maps by introducing mid-training for vision generation, conceptually inspired by mid-training in large language models (Groeneveld et al., 2024). In our setting, maid-tranning is a brief intermediate stage that bridges pre-training (e.g.. diffusion model) and flow map post-training. We instantiate this idea as Consistency Mid-Truining (CMT), a lightweight procedure that leverages trajectories generated by a pre-trained model to produce trajectory-aware initialization. Concretely, CMT trains a model to map any point along a trajectory determined by a pre- pre-trained model, from a prior sample directly to the clean endpoint of exactly that same trajectory in single step. Mid-training with CMT requires no architectural changes, converges quickly, adds only modest cost, and avoids fragile heuristics such as stop gradients, time sampling, and weighting schedules. This trajectory-allge alligned initializer provides a better starting point for flow map post-training than either random or diffusion-basee weight transfer, while also simplifying engineering practices. Most importantly, it significantly reduces the total training cost (in both time and required training data) and improves training stability.
Theoretically, we show that CMT reduce the gradient discrepancy between the oracle and practical flow map losses, providing a stronger and trajectory initializer for the flow map post-training. Empirically, we evaluate on pixel-space benchmarks (CIFAR- 10, FFHQ 64 x 64, AFHQv2 64 x 64, ImageNet 64 x 64) and latent-space high-resolution models (ImageNet 256 x 256/512 x 512) as well as MSCOCO T2I (Lin et al., 2014). Initializing flow map models with CMT improves post-training stability, speeds up convergence, and boosts final quality. CMT achieves new SOTA two-step FIDs of 1.97 (CIFAR-10) 10). 1 1.32 (ImageNet 64 x 64), 1.84 (ImageNet 512 x 512), 2.34 (AFHQv2 64 x 64), and 2.75 (FFHQ 64 64), while redueing training budget (images processed, equivalently backprop steps) and GPU time by up to 98% versus baselines without mid-training. On ImageNet 512 x 512. CMT reaches FID 1.84 with 91.4% less training than the baseline (FID 3.38; Figure 1); on ImageNet 256 x 256, it attains FID 3.34 with 500 less training than MF from scratch (FID 3.43). On MSCOCO T2I, CMT achieves the best FID with ~47% less training.
CMT applies to both CM and MF, demonstrating broad applicability across ODE-based flow map generators. To our knowledge, this work presents the first systematic investigation of mid-training for few-step flow map models in vision generation, establishing CMT as effective approach that significantly reduces training cost while achieving state-of-the-arr quality.
2 PRELIMINARIES AND RELATED WORK
2.1 DIFFUSION MODELS AND FL.WW MATCHING
Diffusion models define forward process that perturbs clean data Xo Pdata into X, 02 + o,E, where e ~ (O, 1) and & E [0, Equivalently, X; Pr (x) = [- a,x0, I)) which induces marginals px(x:) = pr (xo) (xx) Two closely related training approaches are standard.
EDM (Karras et al., 2022) trains a denoiser Dø (x 1) with a preconditioned parametrization by minimizing LDM (0) = Ex,eel ((()) [[[[t)][oxxx,p t)- xul xol At optimum, D(x,,t) = [[x,|x,]] EDM uses 0; = a. for / [0, 7), for large T, the prior Pprior approaches (0, T
2

---

Published as a conference paper at ICLR 2026
Flow Matching (Lipman et al., 2023) fits vector field vo(x,, t) to the conditional velocity of the perturbation: (0) = E, Exo.e wtt) [ t) (a/x0 + o/c) [ At optimum, v(x,, t) E|a/xo + o xr]. A common choice a = 1 - i, or = for 1 E [0, 1] yields a unit-Gaussian prior. Their Relationship. The parametrizations are equivalent; the marginal optimal velocity and denoiser satisfy v(x,t) t) = (<4 - at %f D(x,,t) t) + di Xt, so one can translate between and Do given the de
scheduler. Sampling integrates the PF-ODE, dx. v(x;, (), starting from XT Pprior (Gaussian in both views) down to / = O. Either Vo & v or Do R D can be used to realize the drift.
2.2 FEW-STEP FLOW MAP GENERATIVE MODELING
In this section, we propose a unified view that connects existing formulations of flow map models. Numerical integration of the PF-ODE can be slow, as it requires simulating a system across many small time steps. Ftw models offer a more efficient alternative by directly learning the solution to the PF-ODE's integral: the flow map. (.). This map takes an initial state x at time / and jumps directly to its final destination at time 8:
L (x+o(x))== == Xx + [." v(xu, u)du, v(Xu: u) = (a'--
- au de D(xu: u) + on
L
Xu.
(1)
Special Flow Map: Consistency Models (CM). The CM family adapts EDM's framework and learn a few-step denoiser fo(-, t) that approximates the flow map to the origin, V (-),(-), for any E (0. 77. Training relies on the consistency property: any two points along the same PF-ODE trajectory should map to the same origin. We propose a principled re-interpretution of the CM family objective (Song et al., 2023; Song & Dhariwal, 2024; Geng et al., 2025b; Lu & Song, 2025):
Loracte ( () = Ex,Ex ())(((((() 1), V (xx))
(2)
with d a point-wise distance (e.g.. squared &g or perceptual (Zhang et al., 2018)). At optimum, f(x,,t) = (x,) (Proposition F.1). Since i is unavailable, CM uses a stop- aradierd surrogate from the previous step: (X) = (XI-Xi t At) with At > 0, where x;-At comes from (i) Consistency Distillation (CD); a one-step solver with a pre-trained diffusion teacher, which calls the teacher during training; or (ii) Consistency Training (CT): the analytic estimate Xt-At = Qt + Ot-AiE using the same (Xo, E) as in X, = ayXo + TjE, requiring no teacher calls. Both approaches improve performance by initializing from pre-trained diffusion weights (Lu & Song, 2025; Geng et al., 2025b). In the CT setting, the CM surrogate loss is
(C == xx,,, [w(t)d (fo (x,, t), fo (x- t At))]
(3)
Recent variants (c.g., ECT (Geng et al., 2025b)) refine initialization, time steps, w(t), and d(s-).
General Flow Map. Consistency Trajectory Model (CTM) (Kim et al., 2024) was the first to learn the general flow map i for arbitrary t > s via Go (xe, f, s), minimizing
Corucle-CTMll (0) ( == E.,. pop ( (x)) (Go( s). D (x))]
(4)
As V V,-s is inaccessible, CTM uses stop- stop-gradient target evaluated at Go itself, similar to CM.
More recently, MF (Geng et al., 2025a) builds on the flow matching formulation by modeling the
average drift (x,, t, s) = h(x,, t, s) := - J v(x,,, u) du over an interval [s, t), also following the principled Equation (4). MF constructs a surrogate target by differentiating (t 5s=-s))e s)h(x,,,, t,) Jr v(xu, u) du w.r.t. t, yielding the MF training loss:
EMP () = Ex (w((() (xe, h1 (xr, L, >|
(5)
where the regression target is applied with stop-gradient as h set (x,, 8) = (xx - (1 (v((x,))) t) + 0g In practice, the oracle v(x,,t) t) is approximated either by (i) a pre- trained diffusion model (distillation), or (ii) the conditional velocity d(xu + ofe (training from scratch). CTM and MF share the same framework with equivalent losses up to constant (see Appendix A and Equation (11)), differing only in parameterization and backbone (CTM with EDM, MF with flow matching). We therefore use MF as the representative flow map model V 1--5-
3

---

Published as a conference paper at ICLR 2026
2.3 RELATED WORK
Diffusion Models and ODE Samplers. The PF-ODE enables deterministic ODE sampling (Song ct al., 2021; Lipman ct al., 2023; Liu et al., 2023; Karras et al., 2022). DDIM is a first-order (Euler- type) PF-ODE discretization that reduces steps without retraining (Song et al., 2020). DPM-Solver methods use a log-SNR parameterization with 2nd/3rd-order exponential integrators to solve the PF-ODE in very few NFEs (Lu et al., 2022; 2025), and parallel schemes can further optimize the discretization grid (Sabour et al., 2024; Nguyen et al., 2024).
Few-Step Generative Models. Because stepwise ODE integration is slow, distillation approaches train a student to imitate a multi-step teacher via long jumps (Salimans & Ho, 2022; Luhman & Luhman, 2021; Zheng et al., 2023). CM instead learn direct flow map via consistency property; subsequent work improves training and extends to continuous time (Song et al., 2023; Song & Dhariwal, 2024; Geng al., 2025b; Lu & Song, 2025). CTM learn maps between arbitrary times but rely on adversarial losses (Kim et al., 2024; Lai et al., 2023). Later work removes the adversarial component with new parameterizations and losses while improving few-step fidelity (Frans et al., 2025; Boffi et al., 2024; Sabour et al., 2025). MF (Geng et al., 2025a) trains from scratch without adversarial loss, at the cost of expensive Jacobian-vector products. Despite recent advances (Lai et al., 2025), training flow map models remains computationally expensive and often unstable.
3 CONSISTENCY MID-TRAINING FOR FAST, GENERAL FLOW MAP LEARNING
3.1 PROPOSED PIPELINE FOR FLOW MAP LEARNING
Despite recent advances, large-scale flow map training remains costly, unstable, and configuration- sensitive. The key challenge is the lack of an oracle regression target L r--s: current methods rely on stop-gradients of imperfect models, yielding poor supervision and large deviations from the true flow. To address this, we introduce compact mid-training stage between pre- pre-training and flow map post-training. Specifically, our pipeline incorporates the proposed CMT as mid-training step, providing general and cost-efficient framework for flow map learning:
Stage 1: Pre-Training. It aims to obtain deterministic ODE sampler that transports samples from Ppeior to Pdata. consistent with the marginals of the forward noising process W(; IT) A practical choice is an off-the-shelf pre-trained diffusion model with its PF-ODE solver, as many such models are available (Peebles & Xie, 2023; Karras et al., 2024b; Ma al., 2024). Alternatively, one may use lightweight few-step flow map model supporting deterministic sampling (e.g., MF). We call these variants collectively the teacher sampler.
Stage 2: Mid-Traininn (CMT). Efficiently learn a lightweight, trajectory- proxy of the target flow map with minimal computation and stable convergence, without ad-hoc heuristics. CMT's loss is designed to match the objectives of post-training while using fixed, explicit regression targets supplied by the teacher (see Equation (7) or Equation (8) below). Operationally, CMT learns to jump directly between points on the teacher-generated trajectory of a pre-trained model. Because the targets are fixed and high quality, CMT trains stably and yields a trajectory-aligned initializer.
Stage 3: Post- post-Training. Learn the final few-step flow-map model. Compared to random initialization or initialization from pre-trained diffusion models proposed by literature (Geng et al., 2025b; Lu & Song, 2025), the CMT initializer is trajectory-aligned making post-training more stable, simpler, and faster (as supported by our theoretical analysis in Theorem 5.1 and Appendix F). CMT offers general recipe for significantly cost-efficient flow map learning.
In what follows, we detail the mid-training stage with CMT, first instantiating it for CM (L, and then extending it to the general flow map via MF ( (z)),, with pseudocode Algorithm 1.
3.2 CMT FOR LEARNING CONSISTENCY FUNCTION
Here, we focus CM as the flow map post-training stage. To obtain trajectory aligned initializer for this flow map and to motivate the design of the CMT's mid-training loss, we revisit the CM oracle objective Coracte-CM-
4

---

Published as a conference paper at ICLR 2026
We first propose reinterpretation of CoracleCCM from reverse time generative perspective, under which the objective becomes transparent. Every point x; ~ pr along a PF-ODE trajectory is uniquely determined by its terminal state Xy. Hence, trace its entire trajectory backward. Training then reduces to mapping every point on this reverse path to its single consistent origin in the data distribution Pdata- This yields the following equivalent formulation of the oracle loss; the proof is provided in Appendix F.I.
may sample single terminal state XT Pprior and
Theorem 3.1. If Pprior matches the diffused marginal pr' the oracle loss can be expressed as
Coracle-CM (0) E2 ExT Ppecio (t)d (vr-((x)):) t), (r--((xx)]
(6)
Building on the reverse-time time formulation in Equation (6), the proposed CMT. On the interval [0, T]. we fix a time grid 0 = to < t <... < tu = T with M obtain discrete reference trajectory {xe, ) MM
introduce the training objective of
discretization steps. Given a sample Xy Ppriors by running a numerical ODE solver with the pre-trained diffusion model D (in EDM formulation) as the teacher sampler, anchored XLN XT. The goal of CMT is for fo match any intermediate state & back to its clean origin Etg- Training proceeds by minimizing the following loss:
LCMT-CM = B,E E,Exr PPocio [dfo (Kt: x10)
This objective is discrete approximation of the oracle loss Loracle-CM. since the solver-generated points approximate the true flow map, i.e., &t, ~ MT (XT).
Since the starting states XT Pprior are randomly sampled, the set of possible trajectories can be arbitrarily many. Yet, once a particular Xy is fixed, the corresponding trajectory is uniquely determined. In principle, CMT can thus be trained with arbitrarily many distinct trajectories, avoiding the overfitting issues that arise in standard supervised tasks.
3.3 CMT FOR LEARNING GENERAL FLOW MAP
We now focus on the MF parameterization for the general flow map learning. MF aims to learn the average drift, defined as h(x,, t, s) := t-s fe v(xu, u) du, which aggregates the ODE velocity over the interval [s. 4]. We observe that this quantity can also be expressed through the flow map. Let denote the initial state at time on the same PF-ODE trajectory as X,. Then
1
1 h(x,,t,)) = - Sf." v(xu, u) du S. v(Xa, u) du)
1
@ (TT=((XT) (xT))
Motivated by this decomposition. CMT allows to construct teacher-reference trajectory {x,, from a prior sample Xy ~ Pprior using two possible teacher samplers. The first employs a numerical ODE solver applied to the PF-ODE of pre-trained flow matching model va- Alternatively, since MF supports deterministic sampling, we may use smaller and lightweight MF model to perform multi-step deterministic generation. Although not optimal, this model is much easier train and still yields a valid teacher trajectory. In both cases, the resulting trajectory provides a feasible approximation of the oracle states, Pr-,, = Xt,
The CMT loss for MF then encourages the average drift parametrization he to align with the finite differences between successive reference states:
LCMT MP (0) = Eizj Exy Pprior [X, fz, (z)
(8)
Crucially, in both Equation (7) and Equation (8), our formulation reduces training to standard - regression problem with a fixed target, either Xe or 6- The CMT loss for MP generalizes the CM case. In fact, if we fix 1, 0 in LCMT the loss reduces to learning mapping from every point on the trajectory directly to the clean data, thereby recovering the CM formulation.
'This holds for sufficiently large T as in FDM, or with appropriate noise schedules as in flow matching. Empirically, pr (data-dependent) and Porior (data-free) perform identically, we adopt Pocior in all experiments.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETUPS
Table 1: Sample quality unconditional CIFAR-10 32x32 and class-conditional ImageNet 64x64.
Unconditional CIFAR-IO 32x32
Class-Conditioal ImageNet 64x64
METHOD
NEE (L)
FID (L)
METHOD
NFE (1)
FID (1)
Diffusion Models
Diffusion Models Auto-Guidance)
EDM (Karras et al., 2022)
35
2.01
RIN (Jabri et al., 2023)
1000
1.23
EDM2 (Karras et al., 2024b)
63
1.33
Joint Training
HDM2" (Karras et al., 2024a)
63
1.01
CTM (Kim al., 2024)
1.87
Joint Training
DMD (Yin al., 2024b)
3.77
SiD (Zhou et al., 2024)
1.92
DMD2 (Yin et al., 2024a)
1.28
SiD (Zhou et al., 2024)
1.52
Diffusion Distillation
CTM (Kim et al., 2024)
1/2
1.92 1.73
DENO (Zheng et al., 2023)
3.78
Auto-Guidance Diffusion Distillation
2-Rectified Flow (Liu et al., 2023)
4.85
TRACT (Bertelot et al., 2023)
1/2
3.78 3.32
AYF (Sabour et al., 2025)
1/2
2.98/
1,25
PD (Salimans & Ho, 2022)
1/2
8.34/ 5.55
ECD (Geng et al., 2025b)
1/2
2.24 / 1,50
CMT (w/ ECD) (Ours)
1/2
1.78 1.32
Flow Map Models
Flow Map Models
CD (Song al., 2023)
1/2
3.55 3.55/2.93
iCT (Song & Dhariwal, 2024)
1/2
2.83
CD (Song al., 2023)
6.20
iCT-deep (Song & Dhariwal, 2024)
1 1/2
2.51
iCT (Song & Dhariwal, 2024)
1/2
4.02/3.20
ECT (Geng et al., 2025b)
3.60
iCT-deep (Song Dhariwal, 2024)
1/2
sCT (Lu Song, 2025)
1/2
2.85
ECT (Geng et al., 2025b)
1/2
2.49 1.67
sCD (Lu & Song, 2025)
1/2
3.66
SCT (Lu Song, 2025)
1/2
2.04/
Stable CT (Wang ct al., 2025)
1/2
2.92
sCD (Lu & Song, 2025)
1/2
2.44/ 1.66
VCT (Silvestri et al., 2025)
1/2
3.26
MultiStep-Or CiDdstep-DDDDt (Heek et al., 2024)
1/2
1.90
TCM (Lee et al., 2025)
1/2
2.46 /2055
Stable CT (Wang et al., 2025)
1/2
2
IMM (Zhou et al., 2025)
1/2
3.20/1.98
VCT (Silvestri ct al., 2025)
1/2
4.93/3.07
MF (Geng ct al., 2025b)
2.92
TCM (Lee et al., 2025)
1/2
2.20 1.20
CMT (w/ ECT) (Ours)
1/2
2.74/1.97
CMT (w/ ECT) (Ours)
1/2
2.02/
Datasets & Setup. We evaluate on CIFARIO at 32x32 (Krizhevsky et al., 2009), AFHQQ2 at 64x64, 64, FFHQ at x4464 (Karras et al., 2022), and ImageNet (Deng et al., 2009) at 64 256x256, and 512x512. The low-resolution unconditional datasets (CIFAR10, AFHQv2, FFHQ) follow EDM/ECT//CT protocols (Karras et al., 2022: Geng et al., 2025b; Silvestri et al., 2025). For ImageNet 64 x4 and 512x we adopt EDM2 (Karras et al., 2024b), training the 512x512 case in the latent space of Stable Diffusion (SD) autoencoders. For ImageNet 256x256, we follow MF and SiT (Geng et al., 2025a; Ma et al., 2024), also in the SD latent space. Detailed experimental setup is provided in Appendix B.
Teachers and Solvers for CMT's Mid-Training. Across datasets, CMT employs different teacher-solver pairs based on availability: EDM w/ DPM-Solver++ (Lu et al., 2022; 2025) on (CIFARIO, AFHQv2, FFHQ); EDM2 w/ DPM-Solver++ on (ImageNet 64x64, 512x512); and MF-B/4 on ImageNet 256 x566 DPM-Solver++ uses 16 solver steps and MF-B/4 uses 8 with fixed discretization. We adopt these solvers with their default log-SNR sampling schedulers due their strong theoretical foundations and keep all other settings as simple as possible. For mid- mid-trainin -raittriiing of EDM/EDM2-rellted settings, we apply learned perceptual loss align CMT's predictions with the teacher's s high-fidelity outputs, specifically using LPIPS (Zhang et al., 2018 8) in pixel space and El.atentI PIPS (Kang et al., 2024) in latent space. We use squared (2 loss for MF. We discuss CMT loss function selection motivation in Appendix D.
Post-Training of Flow Map Model. After mid-training. we respectively train a flow map with: ECT on (CIFAR-10, AFHQy2, FFHQ), ECT/ECD on ImageNet 64 x64, MF ImageNet 256x256, and ECD on ImageNet 512 512 for stability in very high dimensions (Lu & Song, 2025). We remark that ECT and MF serve strong representatives of flow map models: ECT builds on EDM's 's backbone, while MF builds on FM's backbone, both of which are widely used in practice. These post-training methods are chosen due public availability and representativeness respective datasets.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison between various CMs given the identical 51.2 million training images budget on AFHQv2 64x64 and FFHQ 64x64. Our CMT achieve the best 1-step and 2-step FIDs.
Unconditional AFHQv2 64
Unconditional FFHQ 64x64
METHOD
NFE (L)
FID (L)
METHOD
NFE (J)
FID (4)
iCT (Song & Dhariwal, 2024)
1 1/2
5.40 2.92
iCT (Song & Dhariwal, 2024)
1/2
5,80 / 4.02
ECT (Geng et al., 2025b)
1/2
3.89 2
ECT (Geng et al,, 2025h)
1/2
5.99 / 4,39
VCT (Silvestri et al., 2025)
1/2
3.84
VCT (Silvestri al., 2025)
1/2
5.47 /4116
CMT (w/ ECT) (Ours)
1/2
2.34/2334
CMT (w/ ECT) (Ours)
1/2
3,89 / 2.75
Metries. We report FID (Heusel et al., 2017), data cost in millions of images (Mimgs) for data efficiency, and training A100 GPU (80GB) time for convergence speed. Specifically, the data cost is computed via batch size per iteration x total iterations, where each batch is randomly drawn from the entire dataset each iteration, i.e. the data cost equals the number of backpropagated inputs.
4.2 MID-TRAINING WITH CMT IMPROVES FLOW MAP POST- TRAINING
In this section, we benchmark CMT against baselines across datasets. Because ECT and related distillation methods (e.g., the distilled variant of MF) typically start post-training from the weights of pre- pre-trained diffusion model, the most direct and fair evaluation of our mid-training strategy is to compare ECT vs. CMT (w/ ECT) and MF vs. CMT (w/ MF). Beyond these direct comparisons, we also report broader results against other baselines in terms of both FID and training cost.
CIFAR-10 and ImageNet 64x64. Table 1 shows that CMT (w/ ECT) attains SOTA with 2-step FID=1.97 on CIFAR-IO, surpassing the teacher EDM's 2.01 with 35 steps and outperforming prior CMs and flow maps. On ImageNet 64x64, CMT (w/ ECT) achieves the best FIDs among all CMs and flow maps; our 2-step FID=1148 is a satisfactory choice over EDM's 1.33 with 63 NFEs. Further, we follow AYF (Sabour et al., 2025) to distill a strong EDM2 with Auto- Guidance to surpass the vanilla flow map model, improving the 1/2-step FID to 1.78/1.32 for CMT (w/ ECD).
For CIFAR-10, the total budget is 51.2 Mimgs (38.4 Mimgs mid-training, 12.8 Mimgs post-training). Under the same 51.2 Mimgs, CMT beats ECT and VCT. sCT uses more budget; under the same $1.2 Mimgs, sCT's 1-step FID is 3.09 vs. our 2.74, demonstrating better data efficiency. Stable CT requires 153.6 Mimgs yet still trails our 51.2 Mimgs results. TCM reaches comparable SOTA but at 332.8 Mimgs. For ImageNet 64x64 CMT (w/ ECT) uses only 12.8 Mimgs (6.4 mid, 6.4 post), whereas ECT and Stable CT use 102.4 Mings, sCT 819.2 Mimgs, and TCM 143.36 Mimgs. Compared to SCT, we save up to 98% training images. Because CMT has lower per-iteration cost than sCT (no expensive JVP), we also cut GPU time by 98% while achieving SOTA. Meanwhile, CMT (w/ ECD) uses only 19.2 Mimgs (6.4 mid, 12.8 post), while vanilla ECD and AYF use 102.4 Mimgs.
Overall, CMT is both SOTA and highly data-efficient, Notably, CMT with ECT/ECD as the post- training flow map consistently outperforms vanilla ECT/ECD under equal lower budgets, under- scoring the importance of our mid-training. Additional CIFAR-I0 10 evidence appears in Section 4.3.
AFHQv2 and FFHQ 64x 64. Table 2 compares CMs under a 51.2 Mimgs budget. Since AFHQv2 and FFHQ are also unconditional, we directly transfer the CIFAR- 10 hyperparameters. CMT achieves the best 1-step and 2-step FIDs, and with ECT as post-training again outperforms ECT at the same budget, highlightnn both hyperparameter robustness and the critical role of CMT across datasets.
ImageNet 512x512. Table 3 reports results along with training costs (Mimgs) for flow map models and CMT, and their comparison with diffusion models. For post-training, we use ECD, making CMT directly comparable to vanilla ECD. CMT substantially outperforms vanilla ECD, again confirming the critical role of mid-training. Overall, CMT achieves the best 2-step FID=1 .884 and a competitive 1-step FID=3.46 at dramatically 93% lower cost than previous sCD The same advantage holds for GPU time, since sCD requires costly JVP computations per iteration. Random samples generated by the trained CMT (w/ ECD) are shown in Figure 3.
Simplicity and Stability of CMT on CM-family Experiments. In mid-training, CMT learns a flow map proxy via an explicit regression target, avoiding goop-gradients, custom time sampling, and handcrafted weights w(),, leading to stable training. Although reference trajectories require a diffusion ODE solver, few-step (~116 methods such as DPM- APM--Solve+++ suffice, and the multistep scheme reuses past states X1 to build later ones, keeping overhead low. On ImageNet
Remark:
7

---

Published as a conference paper at ICLR 2026
Table 3: Sample quality on condiiiool ImageNet 512x512 diffusion models and flow map models. The cost comparisons for flow map models and CMT are measured under millions of training images (Mimgs).
MЕTHOD
NFE (L) FID
METHOD
NFE (1)
FID (U)
Cost (L)
Diffusion Models (" Autoo-Guidacce)
Flow Map Models
RIN (labri et al., 2023)
1000
3.95
ECT (Geng al., 2025h)
9.98 6.28
204.8
HDM2 (Karras al,, 2024h)
63x2
1.81
ECD (Geng et al., 2025b)
1/2
/.378
409.6
EDM2 (Karras et al., 2024a)
63x2
1.25
sCT & Song, 2025)
4.29
204.8
DiT (Peebles Xic, 2023)
250x2
3.04
sCD (Lu Song, 2025)
1/2
2.28
1.88
409.6
Large-DiT (Zhang al., 2023)
250x2
2.52
AYF (Sabour al., 2025)
1/2
3.32/ 1.87
102.4
SYT (Ma al., 2024)
250x2
2.62
CMT (w/ ECD) (Ours)
1/2
3.38 / 1.84
28.8
Table 4: Comparison between CMT and MF on ImageNet 256
Method
Pre- Pre-Training
Mid-Training
Post-Training
Total Time (J)
FID (J)
MF-XI/2 (Rand. Init.)
0
1520 hours
1520 hours
3.43
MF-XI/2 (SiT Init.)
> 1550 hours
0
357 hours
>1520 hours
4.52
CMT-XL/2
38 hours
135 hours
587 hours
760 hours
3.34
64 x 64 and 512 x 512, this yields clear gains: CMT outperforms sCT/sCD while cutting mid- and post-training data and GPU time by 93%-98% (see Appendices and C).
CMT initialization simplifies post-training of ECT/ECD models on {CIFAR-10. ImageNet 64x64/112x512, AFHQv2, FFHQ), removing ad hoc tricks such as Al annealing. loss reweight-
ing, custom time sampling, EMA variants, and nonlinear learning rate schedules. The resulting
pipeline consistently outperforms ECT/ECD basclines and converges faster with minimal engineering.
versus Training GPU
ImageNet 256x256: CMT Enables Flex- ible Teacher Samplers Beyond Diffusion. We test whether CMT's hii-training can non-diffusion teacher sampler, even its quality is low. We compare perfor- mance on larger MF-XL/2 model under three settings: (1) MF-XL/2 (Rand. Init.): post-train only (vanilla MF with random initialization); (2) MF-XL/2 (SiT Init.): pre-train with SIT (Zhu, 2025) followed by post-training with its weights initial- ization; (3) CMT-XL/2: train a small MF- it in mid-
MP from Seorch
300
CMT (Mid)
CMT MP (Port)
1-Siep
200-
100
710
720
00000000
100
A100 (BOG) GPU Hours
B/4 for quick convergence, training as a teacher sampler to generate ODE trajectories for XI/2 with random initialization, then post-train MF-XL/2 ini- tialized from the mid-trained model (see Appendix B.4).
Figure 2: FID training time for vanilla MF and CMT (ours) on ImageNet 256x256. perform mid-training starting from a randomly-initiatie of XL/2 size Icarns to match the deterministic sampler of a weaker, smaller teacher MF-B/4, The resulting mid-trained weights CMT-XL/2 are then used initialize MF-XL/2 post-training. This initialization produces semantically mean- ingful samples early and drives significantly faster conver- gence. With CMT's pipeline, training reaches lower FID in only half the GPU hours compared to MF trained from scratch. MF initialized from SiT also converges fast, but requires more than 1520 hours of pre- treitraining, which exceeds the cost of training MF itself.
XL/2 model, where CMT
Table 4 reports the pre-training, mid- training, and post-training time together with the final FID. In particular, CMT-XL/2 cuts total training time by 50% compared to the other two settings, while achieving even better FID. Even though MF-B/4 is weak teacher (1 / 2 8-step FID 24.47 14.96 13.44), using it in CMT substantially accelerates MF-XL/2 training. which converges faster and achieves better FID than vanilla MF. By contrast, SiT pre-training at XL/2 scale requires very long training and its weights lead to unstable MF post- training (Zhu, 2025), making it impractical as a pre-training mid- mid-training teacher. We also provide qualitative comparisons in Figure 2. After 20 GPU hours, both MF-XL/2 (Rand. Init.) and MF- MF-X1/2 (SiT Init.) still produce noise, whereas CMT already generates semantically meaningful images.
8

---

Published as a conference paper at ICLR 2026
Table 5: Comparison between MF (initialized with random weights or pretrained SiT) and CMT (MF initialized
with CMT mid- trained weights) on the MS-COCO text-to-image (T2I) dataset.
Method
Pre-Training
Mid-Training
Post- Post-Training
Total Time ())
1/2-Step FID (L)
MF (Rand. Init.)
0
60 hours
60 hours
15.97 / 5.42
MF (SiT Init.)
22 hours
0
38 hours
60 hours
15.55 / 5.26
CMT
22 hours
hours
9 hours
32 hours
15.12/ 5.01
Eventually, CMT attains superior FID with only half the total GPU hours compared to training MF from scratch.
These results show that diffusion initialization alone is insufficient and that mid-training is crucial. Using B/4 for pre-training but XL/2 for mid- mid-training further demonstrates that CMT's mid-training is architecture-agnssii since directly learns a map aligned with the teacher sampler.
MS-COCO Text-to-Image (T2I): CMT Accelerates and Improves T2I Generation. We follow the setup of U-ViT (Bao et al., 2023) and train MF with the MM-DiT architecture (Esser et al., 2024) on MS-COCO (Lin al., 2014) from scratch, and we use the validation set for FID evaluation. We consider three MF configurations: (1) MF (Rand. Init.): MF is trained only in the post-training stage, starting from random initialization; (2) MF (SIT Init.): we first pre-train a SiT model and then post-train MF initialized from the SiT weights; (3) CMT: we pre-train SiT, use it as the teacher sampler in the CMT mid- training stage to generate ODE trajectories for an MF model initialized from SiT, and finally post-train MF starting from this mid-trained initializer. Detailed hyperparameters and training settings are provided Appendix B.6, and largely follow the ImageNet 256 256 configuration, since MS-COCO shares the same resolution.
Table 5 shows that CMT also performs well for T2I, with significantly faster convergence: it achieves the best FID while reducing total training time by about 47%. CMT's 2-step FID (5.01) is also close to the teacher SiT's 50-step FID (4.73). We further note that the relatively large I-step FIDs observed across all variants may be attributable to limitations of the dataset rather than to CMT itself: within fixed backbone, CMT consistently improves performance and convergence speed over alternative initialization schemes. CMT should therefore be viewed as general recipe for achieving faster and more stable convergence, not as modification of the backbone architecture. Together with the ImageNet 256 x 256 results, these experiments demonstrate that both smaller MF and SiT diffusion model can serve as CMT's teacher and yield strong performance even in T2I generation, highlighting the generality and effectiveness of CMT across problem settings and model architectures.
4.3 EMPIRICAL ANALYSIS OF THE PROPOSED
SCHEME
Importance of Mid-Training, We further assess the importance of our mid-training stage on CIFAR- 10 using ECT the flow map model of V - To fairly isolate CMT's mid-training, post-training and vanilla ECT share the same hyperparameters (constant At 1/256, learning rate 10 10-4), with all other settings held fixed. We evaluate:
Vanilla ECT: Under 51. 51.2Mimgs budget, obtain the final 1-step / 2-step FID as 3.54/ / 2.12. CMrr Short mid-training for 1. 1.28Mimgs + long post-training for 49.92. 49.92Mimgs. We obtain
the final 1-step / 2-step FID as 3.42 2.11.
CMPlong; Long mid-training for 25. 25.6Mimgs + short post-training for 25.6Mimgs. We obtain the final 1-step / 2-step FID as 3.30 / 2.04.
These results show that longer mid training outperforms shorter mid-training, which in turn improves over mid-training, further confirming the importance of CMT 's mid-training stage.
Ablation Study on Potential Alternatives for Post-Training Initialization. Although we have demonstrated the efficiency of CMT as mid-training scheme, other variants possible. We focus on learning the flow map L e--u and highlight two straightforward alternatives. Following our mid- training design principles, these variants aim to be stable and easy train (e.g., stop-gradient-free regression targets) while using fewer ad-hoc tricks and hyperparameters:
c(L) (0) E,,, p. [[],o(xe,)) (xxx)))
(2) Var (0) = Ex Ppriir [[lfa (XT, T) -x(

---

Published as a conference paper at ICLR 2026
where Xx denotes the estimate t=0 obtained by running the solver with the pre-trained diffusion model starting from the forward perturrbe sample Xt, and Xu is obtained similarly by running backward from a prior sample Xy. We refer to Caar (0) as Slow CMT: it follows the same principle as CMT but uses only the endpoints of the ODE trajectory. ignoring intermediate states, so generating the same amount of training data requires substantially more ODE-solver inference. The loss L (0) is the standard knowledge distillation (KD) objective of Luhman & Luhman (2021).
We compare CMT's mid-training with KD and Slow CMT on CIFAR-10, training all three under the same settings and then post-training ECT for flow-map learning for 12.8Mimgs with each initialization. Comparing CMT with KD, CMT's 1-step 2-step FID (2.74 / 1.97) is significantly better than KD's (3.54/ / 2.19), confirming the benefit of learning intermediate steps. Comparing CMT with Slow CMT, their FIDs are similar (2.74/ / 1.97 vs. 2.75/ 1.98), but Slow CMT's mid-training costs roughly 3x more GPU time because its regression targets cannot reuse intermediate ODE states.
5 THEORETICAL ANALYSIS
In mid-training, CMT learns a reliable proxy of the flow map. which then serves as a well-aligned initialization for post-training. To quantify this effect, we analyze the CM flow map V :z-0 in the post-training stage; the same reasoning applies to general flow maps V such as MF.
We compare how well the surrogate CM objective tracks an oracle objective under three initialization schemes: CMT mid-training (0CMT), a pre-trained diffusion model (0DM) and random initialization (Orand). For simplicity, we consider the squared distance d(x,, y) ||x y and a uniform weight wi(t) = 1. Let Coradee-CM and LCM denote the oracle and surrogate CM objectives (see Equations (2) and (3)). We define the gradient bias B(0)= == V o CM (0)
(M)
and evaluate ((0) the initial parameters f E {fCMT; 0DM- 0rand). ). A smaller (()) means that SGD steps Lcm closely follow those on the oracle objective. The following informal result summarizes the worst-case bias for each scheme.
Theorem 5.1 (Informal Bias Comparison). Fix a tolerance > 0 and a small time step At. Then:
(i) CMT Initialization. If LCMT-CM(@0MM)) < then B(0cMT) O( + +At)). (ii) Diffusion Initialization. If LDM (ODM) < e, then B(0DM) O (s At + /.,oo//a1)
Bx []] (>)ooxx))))) x x=]lē]]
(ii) Random Initialization. For random weights Grand. B(Orana) = O(1)
This comparison explains why CMT's mid-training provides particularly robust starting point: it already yields good proxy to the oracle flow map, so the CM gradient is nearly unbiased up to O(E At?). In contrast, diffusion- additional bias from the forward noising process and from the mismatch between the PF-ODE solution V (x,) and the posterior mean [[xo] xx), while random initialization can be arbitrarily far from the oracle. We present the complete and rigorous version of this result in Theorem F.I. which also covers initialization from a trained consistency distillation model. In that case, the bias contains additional, uncontrolled discrepancy term, and in practice its training often requires extra hoc stabilization, further limiting robustness compared to CMT. We refer to Appendix F for the full ilas-variane analysis and the resulting excess rrikk guarantees.
initialization (Geng al., 2025b) necessarily incurs
CONCLUSION
We introduced CMT, an efficient mid- mid-training stage that learns seojectory-consistter initialization for flow map models from teacher sampler trajectories. This simple, architecture-agnosii step stabilizes optimization, removes reliance on stop-gradient targets and ad hoc time weighting, and accelerates convergence. With CMT as initialization, flow map models such as Consistency Models and Mean Flow attain SOTA two-step FIDs across pixel and latent benchmarks while reducing training data budget and GPU time by up to 98%. The approach makes training of flow map models more efficient and practical, and in principle, applies to a broad class of ODE-based generative models.
10

---

Published as a conference paper at ICLR 2026
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
This work made use of large language models to assist with proofreading and improving the clarity of the writing. All research ideas, theoretical development, experiments, and coding were carried out solely by the authors.
STATEMENT ON REPRODUCIBILITY
We provide all necessary code and models to reproduce CMT's results on CIFAR-10, AFHQv2, and FFHQ. together with the corresponding checkpoints, the http://giibub.com//nyy/ (also included in the abstract). Comprehensive details of our experimental configurations are further provided in Appendix B to ensure faithful reproducibility.
STATEMENT ON ETHICS
As with other generative models, CMT can inadvertently produce harmful inappropriate content (e.g., violent, deepfakes, or derogatory/NSFW materials). These risks are mitigated by enforcing comprehensive safety policies with automated content screening and moderation pipelines that prevent such outputs.
ACKNOWLEDGMENT
We are grateful to our colleagues Christian Simon, Murata Naoki, Yuhta Takida, Nguyen Bac, and Uesaka Toshimitsu from Sony for helpful feedback.

---

Published as a conference paper at ICLR 2026
REFERENCES
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A ViT backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023.
David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbot, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.042244 2023.
Nicholas M Boffi, Michael S Albergo, and Eric Vanden- Mijnenn. Flow map matching. arXiv preprint 20t.:20060075077, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248- 255. IEEE, 2009.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-furst international conference machine learning, 2024.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In International Conference on Learning Representations. 2025. URL https:// openreview net
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13444 2025a.
Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. In International Conference on Learning Representations, 2025b. URL https: / //ooenrevieww.o.m nnt/ http:/ooorrmmmddx/eeeeeeee IdSID.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 15789-15809, 2024.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint (XXi:2403306800 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Allan Jabri, David J Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In International Conference on Machine Learning, pp- 14569,11559...1 PMLR, 2023.
Minguk Kang, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shecht- man, Jun-Ye Yan Zhu, and Taesung Park. Distilling diffusion models into conditional GANs. In European Conference on Computer Vision, pp. 428-447. Sprinner, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 0-226655-26777 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37:52996-533 53001, 2024a.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEÉ/CVF Conference Computer Vision and Pattern Recognition, pp. 24174-24188 2024b.
12

---

Published as a conference paper at ICLR 2026
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probabil- ity flow ODE trajectory of diffusion. In International Conference Learning Representations. 2024. URL https:://///. : openrevi ew net/ f
feDTD.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Naoki Murata, Yuki Mitsufuji. and Stefano Ermon. On the equivalence of consistency-type models: Consistency models, consistent diffusion models, and Pokker-Plannee regularization. In ICML 2023 Workshop Structured Probabilistic Inference &
Generative Modeling, 2023. URL https::/pt..o wowpnoeeiivieeee www.wwwwww....... net/ rumm? i d=wjtGsScvAO
Chich-Hsia Lai, Yang Song, Dongjun Kim, Yuki Mitsufuji. and Stefano Ermon. The principles of diffusion models. arXiv preprint 10.2:::255022/899 2025.
Sangyun Lec, Yilun Xu, Tomas Geffner, Giulia Fanti, Karsten Kreis, Arash Vahdat, and Weili Nie. Truncated consistency models. In International Conference on Learning Representations, 2025.
URL wwtpt:tpp///to /ttp:::://////////////ee/ net. forum? dd=ZEDE.EVEY...
Tsung-Wi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Repre- sentations, 2023. URL https:t/ /ttp://////eeeeeeeeeeeeeeeeeeeee net/
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Confer- ence on Learning Representations, 2020. URL https:: http::/:////peeeeeew net/ forum? id= rkg22aEKDr.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference Learning Representations, 2023.
URL httptttt///// http//ppeeeveiieeeeeeeeeeeeeeeee net / forum? id=XVjTTInwS
Cheng Lu and Yang Song.
Simplifying, stabilizing and scaling continuous-time consistency models. In International Conference on Learning Representations, 2025. URL httpp::/
net met/forumi forum? Ly 11ugy.. Jugy
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pp. 1-22, 2025.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2 1::::201100000 102388 2021.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Saining Xie. SiT: Exploring flow and diffusion based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024.
Bao Nguyen, Binh Nguyen, and Viet Anh Nguyen. Bellman optimal stepsize straightening of flow- matching models. In The Tweifth International Conference on Learning Representations, 2024. URL https: httptttt////ppeeeee/// http////p/e/eeeeeeeeeeeeeee net forum? yvvye2
13

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, segh-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4206 2023.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the s1st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42947-42975, PMLR, 21-27 Jul 2024. URL https: https////occeeiieeeeeeeeoe .mm................mm.m...m..... mmr.. (ress/v235/salee
html.
Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your flow: Sealing continuous-time flow map distillation. arXiv preprint arXiv:2506.14603 2025.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL /tt.ttts////peeeiiiiie
net/ / Eorum? Pawammiid-Tiiie IddIIdIIIgzhhI..
Zekun Shi, Zheyuan Hu, Min Lin, and Kenji Kawaguchi. Stochastic Taylor derivative estimator: Effi- cient amortization for arbitrary differential operators. Advances in Neurul Information Processing Systems, 97:122316-122333 2024.
Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36:4263-4276, 2023.
Gianluigi Silvestri, Luca Ambrogioni, Chich-Hsin Lai, Yuhta Takida, and Yuki Mitsufuji. VCT:
Training consistency models with variational noise coupling. In Forty-second International Conference on Machine Learning, 2025. URL https: /ttp:///pppeeeeieeeeee net forum? id= CMoX0BEsDs.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna-
tional Conference on Learning Representations, 2020.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In
The Twelfth International Conference on Learning Representations, 2024. URL https: / //ppnrevieww... net forum? wiam:ddd-NNyybDDoe
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.
Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021. URL
1d=FxTIG12RRHS,
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. International Conference Machine Learning, pp. 3221 -22522 PMLR, 2023.
Yu-Yun Wang, Zhengyang Geng, and Hongsheng Li. Stable consistency tuning: Understanding and improving consistency models. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025. URL https: opwnreview net corum?
id=5RoPe2Shxx.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. Advances in Neural Information Processing Systemx, 17:47455-47493 2024a.
14

---

Published as a conference paper at ICLR 2026
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6613-6623, June 2024b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In The Thirteenth International Conference on Learning Representations, 2025.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Qiao. LLaMtA-dapter:: Efficient Finetuning of Language Models with Zero-init Attention. arXiv preprint arXiv: 2.33:23303.16199 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 586-595, 2018.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pp. 42390-42402. PMLR, 2023.
Linqi Zhou, Stefano Ermon, and Jiaming Song. Inductive moment matching. International Conference Machine Learning, 2025. URL
net wtf.moormmi/ iid=
pwNSUo7yUb.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang, Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, pp.
PMLR, 2024.
Yu Zhu. MeanFlow: PyTorrch Implementation. https: / /itttbb.b.... thub. com/ //wzzuuuuueere cs/ MeanFlow, 2025. PyTorch implementation of Mean Flows for One-step Generative Model- ing.
15

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS
Introduction
2
Preliminaries and Related Work
2.1
Diffusion Models and Flow Matching
2.2
Few-Step Flow Map Generative Modeling
2.3
Related Work
3
Consistency Mid-Training for Fast, General Flow Map Learning
3.1 Proposed Pipeline for Flow Map Learning
3.2 CMT for Learning Consistency Function
3.3 CMT for Learning General Flow Map
4
Experimental Results
4.1
Experimental Setups
4.2
Mid-Training with CMT Improves Flow Map Post-Training
4.3
Empirical Analysis of the Proposed Mid- Hid-Training Scheme
5
Theoretical Analysis
10
6
Conclusion
10
A Relationship of Flow Map Models
17
Experimental Details
19
B.1 Summary of the CMT Pipeline and Algorithms
19
B.2 CIFAR-10, 10, AFHQv2, and FFHQ
20
B.3
ImageNet 64x64
20
B.3.1
Experimental Details
20
B.3.2
Cost Details
20
B.4
ImageNet 256x256
21
B.5
ImageNet 512x512.
22
B.5.1
Experimental Details
22
B.5.2
Cost Details..
22
B.6
MS-COCO Text-to-Image (T2I)
23
Training Speed and Memory Cost
23
C.1 Empirical Runtime Comparison
23
C.2 Analysis of Computational Cost in CM
23
Discussion CMT Loss Metric
25
16

---

Published as a conference paper at ICLR 2026
Samples Generated by CMT on ImageNet 512x512
26
Theoretical Analysis of CMT
26
E.1
Oracle Loss and CMT's Approximation
26
F2
Initialization Schemes for Flow Map Model Training
28
F3
Prerequisites..
................................
29
F.4
Analysis of Gradient Bias
31
E5
Analysis of Gradient Variance
34
F6
Vars-Varinnee Decomposition
37
F7
Comparison on Optimization Dynamics
38
A
RELATIONSHIP OF FLOW MAP MODELS
The principled objective for learning the general flow map Li- is train neural network Go(X,, t, s) by minimizing Equation (4) (Lai et al., 2025):
CoM == Es>s Expep (t)) ((G, 8), (x))(
In CTM, the network is parameterized in form inspired by an Euler step:
() = 7x + go Xxx (x( t,
Since the true flow map V t- cannot be accessed directly, CTM constructs a surrogate target using its own outputs, in the spirit of soopggradient approximation CM). Concretely, replaces the oracle with an intermediate reference:
t- (x,) Gg- (V)r-(())) s)., for
8,
where (X,, is obtained either by applying few-step solver to a pre-trained diffusion model (distillation), or by using CTM's own parameterization go (x,,,, t, to generate a self-teacher trajec- tory.
In contrast, MF takes a different perspective: instead of directly predicting V ()-,((x)) it parameterizes the network to approximate the average drift along the trajectory:
1 J. v(xu, u) du.
he (x,, t) 8) Ro h(x,,
1,8) =
Conceptually, CTM and MF share the same underlying framework, but differ in how the learned function is parameterized. The relation can be written
V (x:) = 7x +
X₁
v v(xu, u) du
b
23go
J. v(xu, u) du
+ (8
Thus, CTM can be seen as approximating the first form via ge. while MF approximates the second via hg. Their backbone choices also differ: CTM builds on EDM, whereas MF builds on flow matching.
Let
go (xt 1, I -ho 1,8);
17

---

Published as a conference paper at ICLR 2026
and take the distance to be the squared norm d(x, = ||x yl1?. Substituting into Equation (4),
we can expand the loss term as
d( (x),(x,, t, 8), v (x))(()))
[Go(x,,, t) 2 (x+]ll
s
8
8-s
Xt + 8-8 J.
v(xus 2) dey )
xx
+
(x t,
7x +
1 {xx
2
t-s L. v(xn, u) du 11
t
5o (x, 1,)
X1 +
2
12
/1
8
[ tho(x,, t,
X t t-s
v(Xu, u) du
8
2
(x: (xr((x,,,
X - T-8 J. v(xu; u) du
1 (-- v(xu: u)) du
8)²
ho (x(x),,,8)
(10)
Equations (9) and (10) show that the two parameterizaation tightly connected. In particular,
1 t go (x,,t,))
(xt + --tt-o - [ v(xu, u) du)
= ho(x,, t, ----_/ v(xu: u) duy
(11)
Hence, the CTM and MF training losses are mathematically related, differing only by multiplicative constant. Moreover, in both cases setting 0 recovers the CM scenario, where each state is mapped directly to the clean data. Based on this observation, we will focus our theoretical analysis on V 3--00 mostly (Appendix F), noting that the same arguments extend naturally the general L . including the MF case.
18

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B. SUMMARY OF THE CMT PIPELINE AND ALGORITHMS
Algorithm I CMT Pipeline for Fast Flow Map Learning (CM MF)
Input:
flow map type flow fww._wap.tttte.eteeoo...eaaallat wwt.mmap.tttp type E {CM, time grid 0 - to < 4 < ... < tM = T,
pre-trained teacher sampler weights Φb student flow map weights O.
mid-training learning rate Mmidd post-training learning rate /ostt.
Stage 1: Pre-training (teacher sampler).
if teacher sampler not already available then flow Fwww_map__a_____me map_type CM then
Train diffusion model D with standard EDM-style objective. Construct PF-ODE sampler (e.g. DPM-Solver++) from D
else if flow www.maa..yypp.. MF then
Train flow-matching model or smaller MF model h hteacher in the usual way. Construct a deterministic sampler (e.g., a Heun solver with the PF-ODE determined by V&), an MF multi-step solver based heeccher
end if
end
Freeze & and treat the resulting sampler as the teacher sampler TeacherSampler(()) -).
Stage 2: CMT's Mid-Trainin Trainirnn...
Initialize 0 (optionally 0 < Φb when architectures match). repeat
Sample prior noise Xy Pprior-
Generate teacher reference trajectory
{81, 1.M Teache Samplerrrrrrrre ( XT).
if flow_map.type = CM then
b Special flow map V 1
Compute CMT loss as Equation (7)
LCMT-CM (0) = - (fo (Xx.: 4;), xto)
else if flow.map_type
ME then
General flow map V
Compute CMT loss as Equation (8)
8,, R1,
LCMT-MP (0)
Σ ho (xt, tyy
end if
Update CMT parameters <-6 0 Mmmi V VeLCMT- (MM.cMM--MM(())))))
; or < Tmmid (oCCMT-Mr()). until CMT converges (trajectory-aligned initializer obtained)
Stage 3: Post-Training of Flow Map Model.
Use the converged CMT mid-trained weights 0 as the initialization of the flow map model.
if f Flow.map.type CM then
CM-style post-training for Mr e.g. ECT/ECD
repeat
Sample training batch (data or noise) and times t. Compute CM post-training loss (C as in Equation (3).
Update 0 x- 0 Tposs V (O).
until post-training converges
else if flow.map.type MF then
MF post-training for general V (+4
repeat
Sample training batch and pairs 5) with t Compute MF post-traing post-training loss (M)(()) (0) as in Equation (5).
Update 0 0 Mpos V Lmu:(().
until post-training converges
end if
Output: Learned few-step flow map V 1-50 (CM) V Wi-s (MF) with parameters 0.
19

---

Published as a conference paper at ICLR 2026
B.2 CIFAR-10, 10, AFHQV2, AND FFHQ
We use the seriance-preserving (VP) formulation and DDPM++ model structure in Score-SDE (Song et al., 2021), which is also adopted in the teacher EDM diffusion model (Karras et al., 2022).
For the CMT mid-training stage, we utilize third-order DPM-solver++ (Lu et al., 2025) with 16 NFEs to generate the ODE trajectory, achieving an FID of 2. 2.14/2225/2.99 compared to FIDs of 1.97/1.96/2.39 CIFAR-IO/AFHQV///EE
respectively, under an abundant 79 NFEs. The good FID under just 16 steps ensures the sample quality while making CMT fast since the ODE-solver across steps cannot be parallelized without additional care Shih et al. (2023). We the same batch size of 128, 0.2 dropout rate, and Addam optimizer (Liu et al., 2020) as the ECT stage later. We almost keep the same hyperparameters the latter ECT,, but make the following changes. We choose a 2e-4 learning rate for mid-training, which linearly decays to zero until the end of optimization. The EMA 8 0.999 since CMT is stable to ensure faster convergence. The loss metric for CMT is LPIPS (Zhang et al., 2018), and we use the simplest unit weighting.
For the ECT stage, we adopt the same hyperparameters as the original ECT setting (Geng et al., 2025b) on CIFAR-10 10 but keep the At fixed to 1/4096, 1/1024, and 1/512 on CIFAR-10, AFHQv2, and FFHQ, respectively. We use the same le-4 learning rate but decay it linearly to zero until the end of optimization. This simplifies the complicated A% annealing trick in ECT. The choice of AL in our setting is quite straightforward. We search for the smallest A/ that will not trigger loss spike during the first several iterations.
B.3 IMAGENET 64x64
B.3.1 EXPERIMENTAL DETAILS
We use the EDM2-XL (Karras et al., 2024b) model setting.
For the CMT mid-traine hid-training stage, we use a third-order DPM-solver++ (Lu et al., 2025) with 16 NFEs to generate the ODE trajectory. We do not use classifier-free guidance (CFG) to accelerate the trajectory generation. Our 16-NFE FID is 1.56 compared with the EDM2's best 1. 1.33 under 63 NFEs, ensuring a good teacher for mid- -rainirin We use the same hyperparameters, including batch size, dropout rate, Adam optimizer (Kingma & Ba, 2015), etc., as the ECT stage later, but make the following modifications. We choose a 7e-4 learning rate for mid-training, which linearly decays to zero until the end of optimization. The EMA B = 0.9999. The loss metric for CMT is LPIPS (Zhang et al., 2018), and we use the simplest unit weighting. We train for 6.4 Mimgs.
For the ECT stage, we primarily adopt the same hyperparameters al., 2025b) on ImageNet 64x64 with the XI. size and a batch size of 128, while simplifying the following hyperparameters. We keep the A/ fixed to 1/512 instead of the original ECT's s complex annealing trick. We use an initial learning rate of lc-4 decaying linearly to zero at the end of optimization, which is simpler than the quadratic decay in the original ECT and EDM2. Furthermore, we just use a simple vanilla EMA with B = 0.9999 instead of the power function post-hoc EMA in ECT and EDM2. This simplifies various tricks in ECT. We conduct ECT for another 6.4 Mimgs.
the original ECT setting (Geng
For ECD with the Auto-Guidanee (Karras et al., 2024a) augmented EDM2, we start from the mid- trained CMT checkpoint. We keep the At fixed to 1/256 and the learning rate fixed to le-4. We conduct ECD for 12.8 Mimgs.
B.3.2 COST DETAILS
We compare training cost and data budget for ECT, ECD, SCT, AYF, and CMT here.
ECT and CMT require the standard EDM2 diffusion pre-training, with a batch size of 2048 and total of 327680 iterations. Hence, the total training data budget is 67 671088640 xs 671.1 Mimgs. AYF uses batch size of 2048 and total of 524288 iterations, leading to about 1073.7 Mimgs diffusion pre-training cost. sCT requires a TrigFlow diffusion pre-training, with a batch size of 2048 and a total of 540000 iterations. Hence, the total training data budget is 1105920000 1 1105.9 Mimgs.
The total pre-training mid-traing training, and post-training data budget costs of all methods are summarized in Table 6. Our 98% (CMT w/ ECT over sCT) and 81. 11.25% % (CMT w/ ECD over AYF) training
20

---

Published as a conference paper at ICLR 2026
data budget reduction includes both mid-training and post-training. In other words, compare CMT's mid-trame training + post-training total budget with other methods' post-training budget. sCT's TrigFlow-based EDM2 is just reproducing vamilla EDM2, and the teacher diffusion quality is almost the same. And we are focusing on flow map learning, but not the diffusion model pre-training part. Thus, for EDM2-related experiments, including ImageNet 64x64 and 512x512, focus comparing the mid-training + post-training costs.
Method
Pre- Pre-Training Mid-Training
Post-Training
FID (L)
ECT (Geng et al., 2025b)
671.1
0
102.4
2.49 / 1.67
ECD (Geng et al., 2025b)
671.1
0
102.4
2.24 / 1.50
sCT (Lu & Song, 2025)
1105.9
0
819.2
2.04 1.48
AYF (Sabour et al., 2025)
1073.7
0
102.4
2.98 / 1.25
CMT (w/ ECT) (Ours)
671.1
6,4
6.4
2.02 / 1.48
CMT (w/ ECD) (Ours)
671.1
6.4
12.8
1.78 / 1.32
Table 6: ImageNet 64x64: Pre-, mid-, and post-training data costs (in Mimgs).
Furthermore, we summarize CMT's time reduction and speedup compared the baselines in Table where we compare the A100 (80G) GPU time for training ECT, ECD, sCT, and CMT. We compare with ECT and ECD since they are the post-training methods in our CMT. Meanwhile, we compare with the competitive sCT. We conduct experiments to measure per-iteration time for every method, and compute the total training time as total iterations per-iteration time.
Method
Baseline Baseline (hrs)
Ours (hrs)
Reduction
Speedup
CMT w/ ECT ECT
1280
180
85.9%
7.11x
CMT w/ ECT sCT
13312
180
98.6%
73.96x
CMT w/ ECD ECD
1664
308
81.5%
5.40x
CMT w/ ECD
sCD
16000
308
98.1 %
51.95x
Table 7: ImageNet 64x64: Comparison of training time reduction and speedup for four cases: (1)
CMT w/ ECT & ECT; (2) CMT w/ ECT & sCT; (3) CMT w/ ECD & ECD; and (4) CMT w/ ECD & sCD.
We emphasize that the reported best performances of ECT and sCT are achieved by initializing their flow map models from pre-trained diffusion models, which have similar generation quality as the teacher model that CMT uses for trajectory creation. Hence, in our comparison, the teacher model's training cost is excluded. Overall, we observe that flow map training with CMT (including both mid- and post-training stages) achieves an 80%-98% reduction in training time compared training flow map model alone.
B.4 IMAGENET 256x
We follow SiT (Ma et al., 2024) and Mean Flow (Geng et al., 2025a) for this setting.
Regarding MF from scratch, we directly use the default setting in the original MF paper Geng et al. (2025a) and follow the PyTorch Paszke al. (2019) implementation Zhu (2025). The efficient forward-mods JVP Shi et al. (2024) is used to maximize MF training efficiency.
Regarding MF initialized by SiT, we follow Zhu (2025) for a two-stage post-training starting with MF without CFG for stability and then switching to the default MF training with CFG. However, we found that this approach still diverges some point during optimization and cannot be mitigated by changing the random seed and restarting. Furthermore, if one directly tunes the MF initialized by SiT with CFG, then the optimization directly diverges, and the gradient explodes at the very beginning. These observations all point the instability of SiT initialized MF, i.e., the diffusion initialization.
Regarding CMT, the post- toattirining MF hyperparameter is kept the same as vanilla MF except that reduce the batch size from 256 to 64. Since CMT can stabilize training by providing better initialization, there is no need to use a large batch size to stabilize training as in MF from scratch. For pre-training tiny and efficient MF-B/4,
the MF-B/4 training hyperparameters in the
21

---

Published as a conference paper at ICLR 2026
original MF paper Geng et al. (2025a) but change the CFG-related hyperparameters the same as the possttrrimingg For mid-training, we generate the reference ODE trajectory with the pre-trained MF-B/4 with eight uniform steps between 0 and 1. We also use constant learning rate of le-4 and do not use any weighting trick, and use squared fz loss. We use four random samples to generate trajectories, and each sample provides 28 pairs of :x,, With this batch size, we conduct mid- training for 200k iterations. We found that the key is to use the same CFG scale for all the stages. MF with various CFG scales during training has different ODE trajectory. Therefore, it is imperative to match the pre-, mid-, and post-trainl test-trannnng stage CFG scale. Otherwise, one would obtain inferior results due to the trajectory mismatch during different stages.
B.5 IMAGENET 512x512
B.5.1 EXPERIMENTAL DETAILS
ELatentLPIPS. We follow the standard approach to train a VGG (Simonyan & Zisserman, 2015) for LLPPe We train VGG for 100 epochs with SGD. The initial learning rate is 0.1 and decays at the 30th, 60th, and 90th epochs with 0.1 decay rate. The batch size is 256. The resulting VGG achieves 95% topl accuracy on the train set and 64% validation top1 accuracy. Then, this VGG the BAPPS (Zhang et al., 2018) data learn human perception.
should have been fine-tuned However, we do not take this step to ensure a fair comparison with other baselines, i.e., we keep the training data as ImageNet only and do not rely on additional data that other CMs do not.
CMT. We mainly transfer our ImageNet 64 64 hyperparameters since they all follow the EDM2 setting. We highlight the difference below. We do not use dropout stabilize the training. We use the XXL model size.
CMT Mid-Training. We use third-order DPM-solver++ (Lu et al., 2025) with 16 NFEs to generate the ODE trajectory. We do not use classifier-free guidance (CFG) to accelerate the trajectory generation. We choose 2e-4 learning rate for mid-training, which linearly decays to zero until the end of optimization. The EMA B = 0.999. The loss metric for CMT is ELatentLPIPS (Kang et al., 2024), and we use the simplest unit weighting. We train for 12.8 Mimgs with batch size of 128.
(Karras et al., 2024a) model of Size XXL. We keep the Al fixed 1/1024 instead of the original ECD's complex annealing trick. We use constant learning rate of le-4. The batch size is 128. The total training budget is 12.8 Mimgs.
CMT Post- Dost-Training's ECD. We use ECD as post most--raining to distill the EDM2
B.5.2 COST DETAILS
Similar to the ImageNet 64 case, we compare various methods' training data budget cost and training time. Table 8 shows the training data budget, where we achieve 93% lower cost than the sCD and 71% lower cost than the AYF. We report H100 GPU training time in Table 9, where we used better GPU for this higher-di generation task with a larger model. Table demonstrates that CMT (including both mid- and post-training stages) achieves 75%-92.8% reduction in training time compared to training flow map model alone.
Method
Pre-Traine Pre-Training Mid-Training Post- Post-Training
FID (L)
ECT (Geng et al., 2025b)
939.5
204.8
9.98 6.28
ECD (Geng et al., 2025b)
939.5
409.6
8.47 / 3,38
sCT (Lu & Song, 2025)
770.0
0
204.8
4.29/3.76
sCD (L.u & Song, 2025)
770.0
409.6
2.28 1.88
AYF (Sabour et al., 2025)
2147.5
102.4
3.32 1.87
CMT (w/ ECD) (Ours)
939.5
12.8
16
3.38 / 1.84
Table 8: ImageNet 512x512: Pre-, mid- and post-training data costs (in Mimgs).
22

---

Published as a conference paper at ICLR 2026
Method
Baseline
Baseline (hrs)
Ours (hrs)
Reduction
Speedup
CMT w/ ECD ECT
1611.18
403.63
75.0%
3.99
CMT w/ ECD sCT
2339.88
403.63
82.7%
5.80
CMT w/ ECD ECD
4643.99
403.63
91.3%
11.51x
CMT w/ ECD sCD
5591.74
403.63
92.8%
13.85
Table 9: ImageNet 512x512: Comparison of training time reduction and speedup for four cases: (1)
CMT w/ ECD & ECT; (2) CMT w/ ECD & sCT; (3) CMT w/ ECD & ECD; and (4) CMT w/ ECD & sCD.
B.6 MS-COCO TEXT-TO-IMAGE (T21)
The SiT teacher model trained with REPA (Yu al., 2025) for 50k iterations, achieving 50-step FID of 4.73 using second-order Heun ODE solver, and the CFG scale is 2. All other models are also trained with REPA. The SiT teacher additionally serves as the pre-training stage.
Regarding MF from scratch, we mainly reuse the default setting in our ImageNet x56x25 MF experiment. But set the MF's effective CFG scale and the CFG trigger interval [0, 1] since the SiT teacher uses the same CFG hyperparameters. We specifically set 0.5 and w = 1. Regarding MF initialized by SiT and CMT with MF, we use the same MF training configuration for post-training.
For mid-training in CMT, we generate the reference ODE trajectory with the pre-trained SiT with an 8-step Heun solver. We also use constant learning rate of le-4 and do not use any weighting trick, and use squared l loss. We use four random samples to generate trajectories, and each sample provides 28 pairs of (&r, Xt,), inducing total 28 4 112 batch size per iteration.
C TRAINING SPEED AND MEMORY CoST
C.I EMPIRICAL RUNTIME COMPARISON
We report the running speed of CMT, CT, and CD. For ImageNet 512x512, we used single H100 GPU, while for other datasets, we tested on single A100 GPU with 80 GB (8 181920 MiB) of memory. We chose the simple ECT and ECD as representatives for comparison. CMs with additional tricks may incur larger costs. We adopt a second-order Heun first-order Euler solver in CD. The training hyperparameters, especially the batch size, are kept the same as in the main results. For easy speed comparison, normalize our method's speed unit, where larger number means lower speed.
Dataset
Batch
CMT CT (1)
CD-Euler (L) CD-Heun (L)
CIFAR-10
128
0.79
0.98
1.17
AFHQv2 & FFHQ
128
0.85
1.05
1,25
ImageNet 64x64
32
0.80
0.92
1.04
ImageNet 512x512
16
0.68
0.83
0.98
The memory costs of all methods similar, they all involve backpropagation step. But CT has the smallest memory cost, CMT is the second, and CD has the largest memory cost. This is because CT does not require any additional teacher network, while CMT requires one unguided teacher. Lastly, CD requires two additional types nets: guided and unguided teachers.
C.2 ANALYSIS OF COMPUTATIONAL COST IN CM
Empirical Runtime Comparison with CMT, CD, and CT. The important factor in CMT's wall- clock time is the number of ODE solver steps (NFEs): steps along the trajectory sequential, so higher NEE directly increases wall time. While solver-parallelizatie may case this bottleneck (Shih al., 2023), target fast training under prevailing practice by operating in the low-NFE regime.
23

---

Published as a conference paper at ICLR 2026
Throughout, we fix NFE 16, which we found to be sweet spot: it provides sufficiently accurate supervision while keeping CMT only slightly slower than CT. We use the third-order multistep DPM-Solver++ (Lu et al., 2025) for CM-style teachers, it's stable and effective at low NFEs. The multistep scheme also reuses previously computed states Xt, to construct later states & further reducing overhead. Two error sources matter: (i) the distillation fit error due nonzero training loss and (ii) the teacher discretization eπror from small NFE; empirically, (i) dominates.
With NFE 16, DPM-Solver++ attains FID typically within 0.2 of the best large-NFE setting across all datasets, indicating diminishing returns beyond 16. Consequently, CMT with 16 NFEs preserves speed while maintaining competitive quality. Concretely, CMT attains training efficiency comparable to CD (which requires teacher inference) and continuous CT or MF requiring student IVP. Further, CMT's cost is also close to that of discrete CT (Lu & Song, 2025). Per iteration, CMT is only 15%% slower than discrete CT (Geng et al., 2025b), where we used Easy CT and Easy CD (Geng et al., 2025b)'s 8 framework for time evaluation. However, CMT converges faster in far fewer iterations than these alternatives in the entire training loop; thus, the wall-clock runtime is lower. This advantage is clear on ImageNet 64 x 64 and 512 x 512, where CMT outperforms sCT/sCD while reducing training data and GPU cost by 93%-99%999 99%.. We also cut 50% of the training GPU time compared with vanilla MF while outperforming in FID.
Overall, because it provides stronger proxy of the flow-map trajectory during mid- mid-training, substantially accelerates the subsequent flow-map post-training (e.g., CT).
If teacher trajectories are pre-generated, training with CMT reduces to a single backpropagated student evaluation per pair, which is the fastest regime and used by prior distillation work (Zheng al,, 2023). However, pre-generation requires extra preparation and storage; keep the setup simple and comparable, all our experiments run the ODE solver on-the-fly during training.
Theoretical NFEs Comparison with CMT, CD, and CT. We compare CMT, CD, and CT by teacher function evaluations (NFEs), student forwards, and student backpropagations. Costs are term in the loss.
normalized per training pair, where pair is one
In CMT, each teacher trajectory {x,, 1 yields M pairs (Kt, ta) + Xo for = 1....,,, ............ Let M 2 A be the number of steps from [ to lo, the multistep order, and & the NFE cost per bootstrap step used to initialize the first h- 1 history points (e.g., s=1 for Fuler, s=2 for Heun). An explicit k-step solver then incurs one new teacher evaluation per step thereafter. Hence
NFEsmjj = s (k-1) + (M - (k- = M + (s-1)(k-1),
and the per-pait teacher cost for CMT is
Teacher NEEs per pair (CMT) = 1 (s- 1)(-
M
In CD and CT, each pair corresponds single sampled time t. If q denotes teacher NFEs for the one-step teacher update used inside CD (e.g. g=1 Fuler, q=2 Heun), then CD has Teacher NFEs per pair q; CT has none.
The above yields the following per-pair cost summary:
(s 1)(k 1)
CMT: Teacher NFEs 1
Student 1 fwd + bwd,
M
CD: Teacher NFEs =
Student = 1 fwd + bwd,
CT: Teacher NFEs = 0,
Student = 2 fwd + bwd.
We now instantiate the parameters match our experimental setup. With M-16 and & E (2,3).
1,
s=1 (Euler warm-up), E [1.06. 1.12],
CMT teacher NFEs per pair-
1+
k-1 16
8 8=2 (Heun warm-up).
Thus, relative to CD: CMT matches CD when q=1 and s=1; it is only +6%~ 12% higher when g=1, 2=2, k (2,3): and it is cheaper than CD when 2=2 for these M, k. These accounting predictions align closely with the empirical measurements reported earlier in this subsection, further supporting the efficiency of CMT.
24

---

Published as a conference paper at ICLR 2026
In short, we analyze the cost per input-target pair. In CMT, one teacher-generater trajectory yields many pairs by matching intermediate states to the clean target, whereas CD and CT generate each pair independently from sampled time. Per pair, CMT needs one teacher call plus one student forward and one backpropagation; CD needs one teacher call plus the same student cost; CT needs no teacher call but two student forwards and one backpropagation. Hence, CMT is roughly costly as CD and only slightly slower than CT, consistent with our empirical findings (Appendix C.I). Meanwhile, CMT achieves near-unit teacher cost per pair while using a single student forward, making it a lightweight and effective choice for the mid-training stage.
Theoretical NFES Comparison with CMT and Slow CMT Variant. Mathematically, both the CMT loss CC in Equation (7) and the Slow CMT variant Lial are discrete-time approximations to the same oracle loss Lorade CM in Equation (5). However, they differ crucially in how the teacher- generated supervision targets are constructed. In CMT, we first draw XT ~ Pprior: run the teacher ODE solver once from T down to 0 to obtain a full trajectory {x,, J.M and then use the single endpoint 11 shared target for all intermediate states (Rt,, (4) on that trajectory. Thus, one teacher solve yields M supervised pairs (x,.,t) + &... In contrast, in Slow CMT (based on c(U) ) we sample (x,, i) and, for each such pair, independently run the teacher solver from x, ( = 0 obtain xx(x)). Each supervised pair (x,, () + Xo (xx) therefore requires its own teacher integration. The two objectives coincide in the continuous-time limit, but their discrete implementations use teacher trajectories in fundamentally different ways.
Because of this difference, the mid-training cost of Slow CMT is significantly higher for a fixed
number of supervised pairs. Let M denote the number of discretization steps on [0, 7], and let
NFEsua be the teacher function evaluations required for one trajectory. In CMT, a single teacher trajectory is amortized over M pairs, so the teacher cost per training pair is NE NFEsmaj/M / = O(1). In Slow CMT, each training pair requires a fresh teacher solve, so the teacher cost per pair is O(NFEsua)) Consequently, to train on the same number of supervised pairs, Slow CMT would require roughly a factor of M more teacher evaluations (and thus GPU time) than CMT; conversely, under fixed teacher-conee compute budget. Slow CMT can only afford about 1/M many pairs. In our experimental configuration, this leads to an overhead of roughly 3x in wall-clock GPU time, which is what we refer to in the paper when stating that the Slow CMT mid-training stage is about three times more expensive. We did not perform an exhaustive sweep that precisely equalizes GPU hours between the two variants during the review period, but the existing experiments already show that CMT achieves better FID at substantially lower teacher cost, which is exactly the mid-training efficiency advantage we aim to demonstrate.
D DISCUSSION ON CMT LOss METRIC
Loss Metric: LPIPS. The metric is crucial to CMT's performance. The LPIPS metric (Zhang et al., 2018) measuring perceptual similarity is known to align more closely with human vision. Optimizing LPIPS loss helps models generate images that are perceptually similar to the original because it penalizes differences in feature space that aligns better with human visual processing, rather than L2 loss in pixel space. The features are derived from VGG (Simonyan & Zisserman, 2015), which is initially pretrained for ImageNet classification in torchvision (Paszke et al., 2019), and then fine-tuned on human perceptual judgments (BAPPS dataset) to better reflect human judgments of image similarity. CMT generates high-quality supervision signals using high-order multistep ODE solvers, providing accurate and stable labels. To encourage the student model to closely match the teacher's output, we employ the LPIPS loss. Moreover, since CMT provides fixed and stable labels, the training process becomes inherently robust, obviating the need for additional robust loss functions such as the Huber loss used in iCT (Song & Dhariwal, 2024). This also eliminates the burden of tuning extra hyperparameters associated with such losses. While minimizing the Le loss yields outputs with low pixel-wise error relative to the teacher's predictions, it often results in blurry images that fail to capture perceptual fidelity. This is because L2 loss penalizes deviations uniformly across all pixels, disregarding the spatial and structural cues that are critical for human visual perception.
Why ECTACT Uses Huber/L; Loss? The optimization objectives and training dynamics of ECTACT and CMT differ fundamentally. CMT leverages high-quality, fixed teacher labels generated via accurate numerical solvers, providing reliable supervision signal throughout training. In contrast,
25

---

Published as a conference paper at ICLR 2026
ECT and iCT rely self- guidance, where the model learns from its own predictions. In such self-training settings, the use of perceptual losses like LPIPS may introduce additional bias, as the supervision signal is inherently noisy and evolving.
Latent LPIPS Loss. LPIPS operates exclusively in pixel space, whereas latent CM faces the challenge of lacking refined metric and must resort traditional L2. ELatentL (Kang et al., 2024) trains an LPIPS metric the deebencoder-deeedeett latent space. The idea is still to first train VGG (Simonyan & Zisserman, 2015) and then fine-tune it on BAPPS (Zhang et al., 2018) in the latent space. For our latent space experiments, we train a VGG following their setup on the ImageNet datasets, but do not fine-tune on the additional BAPPS dataset ensure a fair comparison with other methods. In other words. CMT do not resort to additional datasets while using the same train set as other baselines.
CMT with MF uses Squared & Loss. This is because MF is general flow map requiring mapping between any time steps, but not just the initial time corresponding to the clean data. Hence, the label in CMT with MF can be noisy data within the trajectory, rendering the LPIPS loss inapplicable. Hence, we resort to the common squared & loss.
E SAMPLES GENERATED BY CMT ON IMAGENET 512x512
To illustrate the visual quality of CMT, we show two step samples generated by CMT(with ECD) trained on 512x512 in Figure 3.
F THEORETICAL ANALYSIS OF CMT
F.1 ORACLE LOss AND CMT's APPROXIMATION
The Minimizer of Equation (2). We first show that the optimizer of Equation (2) recovers the oracle flow map L 1--0-
Proposition F.1 (Oracle CM minimizer), Assume: (i) d: RR x R + 10, oo) satisfies d(y, z) 2 0 and d(y, z) iff y z, (ii) (x,, (lllf((x, t)l < co. Then any minimizer of
Loracle CMa (0) =E, Ex (o())d(fo(x)))) (), v Vo-o(x))]]
satisfies
(x t)
(x))) for a.e. Xt ~ Pt. and
If. in addition, d(.,2) 2) is strictly convex each fixed z, then this minimizer is unique (a.e.).
Proof. Let On T' denote the time distribution of The integrand (fe (x))) is a nonnegative measurable function. Hence
Coraele-CMa (0) 2 0 for all 0.
Choosing fg(x,,t) = 1 (xx, makes the integrand identically zero (since d(z, z) 0), so the infimum of the objective is 0 and is attained by this choice. remains show that any other minimizer must agree with V v almost surely.
Suppose for is minimizer and define the set
A {(t, x²) fe- (x,, [x-)(x))).
On we have d(Lo- (x,, t), L (x,) > 0 by assumption (ii). Since w(t) for Unif[0, -aa.e. t, Unif(o, T) (A) > 0 then by Tonelli or Fubini theorem
Loracle-CM E [w(()d(for (xr, t), & (x-o(x))) ))] 2 E (t
for some 0, contradicting minimality (the minimum value is 0). Therefore U 0. ie., fg- = L Pe-sp holds Un T| x a.e. If d(,, z) is strictly convex (e.g., squared E2), pointwise equality (a.e.) is the only way achieve the minimum, giving uniqueness (a.e.).
26

---

Published as a conference paper at ICLR 2026
VV
XII
X
II NN
IX
II
F
Figure 3: Two-Step Generated Images by CMT. Using the trained CMT (w/ ECD) on 512x512, we achieve the best two-step FID 1.84, at 93% lower cost than previous sCD.
27

---

Published as a conference paper at ICLR 2026
CMT's Loss is Equivalent to Equation (2). We prove Theorem 3.1, which shows that the CMT objective is, up to a discrete-time approximation, equivalent to minimizing the oracle CM flow map loss in Equation (2). We assume that the terminal distribution Pprior coincides with pr. Under this assumption, we will show that the following result holds:
Loracle-CMI (xy))((rr) [d (fg (x))),, (x-o(xx))]-
Proof. We can exploit the semi-groue property of the solution map to express the intermediate distribution Pr as:
v -r- Pr--ttpprrio = f 5(x: Pr- (x)((x)))xx(
Pt = L P-ttaaaa
Using this change of variables in Equation (2), we obtain:
foracle CM (0) = I2 (x(s) (d (fo (xT))(XT))) f), v
1 / d (fo (T) (x (x))xx
E, / (fo (
0(x+ (x-r(xe))prr )
E, / f (fo(
(x))) - ( dxx,,xx
E d (fo( (x)() (d ( o
(xT)
Therefore, the CMT loss approximates the oracle objective Coracle by leveraging a pre-trained diffusion model estimate the solution map (xx) This allows for tractable surrogate the otherwise intractable oracle loss.
F.2 INITIALIZATION SCHEMES FOR FLOW MAP MODEL TRAINING
Let O. We investigate four initialization schemes for the post training stage of CM flow map v learning.
CMT. There exists 1 fCMT such that
LCMT (0 (0cMT) == E,Ex, foowr (Solverr- Solverr-o(x7)) <E,
where solver (xe) denotes the result of running the ODE solver from back to u using the drift of pre trained diffusion model in the PF ODE.
Diffusion Model (DM). Let Do denote the clean prediction of a diffusion model. There exists 0DM such that
(DD()DM) = E,E, ~p: Depsi (xx,t) X
<
General Consistency Distillation (gCD). We define general consistency distillation loss that employs a "soft label" for teacher supervision (Kim et al., 2024). Let E [0, T) be fixed and let Ogen denote the student parameters. We consider
LgoD(O u) E,Ex,op [||eecco (xe, fe,co (Solver,.. w) wlll] <
The loss LgCD includes two important special cases. First, when = - At, it reduces to the conventional consistency distillation objective (Song et al., 2023), where the solver is applied for single step. Second, when 0. it resembles knowledge distillation (Luhman & Luhman, 2021). In this case, by construction of the consistency model parametrization,
fg fo (xolver;-o(xx).0)) 0) Solver;-nnx;))
28

---

Published as a conference paper at ICLR 2026
Random Initialization.
We assume that randomly initialized parameter Grand. satisfies
E ExExerpi [frrndd (xe: < R.
for some constant R > O.
F.3 PREREQUISITES
We first present the summary of assumptions in our individual propositions. Assumption (Data Distribution).
Key Assumptions.
The data distribution Pásta has bounded support and finite
second moments:
= Epeses ||xم| < co.
Assumption B (Smoothness). For 0 = OCMT, 0DM, or ORCD, we assume the following conditions hold:
(i) Bounded Value and Jacobian:
[foll2 V txl: S R, for some constant R
(ii) There exist Lip(fe) such that for all x, E RD and s,t E [0,
lo (x, () foly, I)ll S Lip(fo) (Ixx yll sl)
Assumption (Oracle Flow Map and Solver). We assume the exact flow Li Lest and the solver So using the teacher drift, satisfy the following conditions:
(i) Finite targets: Cy := sup Ex, oo <c.
(i) The exact flow is Lipschitz in state: for some Lip V 2
L xx)( v (y)+ < Lip v |x y|l:
(iii) The solver is Lipschitz in state and time: for some Lip(Solver) 2 1
[Solver (x) - Solver--u(yy]l < Lip(Solver) (||x yll + It - s|)
(iv) The solver Solver, is zero-stable, global order-p solver with 2 1:
sup [Sxlver,-x(xx)) v (x-s(())) = O(At)) 8 s2t. t.
Some Lemmas. We summarize some auxiliary tools that we will use later.
Lemma F.1. Let E RDD be square-integrable and let be any random variable on the same probability space. For any (deterministic) decoder such that F(x:) square-integrable
E [|xx F(x)11 = E [Tr Var(xo xx)] + E [[ x1] - F(x:)
Proof. Write the conditional mean (posterior mean) as
(xx:) == [[x]
and the zero-mean conditional residual as
e = Xo <x(x)),
so that
[E(e|x]] = O.
Then for any F,
Xg F(x:)
(Xo (xx))) + (xe(xx)) F(x:)).
29

---

Published as a conference paper at ICLR 2026
Expand the squared norm and take expectations over (t, xt):
E xxx F(x,)) |1
E [||eli] + E [x))(x)) F(x))l)) 21E ((x)) F(x.)M.
The cross term vanishes by the tower property and the fact that (x(x)) - F(x:) is (xx))u)masuurable
E [e,, (x(x) - F(x)))] = IE E (ee, (x,) - F(xx))/xx
E [E(elxx,, ((xx)) - F(x.))]
E' [(0, xx,) - F(x))) = 0.
For the first term, use the definition of conditional covariance:
Var(xo|x; (x) E jee |xx]
whose trace equals the conditional mean squared residual: Taking expectations over X, yields
Tr Var(xo|X)) = tr eee
Xx
| X
E
||
E [Tr Var(xo|xe)]-
Combining the pieces gives
E E|Ixx F(x))l)) =E [Tr Var(xx E [1))((x)) F(x:)
which is the claimed identity.
Lemma F22 Let the Assumption C hold. Then
[Boolver--(S0Lrrr_sstl) -x,|| (AP)).
Proof. Let P. == Solver; be a numerical solver (using the teacher drift) on a uniform grid with step size At. For any and Xt,
+ -- (xi))) x; -
1 8u o 1- (x:)) Vl P (
(x,))
backward global error
V 5 i- (x+)) V sod V ビ- (x+))
propagation of forward enror
CAP T.|| 1 10 (x+) (x)x(x))| < LCAAP.
Therefore,
[ - -xe|1 = 00 (At)..
In the proofs repeatedly the following inequality, derived from the triangle inequality and the Mauchy-Schware inequality, without stating it explicitly. This inequality allows us to convert bounds in the &g norm into bounds in the squared E2 norm.
Lemma F33. Let be an integer, and {a, A be a sequence of real numbers. Then
N
N
2
a
0
im
30

---

Published as a conference paper at ICLR 2026
F.4 ANALYSIS OF GRADIENT BIAS
We focus on the setting where the distance function is the squared l2 norm,
y) y
and the weight function is uniform, (() = The extension to more general choices of distance or weighting follows the same way. Throughout this section we work with the CM flow map :- analogous statements for other flow maps, such the CTM family, can be derived by following the same arguments presented here.
For convenience, we rewrite Equation (2) in the simplified form
Loracte (0: == [xx, t) 2
(12)
and define the CM training loss as
lCM (0:E) = [ t) - fg (x,-ar, - 1---,,,-00111
(13)
where ş = (t, x,) ~ Unif(o, (t) Pt denotes the training sample, with Unif(O, (t) representing the time sampling distribution (for example, uniform over [0, 2"] or any chosen weighting).
We then introduce the expected objectives
Euracte () := Eg [euracee E)],
(CM == Eg fCM (0; (0:E) E)
which represent the oracle target loss and the CM training loss, respectively. Finally, we define the squared gradient bias
B(0)= V
VolCM(()1
Theorem F.1 (Bias Comparisons). Assume that Assumptions A to C hold with p 2 1. Then the following bias comparisons are valid for the four different initialization schemes (6 OCMr, ODM. ggCD. random initialization Orand.) of flow map model training:
(i) CMT
B(0cMT) = (0 (++++A At2 + AtP).
(ii) Diffussion Model:
B(0DM) = O (8 AP2 +
E,
+ Ex,, [|| \/oo))())) - E[xo x: x
(iii) General Consistency Distillation: For a fixed u E [0, T7, assume in addition that
8n E Ex,<p_
[egcD (Xus u) V vx-o((x)) (xx)
Then
B(0gcD)= =0 0
t.)) (Ea)
(iv) Random Initialization:
O(1).
Proof.
Taking the gradient of Loracle, we obtain the unbiased oracle CM gradient as:
V Voloratte
( E, (xxxxx)((
Vofo(x..t) (fo(xe, t) [x-s((x:)]]
Likewise, the CM gradient is
volcm(() = E2 vofo(xi, (fo(x:, - fo (x,- -A))]
CM is approximating V (x,) with fo- (xx--rr,-- At). CMT and diffusion differ in the initial-
ization of 0. The one-point bias be bounded:
[/Velcu(() - Volonate(()))
< E Vofo(x,,)) f). (000(ffff--
At) v (x:)) [
Ee
Volo 1)122 lifo (xt-se,t- At) v o
G.
Eg ||fo (oo-((x--a (xI- At) - & (
31

---

Published as a conference paper at ICLR 2026
Namely, the deviation with the gradient of the oracle loss is upper bounded as
[Voecm(0) - Velomeie (0)l2 S G Ie [[o (Xr- t - At) - -
In the following, we individually derive the upper bound for different initialization scenarios. Denote :== t At for notational simplicity.
Case 1. CMT We denote p (x,) == Solver,->(xx). Given sample x, ~ Pa. and time t,
define &, = (
For CMT initialization at 0 = CCM,, we have
1 (xx,t') v (x+0(( l|
lfoorr (xV, t') (
< Sooww (xe, 1) foow (xe', 1212 + Soeer (x+, - r (x
+ (I - L [x)+
(I) + (II) + (III.
For (I), by Lipschitzness and the forward parameterization,
(1) 2 Lip((ecMT) (1xx - x=I + |t' - -t), E||xy - 10ar-xx11:-- 1 O(Ar))
since Og - = (a1) and By - B; = ( Hence E[(T²] = =(()))=((A()))
For (III), since 1p : (Xx) MT ( ( (x:)). with Lemma F.2 we have || x O(Att).
Thus,
[[1111) 2] = (())2))
For (II), we first insert &; = PT (x))))
(II) S [feerrr (xe, t) feerr [2, t)|| + feoor (Re,,) To ( (x+T(x)))) (Ila) (IIb.
From Lemma F.2 and Lipschitzness, E((Iaa) O(AtP)) For (IIb), define
g(xr,t) = | (xT-):(x)). t) oo
Then
(Ib)) Pp, (xx))
When the CMT's training expectation is taken with XT Pprior and t Unif(0, 21. we compare
(lxer.)y-pppn x Unit [xx,, t)] to (x) (9 (x,,) t).. Using the coupling x; Ppriors X M (e standard stability of 4 and f yields Lipschitz constant Lip(g) (in xT.) such that
Rg I (g(( - (gg(x t) < Lip(g) (x)((x)) x7 = O(AP))
Hence
B[(11)} Eg(( 22 (
() <++)++++++)+))+))+) ((+O
Collecting the bounds.
1|2
2
E fo forur (xv,t)) - 1 [x)o(())
+ +0( + 30(APP).
Thus,
foowr (xe, (') V (xx) = +
(p 1).
Case 2. Diffusion Model: Let the CM loss be initialized at the pre-trained diffusion model weights
0 0DM. then we have
Pop (x) - L (xx0(xx))l22
|epe (xe)t') - foose (xe, t) + ||eeoss (xe, t) xoll2 + ||xx Mi 1x))112
S || Pera (x),t') - -ennr (xe, t)l1 + | (xe, t) -xol2 + ||xo - V L )13
(I) (II) + (III).
32

---

Published as a conference paper at ICLR 2026
For (I) and (II), we have
EE [Conn (xx, () foom (x (x, A0)13] + IE (xe, xol1
S [ip(ODDM) ||xx- - x(l12] +e
O +e),
following the similar argument in CMT's case.
However, using the pre-trained diffusion model's weight as an initialization induces an additional dis-
crepancy between the data Xo and the reverse-time solution of ODE V Wx-o(Xe), where x is perturbed
from Xo. We will obtain a general upper bound of the term (III) with IE ||xo 2 (xx)2
Applying Lemma F.1 with F L t-1 and then averaging over i, we obtain
Fx, [|xo - V ( (x:)) xx Tr Var(xo) X4, ()] + Rx,, [I (x)((())))) EE X1
xx,1| [1]]-
where the second term means the extra MSE we pay because the flow map is not necessary the Bpyes-optimal estimator (the posterior mean).
Below we compute the upper bound for Ex,x. [ [I Var(xo|x:)] Given an observation x, with fixed, the minimum mean-squared error (mmse)
mtse(t) == inf E [[xx] feown (x)]
where the infimum is all measurable f with finite second moment. The minimizer is the posterior mean E[xo| xr). and
mmse (t) = E |xx E/xo x x1]11]] Ex. Tr Var(xo|x:)].
Since mmse(f) is the minimum risk over all estimators, its value is bounded above by the risk of any specific estimator. Take the linear estimator feour (x,) = (1/a))x,. Using x, = a,xo + o,e and the independence of Xo and
Ex, [T [ <E <E| J. de
of x1 E. El|xx J (a.x0+01e) BB|| ar e12 . EllE
D.
a
Hence mmse(t) I (oi Averaging over t gives the bound:
et ||x0 L (x+o(( S DE,
of
Ex., L (o)) IE [[xxo (x.]x
Therefore, to summarize, we have
Eg [eooe (x(, d') L (xx) 0 + E,
+ Ex,, [11 - E[xo x
Case 3. General Consistency Distillation: Let (x.)) == Solver be the p-th order solver, solving the PF-ODE with the teacher diffusion model's drift.
When initializing at 0 = OgCD, we need to additionally assume that:
8, : Ex. [ (X., u) - v0 (x.)l <0.
General CD at a single u does not control the bias the oracle Va--o: 8n can arbitrarily large even if the General CD is trained well with small The term 8, supplies the necessary anchor at
We use triangle inequality to obtain
focCD (x),, 1') T W
foo (x) fooo (x,, t) 1)112
/eeoo (x,,t) fooon (oi (xx)) will
+ |loeep (Dr-) (Xe), u) (xx)
=(( (II) (III).
33

---

Published as a conference paper at ICLR 2026
For (I),
[e((I) & (i (Ixx - xel12 + A12)
From x, = 0x + 0,c, independence of Xo and 6, we follow the similar derivation as the previous CMT's cases: Hence
E [[( = O(At).
For (II), by the hypothesis
S
B[(M]] S
For (III), we need to bridge from u to O. For notational simplicity, we denote zu == Pz-u((x))
Yu (x.)( Inserting and subtracting the teacher at we will get:
1 (Zu: u) L (x,))l))
lFoOCD (Zu; u) V (y=)ll
I IFOOCDD (Zu: u) fooco (yur u)ll [Io (y's u) L Vn vn-- L uy-) - 4 v
(Mla) + (Tlb) + (MIc)..
Therefore (III) 3 (Iaa) + (IIIb)² + (IIIc))) and, by assumptions
[[([[Ta)]] < Tip (forco El - yuall -all==000s Am))
B[(IITb) 6us [[(IIIc) < Lip" (V) yull 3= O(APP))
Hence,
[[(WI]] O(APP +ww)
We thus conclude that
Egg
||| [[ (x), 1')
L ( O(AI2 + + 0 + +x),
p 1.
Case 4. Random Initialization:
Eg [lferand (xg, t') L Il (xx))
Eg [iee (x,,t') for.na (xe, t)1l2 Eg fo (x,,t)l Ee 4h (x,))1
( + (1))
O(1).
F.5 ANALYSIS OF GRADIENT VARIANCE
Following the same setup as in Appendix F.4, we focus on the case where the distance is given by
d(x,y) = ||x y
w(t) = 1,
and the CM flow map is denoted by 1 The general case can be obtained analogously. For notational simplicity, let
E := (t, x1) ~ Unif(0, T] x pt-
The gradient variance with respective to the expected loss lcM (0; given by
V(0)= = Vare Volcm {)] = Ee V (0: ) Ee[VolcM <) e)]][]] = Tr [e e)).
[
Theorem F.2. Under the same assumptions as in Theorem F.I. The following upper bounds on the
variances hold for different initialization schemes:
34

---

Published as a conference paper at ICLR 2026
1. CMT V(OCMT) O(E + At²)
2. Diffussio Model: V(0DM) min (0(s),
3. General Consistency Distillation: V(0gcD) = O( At3)
4. Random Initialization: v(Orand.) = O(1)
Proof. To analyze the variance, we observe that
v(O) Eg Vofcm(( -
< Eg [ (0,
We compute the gradient of the loss in the gradient variance formula as:
VolcM (0, 5) = e Volo(xx,
where we define the error vector:
e(0) == fo (x, 1) - ra- (x:- Ae, A1) E TR.
Now we bound the second moment by using ||A Frobenius norm of the matrix A, will get
uul2 ||Auull2 < |||] with || denoting the
I 00 e)113 4|l (Volo(x,, ()) e e0))2 s 4 (o
Therefore,
v(0) - 4Eg Volo (x,, 1) e00)
From the assumption that Vofo f)]l - M almost surely, then
v(0)= v EM [00
We now bound Eg [0))() under the four different initializations.
Case 1. CMT: Let . be p-th order solver for the PF-ODE built from fixed drift, and define the forward-backs backward (round-trip) map
x = 4 pr-(( ( (x:)))
R == Pr-t-Ad ( (x
Insert solver's 's round trips in Cocur:
edcar (fecue (x), fecur (x), - (fecxer (xI-Aci At) feomr (Se- At))
(focur (& fecwr (& A()).
Thus, we have
Elle0crrr 112 S Lip(fe (1|x xz ||x--11- xx- + El
(14)
where C, = feeur (x,,t) feexer (x,- At).
To address C. term, we anchor Ct to the solver teacher. Set Sz(x) == Φ7 V (x))())). Then
(x+) == (x:))( and S. (xxx((xx-a)) Ar) = Dr V At at)). We decompose
C as the following:
(Foown (Rt- L - A1) - S. de (x4- + (Sx(x,) - S, se (x(- Ai)) C. = (foorr (), - -.(()))
(8)
(b)
(c)
Because PT Ppriors X, ( (XT) with XT
Pprior.
(a) < Bl|(b) <
Now, we control the teacher drift.
[[(c)]22 (x,) - S S
< Lip(p + (x V A (x--2e))l2,
Lip + Lip V (1|x:
Xt-Arl12 At))
35

---

Published as a conference paper at ICLR 2026
we have B O(At)).
Combining the above bounds, we conclude:
E Ex @0cMT
= 0 (s+ + A(P).
Case 2. Difftusion Model:
Bound I: Training Error Only; No Smoothness. Write
e0DM (FeaM (Xe,() - -x0 (Feuse (xi-sei 1. At) xo)
By ||u 21I $ 2||u||2 2 2||e|| and taking expectation over (1, X Xt At);
- 2E fooar (xx, t)
Xo [1
+
2E [[] (XI-Ai, At) xol13] S 4E,
where the last inequality uses the same training distribution for (t, and (t X( (e.g., t uniform on [tt, 1)). Thus
Copx [1
S 4e.
Bound II: Lipschitz Smoothness; Ar-Sensitive. Assume fepss is Lipschitz in state and time:
fe (xx - ffom (y. s Lip(fe (l|x yl12 + It - s|).
Then
||eppa ll2 S foor (x,, - feose
+ ||fecur (XI- fecue (xI- t At)ll2
$ Lip(femm . (1xx; X1 x1l22 At))
hence by (a + b) < 20² + +1b,
||eepss S Lip L (feest) (1|x, - x AtP).
Taking expectation and using the coupled forward process,
X Xt- = (a: ax-0 + (be - bi-DejE
so with m2 == E x 12 and |||<|| = (and E [ c| = 0),
E| X - Aell 11 = O(113).
Therefore,
E 60 [2] = O(AP))
Taking the better of the regimes yields
Ex c [2 x min {e, Ar2]
(If one averages over f, insert [.] on the second term't bracket; if one prefers a uniform in-t bound, replace the bracket by its sup,.)
Case 3. General Consistency Distillation: With
e0,CD
[eeser (Xt, 1) focrr ( (Pr=)((X),.u) w]] [Ceeser (X:-Ari At) focwr ( (i u)]
A.
Bi-ar
+ [Feexe (Br. (x:). u)
fecue (fe-At-r(X(--r) w)
Cia
by the gCD assumption, we have
A|| =
<E,
so that
3E| 1, 3R|| B, 2 < 65.
36

---

Published as a conference paper at ICLR 2026
For Cr,us using the Lipschitz properties,
|[Ct,all I Lip(fe (1) (xx) Po-tu (x-- -:)1 (x -x-AA-((x--A)))
S Ci
xx- xt- + AI),
hence
E||Ct,ull &
2Lip² (fo LLip (+) xi-aell2 + =AP) = 0(
Combining the pieces,
Bi [leeacn 1 = o
Case 4. Random Initialization: This is straightforward derivation from the assumption:
E [|e(Orana.))s E fl (x,, t)12 + E llfo (x,- At) - 2R.
F.6 BIAS-VARIANCE DECOMPOSITION
For the squared gradient bias and the CM's flow map gradient variance
()))= || Vo (0) VolcM(()) (0) V( = Varg(VolcM(0eee) E)].
Consider the oracle-relative mean squared error (MSE) of CM gradient:
{() Eg VolCM 0 +) V Vo Faracte (0) (0)
Then we have the following mean-squared errors comparison under the four different initializations: Corollary F.3. Under the same assumptions as Theorem F.I, the following CM gradient MSE bounds hold for the four initialization schemes (0 OCMT, ODM, ORCD, Grana).
(i) CMT:
E(fcMr) O (E ([++AP At At))
(ii) Diffussion Model:
E(0TM) = Os+ AL2 E.
+ Ex [1] [[ - E[xo |x,] 1|
(iii) General Consistency Distillation: For fixed [0, 17 assume in addition that
8y = Ex. f0OCD (Xu, 2) L ( 1| co.
Then
E(e_CD) 0 At2 + 5ww)
(iv) Random Initialization:
E(Grand. = -0(1)
Proof. Write the (vector) bias as
b(0) Volcm(0). Veloracle
Then
{() Eg Volcu((OEE Volcs - b(0))l2]
= Ee Volcm((:E) - VolcM + [lb(O)] +2E (ol b()))
(())
The cross term vanishes because Eg V (0 VolcM = 0. hence
E()) (Cove [Cove[VVoccs(0:E]]
Voloracle Volcm (0)
(0)
=B(0)
The remaining steps follow directly by combining the results of Theorems F.1 and F.2.
37

---

Published as a conference paper at ICLR 2026
F.7 COMPARISON ON OPTIMIZATION DYNAMICS
We consider plain SGD on the oracle objective using CM gradients. The iteration is
O x++1 03 v DeVeccMOO (Ox;
& Unif(o, 7) x pr i.i.d.,
with constant stepsize > 0. The expected oracle loss Loracte is assumed L-smooth and to satisfy a Polyak=Lojasiewicz (PL) condition the level visited by the iterates:
V of (0)
V V (6') S L|l0 121
V Voloracle (0)1 2 (looracte (0) (*)
2
for some μ 0. Here,
min loracle min
EE [| t) 1 (x)=0(x))11
0
We use the bias B))), variance V(O), and MSE
( = (O:E) Volorall [0)[[²] -((0) V(0)
as established above. We assume the stepsize satisfies & 1/(4)).
Theorem F.4 (SGD Analysis with Scheme-Specifie Initializations). Assume the conditions Theo- rem F.I, and further assume that ooarte is L-smooth, that the PL(()) condition holds, that the stepsize satisfies y) S 1/(4L), and that & is Lipschitz with constant Lip(E) on the level visited by SGD, Let p 2 1 be the global order of the ODE solver used in pretrainingheachee flows. For each initialization scheme, define
Ao == Eoracle (0o) -E"
Mo = =((0)))
Then, for any K 1,
ElEoracte (0x) (0x [*] & (1 - pm)) Ao + 5 P
Lip(E) VK Ao 5 Mo +
35 Lip(E) (15)
24
4u
Let C(n,K) =
4u Lip(E)² K² Then the initialization lemma (Lemma F:4) and the bias Sers-varaanee/MS-eda at-init bounds assumed in Theorem F.1 imply the following speeff- orders:
CMT:
Ao = O( At2p),
Mo O( A+ - APt).
Thus,
E Coracce(00 1-1 s (1 - (m) O(
21 + Vnk O(E + 1/2)
+ O(E + At + At") + C(n, K).
Diffusion Model (DM): We denote
MDM == Ex,x (xt) E[xo x 11
the deterministic-map versus posterior-mean mismatch. Then
Ao = ((s) +MDM,
Mo Mo=o(s+ A12 AC B(
+ + MDM-
Thus,
E (0x) -pa (11-) (MM)+ + Vnk (O(E) + MoxA 1/2
+
+ (0(s+ A12 + B. + +MDM) + +(1 K).
General Consistency Distillation (gCD):
Ao O(= + d., + At 2),
Mo = O(E + At2 6w))
Thus,
Piooratte (0x) [+] < (1 )u7)* K OlE + 8, + A12) + Vnk O(E + 8, + 2022)122)
+ O(E + At2 + 5w) + C(n,
38

---

Published as a conference paper at ICLR 2026
Random initialization:
Au = O1),
Mo 0
Thus,
1 Corucle (0x) - - - um)* 0() + Vnk O1)) + +11 + K),
All big-O constants are independent of At, E, and K.
All schemes enjoy the same geometric contraction factor (1- (-=);) differences arise solely through the initialization terms Au and Mo. Among them, CMT achieves
;
Pooract (0 (0k)-F] I (1-w7)
which contains no extra irreducible terms (such as MMM or ów). Consequently, while the asymptotic rate is identical across schemes, CMT attains the smallest excess risk (the tightest bound and lowest floor) for any K, up to the common term C11,, K).
The bound My in Equation (15) for each initialization scheme follows directly from Theorem F.I. To obtain a complete upper bound needed for the proof of Theorem F.4, however, we also require bounds on Ao for the four initialization schemes in Equation (15). In Lemma F.4, we establish such bounds for each Ag. We then return to finalize the proof of Theorem R4.
Initialization Excess Oracle Risk. We bound the initial oracle excess risk Earacle (0o) - E" for each of the four iitialiizaiion schemes. We now state and prove the initialization bounds.
Lemma F4. Under Assumptions B and C and following the notations therein, there exist constants C1 C2 < oo that do not depend on Al or such that
Eoracte (oruclee00ca) (CCMT) E" s 2E + C1 At
E Eoracto (MDM) F" s 2E + 2MDM,
Loracto (0 (0xCD) E" < 2 + +6, + +C2 2r
Poracle (aaacl((rann)) Orand.) - E" < 2R 2C+
Here any fixed choice suffices since they are absorbed in C2 in the final rates. In the realizable case E- these are direct bounds on the initialization oracle loss.
Proof. CMT. Let XT Ppriors Xt = ( &t = Solverr-w(XT), and &n oo For fixed 1,
[0owrr (x+;)) V vx-1o(xe)l12 S |oeeerr (Xx, foeeer (3i, 1]]lz + ||eeserr (Xe, x0l12
A2
+
[2 g ( L x
Ag
Ai
By Assumption B, Ai S Lip((oewr))l )|{; x2 Using the semigroup V Vi-0 o V T T and triangle inequality, A3 < 1|x vo (x)) || +
(x) Ve-- (x+) 1|2. By Assumption C,
A₁ < Lip(pp 1) X; xell2 and the second term in Ag also < Lip || - xell2. Now apply
(a b)3 < 20 26 to split (Ai + A₂ + As + A, into 24} + 2(A1 As + A,)2, then again
(u+++w}² 3(u + t + w²) on the second group to obtain
(0 < E
| (2,,t) - x 1 +C' ||x;
xx + 180 $ (x ||2
with == 3(L ) (Lip²((W)) Since the solver is of order p. the last two expectations are ((At2r). Therefore
Eoracte Eoracle(0cMm)) (0CMT) S 28 + 2 A1
Absorbing constants into C₁ and subtracting E gives the claim.
39

---

Published as a conference paper at ICLR 2026
Diffusion Model (DM). By the tower property, for any h(x;),
X hxxx)ll x
|[xxo| x + IE [x(()x))) E|xo x
With h = Don and LDM (M) < DeD Elxo x 11l < c. Thus by |1- S 21/a
ell + 2||c b12 with c = E| xt).
oracle (DDM) E De (xx,t) ( 11 s 2 2MM.
and subtracting L- yields the stated bound.
General Consistency Distillation (gCD). Fix [0, Let x, == Solver and x, == f (Xe). Define (z) := foocp (z, 21) V (z), which is (Lip(focco) + Lip -Lisschizz
by Assumption C and Assumption B. Then for fixed
|1oo (xx, 1) v Vx-oo(x+)112 S
foOLDD (xe; 1) fo (8a: u)|l2 + Fu((x))l2
B1
B2
P ( - Vu-ooxu)]1g
using the semigroup L 1--0 vu- -- By (b) < 2a 262 and then (b+c)² < < 26² +2c?.
Eoracto (OxCD) < 2EB + 4 4EB3 + 4EBB
The first term is controlled by the training loss: EBI LgOD 0 (DD:u <. For B2, by l|a b112 z
(1 + lall + (1 + [bb) with F. (x.)). b = F ., (x.)) - F (x.).
EBI (1 p) E F. (x..) 12 (1 1/p) (Tip
Lip( T 3
x. 11.
Choosing, egg. p and recalling 5, : Fx -p F (x,, 113. p-th order solver (see Assumption C) gives
EBI - 26, + (Lip(fe Lip v )
2p
For B3, by Assumption Cnd Assumption C, EBJ - Lip L At 2p Combining,
Roracto (OgCD) S 28 88 + (2(Lip((o,co)) Lip ()) Lip" (V)
A
Random Initialization. By I|a b112 < 2||a||2 + 2| and Assumption C, Toracle (f (frand.) = -FE| for (x,, t) L 00 (xx) 112 < 2 (x $
22++20
Subtracting 8" in each case yields the claims.
Proof of Theorem F.4.
Proof. Linear Contraction to an MSE floor. By the descent lemma for T-smooth Eoracle
Ln²
(oracte(0x+1) < Eoracle (Oxx n Vloracte VelcM((x:E:)) +
2 (0::
Taking conditional expectation w.r.t. Es, using Egg VolcM(Oni Ex)
Volcm(On) and
E VoocM (Ou: Ex) 1iae=IIVoool (V Tr Cove. (oev...oloo VolcM Oz; Ex)), together with
(Vooratte: VleM) 2 [|Vaaaate || VleM Vioracte||s
and Young's inequality, obtain
E Foraclo (0x+1) 0z] < Toracto (Ox)
37 V (0xra (0x),
where we also used y 5 1/(4T.) to absorb the Tn22 terms into the constants.
40

---

Published as a conference paper at ICLR 2026
Applying the PL inequality eliminate the gradient norm yields
Elforacte( (0x+1) 0z] - - (1-1 (foracle 0x)-e)) + I )
Taking total expectation and unrolling the recursion gives, for any K 2 1,
K
5
orraaee (0x)
E S (1 - pmn) K
(onn)(((oraat)) (foracte (0)) -) + Σ
pm)
(16)
k=
The bounds in Corollary F.3 provide E(0) only at initialization for each scenario. Thus, may need more additional assumptions.
First, a localized MSE stability assumption: there exists neighborhood N of the initialization in which the same order bound holds for E()), and the SGD trajectory remains in N under S 1/(4)).
Then suPoç*<K- (0x) S & with & of the same order as the initialization, which recovers the
floor bound
5
E Coracle
21 & (1 - 1mn)** (Foracle - 2*)
4
In this case, similar bounds can be obtained by applying Lemma F.4 to different initialization schemes of (oracle(On) Et.
Second, a mild continuity control: suppose & is Lipschitz on the level set visited by the iterates, i.e., [())) - E {00) s Lip(E) 0'112.
If. in addition, the step size ensures a bounded path length Σ B - 0x S R (which follows from El 0% l2 n (0p; Ex)ll2 and the same descent argument that bounds the average oracle gradient norm). then
sup 0
(0+)] s E(o)) + Lip(E)
Inserting this into Equation (16) gives data dependenn version in terms of the initialization MSE plus controllable growth term.
Proof of the Mild Continuity Control. Assume that E' is Lipschitz on the level set visited by {Ox}:
|E(0) E(0') Lip(E)| 0'112-
Fix horizon K 2 1. By telescoping argument and Jensen's inequality,
[[()(0.)] < 8(0))+
Σ 10 0,112 s (E(0))+Lipp
Σ B B[[10++1 0,112]
sup OSACK-1
Since 0ʃ+1 0, = - Velcc((;;;))
E 00 - 0, lla| = n/ (O,: & / [[[[Toccu((0;;:))]
Using E| Ve = Volcul : Tr ((,,) and
|V < 2/ &o 2 VolcM Voloructel
we get
E V CM
E;)1
2 V foracte
[0) + 2
-oo00)))-ooommai Coroect + E + I(0))]
<2 V foratte
0 28{E(0,)]
Therefore, by
K-
E 1[110+1--001
< Σ
2 oracto (0,) 2E +2{
1=
K-1
1/2
< k K
2E V foracat (0,)) + + 2
(17)
1=
41

---

Published as a conference paper at ICLR 2026
Summing the one one-step decrease inequality
E ooacle [0,) E8 [oracte (0++)) 2 " (el.orrce(()))) (
from j to K - 1 and using Elforacto (0x) l" yields
5 K-1
[I] MiVVeooraco 0,) S (ooract (0o) (2)
2 (Coracte(O))
Σ
E3
[E(())]]
1=0
j=0
Let EK suposjsk- (O)( Then
K-1
2 < (faracte (0o) -F-) + KE
5
2
Substituting this and E, [[E{(,)) S K& into Equation (17) gives the path-length bound
K-
B[10/+1- -,112a - nvKK ( (Foracte (00) -8²) + 7KEEE) 1/2
1=
(18)
s 2 n (Poracte(()) -)+ + V 12/2
where the last inequality uses Va va+bs V Vo. Combining Lipschitzness and Equation (18) yields
EK s E(0,) + Lip(E) (2 nnK ( (Foracte (0o) (+) ) + VinkE* 122
This is of the form < + 8/8 with = Ex. u E(6,) + 2Lip(EE) V n W -00) and v (7 The inequality S a + v implies S 2u + ?2 (complete-the-square argument). Hence
ΣK < (0.) + 4Lip(E) I (Foracte (0.) - + TL?p(E)?,²??
(19)
Plugging Equation (19) into the pathwise contraction yields
B[ooract (( {*]
5
<(1 -mn)* K (Eoracte(Oo) -F") (2E(0) + (Lip((E) (Foracle -8+ + 7Lip(E)3))
4p
5
5 E(%)
35
=(1 - -m7) (Foracto (0) 2+) + Lip(E) 'K (f0) E*) +
+
Lip(E)2,2K?
2/a.
Ap
This is a data-dependent bound in terms of the initialization MSE E(o)), the loss gap Eoracle (0g) - 8r and the Lipschitz constant Lip(E).
42


Reviews:
[{"review_id": "C008C0afcS", "reviewer": "Reviewer_3ZTM", "summary": "The main contribution of this paper is the introduction of an intermediate training stage for accelerating the training of fast inference generative models for image generation in continuous signals.\n\nCurrent state-of-the-art methods for distilling flow and diffusion models without GAN losses such as ECD [1], and MF [2] typically employ a two-stage training pipeline. In Stage 1 (pre-training), a diffusion model is trained with an x-prediction parameterization [3] or a flow matching model with a velocity-prediction parameterization [4]. In stage 2 (post-training),  the trained diffusion/flow model is distilled by approximating the flow map it defines. However, optimizing the exact objective function for the flow map is intractable, so instead an auxiliary loss is used, commonly involving a stopgrad instance of the current student model which can lead to unstable training. \n\nThis paper proposes a third stage, mid-training, for distillation of flow and diffusion models. The proposed mid-training loss regresses the flow map defined by the pretrained model using pre-sampled trajectories. Finally, the effectiveness of the mid-training stage is validated on several benchmark.\n\n[1] Geng, Zhengyang, et al. \"Consistency models made easy.\" arXiv preprint arXiv:2406.14548 (2024).\n\n[2] Geng, Zhengyang, et al. \"Consistency models made easy.\" arXiv preprint arXiv:2406.14548 (2024).\n\n[3] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.\n\n[4] Lipman, Yaron, et al. \"Flow matching for generative modeling.\" arXiv preprint arXiv:2210.02747 (2022).", "questions": "1. In line 214 it is written that the discretization number of steps $T$ is fixed for the mid-training. First, did you examine how different choices of $T$ affect the training and the final results? Second, in the post-training stage ( i.e., ECD [1]/MeanFlow [2]) a continuous time is used again?\n\n2. How many different trajectories $\\\\{x_{t_0}, x_{t_1}, ... ,x_{t_T}\\\\}$ are used during mid-training and does the time to generate these is taken into account in your mid-training times report?\n\n3. It is claimed in the paper that training of vanilla ECD [1]/MeanFlow [2] remains \"unstable, and configuration sensitive\", how ever it seems that the authors did train a MeanFlow model with relatively close performance to the CMT (the proposed method). Can the authors provide an estimation of how many training attempts hyper-parameters searches are required to obtain the reported results ECD [1]/MeanFlow [2]?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and the method presented clearly.\n\n2. The paper goal - accelerating distillation of flow/diffusion models,  is aligned with current most relevant problems considered in deep learning, potentially making large impact.\n\n3. Introducing a new training stage is a popular method for improving training of neural networks and can easily integrate with existing training pipelines.\n\n4. The empirical validation is comprehensive.", "weaknesses": "1. Regressing a peace wise linear version of the pre-trained model's trajectories was done in previous work e.g., [5], and might be worth mentioning.\n\n2. While the preliminary section covers number of previous work, the main paper does not include a previous work section (instead it is in the appendix) which is somewhat unorthodox for a research paper.. \n\n[5] Nguyen, Bao, Binh Nguyen, and Viet Anh Nguyen. \"Bellman optimal stepsize straightening of flow-matching models.\" arXiv preprint arXiv:2312.16414 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5Yq2Jc22w", "reviewer": "Reviewer_GbKZ", "summary": "This paper proposed a mid-training strategy. The motivation is to alleviate instability in training for the few-step trajectory distillation method, such as the Consistency model (CM) and MeanFlow. Such unstable training is inherent in approximation errors for the trajectory from $t->s$. This paper proposed a CMT loss to first provide a sub-optimal trajectory from $t->s$ with lower approximation errors, which significantly simplifies post-training trajectory distillation.  The experimental results demonstrate that such a mid-training strategy can: 1) achieve the few-step generation built up on the trajectory distillation method, such as CM, 2) boostraps the post-training for CM and MeanFlow.", "questions": "1. A minor question: Can this method be used for a large-scale model such as SD 3.5 and Flux?\n\nTo sum up, this paper proposed a novel mid-training strategy to mitigate the training cost for few-step trajectory-based distillation methods. I have carefully checked the theory proofs and related code. I think there are no major problems. The overall process is novel and sound. The experimental results clearly demonstrate that CMT can reduce the training cost and improve the stability of trajectory-based distillation methods. Therefore, I rate it as accept.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper solves an important challenge about how to train a trajectory-based distillation method in a stable way and at a low training cost.  These distillation methods rely heavily on theory, which is sensitive to approximation errors. The mid-training strategy is a way to remedy this situation.\n\n2. The proposed method is interesting and novel. Such a mid-training strategy provides a good starting point for the trajectory-based distillation method. The experimental results also reveal that the training cost during mid-training remains low compared to that of the direct training of the trajectory-based distillation method.\n\n3. The experimental results demonstrate that CMT with CM can achieve the one-step generation and reduce the training cost.", "weaknesses": "I think this paper does not have significant concerns. I only have minor concerns.\n\n1.  While the author claims that $x_{T}$ is the terminal state. I refer to the code but find that $x_{T}$ actually comes from GT + noise. In this case, to directly claim that $x_{T}$ is the terminal state which differs from \"a fresh data point $x_{0}$ and a noise vector $\\epsilon$\" (205 lines) may mislead the reader. \n\n2. For the choice of the discretization time step. Is there any way to quantify the approximation errors that will be generated in different discretization schedulers?\n\n3. I think this paper is related to the progress distillation [1]. Although the targets of the two methods differ, the author should slightly distinguish them using CMT. \n\n[1] PROGRESSIVE DISTILLATION FOR FAST SAMPLING OF DIFFUSION MODELS. Tim Salimans et al. ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBrgZINUgZ", "reviewer": "Reviewer_12GL", "summary": "This paper proposes a mid-training stage to stabilize the consistency trajectory model training. The proposed method can achieve competitive sample quality while reducing training cost.", "questions": "Did you use Eq. 6 or Eq. 7 in the experiments?\n\nDid you train ECD with general flow map post-training or special flow map post-training? ECD originally only supports the special flow map.\n\nIsn't Eq. 6 the same as ( L_{\\text{var}}^{(1)} ) in L1108? (the slow CMT)\n\n> However, the Slow CMT’s mid-training stage costs 3x more GPU time since generating the regression target in the Slow version is more costly due to the failure to use intermediate points by the ODE-solver.\n\nDo you collect more trajectories for the slow version? Otherwise, the GPU hours should be the same. If we collect the same number of trajectories to match their GPU hours, does the slow version underperform CMT?\n\n( L_{\\text{var}}^{(1)} ) (or eq 6) seems the same as 2-rectified flow loss. If we apply improved techniques for 2-rectified flow from [1], can it improve CMT's performance? It seems that [1]'s FID on CIFAR-10 is already pretty similar to CMT's FID, so it may offer even better initialization for the post training.\n\nTypo: in line 1116; it should be ( L_{\\text{var}}^{(2)} )?\n\n[1] Lee, Sangyun, Zinan Lin, and Giulia Fanti. \"Improving the training of rectified flows.\"", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Evaluation is done on multiple datasets, including a high-resolution dataset (ImageNet 512x512).\n* The method rivals 2-step FID sCD with a 14x reduction in training cost.\n* The idea is simple and easy to implement.", "weaknesses": "One weakness is that 1-step FID on ImageNet 512 is worse than sCD by a non-trivial margin (2.28 vs 3.38). It is understandable as CMT uses fewer resources, but can it match the 1-step FID of sCD if given more compute?\n\nTheoretical analysis (Sec. 5) does not offer significant insight. It says that if we train a model to predict the PF-ODE solutions, it will better approximate the PF-ODE solutions, which is straightforward. I suggest moving this part to the appendix and instead using that space to include contents from the appendix (as it contains some valuable insights and experimental details).\n\nSome experimental setups are not clear. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KCMJF0SUgi", "reviewer": "Reviewer_BkLe", "summary": "Flow map models such as Consistency Models (CM) and Mean Flow (MF) enable few-step generation by learning the long jump of the ODE solution of diffusion models, yet training remains unstable, sensitive to hyperparameters, and costly. Initializing from a pre-trained diffusion model helps, but still requires converting infinitesimal steps into a long-jump map, leaving instability unresolved. This paper introduces mid-training, the first concept and practical method that inserts a lightweight intermediate stage between the (diffusion) pre-training and the final flow map training (i.e., post-training) for vision generation. Concretely, Consistency Mid-Training (CMT) is a compact and principled stage that trains a model to map points along a solver trajectory from a pre-trained model, starting from a prior sample, directly to the solver-generated clean sample. It yields a trajectory-consistent and stable initialization. This initializer outperforms random and diffusion-based baselines and enables fast, robust convergence without heuristics. Initializing post-training with CMT weights further simplifies flow map learning. Empirically, CMT achieves state of the art two step FIDs: 1.97 on CIFAR-10, 1.32 on ImageNet 64×64, and 1.84 on ImageNet 512×512, while using up to 98% less training data and GPU time, compared to CMs. On ImageNet 256×256, CMT reaches 1-step FID 3.34 while cutting total training time by about 50% compared to MF from scratch (FID 3.43). This establishes CMT as a principled, efficient, and general framework for training flow map models.", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Mitigates issues of training instability, hyperparameter sensitivity, and high training costs that plague models like Consistency Models (CM) and Mean Flow (MF)\n\n2. Proposes a compact, principled intermediate stage (Consistency Mid-Training, CMT) between diffusion pre-training and final flow map post-training, providing trajectory-consistent and stable initialization—filling a critical gap in existing training pipelines.\n\n3. The CMT initializer outperforms random and diffusion-based baselines, enabling fast, robust convergence without relying on heuristics.", "weaknesses": "1. Lack of large-scale experiments, e.g., text-to-image, text-to-video. This reduces the credibility of the pipeline.\n\n2. The contribution is quite incremental. \n\n3. I have doubts that the performance gain stems from the unfair experimental setup, since the latent baselines do not use LatentLPIPS in training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)