Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch
Authors: Xu Cai
Yang Wu
Qianli Chen
Haoran Wu
Lichuan Xiang
Hongkai Wen
Venue: NeurIPS 2025 poster ()
Keywords: diffusion
flow-matching
distillation
fast inference
Primary area: deep_learning
TL;DR: We propose a highly efficient distillation algorithm for flow matching models that requires minimal computation and enables few-shot training.

Abstract: We present an ultra-efficient post-training method for shortcutting large-scale pre-trained flow matching diffusion models into efficient few-step samplers, enabled by novel velocity field self-distillation.  While shortcutting in flow matching, originally introduced by shortcut models, offers flexible trajectory-skipping capabilities, it requires a specialized step-size embedding incompatible with existing models unless retraining from scratch—a process nearly as costly as pretraining itself.

Our key contribution is thus imparting a more aggressive shortcut mechanism to standard flow matching models (e.g., Flux), leveraging a unique distillation principle that obviates the need for step-size embedding.  Working on the velocity field rather than sample space and learning rapidly from self-guided distillation in an online manner, our approach trains efficiently, e.g., producing a 3-step Flux <1 A100 day.  Beyond distillation, our method can be incorporated into the pretraining stage itself, yielding models that inherently learn efficient, few-step flows without compromising quality.  This capability also enables, to our knowledge, the first few-shot distillation method (e.g., 10 text-image pairs) for dozen-billion-parameter diffusion models, delivering state-of-the-art performance at almost free cost.

Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch
Xu Cai
Yang Wu
Qianli Chen"
Haoran Wu"
Lichuan Xiang
Hongkai Wen
Abstract
We present ultra-efficient post-training method for shorteutting large-scale pre- trained flow matching diffusion models into efficient few-step samplers, enabled by novel velocity field self-distillation. While shortcutting in flow matching, originally introduced by shortcut models, offers flexible trajectory-skipping capabilities, it requires specialized step-size embedding incompatible with existing models unless retraining from scratch-a process nearly as costly as pretraining itself.
Our key contribution is thus imparting more aggressive shortcut mechanism to standard flow matching models (e.g., Flux), leveraging a unique distillation principle that obviates the need for step-size embedding. Working the velocity field rather than sample space and learning rapidly from self- self-guided distillation online manner, our approach trains efficiently, e.g., producing 3-step Flux <1 A100 day. Beyond distillation, our method can incorporated into the pretraining stage itself, yielding models that inherently learn efficient, few-step flows without compromising quality. This capability also enables, to our knowledge, the first few-shot distillation method (e.g., 10 text-image pairs) for dozen-billion-pare parameer diffusion models, delivering state-of-the-art performance at almost free cost.
Introduction
Recent advancements in accelerating diffusion models Soong et al.] (202Tb). Ho et al. (2020) have significantly reduced the number of denoising steps required for sample generation without compromising quality. For instance, while DDPM Ho et al. [2020) typically requires 1000 sampling steps, DDIM Song et al. [2021a] achieves a substantial speed-up with as few as 50 steps. Nevertheless, as pre-trained diffusion models Podell et al. [2024), Esser al. [202]] BlackForestLabs 2024
continue to scale to (tens of) billions of parameters, the computational cost remains a critical bottleneck for real-world applications. Most recent diffusion models favor Flow Matching (FM)
Lipman et al. [2023], also known as rectified flow Liu et al. [023], due to its simplified Ordinary Differential Equation (ODE) solver and exact likelihood evaluation. FM achieves this by learning velocity field that deterministically maps Gaussian noise to clean data along a (nearly-)straighe flow trajectory. However, detailed in Section 2
only handful of existing works explicitly optimize the
distillation objective specifically for flow matching models.
Among the related works, shortcut models Frans al. 20225] stand out for their unified approach to train both multi-step and one-step diffusion with a step-size embedding, allowing the model to inherently support both modes once properly trained, as further detailed in Section 3.2 The core idea of shortcut models to create direct shortcuts along potentially curved trajectories, by always
"The authors conducted this work independent researchers, pursued as hobby and without institutional atfiliation.
AI Research Center, iHuman Inc..
'Department of Computer Science, University of Warwick. Corresponding author: cai ceitrreeegaall... com.
Project page: wwwptutff...titttt...tt
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

predicting the mean direction between two equally spaced timesteps. The challenge, however, is that the aforementioned property missing in all existing pre-trained FM models. As result, naively applying shortcut methods to standard flow matching would require architectural modifications or complete retraining from scratch to integrate step-size conditioning.
Motivated by this, and inspired by related works on velocity matching (refer to Section 2 for more details), we introduce novel framework for ShoriCutting Flow Matching (SCFM) through a highly efficient distillation paradigm. In contrast to majority existing distillation approaches, SCFM inherits the end-to-end training manner from shorteut models, while differing substantially in distillation principle, how step-size information is incorporated, and training algorithm design.
To elaborate on the high-level idea behind this distillation principle, obvious obstacle to few-step sampling in flow matching models lies in the inconsistency between their theoretical formulation and empirical behavior. We ask: what if the entire non-linear velocity field between noise and data, spanning a long time horizon, could be uniformly forced to a nearly straight trajectory (e.g., via distillation)? In such cases, the explicit step-size parameter may become unnecessary, as the model would now align with the theoretical principles of rectified flow, which naturally supports efficient arbitrary-step sampling. The detailed intuition can be found in Remark With this in mind, we summarize our key contributions as follows:
key insight is that an
We propose SCFM that operates in velocity space, enforcing linear consistency across all timesteps. This consistency is jointly derived from both the teacher and an online-inherited student through a novel dual-target distillation objective. Notably, our approach eliminates the need for explicit progressive distillation; instead, an end-to-end training scheme automatically straightens curved velocity trajectories. In contrast to shortcut models, our method does not rely on an explicit step-size parameter to regulate varying velocities.
SCFM benefits significantly from self-distillation, enabling highly efficient training and entirely removing the need for massive dataset to mimic teacher models-a requirement common to most prior approaches. To demonstrate the novelty and effectiveness of few-shot SCFM, we validate using as few as 10 training samples, yet still achieved competitive performance. To our knowledge, this represents the first successful demonstration of few-shot distillation for large-scale diffusion models.
Our method is designed to generalize to any pre-trained flow matching model. To validate this, we successfully distilled 12B- 2B-parraettrr Flux model BlackForestLabs (2024) into a 3-step sampler within a single day on an A100 GPU, achieving SOTA performance in both quantitative scores and visual quality, without the aid of adversarial distillation (Sauer et al. [2024b standard component in all baselines.
2 Related Work
We follow the categorization in recent survey Fan et al. [0255] existing pre-trained diffusion distillation methods can be broadly classified based on their training objectives: output/distribution based or trajectory-basee -based. Output-based distillation adopts straightforward approach: minimizing the discrepancy between the outputs of the teacher and the student. These methods typically employ sophisticated loss functions. For example, Progressive Distillation (PD) Salimans and Ho [2022 uses
an lg loss, while other methods leverage information theoreetic losses such as KL divergence Yin et al. 2024 or Fisher divergence Zhou et al. [2024), just to name a few.
Trajectory-based distillation, on the other hand, focuses on the transformation path from noisy inputs to clean images. The pioneering work on Consistency Models (CMs) [Song et al. (2023) introduced the idea of mapping any noised data point along the trajectory directly to its clean counterpart, enabling one-step diffusion. More recent approaches extend this idea by targeting near-future noisy states in the trajectory, rather than the final clean image, as the distillation target. This adjustment results in more practical and efficient training strategies for popular pre-trained models such as SDXL Podell
et al.
2024 Notable examples include Luo et al. 2023 Wang et al. 2024 Kim et al.
2024
Ren
al.
2024 Wang et al. (2025). and many more.
More recent and advanced pre-trained diffusion models, such as SD3 Esser et al. (024 and Flux
BlackForestl abs] [2024), adopt flow matching (a.k.a., rectified flow) Liu et al. [2023] as their
underlying trajectory, contrast to previous approaches like EDMs Karras et al. 2022]. The notable advantage of rectified flow is its natural compatibility with few-step diffusion once properly

---

trained. As hinted in Section 1 in practice, however, conditionl models often deviate from this idealized linear trajectory. especially high-noise regions where the mapping often exhibit curvature. Consequently, most of the above mentioned distillation algorithms, which were originally developed for conventional trajectories, are not yet well suited to flow matching models.
Despite this limitation, InstaFlow Liu et al. 2024bb, a successor to rectified flow Liu et al. 2023) introduces two-stage training strategy: pre-training with the rectified flow objective followed by velocity-based distillation. What sets InstaFlow apart is its distillation is carried out in the velocity field, which contrasts sharply with strategies that rely on denoised data targets. Perhaps the most relevant work to ours is the shortcut model Frans et al. (2025), which unifies one-step and multi-step diffusion pre- pre-training by learning in the velocity field and conditioning on step-size size throughout training. Their shortcut principle motivated our approach, which leverages velocity jumping through a novel distillation targets tailored to pre-trained flow matching dynamics.
Other orthogonal approaches for achieving effective flow matching distillation include LADD Sauer ct al. [2024a], a predecessor the well-known Adversarial Diffusion Distillation (ADD) Sauer et al. [2024b]. which introduces a GAN-like objective Goodfellow et al. [2014] based on deceiving a discriminator. Notably, ADD-style methods are orthogonal to most diflusion distillation techniques, as the additional adversarial loss is intended to enhance the alignment between the teacher and student. SANA- Chen et al. [2025] takes a different route by substituting the original flow matching trajectory. Instead, it adopts TrigFlow Lu and Song [2025). generalized trajectory framework that unifies flow matching and EDM under continuous-ting time consistency. Their approach is then combined
with LADD to further improve generation quality.
3 Preliminaries
3.1 Flow Matching Models
Given a data sample Xy D drawn from data distribution D, flow matching (a.k.a, rectified flow) diffusion models learn reverse the noising process through the following continuous-time trajectory:
X = (1 - t)xo + tz,
(I)
where E [0, 1] and z ~ ~N, I). The forward noising process evolves from to t 1, gradually perturbing Xu into z. For notational convenience, we let X1 represent the fully noised version of xg- The corresponding ODE with respect t takes a simple linear form:
0x.
= X0.
(2)
öt
where V: is referred to as the velocity. Given training pairs {xo. }1 our objective is to predict v, (x(,,,t using neural network Ve with parameters 0. We note that Vo(x,, t) denotes the minimal form of the velocity predictor, while a fully parameterized version may be written as Vo (xe, t, w). incorporating a condition (e.g., prompt) and Classifier-Free Guidance (CFG) scale w Ho and Salimans [021].
is unbiased. Consequently, the Mean Squared Error (MSE) of the estimator v, known to minimize the variance, meaning that the estimator with the least variance is considered the best in terms of MSE Finally, the neural network Vo can be optimized by minimizing the following loss function:
Since follows from the normal distribution, the estimator v Elv, Xx, 4
()) Ex Xa N(..)
[ - -v.ll]]
| ((1 t)xo + txi,t) (xi - xo) x=]
where 44 follows from substituting the trajectory II and velocity ( definition into (
The inference procedure solves the ODE backward from f to t 0. An re-tep sampler employs
n + 1 predefined timesteps {tz h. where = to > t1 > ... > fn = 0. Samples are then generated through iterative updates:
X1,.i = XX,- - (t: - t4+1 Vo ts), X ~ ~N(,I)
(5)
with final output Xt., as the generated sample.
3

---

3.2 Shortcut Models
As discussed in Section matching by incorporating a variable step size d (e.g., 1/n). The key property is self-consistency across different step sizes:
2 shortcut models Vx t, d) Frans al. [2025) generalize standard flow
X++2d Xt 2dVe (x,, (,,tt,2))
(6)
2 X² dV'e (x:: t, d) dV'e (X d, d,d), 0 0< ocds
(7)
2
where 6 and reflect the compositional property that one 2d-step prediction equals two consecutive d-steps. Equating 6 and 7 and scale both sides with y leads the self-consistency loss:
Lsc ( Ve (xt,t, 2d) Ve- (x1, (x)))
(
where 0 denotes the stopgrad version of 9 (typically maintained via exponential moving average (EMA)), and the target V/- (x,, t, 2d)) is computed
1 =2oo- Vo- (1-xxx t d, d). 2 Vo- (x,, t, d) +
1
Ve- (x,, t, 2d)
2
(9)
Shortcut models are thus optimized through jointly minimization of:
The flow matching loss 4 (with 0), ensuring basic generation capability.
The self-consistency loss facilitating shorteut abilities via varied step sizes.
During training, step sizes d in uniformly sampled from { s....:3 to promote multi-
scale consistency. To effectively encode rotary positional embedding Su et al. [2244 is employed to preserve the relative scaling relationships between different step sizes.
4 The Shorteut Distillation Method
Inspired by (). and we propose to implicitly train the awareness of in Vo Vo (x,, rather than
including it as an explicit variable in Section 3.2 As hinted in Section [ this naturally encourages
the trajectory to become nearly straight, rather than merely relying on shortcuts along curved path.
A natural adaptation would extend the framework of PD Salimans and Ho [2022] to operate velocity fields instead sample fields, giving rise to an iterative teacher-student distillation process. The key challenge, then, lies in preserving compatibility with standard diffusion architectures while accommodating variable step sizes. To this end, consider a sequence of n time intervals (d; ) (which may be non-uniform) satisfying
Σ d 11
d3 = 4--11111: 44-
(10)
Building on the self-consistency principles from 6 and 7 we derive the velocity space consistency: (d: d 1 (x (x,,,)))
(xr, 4s) d:+ Vo (xr,11)4+1))
(11)
Rearranging terms yields the following form of our distillation target:
dj
di+1
Ve (x (4) = (d, + d Vol (x,,, 1;) + (d, + d=+1)
Vol (x
(12)
where slightly abuse notation by denoting the left-hand side the training target, even though it shares the same symbolic form as the intermediate outputs on the right-hand side. Similar to (9). for the target computed through the teacher and the EMA model, we replace all instances of 0 in (12) with 0* and 0 differentiate the sources.
Equation 12 captures the weighted interpolation of velocity fields across time intervals, forming the foundation for our shorteut distillation process. Next, if one adapts the progressive distillation pipeline, the first stage would begin with fine-grained step sizes d,, where Vo- (xt,, 42) represents the velocity prediction directly from the teacher model. The distillation then proceeds through stages

---

Ve- + Vo- ..... ... with increasingly coarser grained d, However, this naive approach may face challenges in determining optimal transition points between stages, considering the propagation and amplification of approximation errors.
Recall from Section 3.2 that shorteut models ground their learning small step sizes (d 0) via
[log, "/-1) via (8). Such joint optimization is (4) while self- colffbooostrappingggg at larger steps (d E { Tiz0
theocratically equal progressive optimization: first training the model a small step size, and then learning from the previous model at increasing step sizes. Motivated by these insights, we propose
the ShortCut distillation for Flow Matching (SCFM) objective:
Lacfin (0)
Σ (vo(x):)) - Vo- (xxx, N
1
+
Σ 1=++1
(vo(xi.)) Vo- (xx..))
(13)
where Vo- (xx; t) and Va- (xx,)) are calculated through controls the teacher/self-teaching mixing ratio.
12
.N is the total batch size and 0 l: N
Remark 1. The the SCFM loss Lurfm be seen generalization of several existing distillation
methods. When = N. it encompasses a broad class of existing trajectory-basee distillation approaches (see Appendix Bor details); setting h 0 corresponds to a progressive-style distillation scheme. Analogous to shortcut learning, the first term in (13) transfers knowledge from the teacher model a coarser student, while the second term self-distills this knowledge into finer-scale students. This enables implicit progressive self-distillation as suggested in Section [ leading to asymptotic convergence toward one- or few-step sampler within a single training phase.
To build intuition, consider a teacher model with n diffusion steps. The first term trains a student at ~n/2 steps to coarsely rectify the velocity direction. The second term, the same time, enforces consistency between this n 2/2-step student and finer-scale versions (c.g., n /4 steps). By randomly sampling target steps from (n/4, n/8,. ... 1} in the second term, the loss ensures self-consistency across all these finer scales. Interestingly, even if all finer-scale directions are initially wrong, they can still be rectified through the first term via teacher guidance. Consequently, the second term rapidly straightens the overall trajectory, while the first term corrects the direction w.r.t. the teacher.
4.1 Training Details
We update the EMA of the stop stop-gradient parameters 0 using the conventional update rule:
= 0 - (1-4)0, with μ 0.999 by default.
(14)
To enable efficient post-training, we adopt low-rank adaptation (LoRA) Hu et al. 20022) where the model parameters are expressed as O, + AO, with Og denoting the frozen pre-trained weights and A the trainable LoRA. Let A denote the LoRA parameter corresponding to the stopgrad model. Since 0- 0, + A0 we can derive the EMA update rule for the LoRA parameters as:
A
0 6n
00 + (1 1100 Ou
= (00 + A0 ) (1-1) + A0) -0,
(15) where we have used both the update for 0 in (14 and the LoRA decomposition of 0. To further accelerate training, may apply a cyclic restarting strategy to (1)), where A0- is reinitialized with the current A every fixed number of iterations (e.g., every 1000 steps). For a more in- understanding, refer readers the essential Section [5.4] which offers key insights into the final SCFM training strategy that further improves convergence speed. The vanilla training algorithm is provided in Algorithm T located in Appendix A
= -A0 + p)A0,
5 Experiments
In this section, we present empirical studies to evaluate SCFM. We begin by showcasing numerical and visual comparisons, followed by comprehensive ablations in Section 5.4 (including few-shot learning, detailed in Appendix F that explores how to achieve the most effcient and performant SCFM configuration.
5

---

Table 1: Comprehensive comparison with SOTA approaches. Latency is measured with resolution 1024 x 1024 BF16 precision on A100, averaged over 10 runs. The timing specifically reflects
the duration spent within the transformer block loop, excluding pre/post-processingg overhead. We highlight the best and second-best results across all metrics.
Latency (s)
FID J
Methods
Steps
(A) FID
CLIP
(w.r.. base)
Base
SD3.5-Large
32
15.14
18.62
N/A
34.97
Distillation- based
SD3.5I.-Turbo
8
1.95
+7.03 (25.65)
8.18
33.81
SD3.5L-SCFM
8
3.71
+0.32 (18.94)
2.65
33.91
SD3.5L-Turbo
4
0.94
+6.36 (24,98)
6.98
33.03
SDD35L--SCMM
4
+4.45 (23.07)
6.89
33.40
SD3.5I.-Turho
3
0.71
+6.85 (25,47)
7.76
32.25
SD3.5I.SCFM
3
1.31
+5.35 (23.98)
7,41
32.46
Base
F-ux.
32
15.62
27.43
N/A
33.60
Flux-TeaCache TeaCache- 0.6
32
5,36
-2.25 (25.18)
2.18
33.24
Flux-TeaCache TeaCache 0.8
32
4.33
-3.15 (24.28)
3,56
32.77
Flux-Hyper-SD) Flux-TDD
8
3.71
+1.37 (28.80)
3.20
33.46
8
3.71
-0.37 (27 27.06)
4.02
33.17
Distillation- hased
Flux-SCFM
8
3.71
+0.16 (27.59)
2.58
33,76
Flux-Hyper-SD
4
1.80
-0.64 (26.79)
5.45
32.94
Flux-TDD
1.80
-262 (24.81)
5,50
32.57
Flux-Schnell
-6,41 (21.02)
6.76
33.17
Flux-SCFM
4
-0.45 (26.98)
4.50
33.20
Flux- Mix--Hyee-SSD
3
1.33
-1,52 (25.91)
9.65
31.95
Flux-TDD
3
1.33
-4,66 (22.97)
8.26
31.38
Flux Flux-Schnell
3
1.33
-6.58 (20.85)
7.06
33.06
Flux-SCFM
3
1.33
-1.01 (26.42)
6.34
33.10
5.1 Training settings
We conduct experiments filtered subset of the LAION dataset Schuhmann et al. (2022). specifically the LAION-POP dataset[Laion [2024], which contains 600K samples in total with aesthetic scores >0.5. For evaluation, adopt the widely used COCO-300 validation set Lin et al. 2014). All training and evaluation are performed on a single NVIDIA A 100 80GB GPU.
We use two large-scale, high-capacity pre-trained flw-matchinn models teachers: Flux.1 Dev (12B) BlackForestLabs [2024] and SD3.5 Large (8B) Esser
( et al. [2024]. Our method successfully distills a 32-step Flux teacher into 3-step student in under 24 A100 GPU hours, demonstrating remarkable training efficiency. For comparison, progressive-style distillation methods typically require thousands of GPU hours (alimann and Ho 2022 highlighting the practical efficiency of our approach. All experiments are conducted on images resized to approximately 512 x 512 in area. observed performance degradation when [ptimizerLoshchhlloo
Since our method operates in the velocity space,
evaluating at 1024 x 1024 resolution. We use the Adam optimizer Loshchilov and Hutter (2019) with learning rate of 2e -5 batchsize of 16, and set $/4 =0.4 (13). chosen bootstrappingly.
Since Flux.1 Dev, denoted as V(x, ł, w), is already CFG-distilled using an embedded guidance representation, we simply randomly sample the CFG scale from the range the student and teacher share the same w for consistency. In contrast, SD3.5 does not incorporate CFG embedding, instead, we sample CFG values from the range [3.5, 5] (as we aim retain flexibility at inference time by supporting adjustable CFG scales), rather than distilling to fixed value, As result, our method requires CFG- con-conii formulation for all velocity predictors V(x, c) appearing in [22 As in Ho and Salimans [2021]. this can be expressed as:
[0, 8), ensuring both
(16) where 0 denotes the unconditional (null) prompt. During inference. we find that setting the CFG scale within the range [4.5, 6] yields stable and reliable results for both models. Alternative methods
V(x, 1, c) = V(x, I, c) + w (V(x,(,c) 4(x, ()),
"This dataset is publicly available; as specified in Section 5.4 since our method converges rapidly, we used only a random subset (<50% of full dataset).

---

FLUX-Dev
TeaCache-..
HyperSD
TDD
Sannlll
SCEM (Ours)
Stoppt
ultrareshate factosy world where growly is disfed-contin, flot amony condy-cmlored enfored ciouds, forming deromy citrrage stop giont, gentle tutter. Woterfott pour from the sky, nourihhing wbrant end ofr eocsystonss soft pastel polette and intricats detatls evole piayful surreaismi ond boundless imaginztion.
&topps
reolitic painting robot
reflections
3 5.Stpp
fflllo estronout glowlng flowers awllght Crocked viar seft lighti diotuminescent
covered ship touo flowering olien growch serenn, hoanting, survea!
Stopps
MAMAAAN
élegont tounmatine watermeion block oociground, becutful
substartiol
Figure 1: Visual comparisons on Flux: samples from the original teacher and the TeaCache- accelerated variant uses 32 sampling steps.
that rely CFG embedding can reduce inference cost by avoiding a separate unconditional pass (thus halving runtime), but typically require architectural changes and substantially more training from near-scratch. While both approaches are expected to yield similar generation quality, we opt for the simpler, backward-compatible strategy.
5.2 Evaluation Metrics and Baselines
Our evaluation primarily focuses on the fidelity between the student and teacher prodels-specificet how well the student shorteuts (i.e., preserves) the behavior of the teacher. Additionally, we assess the standalone quality of the distillation outputs ensure that acceleration does not come the cost of generating visually compelling and semantically adherence results.
To this end, we deviate from conventional FID Heusel et al. [2017) evaluation against open reference dataset (as commonly done prior works), instead, we propose to compute FID between teacher and student outputs under fixed random seeds. This allows to directly quantify how much fidelity is lost when drastically reducing the number inference steps. This teacher-studens FID serves better proxy for evaluating distillation fidelity, especially when the teacher itself is already a strong pre-trained model. As a complement to the similarity measurement between the student and teacher models, evaluate the visual quality and semantic alignment of the accelerated/distille model using CLIP/Radfinn et al. [2021), equipped with the checkpoints from Koukounas et al.) [2024] the official OpenAl versions (both base and large), due their improved image-tex. text alignment (see Appendix G for detailed comparisons).
Although diffusion distillation is widely studied, to the best of our knowledge, prior work has conducted distillation experiments on recent large-scale diffusion models like Flux SD3.5-apan from the official releases of Flux-Schnell BlackForestLabs 22024) and SD3.5-Turbo Esser al.

---

SD&S-Large
- -urbo
SCEM (Ours)
SD35-Large
SD35L-TTurb
SCEM (Ours)
8 Stoppe
Stepp
of halding brass that transforms computer chi,, symrsticing the
Avbrant nestrr, feeninng the caln surfor arder bright sunth reflertons tygllgher on plms, gentlr bobwn light water
connection betwnns hutron inteligence criiicaal inteligence, dark dsckground of worid!
Stope p
Steps
squared planet suffore borkgmunt plowin
hot sprigg.
bright
tiec'
siogulor
acneost
Figure 2: Visual comparisons on LD3.5-Large: samples from the original teacher uses 32 sampling steps.
2024 We further note that recent trajectory-based distillation methods such as HyperSD Ren et al. 2024 and TDD Wang et al.) (2025) were subsequently applied to Flux (and thus were not included in the official benchmarks reported their papers): for fair comparison, we use their publicly released checkpoint weights to establish baselines. Notably, all these baselines have utilized ADD/LADD for improved performance, whereas ours dose not. On the other hand, we also include TeaCache Liu et al. [2024a] baseline for Flux using their publicly released code TeaCache is training-fres free approach that accelerates inference by selectively skipping redundant timesteps based on output fluctuation, rather than directly reducing the number of steps. As TeaCache introduces quality-latenc trade-off via a threshold parameter, we evaluate their fastest (0.8) and second second-fastest (0.6) config aonfigurations as provided in their official Flux implementation.
5.3
Analysis
We present our main evaluation results in Table focusing on the key metrics discussed above. In particular, we adopt AFID to quantify the distributional shift between the student and teacher models with respect common fixed validation dataset. To further strengthen this comparison, construct a teacher- generated reference dataset using fixed CFG values and random seeds. This enables direct, controlled measurement of tedent-teacher fidelity, isolating the impact of distillation from other sources of variance. Across all three evaluation metrics AFID, FID, and CLIP our proposed method consistently achieves the best performance.
also
we did not perform CFG embedding distillation for SD3.5; as a result, method requires twice the number of function evaluations during inference compared CFG- distilled variants like SD3.5-Ti S-rurbo. Nonetheless, our framework is fully compatible with CFG embedding distillation, which we leave for possible future exploration. While TeaCache-0.6 achieves a competitive FID score, it lags behind our method in inference speed and CLIP score. Moreover, our slowest distilled 8-step student outperforms the fastest TeaCache-0.8 in both speed and generation quality-desitee despite requiring only few hours of training oofferng a fair and highly efficient alternative this training-fre free approach.
As noted in Section 5.1
Visual comparisons for Flux are depicted in Figure Our method, SCFM, maintains the highest fidelity to the 32-step teacher, particularly in the 8-step student, while simultaneously improving prompt adherence- eevieeence-evidentt evident in the 3- and 4-step students. For instance, SCFM better captures details like "robot painting" and "glowing flowers". Similar trends are seen in the SD3.5 results in Figure 2 where the 3- and 4-step students accurately capture prompt-specific concepts like "squared"' "bison" and "fox", sometimes better than the teacher. On the other hand, a closer examination of our few-step samples reveals noticeably improved clarity, detail, sharpness, and composition compared to other
'We omit the SD3.5 version, as TeaCache has not been extended to support this model at the time of submission, see https: /ttgttttttttteeeeeeeeeeeeeee.e com/ commaall-vi1ab//eaaaaaai.c

---

methods. While the image quality is on par with Flux-Schnell, our approach better preserves the characteristics of the original Flux-Dev model, as evidenced by the lowest fidelity score and superior visual consistency with the teacher. Additional visual results are provided in Appendices H and
5.4 Further Ablations and Discussion
Given the simplicity of the vanilla SCFM introduced in Section f several promising avenues for further exploration naturally emerge. The following ablation studies trace the empirical motivations behind the final version of our SCFM training algorithm:
1. As early reference to Appendix approaches, we consider an ablation in what if we mix the teacher and stopgrad student solvers in the first term of (T3). Does this impact overall performance or training speed? As shown in Appendix C we observe slight improvement in quantitative results. This indicates that the training scheme is robust and amenable further refinements.
B where we compare our method with other distillation
2. As hinted at the end of Section 4.1 employing cyclic restarting for the stopgrad model can significantly accelerate convergence, confirmed by CLIP score progression curves in Appendix D However, the strategy may become detrimental during very late-stage training, nevertheless, we find that an 8-step student safely converges by approximately the 2000th crration-eeqivallee to around 10 A100 GPU hours when distilling Flux in our setup.
3. The above findings suggest that a rapidly accumulated stop- graggradient model facilitates faster convergence. Building on this and motivated by the first point, we propose mixing teacher and stopgrad models with different EMA decay rates in (13): a "fast" model with a lower decay parameter (μ- 0.99) and "slow" model with a higher decay parameter (μ 0.999). This dual-EMA strategy eliminates the need for manual cyclic restart, instead enabling a fully automatic mechanism that leads to even faster convergence. With this setup, the 8-step student safely converges by around the 1000th iteration (~5 A GPU hours), and the overall model reaches 3-step performance in under 24 GPU hours. Further details are provided in Appendix E
4. Inspired by the observations above, we note that SCFM is already highly efficient to train: by the 1000th iteration, the model has only seen approximately 16k images. This naturally raises an interesting question of whether few-shot distillation is feasible. In Appendix FF) we explore this by training on dataset of only 10 images (simultaneously using this the batchsize). and find that the results remain comparable to those achieved using the full training set.
5. Nevertheless, there remains room for additional, less critical ablation studies- such as investigating the effect of the teacher-student mixing parameter k. which, as briefly mentioned in Section 4 serves to interpolate between trajectory-bast based and progressive-style distillation strategies. However, since the preceding ablations already demonstrate that accelerated self- distillation (i.e., non-zero k) leads to faster convergence, we consider this particular experiment to be of lower importance and do not explore it in detail.
We note that the above explorations primarily affect training speed, with negligible impact on the final converged results. Therefore, all the tables and figures presented in the experimental section remain valid and unaffected.
Conclusion and Future Work
In this work, we presented SCFM, an efficient distillation method that accelerates pre- vre-trained flow matching diffusion models to very few steps. Experimental results demonstrate that method best preserves many-step teacher characteristics. Compared to previous works, the computational and training data requirements is almost free, providing compelling alternative to training-free acceleration methods while delivering significantly better results. Future directions may include:
Preserving model creativity is key aspect of distillation quality (i.e., producing diverse samples under different random seeds). Our experiments show that our method retains many-step teacher characteristics, including variability, though quantitative metric still lacking. This may stem from our focus on velocity trajectories, whereas common approaches directly predict clean samples ge. [Sng et al. [202).. J. Yin et al. (2024)) and yield less diversity. This also indicates that our method generalizes better in few-shot settings, being less affected by the scarcity of
g.,

---

training samples. Future work may combine both approaches to enhance sample quality and variability.
While our approach eliminates the need for step-size embeddings in shortcut models with a distinct learning paradigm, it is nonetheless encouraging to see the development of highly efficient algorithms that converts pre-trained flow matching into exact shortcutting.
Since ADD-style methods can be readily integrated into our framework to potentially enhance few-step generation, or even enable one-step generation, exploring a velocity-spant variant of LADD may be promising direction for enhancing velocity consistency.
Our method is designed to be moddllityagnooste and applicable flow matching in diverse domains, such as 3D, video, audio and etc. It would therefore be valuable to explore how practitioners might adopt the algorithm, possibly with domain-specific modifications.
References
BlackForestl.abs, Flux. htttps///tthubb.co/bblaakk//oos
2024.
Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Enze Xie, and Song Han. Sana-sprint: One-step diffusion with continuous-time consistency distillation. arXiv preprint arXiv:2503.096// 2025.
Patrick Esser, Sumith Kulal, Andreas Blatimann, Rahim Entezari. Jonas Müller. Harry Saini. Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Xuhui Fan, Zhangkai Wu, and Hongyu Wu. A survey on pre-trained diffusion model distillations. arXiv preprint 2rXiv22502.08364 2025.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrevie: wwweet//oormmmidd1111e....
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL http:ttttt//pocceeeingee/ neterinerrppp ips. cc/ pdf
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update nule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL http::tt////ppereriii net:/orummid--uBAAN.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL hntps://opnrreieww.ee net.ieewwweeeeeeeeee mettfoorumicc nZeVKeeFYf9
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. The Twelfihh International Conference on Learning Representations, 2024. URL htt.lcppee/poppeeeeeeeeee/////ro
10

---

Andreas Koukounas, Georgios Mastrapas, Sedigheh Eslami, Bo Wang, Mohammad Kalim Akram, Michael Günther, Isabelle Mohr, Saba Sturua, Nan Wang, and Han Xiao. jina-clip-v2: Multilingual multimodal embeddings for text and images. arXiv preprint arXiv:2412.08N8 2024.
LaionPop. Laion pop: 600,000 high-resolution images with detailed deseriptions. htw.giigaaaccc.o///aaa/a//////// httpa///o...aaaaaaeaa/aa///aaaaa
https://
2024.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411.19108 2024a.
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Kleventh International Conference on Learning Representations, 2023.
Xingehao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. Instaflow: One step is enough for high-quality diffusion-basee text-to-image generation, 2024b. URL https:tts://axxi org/
abs abs/2309 06380
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International
Conference on Learning Representations, 2019. haalaatp:e//ppeeeeeveeeeee/tttrp
Bkg6
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreview. wwt.:neeeeetoootdtddtlllaaaeee
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference, 2023. Uttthtts:: //attis://rxivv org/ (bb/0331004337) 04378
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL
httpps://openreviex wwwss/ffoummidde htrt//forrmmmdddd1122285ggg
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PmLR, 2021.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-SD: Trajectory segmented consistency model for efficient image synthesis. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreviess
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 2=Xxi:2202.005/2 2022.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. SIGGRAPH Asia 2024 Conference Pupers, SA '24, New York, NY, USA, 2024a. Association for Computing Machinery. ISBN 979840071 1312. doi: 10.1 /555///366602226877255555 U wtt.hhttts:: :/dttpss:///i..... org//
10. 1141/3680552 36675586666528868662555

---

Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87-103. Springer, 2024b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 15:25278-25299 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a. URL htttps:/ het./forumnid-sttlgiarittt
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- basee generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. U.LLattss//pppmnooee...ee///oot
PXTIG12RRHS
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 3221 1-32252. PMLR, 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Cunzheng Wang, Ziyuan Guo, Yuxuan Duan, Huaxia Li, Nemo Chen, Xu Tang, and Yao Hu. Target- driven distillation: Consistency distillation with target timestep selection and decoupled guidance. Proceedings of the AAAI Conference on Artificial Intelligence, 3977)7719--767,, Apr. 2025. doi: 10. v39../ 32820.. URL attps:://op aaai. org/ index pwwp 32820
Fu- Yun Wang, Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, Xiaogang Wang, and Hongsheng Li. Phased consistency models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httts::/
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CV conference computer vision and pattern recognition, pages 6613 36233, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable ceffectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are all validated by experiments in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We have listed some of the limitations, such as pursuing one-step generation, in 6
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
Our method dose not need assumptions or proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question:
Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided all the experimental details Section 5.1 and algorithmic details in related appendices.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
Code is available at shhrtcuttfm github..hh
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wwwpommmsssiooooooooooooo oniiioicnliyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiipe.cc/pullic/uuiii gwtp://gggee//eee CoodeSumiiii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper dose not include error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwrr..coooaaalllcccccaaa
Answer: [Yes]
Justification: We have reviewed the code of ethics and ensured that our work complies with all outlined guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
To our knowledge, this work does not pose any direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited all the original owners of the related assets in Section
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprssiiiitt. thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs are needed in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Pseudo Code of Vanilla SCFM
First, let Ln Linspace(1, 1, 0, + 1) denote the of equally spaced discretized timesteps. decreasing from 1 to O, where corresponds to the target number of teacher steps.
Prevalent pre-trained flow matching models often avoid using uniformly spaced discrete ODE steps. instead applying timestep shifting to improve sample quality |Esse et al. (2024), BlackFoorest [224]] In some cases, the shift may even dynamically depend on image resolution. In this paper, focus exclusively on the non-dynamic shift, where the shifted timestep is defined as:
st
(t) = 1+ (s 1)t
8
(17)
For the shift parameter in (17). we sample uniformly from the range [2.5, 4.5] for both Flux and SD3.5-Large. Intuitively, a larger shift value s concentrates more sampling steps in the high-noise region (early timesteps). and fewer in the data region (late timesteps), which typically leads to improved empirical performance compared to uniform scheduling.
While one could design shift distribution tailored to the number of ODE steps, we observe that uniform sampling within this range performs comparably to more sophisticated strategies. Nonetheless, this remains an interesting direction for future exploration. The overall training procedure for the vanilla SCFM is summarized in Algorithm
Algorithm 1 Vanilla SCFM Training
1: Input: Teacher parameters 0" trainable student parameters 0, stopgrad parameters 0 +- O, discretized timesteps L,, number of teacher- teacher-guided samples and batchsize N
2: while not converged do
3: Sample data Xg ~ D, and noise X ~ (N,
4: Sample shift value E [2.5, 4.5]: VI E Ln apply shifting L +- S.(1) as in 11
5: for first & elements in the batch do
6: Sample three consecutive t1, f2, it from L,, with fixed skip of
7: for remaining N - elements in the batch do Sample three consecutive t,,t2 ty from Ln with skip randomly drawn from
8:
(2.4. (2.4...., T/43
9:
Xt, < (1 f f1x1, then perform forward pass obtain Vo(x,, f1)
10:
Compute targets Vo. (Xe,:t) f1) and (x-(xx,; (1) using (12), with either the embedded CFG or the variant in (16). Specifically, set d; 11 and di+1 +- 13- Backpropagate 0 using loss (13) with (x( 4.). Vo- (Xt,, (1) and Vo ;
11:
12:
Update the EMA model 0 according to [4)
13:
Optional: cyclic restarting is enabled, reinitialize 0 + at fixed iteration intervals
B
Closer Comparison to Other Trajectory-based Distillation Methods
As discussed in Section 2 shortcut distillation represents a novel class of trajectory-based distillation methods that operates in velocity space rather than directly regressing ODE-simulated samples. Existing trajectory distillation methods typically employ the following sample-space mappings:
tx) + x,, E (0,t),
(18)
which also requires ODE solving along trajectories. Unlike our velocity-space approach in (12).
conventional methods Luo et al. 20233 Kim et al. [2024] enforce consistency through mixing teacher and student solvers:
Xt fo- (Xe,41> 1++) = fo- (fe- (X2, ti), 14+1).
(19)
Intuitively, this formulation injects teacher knowledge through the inner teacher ODE solver, and mixes it with the outer (stopgrad -oopgrad-)studeent solver. The student thus learns to generate outputs from arbitrary intermediate states produced by the teacher.
In contrast, shorteut distillation (9) employs consistent solvers for both steps steps-either using the teacher Xta+a for (fe- (xe, f,), 441), the student Xt;-2 -fo- (fo-(xee tx), 1,41,, and further combines
20

---

their outputs through self-distilled bootstrapping. This preserves internal solver consistency while progressively transferring knowledge from teacher to finer students. Finally, a generalized distillation loss can be written as
(()) (x) 14), (x (++1))
(20)
In light of this loss formulation, we proceed to elaborate on the key differences in greater detail.
Remark 2. For flow matching models, the ODE in (18) simplifies to (5). Henceforth, some previous works may arrive at objectives resembling D[(d, + d2-11) (Xxe, 44), d, Vo- (Xe,, (:)
d,-1 Vo (Xtqr f, which superficially resembles our 12). However, crucial differences exist:
ID in existing works is measured in sample space rather than velocity space. Unless sample- wise/adaptive weighting function proportional to 1/(d, + d,+1) is explicitly incorporated, this formulation fails align with the scaling behavior inherent to flow matching objectives. To our knowledge, no prior work has proposed such weighting scheme specifically tailored flow matching models, nor is there clear empirical evidence supporting its use.
Numerically, since d; E (0, 1] in (TO), errors computed in sample space are typically 10-100x smaller than those in velocity space. As a result, they require extremely small learning rates (e.g.. 10 9) and significantly more training iterations. For example, (Ren et al. 2024 Section 4.1] reports that each distillation stage for a 2.6B-parameter SDXL requires approximately 200 A100 GPU hours. Some works also consider alternatives to the standard & loss for D, such as the Huber loss, { loss, or perceptual losses like LPIPS Zhang al. [018 This is usually motivated by the observation that sample-space losses often exhibit greater instability. As result, more robust loss functions- -uunn as the Huber loss are sometimes preferred due to their reduced sensitivity to outliers (e.g., see Chen et al. 2025). In our case, we adopt the standard & loss, which preserves the optimality of unbiased estimation in velocity fields, as hinted in Section 3.1
Nonetheless, as shown in Appendix C we explore the possibility of mixing only the teacher solver with the student- following the practice of some prior works- and observe nearly identical results with slightly faster convergence. This finding motivates deeper investigation into how to best leverage self-distillation, as further explored Appendices) and D
C Ablation on The First Term of Loss
13
As hinted in Appendix B and in the first point of Section 5.4 we explore an alternative formulation in which the solvers used in the first term of (13 are interleaved or mixed, as described in (19). while keeping the second term unchanged. More formally, similar to Equation (12), the modified target Vo- (Xx,, t,) is computed as
d, Vo (X (4) = (d, + di+1) Vo- (x),,t) + (de + dz41) V Vo (x11))
di+1
(21)
This formulation linearly blends teacher and EMA student together, enabling faster knowledge transition to the second term of Loeef-- As an evident in Figure 3] this approach (denoted as vanilla-mix) yields slightly faster convergence compared to the vanilla SCFM formulation. This observation suggests that employing a more frequently updated EMA student model may facilitate better knowledge propagation from the teacher.
To further investigate this idea, we introduce a cyclic restarting strategy for the student EMA model. In this scheme, the EMA accumulator is periodically reset at fixed iteration intervals, allowing the student forget stable or overly smoothed knowledge and adapt more quickly to the evolving teacher signal. This leads to even faster convergence, as discussed in detail in Appendix D
D Ablation on Cyclic Restating
Cyelic restarting included as an optional component in Algorithm (highlighted blue), designed to accelerate training by periodically resetting the EMA student model. In this section, we present experimental evidence demonstrating the effectiveness of this simple yet practical strategy. Specifically, we compare the vanilla version of Algorithm | against two variants: one with an
21

---

Algorithm 2 Fast-Slow EMA SCFM Training
Input: Teacher parameters 0" trainable student parameters fast stopgrad parameters 0 < with j 0.99, slow stopgrad parameters 0 <- with μr 0.999, discretized timesteps Ln,
number of teacher-guided samples k. and batchsize N
2: while not converged do
3: Sample data Xo ~ DD and noise X1 ~ (0,
4: Sample shift value
[2.5, 4.5|; Vt Ln apply shifting St(t) as in 5: for first & elements in the batch do
6: Sample three consecutive f1, 22, t3 from L, with fixed skip of
7: for remaining N - k elements in the batch do Sample three consecutive 11, 11,12,4, 13 from In with skip randomly drawn from (2.4...., T/4)
8:
9:
Xt, t (1-111)) f1)x0 + txii, then perform a forward pass to obtain (x bt)
10:
Compute the targets Ve- (Xty,4) and Vo- (Xe, (2) using (21 and
Remaining
calculations follow same as Line of Algorithm
11:
Backpropagate 0 using loss (13) with ( 41), Ve- (Xts: (1) and (x-(x))
12:
Update the EMA models 0 and 0 according to [4]. each using its respective decay rate
aggressive restarting schedule that resets the EMA every 500 iterations (referred cyc-500), and another with a more moderate schedule that resets every 2000 iterations (denoted as cyc-2000).
As shown in Figure 3 both variants significantly improve convergence speed the vanilla baseline. The cyc-500 variant exhibits the fastest initial convergence due to its rapid adaptation, while cyc-2000 achieves a more balanced trade-off between stability and speed. These results confirm that periodic forgetting of stale EMA knowledge can facilitate quicker adaptation and reduce training time.
However, discussed in Appendix SenndiixDnndd and illustrated in Figure overly frequent resets (e.g., cyc-500) can lead to performance degradation by hindering late-stage convergence. This highlights the need for careful tuning of the restart interval, which may vary depending on the specific distillation setup. To address this issue, and inspired by the findings in Appendix C we propose a fully automatic fast training algorithm, presented in Appendix
E Final Version: Ablation on Dual Fast-slow EMA
We retain the computation Ve- (xt,, f:) as defined in (11). but introduce a fast EMA model, denoted by o+, into the second term the SCFM loss. This modification still preserves the solver consistency in terms of self-distillation, with the key distinction being that and slower EMA model. Specifically, the revised computation of the fast-slow target Vo given by
mix outputs from both a faster
d, Vo (x), = (d, d,+1) Vol (x), + di+1 Vo (xta1s
(di di41)
(22)
The resulting dual stopgrad EMA training algorithm largely mirrors the structure of Algorithm I hence, highlight only the modified components in red for clarity and ease of comparison.
As shown Figure Algorithm 2 achieves even faster convergence--upperffooiii restarted variant of Algorithm that resets every 500 training iterations. Most importantly, an additional benefit is that this fast training procedure is fully automatic and avoids the late-stage performance degradation seen with the manual restarting strategy, demonstrated in Figure
the aggressively
22

---

34
32
ClipScore
-43
29
Treining Method
Varillo
28
be-mi
Cyclo-000
Cydlo-2003
Eina
24
2503
5003
7000
10003
12900
16000
17500
22000
Iteration Steps
Figure 3: CLIP score progression across training methods, evaluated 20 random image-text pairs from the COCO-30k validation set. Experiments are conducted on Flux and evaluated on the 8-step student.
96
53
Score
Training Melhod
20
Vardaa
Verii-etic
Cyclo 500
Uyt1o-2000
Dux
30000
S2000
54000
30020
38002
42000
Iteration Steps
Figure 4: Same setting Figure focuses exclusively on the convergence behavior during the later stages training.
F Ablation on Few-shot Learning
In this section, we present an interesting ablation study to evaluate the effectiveness of SCFM in few-shot learning setting. As shown in Table distilling an 8-step student model using dataset of only 10 images results in only minor differences in CLIP score, along with slight increase in FID and AFID compared the baseline trained on the full dataset. However, these quantitative differences do not fully capture the performance of the model. As illustrated in Figure the few-shot distilled model demonstrates only modest degradation in preservation ability. the mean time, generated samples maintain high image quality and strong prompt alignment, aligning well with the metrics in Table 2 also worth noting that preservation ability may not always be a critical goal coal-aspyc in cases where the objective is to guide the student model to learn from specific, small dataset, or to acquire general generation capability rather than closely imitating the teacher model.
Overall, this suggests that while the shortcutting ability SCFM may be moderately constrained when training data lacks sufficient coverage of the teacher's ' knowledge, the overall performance remains surprisingly robust. Such setting could particularly valuable when collecting large- scale data is impractical, e.g., 3D, video even robot data. In such cases, even a limited set of self- delfraeeee or synthetic samples might be sufficient to produce a strong distilled model. more thorough exploration of these data-efficient regimes is left for future investigation.
23

---

Table 2: Qualitative evaluation of few-shot SCFM trained Flux. All samples are generated using steps. The experimental setup follows the same configuration as in Table
AFID4
FID
CLIP
Flux. 1-Dev
27.43
N/A
33.60
SCFM
+0.16 (27.59)
2.58
33.76
Few-shot SCFM
-0.3 (27.13)
2.94
33.75
Flux-Dev
SCFM
Few-shot SCFM
Figure 5: Side-by-side comparison on few-shot trained v.s. regularly trained SCFM. Teacher samples use 32 steps, whereas others use 8.
G Comments on CLIP Score
In this section, we elaborate our choice to use the CLIP checkpoints provided by Koukounas et al. [0024) -named Jina-rather than the official OpenAl versions. Although OpenAl's CL.PP models are widely adopted, we found that their scoring often fails to align with human perceptual semantic

---

judgments-pariinnnn particularly in the case of low-qualits synthetic data. While our validation set contains numerous examples. representative case Sigure@@dmmossterrn that the Jina checkpoint the only variant among those tested that produces CL.IP score ranking consistent with human preference. In contrast, the OpenAl models fail to capture key elements from the caption, such as "grass", mistakenly favoring an image that shows green land but no actual grass. This suggests that the model may be overly influenced by coarse visual features. Such discrepancies indicate that the Jina checkpoint provides better calibration for text-image similarity, making it more appropriate choice for our evaluations. Nevertheless, all CLIP scores obtained Table Jses this model consistently.
Flux-HyypeSD-3SSteps
Flux-SCFM-3Steps
Human Preference
Clip-base-patch-16 Clip-base-patch-32
35.05
32.28
34.62
32.16
Clip-largo-patch-11
28.56
26.96
Clip-large-patch-1--33 14-333 14-336 Jina-clip-v2
30.09
27.59
33.0
35.0
Figure 6: The caption used to generate the two images above is: "A group of zebras grazing in the grass."
H Additional Results for Flux
See the visual comparisons in Figures 7 and where we only include the TeaCache-0.8 results in Figure 7 as it offers inference speed closest to that of 8-step distilled models (sce Table
25

---

FLUX-Dev
TeaCache-88
HyperDD
TD0
FLUX- FLUX-Scmell
SCEM (Ours)
Greereetc
metailic scoies fiicering holographic
fuscin-fussig myehic futuritic
Ultrorealstic stock photo of Elthyonk/ Imint meditating omietr the rutss of accient, ethereal city the Astrol Plane. Their pleoming onnor reflects stronge, otherworlsly light they wrestle confict-dutty ambtiio, discipline lie esudes mystical brooding intersity, symbolztng loyalty Visskih, despite
A cute sauored planet with blue colored surface surrondd o soft plak fuud space Background with glowing stors the whele scone has whimskcl and adorable wibe with bright shny and brillont surface on the platet fecturing singular charming house
Ultrarealstic Drungeman," figur orange segenents, playfully polishel wooden surfece, casting seft shadow, surrounded by scottered juice droplets. Wares light echonaes his juicy textures, evoking chisheed wender and creativity perrfctt for children's media playful adi.
syes
photo coptures o mossive cotolydic unit twilight, intricate network pipes vessels glowing agoinst fading billws from vents, Ninting at the intense processes within, while the surounsting londscape underscores the uni's entical rale in energy produrtion and modern industry.
IPhone 13 Fro Mor Acoting in caberpunk vod, Coomume frome glowlng with ncon crcunts. Synamic iplnd projects holcgraphic ciuscopee Modie Bock reveols hidden Apple reflections. Camera lenses refrect floating lihhting,
reality pround No.
Figure 7: Additional visual comparisons for Flux under 8 steps.
26

---

FLUX-Dev
SD
TDD
SCFM (Ours)
group of street ortist spray painting vibrant mural that seamleszly incorporates hole in weathered brick wall. Bold colors explode against the crumbiing surface, blending decay with imagination. power with futuristic tech.
An aged ferryhoat glides through still waters, its weathered etched with the passage of time. Rust troces railings, and sun bleached point peels like pages from long-forgotten logbook. Overhead, seaguils spiral and cry, animating the fleeting energy, Ught dances across the surface, catching every dent, rivet, and memory embedded in the versel's frame.
The Fluorescent Forest, Laves Whispers in the Wind Comera Wideongle lens, eyelevel shot Focal Length 24mm Render Settings Highquality occlusion, motion
194305, 120642, Shole Green 165304 Framing The Puuorescet forest, loves whispers the wind, where listen to the breeze, hearts resonating with the hushed secrets, forever attuned to the harmony of loves sacred hymn
textures, dynamic lighting, global illumination,
portrait teenage fairy magical forest, midsummers night dream, fairy mogic, fairy cors, poloroid, foirytale poverty fashion, tangled unwashed hair
Jopanese style sushi restouront cartoon with soft and Junny with with white background
A futuristic, sustainable city golden nour-sunrise casting worm glow over a skyline of flowing, argonic architecture wrapped in lifts distance,
greenery. through metropolls flying cars glide drone through the foreground, blending nature and innovation visionary cover shot
capped
Figure 8: Additional visual comparisons for Flux under steps.
27

---

FLUX-Dev
SD
Schnlll
SCFM (Ours)
Bokeh, Electric Colors, Accent Lighting, Uightring, inferno, insunely detailed and intricote, hypermoxiimalist, elegant, ornate, hyper reolistic, detailed, phoenix with wings of fiane_firont magical atmasphere.
Craft on image of person walking on winding poth through beautiful landscape, with the seasons changing around them from spring winter. Behind the person, the should fade sepio symbolizing memories and experiences, while the path ahead remains vibrant and colorful, representing the present and future.
Aeriol shot stunning greek fontasy polace with lots towers and spires, lorge windows ond airy quality and feel, paloce is in the center sprawling city full of librrries, shops, and bustting activity, background is rolling mountoin ronge, farge fluffy clouds in the skyUse ethereal lighting and light colors create idyllic atmosphere, sky, Otherworidly, Mendozo, Jordan Grimmer, Ivan Shishkin, An Jung twon, HD, Cinematic Lighting, Picturesque, Chioroscuro, light roys, Hdr, 8k.
realistic, Magic, Decorated, Art by Eddie
20 onimated ice queen cot with crystal crown, glowing blue eyes, and sleek, snow-white coot. She stands atop glistening ice ciiff overlooking vast frozen kingdom-spiin costles, glowing glaciers, snow pine forests lights ripple the twilight sky, while deficote frost patterns donce the Regoi, megical, and cinematic in
A professional picture, landscape of mountoins with beautiful water foll, sun lights in the background, 8k, ultra high quality.
Digital art of young girl reading on enchanted forest, inspired by Arthur Rachham's 's whimsical style. Warm sunlight filtess through lush fairies, glowing firefiles,
greenery (saft greens, brawns, golden yellows). and curious foxes. Delicate olitter accents add magic-perfee for child's dreamy escape.
centers her, hand,
Figure 9: Additional visual comparisons for Flux under steps.
28

---

I Additional Results for SD3.5L
See the visual comparisons in Figure 10
SD35sLamp
333.II-Turie
SCEM (Ours)
SD35-Lare
S03.5-TTuh
SCEM (Ours)
Olynpe National Part, Wourington and Alustratoo, toer the mahprampe mighaateaaamddinn wrth the Image.
lete wheel of pleaming wood and politted bruss suptured mia poise pefore aph Shong denat nowah five creftmmmmitp.. hys erop namberss and the rener stlve Prans-yytbalnng rut, neserd and the theil unsertavry, fiam gambling fmoree
detas's imphoste lnvel driat
Stops
Alhorealite Wrangemen
witmmiia
fiom orange segnien, caried peek, and clras asit wilh dlss eys and Neaf sestands
lalodoucopi multficctco resemrlls escroscoppc ridge
alayfuly politedd moodon aurlce dusting soft chadde, sumpunded by scuttered sloes.
Steps
portrett of happy in hage ruciser explasion Acrip
wagss cioscape, shop focui foce, ahoc hasseblld starpon tigh resovtion staro
araun the
Ropereallaii stock piooo coptares majestic prdff anow copped
citf Aingdlati distonce.
Ultorveattsicc night eusilty. High res, desaledd, Zhougligle Aotiona Row, Outerspace sky.
Colougha Cosmir Nebulo Goos Sty, close New
Litrowallats, high qustiy, Jigh reg detalled dag mekinng or cnllefffl paint
D
Steps
Cute bey, Antr fooiing on the store stow, brantful lpsteesrg pointipg etyle ryse obe Romyysk
3dhuger roalstic oome ort d fightigg cuts cot neuving headsheres by 4j tools noon ligns music bcoutjul brunctte belind.
simu fylngg mountoin spring, notiona!
Egypt
peopeopent
pesenating
Figure 10: Additional visual comparisons for SD3.5-Large
29


Reviews:
[{"review_id": "QPmfmezg4l", "reviewer": "Reviewer_qe9Q", "summary": "The authors consider shortcut models and note that while they can be used to obtain a few-step flow model, they require training from scratch, since they require a specialized step-size embedding incompatible with existing diffusion/flow models. To address this problem, the authors propose a novel distillation objective that can be used for “shortcutting” a model without using additional “step size” embedding and thus is applicable for fine-tuning of already pretrained large text-to-image models. The authors evaluate their method together with LoRA decomposition of parameters to distill SD3.5L and Flux models.", "questions": "No", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe authors propose to extend the “shortcutting” procedure to the use of pretrained models, which do not have \nThe authors support the proposed method by testing it on distilling SD3.5L and Flux models.\nThe proposed method (together with using LoRA) is efficient and allows for efficient distillation of large models like Flux into a few-step generator under 24 A100 GPU hours.\n\n**Weaknesses:**\n- The main benefit of the proposed algorithm is that it is applicable to the original model without architectural modifications. However, it is not clear whether a direct addition of the step-size conditioning with randomly initialized parameters to the pretrained teacher model (+ using EMA for teacher, as in the proposed method) will be less efficient or provide worse quality of generation. Furthermore, the direct addition of the embedding conditioning does not require\n- Notation consistency. The authors start to derive the self-consistency principle using notation V_{\\theta} and then change it to V_{\\theta_{-}} (Eq. 11 and Eq. 12). I think it would be better to use a general notation for the velocity in this derivation and introduce parametrization of teacher/EMA later, when discussing the algorithm to make notation consistent.\n- Notation of sums in Eq.13 is ambiguous. There are two sums, but it is hard to understand the summation of what they describe, since there is no index of summation in the sums and in the expressions of these sums. In essence, $N$ is a batch size, and $k$ is the number of elements from the batch size used for the first part, and $N-k$ is the number of elements used for the second term. It should be written on the top of sums as their limit, and also a summation index should be introduced (e.g. $\\sum_{n=1}^{k}$, $\\sum_{n=k+1}^{N}$ and $n$ is used in the loss expression). Since it is the main expression of the paper, it is critical to provide it in the most clear form.", "was_revised": true, "final_justification": "I keep my original score. During the rebuttal, the authors answered my questions.", "rebuttal": "We are very grateful to the reviewer for the feedback, and very pleased to have received positive reviews. \n\nQ1. *A direct shortcut implementation*  \nA1. We have attempted to implement the original shortcut method for comparison. However, we observed that convergence was significantly slower and the resulting performance was not competitive with any other baselines (i.e., it failed to produce reasonable outputs at low sampling steps, such as 8-12).  Henthforce, applying the original shortcut formulation to large models such as FLUX or SD3.5 appears to require substantial computational resources and/or non-trivial pretraining techniques, and we are concerned that our current implementation is not sufficient to support a fair comparison.  \n\nQ2. *Notation consistency/ambiguous*  \nA2. Thank you for the suggestion! We are now aware that some notational inconsistencies and ambiguities are indeed present, and we will do a careful revision in the next version of the paper to ensure clarity."}, {"review_id": "QUE0xkKi9k", "reviewer": "Reviewer_VfPo", "summary": "This paper integrates the self-consistency loss from shortcut flow matching ideas into model distillation in the velocity space of flow matching models. The authors additionally uses an implicit step size approach to improve training efficiency. The method is evaluated on certain latest text-to-image models.", "questions": "1. See the \"Weakness\" above. \n\n2. Does the training of the proposed model always initialize the model weights from the pre-trained model? Are prompts used in the test time different from the training time? Especially, when you talk about training with only 10 images and achieving good results, do you also use the same 10 prompts/images for testing or do you use different prompts/images?\n\n3. What is the model performance on versions of Stable Diffusion (e.g. 1.5) where there are already established baselines?\n\n4. Why does the number of steps stop at 3? Why not lower?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n-- Distillation a large pre-trained flow matching diffusion model into a model with fewer steps is an important research direction.  \n\n-- Incorporating the self-consistency loss from shortcutting flow matching ideas into distillation is nice and makes a lot of sense.\n\n-- The method is evaluated on certain latest text-to-image models. \n\n**Weaknesses:**\n\n-- The proposed model integrates the self-consistency idea from shortcut flow matching approaches but with an implicit step size. In particular, previous shortcut flow matching model trains for the look-ahead velocity V(x_t, t, d) where d is the step size. This does not align with a pretrained model. This paper instead proposes to train for a single velocity of the form v_\\theta(x_t, t) for all step sizes (what they refer to as implicit step size $d$, based on my understanding). I find this somewhat problematic: In general, this is ill-defined, because if the trajectory is not straight, then V_\\theta(x_t, t, d) may vary with different $d$; that is, there does not exist a single V_\\theta(x_t, t) for all different step size. The paper did try to motivate their method by assuming that we can straighten curved velocity trajectories. However, I am not sure how justified this is, especially since the teacher model in the first term of Eqn(13) may not have straight trajectories. Furthermore, as this paper states in lines 86-87: \"In practice, however, class-conditional models often deviate from this idealized trajectory, especially in high-noise regions where the mapping often exhibit curvature\". \n\n-- I feel that the proposed model can be more comprehensively evaluated, such as comparing with popular distillation methods, e.g. reflow, progressive distillation, consistency distillation, on the pre-trained models where there are already established baselines. I would imagine that this should not be hard given the proposed method is reasonably efficient. \n\n-- In Table 1, the $\\Delta FID$ of the newly proposed SCFM often is not the best for Flux related models. For SD3.5 model, the SCFM doesn't see to have significant improvement for low step sizes, but with a higher latency.", "was_revised": true, "final_justification": "Thanks for the responses. Nevertheless, I think that training a single $V_\\theta(x_t, t)$ across all $t$ will theoretically require straight trajectories, while training different $V_\\theta(x_t, t)$ parameterized by $t$ only requires local straightness. I will maintain my rating.", "rebuttal": "We thank the reviewer for the positive feedback and these constructive questions, which are valuable for clarifying key aspects of our work.  We hope that our responses help clarify your questions and showcase the potential of our work. If so, we kindly invite you to consider raising your score :).\n\nQ1. *$V_\\theta(x_t, t)$ can be ill-defined*  \nA.1 To clarify, training a single velocity model $V_\\theta(x, t)$ across multiple step sizes (i.e., without explicit $d$-conditioning) is well-defined within the framework of rectified flow training.  Specifically,  the 1-rectified flow model is derived from a family of $k>1$-rectified flows, each inducing **different** (and possibly curved) velocity fields.  While each $k \\ge1 $ version has its own teacher and potentially distinct $V_\\theta$, in practice, one may recover the 1-rectified version either by (1) gradually updating the model as $k$ decreases, or (2) training a unified model across all $k \\ge 1$ trajectories (**as we do**). These two approaches, when fully optimized, should theoretically yield similar outcomes, and justify each other.  \n \nGiven this view, we believe it is more precise to reframe the concern: rather than “assuming we can straighten curved velocity,” our method encourages the model to learn to straighten trajectories from curvature.  In this sense, straightness is not assumed: it emerges as a learned property from training using our proposed training objective. This is achieved by rectifying curved trajectories as much as possible. Indeed, in worst-case scenarios, e.g. regions with extreme curvature, the training loss can become large and may require special treatment.  In these cases, as the reviewer pointed out, the model may struggle to align all the $V_\\theta(x, t)$ consistently. A more comprehensive justification of such behavior likely requires further theoretical investigation in future work.  \n\n\nQ2. *Why not lower steps*  \nA2. Based on our understanding on existing low-step generation methods, achieving high-quality images with extremely low sampling steps requires two key components: **core algorithm** and **image-level feature learning**.  Current one/two-step generation methods rely heavily on optimizing feature learning through additional modules such Adversarial Distillation (ADD), which injects richer discriminative visual features into the model. This reflects the fact that reducing sampling steps is not solely a matter of core algorithm design (**which is our focus**), but also requires enhanced perceptual understanding.\n\n\nFor this reason, all competing baselines focus on ADD to overcome lower step challenges , and may sometimes step in their core algorithm contribution (*something we aim to avoid in our approach*).  Nevertheless, we hypothesize that there may exist more elegant alternatives to replace ADD by focusing on more straightforward image feature learning. We are actively exploring this direction and hope to report promising results in the near future.  \n\nAdditionally, we envision SCFM as a lightweight yet vital *post-training refinement* procedure (analogous in spirit to RLHF), that can significantly reduce sampling steps while preserving or even improving generative quality.  \n\n\nQ3. *Not comprehensively evaluated; Other baseline models/ distillation methods*  \nA3. Thank you for the suggestion. There are several reasons why we did not include additional experiments with methods such as ReFlow, Progressive Distillation (PD), and Consistency Distillation (CD): \n- These distillation techniques have so far only been implemented on EDM-based models, while our method is specifically designed for flow-matching approaches. As a result, a direct and fair comparison is not feasible. The same reasoning applies to SD1.5, which is not pretrained using flow matching and therefore not directly comparable to our setting.\n- We believe the most meaningful comparison would require a benchmark consisting of toy-level pretrained flow-matching models that could then be distilled using PD, CD, and other approaches. Creating such a benchmark would likely need broader community support to establish a shared set of FM-pretrained models.\n- Our work focuses on large-scale pretrained flow-matching models. Implementing and evaluating all baselines at this scale would require computational resources that we currently cannot afford.\n- We note that these distillation techniques have already been shown to perform worse than the baselines listed in Table 1 in their original papers. It is therefore reasonable to expect that our method would also outperform them under similar conditions.  \n\n\nQ4. *Some results are not the best*  \nA4. There may be some misunderstanding in our presentation, but as shown in Table 1, our method consistently *outperforms* all baselines across the reported metrics. The doubled latency in the SD3.5 experiments, as explained in lines 222–226, results from skipping the CFG distillation (unlike SD3.5L-Turbo which includes it), and consequently performing inference with double batch sizes.  We will revise the phrasing to clarify this distinction and avoid any further confusion.  \n\n\nQ5. *Model weights initialization; training/testing prompt difference*  \nA5. The model is trained using LoRA, with the $B$ matrix initialized to zero. Therefore, in all our experiments, the model weights are indeed initialized from the pretrained backbone. \n\nRegarding training data, the training and testing prompts are **different**, in other words, the model is evaluated in a fully **blind** setting. The few-shot learning experiments are intended to show that a **general distillation capability** can be obtained using a small, randomly selected dataset, or even one generated by the model itself. The benefits are obvious: (1) unlike all other methods, we do not require big training sets especially for uncommon modalities; (2) less computational resources; (3) appealing flexibility for model expansion (see our response to Reviewer SAAF for more details); (4) customizable to any self-maintained/private data requirements, and etc."}, {"review_id": "Ek7ZHkj4VK", "reviewer": "Reviewer_SAAF", "summary": "The paper proposes a method to conduct short-cut distillation on pre-trained Flow Matching based diffusion models. The goal is to get rid of the explicit variable which causes the original model structure to be modified. The method is done through a weighted sum of flow matching loss and a self-consistency loss. Experiments show that the proposed method achieve best performance compared to baseline methods and maximally achieves similar output to the ones generated by the teacher model.", "questions": "1. How does the proposed method work under extremely low time steps such as 1\n2. What's the author's plan for releasing the code and checkpoints for further evaluation? \n3. For the rest of my questions, please see the weakness part.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n1. The paper is well written and easy to follow.\n2. The proposed method is simple and effective with low training cost\n3. The final performance is strong even without ADD/LADD\n4. The presented method requires no changes to the original model\n\n## Weakness\n1. There is no comparison with the original short-cut model.\n2. Some ablation studies are only briefly discussed without conducting the actual experiment such as integrating with ADD or LADD.\n3. The author mentioned that the proposed method is not limited to image modality but no experiments are conducted to show the effectiveness of the proposed method in other modalities such as video.\n4. The evaluated models (SD-3.5 Large 8B and Flux.1 Dev 12B) are limited, it will be good show that the proposed methods work for other open-source models too such as smaller models and compare with more distillation methods", "was_revised": true, "final_justification": "The paper proposed a way to accelerate model inference by shortcutting the model which doesn't require extensive re-training. The achieved performances are strong. Therefore I maintain my positive rating and kindly ask the authors to release their code as promised.", "rebuttal": "We are grateful to have received positive feedback.  Some of our responses overlap with those addressed to other reviewers, and we provide cross-references for clarity and conciseness.  Please feel free to raise your score if you find that your concerns have been addressed or if you see new strengths, interesting insights, or future potential in our work :) .  \n\nQ1.*Compare to original short-cut model*  \nA1. We attempted to adapt the original shortcut approach to our distillation setting, but it yielded poor convergence and low-quality outputs.  For details, we kindly refer the reviewer to our response to Question 1 from Reviewer qe9Q for a detailed explanation.  \n\n\nQ2. *Lower steps / ADD*  \nA2. Please see our response to Reviewer VfPo, Question 2, where we address the reasons in detail.  \n\nQ3. *Other baseline models/ distillation methods*  \nA3. A detailed discussion of this can be found in our response to Question 3 from Reviewer VfPo.  \n\nQ4. *Code release*  \nA4. We commit to releasing the code following the final decision on the paper. We would also appreciate some time to further develop and refine it to ensure it is released in its best possible form. Please also see our response below for more information.  \n\nQ5. *Experiment with other modalities*  \nA5. Related to the code releasing question, we are planning to release a general-purpose training library based on our algorithm. This library is designed to support a wide range of customized modalities and models, rather than implementing each modality separately, which we find inefficient. To our knowledge, this would be the first off-the-shelf codebase that supports such flexibility. For example, in the case of video, our method requires only a few self-generated clips. It also supports distillation for modality-conditioned variants, such as the recently released FLUX-Kontext."}]

Author rebuttal:

Meta-review: Tl;dr: Based on reviews, rebuttal and ensuing discussion I recommend accept.

### Paper summary

This paper introduces an efficient distillation method, termed ShortCut distillation for Flow Matching (SCFM) based pre-trained models. The core claim is obviating the need for explicit step-size conditioning. A new objective function based on the self-consistency principle is proposed. Reasonable empirical validation is provided, demonstrating that the method can distill very alrge models.

### Strengths and weaknesses

Strengths: 1) Efficiency: Both distillation time and data efficiency. 2) Results: Experimental results are appealing, 3) Simplicity: No model changes are required, results shown on two large models.

Weaknesses: 1) Theory: It is optimistically assumed that the velocity field will get straightened rather than explicitly being targeted or guaranteed. 2) As Reviewer VfPo notes, the existence of a single velocity field V(x_t, t) for all step sizes is not obvious. Author’s clarify this as an empirical observation. 3) Ablation: missing comparison with original shortcut model (R: SAAF) would make paper stronger.

### Decision justification

Main reasons are novelty, promising empirical results and practical significance. Important problem of slow sampling speed of diffusion models is tackled with a method that has low computational overhead. All three reviewers were satisfied with author responses and provided positive ratings.
Decision: Accept (poster)