Title: Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion

flow-matching

distillation

fast inference
Abstract: We present an ultra-efficient post-training method for shortcutting large-scale pre-trained flow matching diffusion models into efficient few-step samplers, enabled by novel velocity field self-distillation.  While shortcutting in flow matching, originally introduced by shortcut models, offers flexible trajectory-skipping capabilities, it requires a specialized step-size embedding incompatible with existing models unless retraining from scratch—a process nearly as costly as pretraining itself.

Our key contribution is thus imparting a more aggressive shortcut mechanism to standard flow matching models (e.g., Flux), leveraging a unique distillation principle that obviates the need for step-size embedding.  Working on the velocity field rather than sample space and learning rapidly from self-guided distillation in an online manner, our approach trains efficiently, e.g., producing a 3-step Flux <1 A100 day.  Beyond distillation, our method can be incorporated into the pretraining stage itself, yielding models that inherently learn efficient, few-step flows without compromising quality.  This capability also enables, to our knowledge, the first few-shot distillation method (e.g., 10 text-image pairs) for dozen-billion-parameter diffusion models, delivering state-of-the-art performance at almost free cost.


Peer reviews:
comments: SUMMARY:
The paper proposes a method to conduct short-cut distillation on pre-trained Flow Matching based diffusion models. The goal is to get rid of the explicit variable which causes the original model structure to be modified. The method is done through a weighted sum of flow matching loss and a self-consistency loss. Experiments show that the proposed method achieve best performance compared to baseline methods and maximally achieves similar output to the ones generated by the teacher model.

STRENGTHS AND WEAKNESSES:
## Strength
1. The paper is well written and easy to follow.
2. The proposed method is simple and effective with low training cost
3. The final performance is strong even without ADD/LADD
4. The presented method requires no changes to the original model

## Weakness
1. There is no comparison with the original short-cut model.
2. Some ablation studies are only briefly discussed without conducting the actual experiment such as integrating with ADD or LADD.
3. The author mentioned that the proposed method is not limited to image modality but no experiments are conducted to show the effectiveness of the proposed method in other modalities such as video.
4. The evaluated models (SD-3.5 Large 8B and Flux.1 Dev 12B) are limited, it will be good show that the proposed methods work for other open-source models too such as smaller models and compare with more distillation methods

QUESTIONS:
1. How does the proposed method work under extremely low time steps such as 1
2. What's the author's plan for releasing the code and checkpoints for further evaluation? 
3. For the rest of my questions, please see the weakness part.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper integrates the self-consistency loss from shortcut flow matching ideas into model distillation in the velocity space of flow matching models. The authors additionally uses an implicit step size approach to improve training efficiency. The method is evaluated on certain latest text-to-image models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n-- Distillation a large pre-trained flow matching diffusion model into a model with fewer steps is an important research direction.  \n\n-- Incorporating the self-consistency loss from shortcutting flow matching ideas into distillation is nice and makes a lot of sense.\n\n-- The method is evaluated on certain latest text-to-image models. \n\n**Weaknesses:**\n\n-- The proposed model integrates the self-consistency idea from shortcut flow matching approaches but with an implicit step size. In particular, previous shortcut flow matching model trains for the look-ahead velocity V(x_t, t, d) where d is the step size. This does not align with a pretrained model. This paper instead proposes to train for a single velocity of the form v_\\theta(x_t, t) for all step sizes (what they refer to as implicit step size $d$, based on my understanding). I find this somewhat problematic: In general, this is ill-defined, because if the trajectory is not straight, then V_\\theta(x_t, t, d) may vary with different $d$; that is, there does not exist a single V_\\theta(x_t, t) for all different step size. The paper did try to motivate their method by assuming that we can straighten curved velocity trajectories. However, I am not sure how justified this is, especially since the teacher model in the first term of Eqn(13) may not have straight trajectories. Furthermore, as this paper states in lines 86-87: \"In practice, however, class-conditional models often deviate from this idealized trajectory, especially in high-noise regions where the mapping often exhibit curvature\". \n\n-- I feel that the proposed model can be more comprehensively evaluated, such as comparing with popular distillation methods, e.g. reflow, progressive distillation, consistency distillation, on the pre-trained models where there are already established baselines. I would imagine that this should not be hard given the proposed method is reasonably efficient. \n\n-- In Table 1, the $\\Delta FID$ of the newly proposed SCFM often is not the best for Flux related models. For SD3.5 model, the SCFM doesn't see to have significant improvement for low step sizes, but with a higher latency.\n\nQUESTIONS:\n1. See the \"Weakness\" above. \n\n2. Does the training of the proposed model always initialize the model weights from the pre-trained model? Are prompts used in the test time different from the training time? Especially, when you talk about training with only 10 images and achieving good results, do you also use the same 10 prompts/images for testing or do you use different prompts/images?\n\n3. What is the model performance on versions of Stable Diffusion (e.g. 1.5) where there are already established baselines?\n\n4. Why does the number of steps stop at 3? Why not lower?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider shortcut models and note that while they can be used to obtain a few-step flow model, they require training from scratch, since they require a specialized step-size embedding incompatible with existing diffusion/flow models. To address this problem, the authors propose a novel distillation objective that can be used for “shortcutting” a model without using additional “step size” embedding and thus is applicable for fine-tuning of already pretrained large text-to-image models. The authors evaluate their method together with LoRA decomposition of parameters to distill SD3.5L and Flux models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe authors propose to extend the “shortcutting” procedure to the use of pretrained models, which do not have \nThe authors support the proposed method by testing it on distilling SD3.5L and Flux models.\nThe proposed method (together with using LoRA) is efficient and allows for efficient distillation of large models like Flux into a few-step generator under 24 A100 GPU hours.\n\n**Weaknesses:**\n- The main benefit of the proposed algorithm is that it is applicable to the original model without architectural modifications. However, it is not clear whether a direct addition of the step-size conditioning with randomly initialized parameters to the pretrained teacher model (+ using EMA for teacher, as in the proposed method) will be less efficient or provide worse quality of generation. Furthermore, the direct addition of the embedding conditioning does not require\n- Notation consistency. The authors start to derive the self-consistency principle using notation V_{\\theta} and then change it to V_{\\theta_{-}} (Eq. 11 and Eq. 12). I think it would be better to use a general notation for the velocity in this derivation and introduce parametrization of teacher/EMA later, when discussing the algorithm to make notation consistent.\n- Notation of sums in Eq.13 is ambiguous. There are two sums, but it is hard to understand the summation of what they describe, since there is no index of summation in the sums and in the expressions of these sums. In essence, $N$ is a batch size, and $k$ is the number of elements from the batch size used for the first part, and $N-k$ is the number of elements used for the second term. It should be written on the top of sums as their limit, and also a summation index should be introduced (e.g. $\\sum_{n=1}^{k}$, $\\sum_{n=k+1}^{N}$ and $n$ is used in the loss expression). Since it is the main expression of the paper, it is critical to provide it in the most clear form.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}