Title: Improved Training Technique for Shortcut Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative models
Abstract: Shortcut models represent a promising, non-adversarial paradigm for generative modeling, uniquely supporting one-step, few-step, and multi-step sampling from a single trained network. However, their widespread adoption has been stymied by critical performance bottlenecks. This paper tackles the five core issues that held shortcut models back: (1) the hidden flaw of compounding guidance, which we are the first to formalize, causing severe image artifacts; (2) inflexible fixed guidance that restricts inference-time control; (3) a pervasive frequency bias driven by a reliance on low-level distances in the direct domain, which biases reconstructions toward low frequencies; (4) divergent self-consistency arising from a conflict with EMA training; and (5) curvy flow trajectories that impede convergence. To address these challenges, we introduce iSM, a unified training framework that systematically resolves each limitation. Our framework is built on four key improvements: Intrinsic Guidance provides explicit, dynamic control over guidance strength, resolving both compounding guidance and inflexibility. A Multi-Level Wavelet Loss mitigates frequency bias to restore high-frequency details. Scaling Optimal Transport (sOT) reduces training variance and learns straighter, more stable generative paths. Finally, a Twin EMA strategy reconciles training stability with self-consistency. Extensive experiments on ImageNet 256x256 demonstrate that our approach yields substantial FID improvements over baseline shortcut models across one-step, few-step, and multi-step generation, making shortcut models a viable and competitive class of generative models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework called the Improved Shortcut Model (iSM), which aims to overcome the limitations of existing shortcut generative models. While these shortcut models offer efficient few-step generation, they often suffer from issues like fixed guidance strength, high training variance, and a bias towards low-frequency details, which can compromise flexibility, stability, and sample quality. iSM addresses these challenges through several key innovations: it incorporates dynamic guidance by making guidance strength a model input and optimizing its application; it employs a multi-level wavelet loss to better capture crucial high-frequency details; it utilizes scaled optimal transport matching to stabilize training and straighten the generation path; and finally, it implements a dual EMA (Exponential Moving Average) strategy to effectively tackle target lag issues that arise during training.\n\nSTRENGTHS AND WEAKNESSES:\n### **Summary Of Strengths:**\n\n- The paper's core contribution lies in its proposed iSM framework's ability to significantly improve the FID scores of shortcut models when generating images in single-step or few-step (e.g., 4-step) settings. This means it achieves better image quality while maintaining high efficiency, effectively solving the performance degradation issue that shortcut models typically face at low sampling steps.\n- By introducing Integrated Guidance and interval guidance, the model allows users to more flexibly adjust guidance strength during inference to balance diversity and fidelity without retraining the model. This overcomes the limitation of previous shortcut models having fixed CFG scales.\n- This work doesn't just tackle a single problem; it systematically addresses multiple critical technical challenges that previously hindered shortcut model performance. It achieves this through various innovative methods, including multi-level wavelet loss, scaled optimal transport matching, and a dual EMA strategy, thereby concurrently resolving issues like frequency bias, high training variance, and self-consistency target lag.\n\n### **Summary Of Weaknesses:**\n\n- Experiments were primarily conducted on ImageNet 256x256. While this is a standard benchmark, a more comprehensive evaluation is needed to understand how the method performs on other types of datasets or at higher image resolutions.\n- The paper lacks a comparison with SOTA multi-step models. This would be beneficial for researchers to understand the performance gap between few-shot and traditional text-to-image models, such as Stable Diffusion 2.5.\n- It appears that the main body of this paper does not include a section on experimental setup, and there is no report on GPU resources in the appendix either. Providing more detailed setup information in the main text would be beneficial for understanding the experiments.\n\nQUESTIONS:\n- Can the authors provide scaling law experiments for larger models?\n- Can the authors add experiments comparing their method with SOTA models across 1 to 50 steps?\n- Can the authors provide the complete training resources and more detailed, reproducible experimental steps?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes several techniques to improve training of shortcut models. Specifically, they introduce 5 improvements: (1) integrate guidance scale as an input to the network, (2) interval guidance, (3) multi-level wavelet loss, (4) twin ema that separates the model weight for inference and self-consistency target, (5) OT matching. By incorporating all of these five components, the authors show shortcut model training can be greatly improved compared with the vanilla model training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is generally well-written and easy to follow.\n- Several components looks novel, e.g., applying OT over aggregated multiple minibatches.\n- The improvement compared with the vanilla shortcut model seems quite significant.\n- The authors provide component-by-component analysis.\n\n**Weaknesses**\n- There's no detailed explanation of how sOT is performed. Could the authors provide more detailed explanation (e.g., algorithm)?\n- OT will require additional computational and memory burden. The paper mentioned it but it would be great if more explicit analysis (e.g., by numbers) were provided.\n- I wonder if Multi-Level Wavelet Function is scalable to resolution, e.g., if it still works with ImageNet 512x512.\n- Interval guidance improves the performance, but at the same moment, it also induces a new hyperparameter $t_{\\text{interval}$ that should be set at training. While the authors show 0.3 is optimal in their experiments, but experiments on other setups (e.g., other datasets or higher resolution datasets) would be make the paper more stronger by showing such values are robust to different setups.\n- The paper only uses FID for quantitative evaluation. Other metrics (e.g., IS, Precision, Recall) can be provided.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce multiple numerical tricks to make class-conditional consistency models competitive with standard class-conditional generative models. The tricks are the following: a wavelet-based loss, a ¨scaled¨ optimal transport flow matching, and leveraging EMA networks with multiple decays (a slow decay for inference, and a large decay to anchor the consistency model).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is easy to follow and tackles an important problem\n\nWeakness:\n- Given the performance improvements (all methods perform very similarly), I would recommend to add other performance metrics, such as the FID with DINOv2 embedding (as recommended in [1], instead of the inception network), maybe also the precision and recall metrics [2]?\n\nQUESTIONS:\n- ¨A critical, previously overlooked issue in shortcut models is the\naccumulation of guidance effects.¨ Is this a well known problem? If yes can authors provide a reference on it? if not, is it a contribution?\n\n\nAblation study.\nI love the fact authors provide a detailed ablation study, however I have some questions about it:\n- Some of the performance gain claims seems rather small and might not be significative, especially for the ¨augmented OT¨ batch parameter K. To the best of my knowledge, OT flow matching makes bring empirical improvement when one can only use a small number of inference steps (NFE<10, which makes sense in your setting), and does not bring improvement with a large number of inference steps (NFE> 50). However, computing OT pairing in the 256*256 image space should not make sens: the dimension is too large compared to the number of samples to give meaningful information. I wonder if authors have any kind of intuition on this?\n- If understand well your benchmark, GANs is still the best 1-step generation model for conditional generation?\n\n\n\n- Are Section 4-6 contribution of the paper?\n\n\nMulti-level Wavelet Function:\n- Could authors provide additional information on the ¨multi-level Wavelet Function¨? Is it mainly a modification of the loss?\n- Do you use a latent space? as in [3]?\n\n\n[1] Stein, George, et al. \"Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.\" Advances in Neural Information Processing Systems\n\n[2] Sajjadi, Mehdi SM, et al. \"Assessing generative models via precision and recall.\" Advances in neural information processing systems 31 (2018).\n\n[3] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an Improved Shortcut Model (iSM) to address key limitations of existing shortcut models in generative modeling. The identified issues include fixed classifier-free guidance (CFG) scales limiting inference flexibility, high training variance from random noise-data pairings, frequency bias toward low-frequency components, and conflicts between EMA updates and self-consistency objectives. To tackle these, the authors propose a unified framework incorporating: (1) Integrated Guidance for dynamic guidance scaling at inference; (2) Multi-Level Wavelet Loss to mitigate frequency bias; (3) Scaling Optimal Transport (sOT) Matching to reduce training variance; and (4) a Twin EMA strategy to balance self-consistency and training stability. Extensive experiments on ImageNet 256×256 demonstrate that iSM achieves significant FID improvements across on1e-step, few-step, and multi-step generation, outperforming baseline shortcut models and comparable methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper identifies both known (e.g., fixed CFG scales) and overlooked (e.g., accumulated guidance effects, EMA-self-consistency conflict) limitations of shortcut models, providing a clear motivation for the proposed improvements.\n2. The proposed four core components in this paper (Integrated Guidance, Multi-Level Wavelet Loss, sOT, Twin EMA) are complementary, forming a cohesive solution that addresses multiple issues simultnaeously.\n3. The authors conduct thorough experiments, including quantitative comparisons with state-of-the-art models, ablation studies for each component, and analysis of key hyper-parameters.\n4. The proposed iSM supports dynamic guidance scaling and reduces inference time by ~50% compared to standard CFG, improving real-world applicability for resource-constrained scenarios.\n\n\nWeaknesses:\n1. All experiments are conducted on ImageNet 256×256 and this paper lacks further solid experimental evaluations on larger-scale experiments, such as ImageNet 512x512, or other text-to-image experiments. It remains unknown how well these techniques and hyper-parameters perform on more complex generation tasks.\n2. Though the proposed method iSM reduces inference steps, the performance is still limited compared to other cutting-edge models, such as REPA-XL/2 (FID=1.42) (compared to iSM-128NFE, 1.88 FID). In addition, it still lags behind multi-step methods (FID 5.27 vs. 1.88 in 128 steps). The gap is significant. \n3. The proposed iSM is tested only with the SiT-XL/2 architecture; its compatibility with other backbones (e.g., DiT, U-ViT) is unproven, limiting its generalizability.\n\nQUESTIONS:\n1. How about other metrics for comparison, such as LPIPS or IS?\n2. The Scaling OT (sOT) method is very interesting. Have you investigated the effect of the update frequency? You currently recompute it every K batches; would performance change if it were recomputed more or less frequently, and how does this interact with the scaling factor K?\n3. The accumulated guidance problem is formalized for N=128 base steps. Does this problem persist for other base step sizes (e.g., N=64 or N=256)? How does iSM’s Integrated Guidance adapt to varying N?\n4. Regarding the Twin EMA strategy, the choice of a fast decay rate of 0.95 for the target network seems relatively low compared to the slow rate of 0.999. Could you provide more intuition on how you arrived at this value? Is there a risk of the target network becoming too noisy or unstable with such a fast update?\n\nLIMITATIONS:\n1. The gap between one-step and multi-step is still significant, which is against the claim of this paper.\n2. Lack large-scale experiments to further support the claims of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}