Title: Gradient Variance Reveals Failure Modes in Flow-Based Generative Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: CFM

Rectified Flows

Generative AI
Abstract: Rectified Flows learn ODE vector fields whose trajectories are straight between source and target distributions, enabling near one-step inference. We show that this straight-path objective reveals fundamental failure modes: under deterministic training, low gradient variance drives memorization of arbitrary training pairings, even when interpolant lines between training pairs intersect. To analyze this mechanism, we study Gaussian-to-Gaussian transport and use the loss gradient variance across stochastic and deterministic regimes to characterize which vector fields optimization favors in each setting. We then show that, in a setting where all interpolating lines intersect, applying Rectified Flow yields the same specific pairings at inference as during training. More generally, we prove that a memorizing vector field exists even when training interpolants intersect, and that optimizing the straight-path objective converges to this ill-defined field. At inference, deterministic integration reproduces the exact training pairings. We validate our findings empirically on the CelebA dataset, confirming that deterministic interpolants induce memorization, while the injection of small noise restores generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the causes of failure modes behind Rectified Flows and Schrödinger Bridge. This paper introduces a gradient-variance diagnostic to measure the transport errors of the learned vector fields. Utilizing the Gaussian transport, the paper shows that the neural networks fail to obtain strict convergence. Moreover, the paper proves that Reflow with noiseless interpolants stagnate when arriving at straight couplings, and memorize when trained on deterministic couplings. The paper also provides a counterexample to show that 1-Reflow could fail. To address to these drawbacks, the paper claims that noise can alleviate these two effects. The paper experiments on synthetic dataset and CIFAR-10 to support this claim.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The perspective to study gradient variance to provide understandings on the limitations of neural networks and Rectified Flows is novel.\n2. The paper provides the counter example to show that 1-Reflow may fail in specific cases, which contributes to the study of Rectified Flows.\n\nWeaknesses:\n1. The clarity of the paper writing can be improved. The title focuses on the “gradient variance”. However, in section 4, most of the claims do not closely relate to the “gradient variance”, making the focus of the paper being obscured. In line 177-183, the key takeaways include multiple serial code (A)s and (B)s. In Proposition 1, the text and the formulas in line 157-159 are mixed together with multiple “and”s.\n2. It remains to be examined how much drawback reflected by the synthetic Gaussian transport counter examples will truly appear in general flow matching methods, which is widely used in various fields with acceptable results.\n3. The paper only presents potential understandings on the stagnation and memorization issues, but it does not provide new methods or ideas to tackle the issues while preserving the benefits of noiseless methods like Rectified Flows. Preferring stochastic flow matching is not a permanent solution, since deterministic and straight paths enjoy other advantages like faster simulation and smaller discretization errors. Moreover, the novelty of utilizing noise in generation is not that significant.\n4. It would be beneficial to experiment on more datasets to show the of the broad applicability of the proposed gradient variance study.\n\nQUESTIONS:\n1. The paper claims in the Gaussian setting that MLP cannot approximate without error. Is this really a critical issue? In experiments, other types of errors like discretization and sample errors also affects the final results, the approximation error could not be the main influencing factor.\n2. What is the main point of Section 3.2? If only an observation of the sample variance of the SBM is provided, it seems that this part is not strong enough to support the main findings and could be removed.\n3. The second last row of the caption of Figure 3 states that ”confidence intervals narrow with higher $\\sigma$”. This seems to be not apparent in Figure 3 since the shaded regions seems to be the same width across different $\\sigma$s?\n\nLIMITATIONS:\nThe paper addressed most of the limitations. As a supplement, it remains to be examined how much drawback reflected by the Gaussian transport will appear in general flow matching methods, which is widely used in various fields.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the common causes of model failures in class of the state of the art (SOTA) fixed point conditional generative models: Rectified Flows (ReFlows), Schrödinger Bridge Matching (SBMs). The main analysis lies in the statistical investigation of the gradient variance to decide whether the resulting vector field is optimal or not. The authors evidently demonstrate counterexamples when the low gradient variance does not coincide with the optimality of the vector field. The presented examples of SOTA models fail to learn exactly the transport between two angled Gaussians. The authors paid special attention to ReFlow model indicating its stagnation on straight couplings tending to memorize deterministic pairings. The noisy interpolants can mitigate these issues on toy examples. The empirical verification of the claimed results provided for conditional flow matching on CIFAR-10 dataset.\n\nSTRENGTHS AND WEAKNESSES:\nThe work provides some rigorous theoretical results on counterexamples for conditional flow matching (CFN) including SBMs and ReFlows.\nThe experiments on Gaussian transport and CIFAR-10 support the claims in general.\nThe work reconsiders very common assumptions in CFNs providing counterexamples.\n\nQUESTIONS:\n1) How ReFlows would behave on datasets given the varying dataset size, dimension, noise?\n2) Can additional regularization (eg. KL on distributions) and learning rate/batch size scheduling avoid memorization effect or suboptimal vector fields?\n3) Can you provide some more (counter)examples on CFN suboptimality while achieving a stationary point on loss curve for SBM?\n\nLIMITATIONS:\n1) The work needs extended experiments with varying dataset sizes, dimensions and (likely) domain types to verify the consistence of provided claims.\n2) The work lack some thorough analysis especially towards SBMs and optimal transport (OT).\n3) The work indicates concrete limitations but hardly discuss how to fix it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the fundamental limitations of flow-based generative models, particularly Rectified Flows and Schrödinger Bridge Matching, through a novel gradient variance analysis framework. The authors introduce a diagnostic tool that measures how learned vector fields induce transport errors by analyzing the variance of gradients during optimization. Their main theoretical contributions include proving that standard neural network architectures cannot exactly represent even simple Gaussian-to-Gaussian optimal transport, demonstrating that Rectified Flows can stagnate when reaching straight couplings and memorize deterministic pairings rather than learning optimal transport, and showing that stochastic interpolants help break these failure modes. The work challenges the common assumption that straight transport paths are always optimal and provides both theoretical insights and empirical validation on synthetic Gaussian data and CIFAR-10.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- I find the theoretical framework quite compelling. The gradient variance diagnostic is genuinely novel and provides insights that pure loss analysis misses - the counterintuitive finding that low variance doesn't guarantee optimal transport is particularly valuable. The mathematical rigor is impressive, especially Lemma 1's proof that MLPs cannot exactly represent the matrix inverse terms in optimal transport vector fields, and Proposition 2's formal characterization of memorization in finite datasets.\n- I think the paper makes important corrections to recent literature. Counte-rexample 1 effectively refutes claims about 1-Reflow sufficiency using rotation-based transport maps, and the theoretical explanation of why stochastic interpolants help (by breaking bijections) provides much-needed understanding of an empirically observed phenomenon.\n- The experimental design is thoughtful, particularly the synthetic Gaussian experiments that allow for analytical verification of theoretical predictions. The CIFAR-10 forward/backward asymmetry analysis provides interesting practical insights about sampling stability.\n\n# Weaknesses\n- My main concern is the limited experimental scope. While the Gaussian experiments are convincing, only CIFAR-10 is used for real-world validation. Given the paper's claims about fundamental limitations of flow models, I would expect evaluation on additional datasets to strengthen the generalizability argument. However, given the theoretical nature of this paper, I think this is more of a minor concern.\n\n- The practical impact feels somewhat incremental. While the theoretical understanding is valuable, stochastic interpolants were already used in practice (SBM), and the main practical recommendation (use noise) was already known. The gradient variance diagnostic, while theoretically interesting, doesn't clearly translate to improved training procedures.\n\n- From a clarity perspective, some technical sections are quite dense. The connection between gradient variance and actual transport quality could be made more intuitive. Figure 2's schematic helps but could be more detailed about the mechanisms involved.\n\nQUESTIONS:\n- Can you provide results on additional real-world datasets beyond CIFAR-10? Your theoretical claims about fundamental limitations would be much stronger with broader empirical validation. Specifically, how do your findings hold for higher-resolution images, different data modalities, or more complex transport scenarios?\n- While the gradient variance diagnostic is theoretically interesting, how can practitioners actually use this tool? Can you provide concrete guidelines for when the diagnostic suggests switching to stochastic interpolants or adjusting noise levels? A practical algorithm that leverages your insights would significantly strengthen the contribution.\n- Comparison with recent advances: How do your findings relate to more recent developments in flow matching, such as consistency models or recent improvements to Rectified Flows? Have any of these approaches inadvertently addressed the memorization issues you identify?\n\nLIMITATIONS:\nYes, the authors adequately address limitations in their conclusion, acknowledging the restricted experimental scope to CIFAR-10 and the preliminary nature of their SBM analysis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes multiple theoretical results related to vector field learning.  They show that various ML architectures cannot approximate an optimal transport vector field without error. Next, they assert that low loss gradient variance does not necessarily correspond to an optimal learned vector field.  They provide a series of observations about the impact that stochastic interpolants have on the gradient variance, as compared to deterministic interpolants.  Finally, they present theoretical results on rectified flow models, showing that deterministic interpolants are susceptible to memorization.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n- The paper is written with mathematical rigor in definitions and derivations.\n- The idea of employing variance of the loss gradient as a “diagnostic” for measuring the faithfulness of learned vector fields to the true optimal transport is creative.\n- Insightful result that the introduction of noise via stochastic interpolants can ameliorate certain issues such as stagnation. \n\n__Weaknesses:__\n- The significance of the paper’s main results are not clear.  While the theoretical exposition is thorough, the paper would benefit from a more assertive stance on what these multiple results collectively signify.  After reading this paper, I do not come away feeling a greater sense of insight about the question posed in the title of this work.\n- Empirical results are evaluated only on a limited dataset (CIFAR-10).  Additional empirical validation on larger, more diverse data domains would highlight the findings more.\n- There is an overall lack of “connective tissue” in the writing.  Lemmas and propositions are offered without enough context, and there is very little flow between sections 3 and 4 (and their subsections).  It feels like an ad hoc assortment of observations rather than an intentionally constructed set of arguments supporting a central thesis.\n\nQUESTIONS:\n- In L137, it is not clear to me why rotation of the vector field is introduced.  There is no explanation or background provided in the text.  Can you please elaborate on why this is included?\n- What does Lemma 1 have to do with the central question that the paper is posing (i.e. what does this tell us about the optimality of vector fields)?  The lemma states that various ML architectures cannot perfectly approximate an optimal vector field — could you please clarify why this is relevant?\n- Proposition 1 — it’s unclear to me what the significance of this proposition is.  Bounds are derived for the variance of gradient.  However, the takeaways offered in the text (L162-L166) almost seem to assert that the variance of the gradient does not reliably indicate transport optimality.  Then what is the insight?\n- Section 4 — what does this section have to do with analysis of the gradient variance?\n\nLIMITATIONS:\n- Results would be more convincing from an empirical perspective if larger, more complicated datasets were used beyond CIFAR-10.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}