Title: High-Order Flow Matching: Unified Framework and Sharp Statistical Rates
Venue: NeurIPS 2025
Accepted: yes
Keywords: Flow Matching

Generative Model

Generative AI

Minimax Optimal
Abstract: Flow matching is an emerging generative modeling framework that learns continuous-time dynamics to map noise into data.
To enhance expressiveness and sampling efficiency, recent works have explored incorporating high-order trajectory information. 
Despite the empirical success, a holistic theoretical foundation is still lacking. 
We present a unified framework for standard and high-order flow matching that incorporates trajectory derivatives up to an arbitrary order $K$. 
Our key innovation is establishing the marginalization technique that converts the intractable $K$-order loss into a simple conditional regression with exact gradients and identifying the consistency constraint.
We establish sharp statistical rates of the $K$-order flow matching implemented with transformer networks. With $n$ samples, flow matching estimates nonparametric distributions at a rate $\tilde{O}(n^{-\Theta(1/d )})$, matching minimax lower bounds up to logarithmic factors.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper construct a theoretically unified k-order flow matching framework which has several high-order FMs and original FM as its special cases. This paper further establish the sample-complexity bound of k-order FM under the usage of transformers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n \n- The theoretical derivations and the formulation of this paper are rigorous from a mathematical perspective.\n- The paper is well written, with a clear and logical structure that makes it accessible and easy for readers to follow.\n- It represents a significant milestone by being the first to construct a unifying framework for high-order Flow Matching (FM) methods. This framework encompasses several previous methods, such as the HOMO approach proposed in reference [1], as special cases. Such a unifying structure is expected to substantially deepen our comprehension of high-order generative models.\n \nWeaknesses:\n \n- The analysis of transformers appears to be somewhat restricted. The paper directly incorporates the complexity results of transformers from prior studies into its own analysis without further in-depth exploration. Conducting a more in-depth PAC learning theory analysis specifically related to high-order FM would greatly enhance the theoretical depth and comprehensiveness of the study.\n- While this paper makes important contributions to the theoretical understanding of FMs, its practical implications are somewhat limited. The stability of the model when the order k exceeds 3 cannot be guaranteed. As pointed out in reference [2], a severe oscillation phenomenon will emerge when higher-order flow matching models are employed.\n \n[1] Chen B, Gong C, Li X, et al. High-Order Matching for One-Step Shortcut Diffusion Models[J]. arXiv preprint arXiv:2502.00688, 2025.\n\n[2] Barrett J H. Oscillation theory of ordinary linear differential equations[J]. Advances in Mathematics, 1969, 3(4): 415 - 509.\n\nQUESTIONS:\n-\tCan the analysis of the transformer in the paper be directly applied to the U-Net and Dit? These two models are commonly employed in current SOTA models like FLUX or SD-3.5.\n-\tThe other one of my concerns is about the understanding of the choice of k. To illustrate the practical usage or limit of the high-order FMs proposed in the paper, it may be beneficial to train several high-order FMs with different k on synthetic data like a 2d chessboard to see the ability of high-order FMs in various choices of k. What will be the best choice of k in practice, the larger the better, or is there a best one?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a theoretical framework for high-order flow matching: models that extend standard flow matching by incorporating higher-order trajectory information. Given recent empirical work on more expressive continuous-time flows, the authors propose a general method that models derivatives of the flow up to an arbitrary order. They formulate a marginalisation technique that transforms the otherwise intractable higher order loss into a tractable conditional regression objective with exact gradients, generalising the flow matching marginalisation technique to higher velocity fields. They also derive sharp statistical convergence rates for high-order flow matching when implemented with transformer architectures by generalising flow matching bounds to an arbitrary order velocity. In addition to the technical contributions, the paper and supplementary material serve as a comprehensive reference on the theory of flow matching. It consolidates existing results on convergence and extends them into a high-order setting, making it a valuable resource for researchers.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\nThe paper is technically strong and builds on foundational work in flow-based generative modelling. The extension to high-order flow matching addresses practical needs in current generative models for greater expressiveness and stability. The high-order marginalisation trick is an elegant formulation which follows the flow matching version in the first order. The statistical analysis is rigorous and thorough. It will be interesting (in further research) to see empirical studies on how increasing the order affects performance or training stability.\n\nClarity:\nThe paper is clearly written and structured. The progression from standard flow matching to its high-order generalisation is easy to follow. Some parts of the statistical results in Section 4 are quite dense and could be more accessible with illustrations, like plots visualising bounds in with respect to order or feature dimension.\n\nSignificance:\nThis work can serve as a central theoretical reference in the flow matching literature. Even without experimental validation, the theoretical results are crucial for researchers developing or analysing continuous-time generative models. It provides a clear direction for further work on scalable training and evaluation of high-order flow models. The focus on transformer architectures and to provide detailed statistical rates for this specific architecture is significant.\n\nOriginality:\nThe paper brings together existing ideas in a novel way. While at least two high-order flow models have been proposed recently, this is the first work to systematically formalise the theory behind them and relate them to a general framework for higher order flow matching. The high-order marginalisation technique is an original contribution that improves the applicability of these models in the future.\n\nQUESTIONS:\nCould other architectures (e.g., MLPs, CNNs) could potentially achieve similar rates? to what extent your results depend on the specific architecture class?\n\nThe theoretical rates themselves do not currently improve with increasing order. Are there any particular settings where you suspect the theoretical gap might close? besides errors in numerical methods.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces High-Order Flow Matching (HOFM), a generalized theoretical framework that extends standard flow matching to incorporate trajectory derivatives up to arbitrary order $K$. To this end, the authors work with a $K$-order velocity field that captures higher-order temporal dynamics and develop a high-order marginalization technique to convert the intractable $K$-order loss into a tractable conditional regression. They claim to establish statistical rates for K-order flow matching with transformer networks, asserting $O(n^{-\\Theta(1/d)})$ convergence rates that match minimax lower bounds up to logarithmic factors. The framework allegedly subsumes standard first-order flow matching ($K=1$) and claims to provide theoretical grounding for recent high-order approaches like FlowMP and HOMO. However, several of these claims are questionable due to unrealizable assumptions and missing critical implementation details.\n\nSTRENGTHS AND WEAKNESSES:\n**Potential Strengths** (if correct):\n\n- Claims to provide a unified theoretical framework that generalizes standard flow matching to arbitrary order K\n- Attempts to establish rigorous statistical analysis with sharp approximation, estimation, and distribution learning rates\n- Claims to show near-minimax optimality of the proposed estimators\n- Attempts to connect various existing high-order flow methods under a single theoretical umbrella.\n\nHowever, the validity of these contributions is questionable given the reliance on unrealizable assumptions (e.g., Assumption 4.1) and the extremely long appendix that makes verification of proofs difficult.\n\n**Weaknesses:**\n\n*Critical consistency issue:* The higher-order velocity fields must satisfy total derivative constraints (Equation 3.3), but the paper fails to explain how this consistency is enforced in practice or during training\n\n*Flawed assumptions:* Assumption 4.1 contains impossible bounds where $C < q(x_1) ≤ C_1\\exp(-C_2|x_1|^2/2)$, which would require the lower bound $C$ to be less than the upper bound for all $x_1$, creating mathematical inconsistencies.\n\n*Lack of empirical validation:* No numerical experiments are provided to validate the theoretical claims or demonstrate practical benefits\n\n*Missing credit:* The paper fails to acknowledge stochastic interpolants alongside flow matching and rectified flows, despite these being essentially equivalent approaches proposed simultaneously.\n\n*Unclear practical implementation:* The paper doesn't address how to ensure the learned velocity fields maintain the required derivative relationships during optimization\n\nQUESTIONS:\n- How do you ensure the total derivative constraints (Equation 3.3) are satisfied during training? \n- Is there a regularization term or constraint optimization approach?\n- Can you provide a corrected version of Assumption 4.1 that doesn't contain contradictory bounds?\n- Why are there no experiments showing the practical benefits of higher-order matching compared to standard flow matching?\n- How does the computational cost scale with order $K$, and when does the higher accuracy justify the increased complexity?\n- What happens when the learned velocity fields violate the consistency constraints. Does the method still converge? Does it introduces biases?\n\nLIMITATIONS:\n- The framework requires K-times differentiable flows, which may be restrictive in practice\n- There is no discussion of computational complexity scaling with order $K$\n- The statistical rates do n0t show significant improvement with increasing $K$, questioning the practical value\n- Missing analysis of stability and error propagation in high-order numerical integration schemes\n- The paper does n0t address how to choose the optimal order $K$ for a given problem\n- No comparison with existing high-order methods in terms of computational efficiency", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a unified framework for flow matching by extending it to high-order flow matching, which incorporates trajectory derivatives with respect to the time variable up to an arbitrary order $K$. The authors analyze several statistical properties of this general framework, including its universal approximation property (Sec. 4.1), the corresponding estimation error rate (Sec. 4.2), the estimation of the 2-Wasserstein distance between the empirical and ground-truth solutions (Sec. 4.3), and its minimax property (Sec. 4.4). The paper does not include any experimental results.\n\nSTRENGTHS AND WEAKNESSES:\nW1. I believe the paper should more clearly emphasize how its contributions differ from previous works such as standard flow matching and [1]. At present, I do not see a meaningful distinction between the proposed method and these prior approaches. In particular, the paper should better explain the role and significance of each theorem, and clarify how each result specifically relates to the extension to higher-order flow matching.\n\n- In Section 3, the authors derive the high-order flow matching objective. However, this seems rather straightforward: by definition, $\\frac{d^k}{dt^k} \\psi$ is simply the velocity field of $\\frac{d^{k-1}}{dt^{k-1}} \\psi$. Given this, the resulting formulation appears to be a direct and obvious extension of standard flow matching, without clear additional insights.\n\n- In Section 4, the paper presents several statistical properties. For example, Section 4.1 shows a universal approximation result, but this appears to be a direct application of the findings in [1]. In Section 4.2, Theorem 4.2 states a $K$-order velocity estimation rate, but the result does not actually depend on $K$ in any meaningful way—it simply reflects a general property of flow matching. I do not see why this theorem is significant in the context of higher-order estimation. Similarly, the estimation result for the 2-Wasserstein distance in Section 4.3 is a standard statistical bound that does not reveal any unique property tied to the higher-order framework. Overall, these theoretical results feel like a straightforward collection of known properties rather than new insights specific to the proposed extension.\n\nW2. The main claim of the paper is that it extends flow matching to KK-th order derivatives. If this is the central contribution, then the paper should demonstrate—either theoretically or empirically—why this extension is meaningful. In its current form, there are no examples, experiments, or concrete settings showing how or why using higher-order flow matching improves performance over standard flow matching. I strongly encourage the authors to include empirical evidence or clearer theoretical justification that illustrates the practical or conceptual benefits of the proposed higher-order estimation.\n\nReferences\n[1] On statistical rates of conditional diffusion transformers: Approximation, estimation, and minimax optimality, ICLR, 2025\n\nQUESTIONS:\nThe paper includes a helpful summary of its contributions in lines 42–59, which makes it easier to follow. However, it remains unclear what exactly the authors improve upon relative to prior work. In particular, I could not find a clear explanation of why one should use high-order flow matching, or what statistical advantage it offers compared to standard flow matching. Overall, Section 4 reads more like a collection of general statements rather than providing concrete justification for the proposed extension. Could the authors please clarify these points and better articulate the practical or theoretical benefits of the high-order formulation?\n\nLIMITATIONS:\nPlease refer to Strength and Weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a unified theoretical framework for high-order flow matching (HOFM), extending the standard FM approach to incorporate higher-order trajectory derivatives. The key contributions include generalized formulation and providing convergence rates for transformer-based HOFM, showing near-minimax optimality in distribution estimation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Overall the paper is well organized despite its length \n- It provides a comprehensive mathematical foundation for high-order flow matching within a unified framework.The proofs are detailed and technically sound\n- The provided analysis aligns with modern architectures (transformers) and applications, making it relevant to real-world generative modeling\n\nWeaknesses:\n- The main weakness lies in a lack of empirical validation. While the theory is robust and sound, the paper does not include experiments to validate the claimed benefits of high-order dynamics (e.g., improved sampling efficiency or training stability). A carefully designed experiments (even on toy datasets) to demonstrate the theory would significantly strengthen the paper \n-  Limited Insight into $K$-dependence is provided: The framework does not theoretically explain why using higher $K$ should improve performance, leaving concerns on the practical advantages of high-order terms\n- There is a lack of discussion on whether the provided framework generalizes popular probability paths, such as optimal transport paths and diffusion based paths, making its relationship to established FM variants unclear. A deeper discussion of design choices and compatibility would strengthen the paper’s impact \n- Complexity of implementation is not studied. The high-order ODE system and transformer architecture may be computationally costly, but this is not discussed in depth \n- The sub-Gaussian and Lipschitz assumptions may not hold for real-world datasets, limiting the applicability of the framework\n\nQUESTIONS:\n- Does higher $K$ always help to improve performance? Are there trade-offs (e.g., computational cost vs. accuracy)? How should we choose $K$ in practice? \n- How does the general method scale with dimension $d$ or order $K$? The rates provided depend on $d$, but no empirical scaling analysis is provided \n- Can we extend the framework to incorporate noise (e.g., for diffusion based sampling)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}