High-Order Flow Matching: Unified Framework and Sharp Statistical Rates
Authors: Maojiang Su
Jerry Yao-Chieh Hu
Yi-Chen Lee
Ning Zhu
Jui-Hui Chung
Shang Wu
Zhao Song
Minshuo Chen
Han Liu
Venue: NeurIPS 2025 poster ()
Keywords: Flow Matching
Generative Model
Generative AI
Minimax Optimal
Primary area: deep_learning
TL;DR: We present a unified theoretical framework for standard and high-order flow matching and prove their minimax optimality.

Abstract: Flow matching is an emerging generative modeling framework that learns continuous-time dynamics to map noise into data.
To enhance expressiveness and sampling efficiency, recent works have explored incorporating high-order trajectory information. 
Despite the empirical success, a holistic theoretical foundation is still lacking. 
We present a unified framework for standard and high-order flow matching that incorporates trajectory derivatives up to an arbitrary order $K$. 
Our key innovation is establishing the marginalization technique that converts the intractable $K$-order loss into a simple conditional regression with exact gradients and identifying the consistency constraint.
We establish sharp statistical rates of the $K$-order flow matching implemented with transformer networks. With $n$ samples, flow matching estimates nonparametric distributions at a rate $\tilde{O}(n^{-\Theta(1/d )})$, matching minimax lower bounds up to logarithmic factors.

High-Order Flow Matching: Unified Framework and Sharp Statistical Rates
Maojiang Su*t
Jerry Yao-Chieh Hu*t
Yi-Chen Lee
Ning Zhu
Jui-Hui Chung
Shang Wul
Zhao Song
Minshuo Chen
Han Liu
Northwestern University
National Taiwan University
University of Glasgow Simon Institute of Computing, UC Berkeley
*rrnneton University
{emj. jhu, shangwu2028)@u hww.://www.tttttttttttetooommm edu, b10202055Bntu, edu.
zhuning05198gmail.. com, juihui@princeton edu, (minshuo.chens hanliuu)northvestten.
Abstract
Flow matching is an emerging generative modeling framework that learns continuous-time dynamics map noise into data. To enhance expressiveness and sampling efficiency, recent works have explored incorporating high-order tra- jectory information. Despite the empirical success, holistic theoretical foundation is still lacking. We present a unified framework for standard and high-order flow matching that incorporates trajectory derivatives up arbitrary order K. Our key innovation is establishing the marginalization technique that converts the in- tractable K-order loss into simple conditional regression with exact gradients and identifying the consistency constraint. We establish sharp statistical rates the K-order flow matching implemented with transformer networks. With sam- ples, flow matching estimates nonparametric distributions a rate Ö(n -((1/d)) matching minimax lower bounds up to logarithmic factors.
Introduction
We present unified theoretical framework and establish sharp statistical rates for standard and variant flow-man m---hhitchn generative models with high-order velocity fields. rigorous theoretical understanding of such models is crucial in the current of rapidly advancing generative AL. Flow-based generative models, particularly those employing Flow Matching (FM) principles [Lipman et al., 2022, Liu et al., 2022], have emerged powerful class of methods, achieving state-of-the-ar performance across diverse domains such as image, speech, and video generation [Esser et al., 2024, Le et al., 2023, Polyak al., 2024]. Standard flow matching has focused learning first-order trajectory dynamics by matching the instantaneous velocity field [Lipman et al., 2022, Liu et al., 2022, Lipman et al., 2024, Gat et al., 2024, Chen and Lipman, 2023].
However, there is growing interest in leveraging richer dynamical information, such as high-order time derivatives of the trajectory. with the intuition that this could lead more expressive models, smoother generation paths, improved physical plausibility, or more efficient sampling strategies. This trend is evident in recent empirical works. For instance, High-Order Matching for One-Step Shorteut Diffusion (HOMO) [Chen et al., 2025] and Force Matching (ForM) [Cao et al., 2025] have shown that supervising on acceleration and jerk leads to improved smoothness, stability, and precision in generative tasks, particularly in high-curvature regions where first-order methods falter.
Egqual contribution. Version: October 28, 2025. Please arXiv for the full and latest version.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite these promising empirical explorations into high-order dynamics, there lacks a comprehensive theoretical framework that incorporates derivatives up to an arbitrary order K. Rigorous understanding of its statistical properties is also missing. This paper addresses these gaps by introducing High- Order Flow Matching, a generalized theoretical framework for flow-based generative modeling. Specifically, High-Order Flow Matching defines a K'orddr velocity field fc. This field is constructed by concatenating K individual d-dimensional column vector fields u² ,.,,.,, u* Each u component is designed to capture aspects of the flow dynamics, with w² representing the primary velocity and w (with 1) capturing higher-order temporal information of an underlying flow. To complete the theoretical foundation of High- Oighr Flow Matching, analyze its statistical rates when implemented with transformers [Vaswani et al., 2017] to align with modern developments in practice.
Contributions. Our contributions are two-fold:
High-Order Flow Matching: A Unified Theoretical Framework. We present a unified frame- work for Flow Matching models. We first introduce the flow ODEs of any order (Definition 3.1) and the mass conservation formula (Theorem 3.2). A key technical innovation is the high-order marginalization technique (Theorem 3.3). This approach, incorporating consistency constraint, leads to a tractable loss for K-order flow matching (Theorem 3.4). We then prove that High-Order Flow Matching subsumes standard first-order Flow Matching (when K 1, Proposition 3.1) and provides a unified theoretical foundation for understanding emerging high-order flow model approaches. For example, the objective in HOMO [Chen al., 2025], which target velocity and acceleration, are instantiated by High-Order Flow Matching for K 2.
Statistical Rates for High-Order Flow Matching with Transformers. We provide the first rigorous statistical analysis of the High-Order Flow Matching framework when implemented with transformer architectures. We establish sharp approximation rates for transformers learning the K velocity components u' ... (Theorem 4.1), derive corresponding estimation error rates (Theorem 4.2), and further provide end-to-end distribution estimation rates under the 2- Wasssersteein metric (Theorem 4.3). In addition, we show that these rates are nearly minimax optimal up to logarithmic factors (Theorem 4.4). Importantly, our rates match the established near-minimax optimal rates of standard flow matching [Jiao et al., 2024, Fukumizu et al., 2024].
Section 2 reviews preliminary concepts about standard flow matching. Section details the High-Order Flow Matching framework, its properties, and its connections to existing methods. Section 4 presents statistical results. Section 5 summarizes our work and discusses the implications of our findings. The appendix includes the supplementary theoretical backgrounds (Appendix B). the detailed proofs of the main text (Appendices C to G), the statistical rates for standard first-order flow matching transformers (Appendix I) and its proof (Appendices J to N).
Organization.
Notation. We denote the index set (1...., 1} by [7]. Let az(il denote the i-th component of a vector r. Let Z denote integers and Z4 denote positive integers. Given random variables X and Y with marginal densities Ng and Hy respectively, we denote the 2- 2-Wasserstein distance between ba and by by W2(mz: Hyy). Given matrix Z E R L Z||2 and //|F denote the 2-norm and the Frobenius norm. Let & E Rdd be column vectors for le E [K]. we denote col(u' ,..., k) E R&dd as the vertical concatenation of w!, u!....,u* k Let Div be the divergence operator.
2 Preliminaries
In this section, we provide a high-level overview of the Flow Model and Flow Matching (FM).
Flow Model. The flow model transforms Xo = zo from a source distribution /' (c..g., the Gaussian distribution) into samples X, = x1 from a target distribution Q. flow sb [0, 1] Rd - I is time-dependent mapping implementing vb (f, + (z)).. The flow model is continuous-time Markov process (Xt)ostg1 defined by applying a flow th to the random variable X₂- ~ P:
X, = v.
t E [0.
On the other hand, a time-dependent velocity field u [0, x| - Rd implementing u (t, + un defines a unique flow vi via the following ordinary differential equation (ODE):
dıh dt
= (z)((())) with initial condition vo(x) = x.
(2.1)

---

Given flow 1i, the marginal probability density function (PDF) of flow model X; (x(x)) Pt is a continuous time probability path (prJosegi- The probability path p: follows push-forward equation:
Joi 0x
Pr = (o()) == (o (a)) det
(2.2)
Further, by the equivalence of flows and velocity fields [Lipman et al., 2024], given invertible C* diffeomorphism the there exists a unique smooth conditional velocity field n, taking form:
d
2e (x)) ( (x)), with v
dt 2/2
(2.3)
For arbitrary probability path pt, we define velocity field ur that generates pt. if its flow the satisfies (2.2). Continuous Normalizing Flow [Chen et al., 2018) models the velocity field u, with a neural network u". Once we obtain well-trained u², we generate samples from solving ODE (2.1).
Flow Matching. Instead of training flow model by maximizing the log-likelihood of training data [Chen et al., 2018) 8], flow matching [Lipman et al., 2022] is simulation-free framework to train flow generative models without the need of solving ODEs during training. The Flow Matching objective is designed to match the probability path (pt)ogest which allows us to flow from source Po P to target p1 = Q. Suppose un generates such probability path pt, the flow matching loss is
(FM( = E t,Xi-p:
(X,, un (111
(2.4)
where f ~ U0, 1], u is a neural network with parameter 9. Flow Matching simplifies the problem of designing a probability path pt and its corresponding velocity field oe by adopting conditional strategy. Formally, conditioning on any arbitrary random vector E IRm with PDF pz, the marginal probability path Pa satisfies
(r(x)= = f (=(xl=))p(()ddz.
(2.5)
Suppose conditional velocity field (x|2)) generates Pt (x/l2), Lipman el. [2022] show that following marginal velocity field u, generates marginal probability path pt under mild assumptions:
Uy (x) = (=(=)<))2)(((()))dd with (2)1(2(1) = pr(x(=)p2(2)
(2.6)
(x(2)
where the second equation follows from the Bayes' rule. Combining above, the tractable conditional flow matching loss LCFM, which satisfies V VoLCFM VoLFM (0). is defined as:
(CrM(() X.2--x,X:p..111
E
u | tX,,t) (X,,2))13
(2.7)
Affine Conditional Flows. The conditional flow matching loss works with any choice of conditional probability path and conditional velocity fields. In this paper, we consider the affine conditional flow with independent data coupling following [Lipman et al., 2022, 2024]:
(x(())) = 1100 + o
(2.8)
where μt, oz [0, 1] + [o, 1] are monotone smooth functions satisfying
Hg oi μi oo 1, and diee da, 0 for tE (0, 1).
(2.9)
dt dt
Setting Z = X1 Q. Xo 1). the flow v. induces the probability flow (X) N (MrX1,0711 and velocity field
u (x(or) = (v. (x(a)) 0,(x - (rTti) + ju
(2.10)
de
Further, using the law of unconscious statistician with X. (ioo))))), the conditional flow matching loss takes the form
(o) No0
E
[Ilu to) -(4xx + x1
(2.11)
3

---

In practice, for collected i..d. data points { (2.1 1) is implemented with Monte- Carlo simula- tion. To avoid instability, we often clip the interval [0. 1] with to and T. Namely, for any velocity estimator u., we consider the empirical loss function LcrM(u"):
(CPM(u") 1
1 T lo No-N(
E [llu (uea; + o,)o,) - (Wiر + dx [j
(2.12)
Transformers. Throughout the paper, we parameterize u by transformers. Due to space limit, we defer formal definition of transformer networks Appendix
3 High-Order Flow Matching
This section extends the flow matching framework in Section 2 to incorporate high-order trajectory information. Recall that these high-order dynamics are proven be relevant to further improving the performance and stability of flow matching. Specifically, in Section 3.1, we first define a high-order velocity field fe using an ODE system and subsequently prove its equivalence the mapping flow the (Theorem 3.1). Furthermore, we derive the corresponding Liouville's ' equation (Theorem 3.2), which demonstrates mass conservation for this high-order system. Building on this foundation, Section 3.2 addresses the learning objective. We first propose the high-order Flow Matching loss (Definition 3.2).
However, similar to flow matching (Lipman al., 2022], direct optimization is intractable. To address this, we establish the high-order marginalization trick under consistency constraint (Theorem 3.3). The method allows to derive tractable high-order conditional flow matching loss that preserves the original loss's gradients (Theorem 3.4). Section 3.3 clarify how that High-Order Flow provides unifying theory. Specifically, we demonstrate that high-order flow matching subsumes existing flow-based generative modeling techniques, with standard Flow Matching serving as a foundational instance within our framework.
3.1 High-Ordes Flow Model
For E [0, 1]. let the and Pt be the time-dependent flow mapping and probability paths follows Section 2. Instead of using velocity field u, to construct flow th via the ODE (2.1), we propose using
K-order velocity field fe Rdd - Kd to construct t/e
Definition 3.1 (High-Order Velocity). Let E [0, 1). flow t/2 can define K-order velocity field ft Rk Rd via the following ODE:
del (x(()) de (a))
u (1 t)
u² (x) 00)
d di Vr
= =s((V)) with vo(x) = x,
(3.1)
t (x)
K (x. 1)
where y1. = col(v)(()) 4 2h (z),
- (x)) = col(x; (0) xi
x (K-1)) E IR kd and wt
gkd 1] Rd is k-th order velocity field for all k E [K]. Moreover, notice that x (Xn) is random variable since Xu ~ p. Then, the extended state variable of order K is the random vector
d²
Y, = =o xx
X x((- E IR. kd
with
x(() = dt ve(xx)
(3.2)
x!!
For k .,..., K 1, define p Rd + R the probability density function of xx() Denote P: : Rdd + IR as the probability density function of Y, = [X ......!!! (K--)) DD at time t. For simplification, we define Y, satisfy d Y, = f((Y.) (3.1) and (3.2) hold.
Remark 3.1 (Total Derivative Constraints). The ODE (3.1) imposes a sequence of total derivative
constraints on the velocity fields u! (xx (0) ,t),..., u (z; (0) t), for any & E [K]:
= (x( (0) .)) de
d (x)
d =
- (x.
(0)
t)
① u (x (0) f) + Vu (x (0)
t) u (x (0) .t,,
dt
at
(3.3)

---

where u² (xxx () "
This recursive relation reveals that the velocity fields induced by the flow Uh are not independent, but instead coupled through the structure of the ODE via (3.3).
Remark 3.1 guarantees the equivalence between flows th and K-order velocity field fx.
Theorem 3.1 (Flow-Velocity Equivalence via ODE). Define the class of structured k-order velocity fields those of the form:
(K (K-1)) 1) where w² RKd [0, 1] + locally lipschitz in yt and continues in for any k [K]. Suppose .t..... u (xt (0) ,t) satisfy total derivative constraints (3.3). Then, for any initial condition E Rd the ODE at Ur - (Wve) exists unique local solution ye, which defines a
(B)) = col(u' (x) t)....,,"*
(0)
"x (x, (x,",,) t)) E Rd yi col(2,
E RRdd
the velocity fields u² (a: (0)
K K-times differentiable flow (x(x)) = 2i and satisty de
d (x)) = It (k) for all [K].
Conversely, any K K-times differentiable flow 1/2 R2d + IRd defines a velocity field ft via (3.1).
Proof. Please see Appendix C.I for a detailed proof.
Recalling from Section and the flow-velocity equivalence established in Theorem 3.1, the K K-order velocity field fo governs the evolution of the probability density Pt for the K-order state Y, The precise relationship describing this evolution is captured by the mass conservation formula:
Theorem 3.2 (Mass Conservation of High-Order Flow). Let y1
(x ....., (K 1 RR Let [t) E Rkd, where w² (z t) is locally Lipschitz and integrable for all & E [K]. pe R + R be a time-varying probability density over the extended state Y, I Kd follows Definition 3.1. Then the following statements are equivalent:
x1
velocity field (3r) = (u (x) (0) t)..... n (x (0)
1. The pair (St; satisfies the Liouville's equation on the extended space:
① σ. (i(1) + V (pe(v))e(v) = 0, for all / E [0, 1).
2. Following Definition 3.1, the probability law of Yr evolves under the flow:
d Y, = f(Y)), with Yo po:
Yi ~ Pt.
(3.4)
dt
For some arbitrary probability path Pr. we define f. generates Pe if (3.4) holds.
Proof. Please see Appendix C.2 for a detailed proof.
3.2 High-Order Flow Matching
To model the Koorder velocity field f., introduce following high-order flow matching loss:
Definition 3.2 (High-Order Flow Matching Loss). Let fa denote the ground truth K K-order velocity field and f be its estimator parameterized by neural network. Let pt be the probability density function of Y, Then, the K- Koorder Flow Matching objective minimizes the following regression loss:
(0) = E L.Yippt
[ (()))
where dissimilarity measure between vectors, such as the squared norm
Similar to standard flow matching, the ground truth velocity fe is intractable. To address this, we adopt the conditional flow matching loss train our model, leveraging the equivalence between the flow matching loss and its conditional counterpart. As preliminary step, we introduce the marginalization trick for high-order flow matching.
Theorem 3.3 MMarginalization) Recall that for some arbitrary probability path Pt. fc generates Pt Y Pe for all [0, 1). Let Z be random variable, if ft(x)2) is conditionally integrable and gen- erates the conditional probability path pe(-[z), then the marginal velocity fi := generates the marginal probability path p..
5

---

Proof. Please see Appendix C.3 for a detailed proof.
Now we are ready to prove the higher version the equivalence between the flow matching loss and conditional flow matching loss. We first define the tractable K-order conditional flow matching loss:
(C) (.Z.YemP1z Jf (Y))) (X)))
E
(3.5)
Following Lipman et al. [2024], we specify the dissimilarity metric ((: Bregman divergence, which measures the distance between vectors u, Rkd D(u, v) :- ((u) ( + (u
[ where 4 : I kd + IR is strictly convex function defined on a convex domain ! C I kd
Bregman divergences possess key property allowing interchanging gradients and expectations (Holderrieth et al., 2025, Lipman et al., 2024]:
V D(E())))))) (
for any random vector EE ERdd
(3.6)
This property implies that the gradients of the flow matching loss and the conditional flow matching loss are identical, making the two objectives equivalent for training.
Theorem 3.4 (Gradient Equivalence of Losses). Let the Flow Matching loss C be defined in Definition 3.2, and the Conditional Flow Matching loss LERM be defined as in (3.5). Then, when D(..)) is Bregman divergence, the gradients of the two losses coincide:
( = VC (O).
Proof. Please see Appendix C.4 for a detailed proof.
We now consider training the model using the pre-constructed conditional flow the (x1) described in Section 2. By the equivalence between flows and high-order velocity fields (Theorem 3.1), there exists unique smooth conditional K K-order velocity field fa such that the conditional trajectory y satisfies the ODE: d dt ft fVt),, in accordance with (3.1), Following Definition 3.1, we specify
ve(x σT) 1, ogæ, which induces family of k-th order velocity fields w². By Definition 3.1, for & E [K]. we have
d
d
w (zt .t)
(x()..
(By Definition 3.1)
di
de
Because th is an invertible diffeomorphism, we define x' = 1/ (x) and obtain
u (+)(())) = u² = d
dt ( (
Extending this to the conditional setting, the conditional k-th order velocity field becomes
d
(x, (z, + = des (/eb (x [
(3.7)
Combining the results above, we now revisit the tractable training loss by setting = x(() q
(0)
E
(By (3.5)
t, T.Xio>ng.Yorp Yomp,
xx fo: (1x(
For further simplifications, we adopt the squared & norm as the Bregman divergence. Let x² denote the k-th order velocity field, and u be its estimator parameterized by neural network. Denoting the distribution of the k-th order state as x((A)) ~ ptt the training objective becomes
LK (0)
E
[11((((1(()) (Y)
(By (3.5)
t.x(00 ~g Yimp
(1)
ux
K
E 1,x(0)
Σ
I (Y,, r (Y.,
(By Definition 3.1)
(1))))))
xx (o
"The marginal velocity ft implies consistency constraint: ut f (v/2) pr(2)))dz for all & [K].

---

d due (x, - u (x," 1)]]
(By (3.7)
1 mq
x ppp (1
K
d the (x 1x10) w# (x, .))1
E 1= (, xg ) x(#-1) are determined by X via the relation x²()
(3.8)
The intermediate states x(") d' des (/(x)) x=(x)) Therefore, the inside expectation only needs to be taken over x!
Now, we consider the affine conditional flow (z((=/1)) = (ad1 + ogæ follows Section 2. Applying (3.8), the high-order conditional flow matching loss takes the form
CK (O) =
Σ
[11C4 (8) x + of of x - -u 1: (x.) ,
In practice, we train the general high-order velocity estimator u.. k.0 with i.i.d samples { by optimizing the empirical high-order conditional flow matching loss:
K
[I(ut + to, (2) xx0) * (x (0) 113 (3.9) T to to _0
CFF =
E
k=1
A significant theoretical consequence of learning the complete K-order velocity field f. is the ability to employ high-order numerical integration schemes for sampling. For instance, to solve the ODE (3.1), we use K-th order Taylor expansion with step size h for the numerical integration:
h2 i+h x, + hu (x. t) + u
h
2.0 (x
(x (x()),
K.0 (x
(3.10)
2!
K!
3.3 Unified Perspective High-Order Flow Dynamies
We show that K-order flow matching framework offers significant unification perspective and theoretical foundation on existing flow-based generative modeling. Firstly, our framework subsumes standard first-order Flow Matching [Lipman al., 2022] direct special case.
Proposition 3.1 (Reduction Standard First-Order Flow Matching). When K 1, the entire K-order flow matching framework, including the governing ODE, the probability path definition via the continuity equation, and the K-order flow matching objective, becomes precisely equivalent to the standard first-order Flow Matching framework as detailed in [Lipman et al., 2022, 2024].
Proof. Please see Appendix C.5 for a detailed proof.
Proposition 3.1 establishes our K-order framework as a strict generalization standard first-order K-order framework provides
Flow Matching. Beyond encompassing established methods, robust theoretical structure for understanding models that leverage high-order trajectory dynamics.
For instance, HOMO framework [Chen et al., 2025] defines its training objective ([Chen et al., 2025, Definition 4.3]) by matching network predictions against the true velocity and acceleration jectories. Removing the regularization term (aligns with our total derivative constraints Remark 3. 1), their loss is also a direct instantiation of our K-order framework's 's objective (Definition 3.2) for K = 2. Furthermore, while the Force Matching (ForM) model [Cao al., 2025] introduces specific relativistic constraints, its fundamental generative mechanism involves matching a target "force" field ([Cao al., 2025, Definition 4.1.). Given that force is proportional to acceleration, if separated from its relativistic regularization, aligns with matching the second-order information captured within our K framework.
tra-
In summary, the K-order flow matching framework serves as a unifying theoretical structure. not only subsumes standard flow matching but also provides formal grounding for models that have intuitive or empirical benefits of incorporating richer, high-order dynamical information. The subsequent statistical analysis in Section 4 builds upon this unified perspective.
7

---

Statistical Rates of High-Order Flow Matching Transformers
This section characterizes sharp statistical rates for K Korder flow matching transformers. Building on Section 2 and Section 3, we consider the case of affine conditional flow with independent data coupling. We focus on transformer architectures as Flow matching (FM) with transformers powers today's best generative models, including MovieGen [Polyak et al., 2024] and Voicebox [Le et al., 2023] by Meta, and Rectified Flow [Esser et al., 2024] by Stability AL. Section 4. and Section 4.2 establish bounds for the approximation and estimation of the K -order velocity. Based on the K K-order velocity estimation rates, Section 4.3 analyzes the distribution estimation rate under the 2- 2-Wasserstein metric. Finally, Section 4.4 presents the nearly minimax optimality of the K-order velocity estimators.
Transformers. We defer standard definition of transformer to Appendix due to the page limit.
4.1 High-Order Velocity Approximation
To establish statistical theory for K-order flow matching transformers, we first investigate an approximation theory for the K Korrder velocity under sub-Gaussian assumption. In particular, we characterize the regularity of the target density function ((x1) with Hölder smoothness, defined by:
Definition 4.1 (Hölder Space). Let E _ and let B ki + 7 denote the smoothness parameter, where 1 [ and y [0, 1). Given function _R I, the Hölder space H (12)) is defined the set of a-differentiable functions satisfying: H. (IR") { Rdd R ( )o).
where the Hölder norm |I (B()))) satisfies:
1 f(x) 8 f(x')
[[R))) = Σ sup 10th f(z)] + max
sup
a:||al| a =h xØ
| - I'
[|a]]]
Also, we define the Hölder ball of radius B by H (RL, B) = ( Rd + R [1 < BJ.
With Definition 4.1, we state our assumption on the target density function q(x1):
Assumption 4.1 (Sub- Gaussian Property and Hölder Smoothness of Target Distribution). The target distribution g(x1) ) E 7e (Rd., B). Further, there exist two positive constants C1 and C2 such that ((x1) S C1 exp( - C a 12/
Assumption 4.1 provides tail bound for the approximation error, and leverage it to address the error outside the bounded domain where our transformer approximation applies. We now present the approximation theory for high-order flow matching transformers.
Theorem 4.1 (K (ordrder Velocity Approximation with Transformers). Assume Assumption 4.1. Suppose the k-th order velocity field x² (z, t) is Li-Lipschitz for all k E O,... K 1 in dd-distacce. Let E (0, 1) be the precision parameter satisfying < O(N -) for some N E N and smoothness parameter O. Then, there exists transformers u1.8 ( (s, t), ................................ k.8 (x. t) Thh... h.s.r such that for any Rd and [0, 1), it holds:
T E/
Σ
K
JRs. [l,=0(z, 11
[
- (x, (x, pe(x)dedt = O(N (log N) 2 -).
k=1
Further, for all &e E [K], the parameter bounds in transformer network class satisfy
CKQ, C2.0 = O(> NNN0(22++++) (log 2)2d+1>; Cov, C2 O(N-3);
CF C20 = 0 Vlo N Lx-1); CE = O()) CT = O(Lx-))
where --1 = 0 log 2y14+3 is the unverse-temperant hides all polynomial factors depending on d,,, d, L,, B, C1, O2.
scaling in the softmax function and ((-)
Proof. Please see Appendix D for detailed proof.
4.2 High-Order Velocity Estimation In this section, we apply the approximation results Section 4.1 to derive K-order velocity esti- mation rates (Theorem 4.2), Given set of i..d samples {x, J. train transformer networks
8

---

u 1.
u K.0 by minimizing the high-order empirical conditional flow matching loss (3.9);
CK CFM
T T to No.N(
1
E
[ (k) + 0² x² - u (x, t) dt.
We evaluate the performance of estimators u 1.0 u K.0 through the K-order flow matching risk:
Definition 4.2 Frigh-Order Flow Matching Risk). Let u be the estimator of the k-th order velocity field u². Let @ be the collection of parameters of u ..... uK. We define the flow matching risk Rx (+) as the sum of the expected mean- mean-squared difference between h.e and w²:
K
Rx((O)= = Σ
E wmpp
[1lu* (x, us (x, 1)1 di,
T
to
k=1
where the density function Pi represents the probability density function of x (Definition 3.1).
Further, assume the path coefficients of the affine conditional flow preserve regularity.
Assumption 4.2 (Path Regularity). Consider the affine conditional flow t/ (x(xx()) = ,, Øgt, the k-th derivative of path coefficients o, and 14 continuous [to, 7). where to, T [0, 1].
Assuming k-th order velocity Lipschitz continuity and affine path regularity (Assumption 4.2), the following theorem presents the upper bounds on estimation error R (9) with sample size
Theorem 4.2 (High-Orde Order Velocity Estimation with Transformer). Assume Assumption 4. and Assumption 4.2. Let @ E ETA Thh... be the estimator of the k-th order velocity field u trained by minimizing the high-order empirical conditional flow matching loss (3.9), Let θ be the collection of parameters k for E [K]. Suppose the k-th order velocity field (x, t) Lx Lipschitz for all k =0,... O...., K Suppose we choose the transformers in Theorem 4.1, then
Rx (+) 0(n Tha (log 11) 10dz)
E
where d is the feature dimension.
Proof. Please see Appendix E for detailed proof.
4.3 High-Order Distribution Estimation
Based on the K-order velocity estimation result in Theorem 4.2, we further analyze the distribution estimation rate for K K-order flow matching transformer. The next theorem presents the upper bounds on the expectation of 2- Wasserstein distance between the target and estimated distribution induced by estimators w K.9 trained by optimizing the empirical conditional loss (3.9).
Theorem 4.3 (High-Order Distribution Estimation under 2- Wasserstein Distance). Assume As- sumption 4.1 and Assumption 4.2. Let P/K be the estimated distribution at time T. Then, it holds
WW2 PK = 0(n T (log n nd)
where d the feature dimension.
Proof. Please see Appendix for a detailed proof.
4.4 High-Order Minimax Optimal Estimation
We show that the Korrder flow matching transformers achieves nearly minimax optimal rate:
Theorem 4.4 (Minimax Optimality of High- Orderr Flow Matching Transformers). Assume that the target density function satisfies ((x1) E ([-1,,1 f B) and ((x)) 2 C' for some positive constant C. Then, under the setting of 18d( + 1) d, + 23, the distribution estimation rate of flow matching transformers presented in Theorem 4.3 matches the minimax lower bound of Hölder distribution class in 2- Wasserstein distance up log and Lipchitz constants factors.

---

Proof. Please see Appendix G for a detailed proof.
Remark 4.1 (Comparison with Existing Works), Flow matching with ReLU networks is nearly Munimax-optimal on Besov densities in W2 [Fukumizu et al., 2024], and kernel methods achieve comparable rates in W1 [Kunkel and Trabs, 2025]. We extend these results to all orders K and to the major powerhouse in practice: transformer architectures. Our analysis proves that flow-matching transformers attain near-minimax rates on Hölder densities in W2 with assuming Lipschitz velocities, subsuming the first-order case at K = 1. Please see Appendix for details.
5 Discussion, Limitation, and Open Question
Section 3 and Section 4 establish unified theoretical framework for High-Order Flow Matching and offer a sharp statistical analysis of High- Flow Matching transformers. As discussed in Section 3.3, this framework subsumes the not only original first-order [Lipman et al., 2024, 2022] but also many high-order flow matching models [Chen et al., 2025, Cao et al., 2025]. Furthermore, the established sharp statistical rates provide rigorous support for all models under this unified framework. This broad theoretical guarantee, covering both first-order and high-order approaches, helps explain the empirical success of the high-order flow models.
While our analysis provides foundational statistical guarantees, the compelling empirical evidence and our current theoretical framework present an intriguing open question: it does not elucidate significant improvement in statistical rates with increasing order K. In addition, while our framework offers unified perspective for numerous empirical studies, these often assume the validity of the consistency constraint within the marginalization process (Theorem 3.3). Our research indicates that the general validity of this constraint, or indeed the derivation of similar conclusions under broader conditions, remains an open question. We identify three primary directions for future work stemming from these considerations: (i) Sampling Efficiency: The High-Orde Flow Matching framework enables the use of a K-th order Taylor expansion sampler. This sampler achieves a local truncation error of O(hK+1) per step, with all K velocity components w/ evaluable in parallel. Future empirical work should investigate whether this high-order accuracy per step translates into practical benefits, such requiring fewer function evaluations for target sample quality or faster convergence to high-fidelity samples. (ii) Stable Approximation Error Propagation: In standard flow matching using Runge-Kutta Methods, the sequential nature means approximation errors in us evaluations may propagate and amplify within a single step as they influence subsequent intermediate calculations. However, our K Korder flow matching approach solves the ODE without this feedback loop, which might leads to more stable error propagation. (ii) Relaxing the Consistency Constraint: significant direction for future research involves exploring methods to either remove or relax the consistency constraint highlighted in Theorem 3.3.
6 Concluding Remarks
In this work, we introduce High-Order Flow Matching, generalized theoretical framework for flow-based generative modeling. Specifically, we characterize the relationship between flow tht, K-order velocity field ft. probability path pt through governing ODE and mass conservation formula (Definition 3.1 and Theorem 3.2). Then we purpose the K-order flow matching loss and establish a tractable equivalent conditional K -order flow matching loss (Theorem 3.4) via high-order marginal- ization trick (Theorem 3.3). Further, we prove that High-Order Flow Matching subsumes standard first-order Flow Matching for K = (Proposition 3) and providing unified theoretical foundation for understanding emerging high-order flow model approaches such as HOMO [Chen et al., 2025]. Our second primary contribution is the first rigorous statistical analysis of this High-Order Flow Matching framework when implemented with transformers. We establish sharp approximation, estimation, and distribution learning rates (Theorems 4.1 to 4.3), and demonstrate their mear-minimax optimality up to logarithmic factors (Theorem 4.4).
Related Work. We defer an extended discussion on related work to Appendix A due to page limits.
Acknowledgments
JH would like to thank Sitan Chen, Mimi Gallagher, Sara Sanchez, T.Y. Ball, Dino Feng and Andrew Chen for valuable conversations; David Ting-Chun Liu, Sophia Pi and Mingcheng Lu for pointing out typos; Mingcheng Lu, Weimin Wu, Yibo Wen and David Liu for collaborations related topics; and the Red Maple Family for support. The authors would like to thank the anonymous reviewers and program chairs for constructive comments.
10

---

Lastly, JH dedicates this work to the memory of Prof. Bi-Ming Tsai, who passed away during its camera-ready preparation (Oct 2025). Her teachings on health, cooking, good taste in food, and the art of thoughtful living continue to inspire JH.
JH is partially supported by the Walter P. Murphy Fellowship. Han Liu is partially supported by NIH R01LM1372201, NSF AST-2421845, Simons Foundation HPPSAA--00105111 Abb Dolby and Chan Zuckerberg Biohub Chicago Spoke Award. This research was supported in part through the computational resources and staff' contributions provided for the Quest high performance computing facility at Northwestern University which jointl supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
References
Michael S Albergo and Eric Vanden-Eijnded Building normalizing flows with stochastic interpolants. The Eleventh International Conference on Learning Representations, 2022.
Silas Alberti, Niclas Dem, Laura Thesing, and Gitta Kutyniok. Sumformer: Universal approximation for efficient transformers. In Topological, Algebraic and Geometric Learning Workshops 2023, pages 72-86. PMLR, 2023.
Joe Benton, George Deligiannidis, and Arnaud Doucet. Error bounds for flow matching methods. Transactions on Machine Learning Research, 2023.
Yang Cao, Bo Chen, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. Force matching with relativistic constraints: physics-inspired approach stable and efficient generative modeling. arXiv preprint arXiv:2552.081555 2025.
Bo Chen, Chengyue Gong, Xiaoyu Li, Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. High-order matching for one-step shortcut diffusion models. arXiv preprint orXiv:2502.00688, 2025.
Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries. The Twelfth International Conference on Learning Representations, 2023.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Benjamin I. Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793-5831. PMLR, 2022.
Patrick Esser. Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-furst international conference on machine learning, 2024.
Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprinl arXiv:2403.1 1968, 2024.
Kenji Fukumizu, Taiji Suzuki, Noboru Isobe, Kazusato Oko, and Masanori Koyama. Flow matching achieves almost minimax optimal convergence. arXiv preprint arXiv:2405.20877 2024.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 133345-133385, 2024.
Thomas Hakon Gronwall.
Note on the derivatives with respect to a parameter of the solutions of a system of differrntial equations. Annals of Mathematics, 20(4):292-296, 1919.
Emst Hairer, Syvert P. Nørsett, and Gerhard Wanner. Solving Ordinary Disferential Equations I: Nonstiff Problems, volume 8 of Springer Series in Computational Mathematics. Springer- Verlag, Berlin, Heidelberg, second revised edition edition, 1993. ISBN 778-3-540-56670-0. doi: 10. 00.1007/978-35440778

---

Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. In The Thirteenth International Conference on Learning Representations, 2025.
Jerry Yao-Chich Hu, Weimin Wu, Zhuoru Li, Sophia Pi, Zhao Song, and Han Liu, On statistical rates and provably efficient criteria of latent diffusion transformers (dits). Advances in Neural Information Processing Systems, 38:31562-3168 -36628,, 2024.
Jerry Yao-Chieh Hu, Hude Liu, Hong-Yo Yu Chen, Weimin Wu, and Han Liu. Universal approximation with softmax attention. arXiv preprint arXiv:2504. 15956, 2025a.
Jerry Yao-Chieh Hu, Wei-Po Wang, Ammar Gilani, Chenyang Li, Zhao Song, and Han Liu. Funda- mental limits of prompt tuning transformers: Universality, capacity and efficiency. Proceedings of the Thirteenth International Conference on Learning Representations, 2025b.
Jerry Yao-Chieh Hu, Weimin Wu, Yi-Chen Lee, Yu-Chao Huang, Minshuo Chen, and Han Liu. On statistical rates of conditional diffusion transformers: Approximation, estimation, and minimax optimality, In Proceedings of the Thirleenth International Conference on Learning Representations, 2025c.
Guillaume Huguet, James Vuckovie, Kilian Fatras, Eric Thibodeau- Pablo Lemos, Riashat Islam, Cheng- Liu, Jarrid Rector-Brooks, Tara Akhound- -aaooun,-SSdegh,,. Michael Bronstein, Alexander Tong, and Avishek Joey Bose. Sequence-augment se(3)- matching for conditional protein backbone generation. The Thirty-eights Annual Conference on Neural Information Processing Systems, 2024.
Haotian Jiang and Qianxiao Li. Approximation rate of the transformer architecture for sequence modeling. Advances in Neural Information Processing Systems, 37:68926-68955, 2024.
Yuling Jiao, Yanming Lai. Yang Wang, and Bokai Yan. Convergence analysis of flow matching in latent space with transformers. arXiv preprint arXiv:2404.02538, 2024.
Tokio Kajitsuka and Issei Sato. Are transformers with one layer self-attention using low-rank weight matrices universal approximators? The Twelfth International Conference on Learning Representations. 2023.
Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations, 2022.
Lea Kunkel and Mathias Trabs. On the minimax optimality of flow matching through the connection to kernel density estimation. arXiv preprint arXiv:2504. 13336, 2025.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36: 1400441003, 2023.
Yingyu Liang, Zhizhou Sha, Zhenmei Shi, Zhao Song, and Mingda Wan. Hofar: High-order augmentation of flow autoregressive transformers. arXiv preprint arXiv:2503.08032, 2025.
Yaron Lipman, Ricky TQ Chen, Heli for generative modeling. The Eleventh International Conference on Learning Representations, 2022.
Maximilian Nickel, and Matt Le. Flow matching
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. arXiv preprint 24Xi:24122.06644 2024.
Hude Liu, Jerry Yao-Chich Hu, Zhao Song, and Han Liu. Attention mechanism, max-affine partition, and universal approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and trans- fer data with rectified flow. The Eleventh International Conference on Learning Representations, 2022.
12

---

Jonathan Niles- Weed and Quentin Berthet. Minimax estimation of smooth densities in wasserstein distance. The Annals of Statistics, 50()):519-1150, 2022.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference Machine Learning, pages 26517-26582. PMLR, 2023.
Sejun Park, Jaeho Lee, Chulhee Yun, and Jinwoo Shin. Provable memorization via deep neural networks using sub-linear parameters. In Conference on learning theory, pages 3627-3661. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Tho Ma, Ching- Yao Chuang, al. Movie gen: A cast of media foundation models. CoRR, 2024.
Maojiang Su, Mingcheng Lu, Jerry Yao-Chiee Hu, Shang Wu, Zhao Song, Alex Reneau, and Han Liu. A theoretical analysis of discrete flow matching generative models. arXiv preprint arXiv:2509.22623, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017
Konstantin Yakovlev and Nikita Puchkin. Generalization error bound for denoising score matching under relaxed manifold assumption. arXiv preprint a 13662, 2025.
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-ssqqeecce functions? International Conference on Learning Representations, 2019.
Tong Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2(Mar):527-550, 2002.
13

---

Appendix
A Related Work
16
Supplementary Background: Transformer Block
17
B.1 Transformers
17
B.2 Lipschitzness of Transformer Network
18
Proofs in Section 3
19
C.1 Proof of Theorem 3.1
19
C.2 Proof of Theorem 3.2
20
C33 Proof of Theorem 3.3
21
C.4 Proof of Theorem 3.4
21
C.5 Proof of Proposition 3.1
22
D Proof of Theorem 4.1
23
D.I Auxiliary Lemmas
23
D.2 Main Proof of Theorem 4.1
26
E Proof of Theorem 4.2
28
E.1
Preliminaries
................................
28
E.2 Auxiliary Lemmas
................................
29
E,3 Main Proof of Theorem 4.2
Proof of Theorem 4.3
37
G Proof of Theorem 4.4
39
H Preliminaries: Universal Approximation of Transformers
40
I Statistical Rates of Flow Matching Transformers (FMTss
54
LI
Velocity Approximation: Generic Hölder Smooth Data Distributions
54
1.2
Velocity Approximation: Stronger Hölder Smooth Data Distributions.
55
L.3
Velocity Estimation and Distribution Estimation
56
14
Minimax Optimal Estimation
58
J
Proof of Theorem I.1
59
J.1
Auxiliary Lemmas
59
J.2 Velocity Approximation Bounded Domain
62
J.3 Main Proof Theorem I.I
69
K
Proof of Theorem L.2
74
K.1
Auxiliary Lemmas
74
K.2 Velocity Approximation on Bounded Domain
76
K.3 Main Proof of Theorem 1.2
80
Proof of Theorem I.3
83
L.1
Preliminaries
83
L,2 Auxiliary lemmas
84
L.3 Main Proof of Theorem I.3
97
M Proof of Theorem L.4
102
M.1 Auxiliary Lemmas
102
M.2 Main Proof of Theorem 1.4
102
N Proof of Theorem L.5
105
O Experimental Validation
106
14

---

0.1 Experimental Setup
106
0.2 Results and Discussion
106
15

---

Impact Statement
This theoretical work advances the fundamental understanding of flow matching generative models and presents no foresecable negative social impacts.
A Related Work
In the following, we discuss the recent success of the techniques used in our work. We begin with the universal approximation theory of transformers. Then, we discuss the recent theoretical progress in flow matching framework, including approximation, estimation and minimax optimality theories.
Universality of Transformers. The universality of transformers refers to their capability to approx- imate arbitrary sequence-to-sequenn functions with any desired precision. It is key to connect flow matching Transformer architectures with meaningful statistical estimations in this work. Yun et al. [2019] first prove this capability with deep stacks of self-attention and feed-forward layers through the idea of contextual mapping by assuming a minimal separation among all hidden represen- tations. Subsequent work by [Alberti et al., 2023] extend the guarantee to variants that employ sparse attention mechanisms. Building upon these works, Hu et al. [2025b], Kajitsuka and Sato [2023] show that a transformer block with a single self-attention layer is sufficient to achieve universal approximation with refined contextual mapping techniques. Beyond contextual mapping, Jiang and Li [2024] derive explicit Jackson-type approximation rates for single-layer, one-head Transformers, with errors governed by the low-rank structure of the target attention kernel and by head/FFN budgets. Hu et al. [2025a]. Liu et al. [2025] show sequence-tor -eguence-oo-eqquence (attention-onlly via interpolation and max-affine partition constructions. In particular, Liu et al. [2025] prove that a single-head self- or rross-attention module achieves universal approximation under L for continuous targets and extends to Lp
sequence universality of minimal Transformers
Remark A.1 (Approximation with High- vs Low- Low-Tempeatuure Softmax Transformers). We remark that, the current statistical rates rely on -egperreper region of softmax attention (i.e., Softmax, (-) with small X, as in Lemma H.2, Theorem H.1 and (H.25)) to cancel the pouble-exponentiie factor reported in [Hu et al., 2024, Remark 3.4]. Is it possible to circumvent with above mentioned different approximation results. For example, Hu et al. [2025a] establish universal approximation in the low-temperature regime (large A). suggests that analogous statistical guarantees may also hold without resorting to high-temperature softmax scaling. We leave this for future explorations.
Flow Matching and High-Order Flow Matching. Flow Matching generative modeling [Lipman et al., 2024, Gat et al., 2024, Chen and Lipman, 2023, Lipman et al., 2022, Liu et al., 2022] has advanced the state-of-the-art in various fields and applications, including images [Esser et al., 2024]. . speeches [Le et al., 2023], audios [Polyak et al., 2024] and biomedical data [Huguet al., 2024]. These standard flow matching frameworks learn first-order trajectory dynamics (velocity field) to smoothly transport a simple source distribution the target data distribution. However, there is a growing interest for the role of high-order dynamics in generative modeling with improved accuracy and efficiency, which has been applied in various empirical explorations. For instance, Cao al. [2025] integrate special relativistic mechanics to enhance the stability of generative modeling by supervising on second-ordes dynamics (acceleration) to ensure sample velocities remain bounded within a safe limit. Similarly, Liang et al. [2025] also augment flow auto-regressive transformers with second-order supervision by capturing complex dependencies through high-order dynamics.
Statistical Rates and Minimax Optimality of Flow Models. Benton et al. [2023], Albergo and
Vanden- [2022] measure the convergence of flow models by the T-grrisk of the velocity field but omit explicit convergence rates. Jiao et al. [2024] work in the latent space of an autoencoder and derive explicit convergence rates for flow models; however, they do not consider the smoothness of the target density class. Su et al. [2025] establish statistical rates for discrete flow matching by deriving a model-aggostic intrinsic error bound. Fukumizu et al. [2024] demonstrate that flow matching achieves nearly maniimax-opiima distribution estimation rates in Besov density function spaces under the 2- Wasserstein distance using ReLU network architectures. Kunkel and Trabs [2025] establish similar results under the I-Wasserstein distance by employing the kernel density estimators. this work, we provide the first theoretical evidence of the minimax optimality of any order flow matching using transformer architectures, and our results recover the first order case as special instance. Notably, we show that flow matching transformers (FMTs) achieve nearly minimax optimal rates
16

---

in Hölder density function spaces under the 2- --Wasersstenn distance without imposing the Lipschitz continuity assumption on the velocity field. Please see Appendix for a detailed analysis.
B
Supplementary Background: Transformer Block
In this section, we introduce the transformer architecture and its Lipschitzness property. Appendix B. provides formal definition of the transformer network class that we use throughout the paper. Further, Appendix B.2 shows that transformer block is Lipschitz continuous over compact domain.
B.1 Transformers
Our notation follows [Hu et al., 2025c, 2024]. To begin with, given a matrix Z E R we denote the i-th column and the j-th row by Z. and Zy: respectively.
Transformer Block. Let F(SA) _d Rdx denote the self-attention layer. We use h and to denote the number heads and hidden dimension in self-attention layer, and then we have
(SA) (2)
[oftmaa Σ w (WiZ) Softmax ((W) () (Wb2)), (WQ)),
(B.1)
where Softmax(() is the column-wisd softmax function, WV, W W Rx and W Raxs are the weight matrices. Let r be the MLP dimension. Then, we define the feed-forward layer:
(FFF) (Z) = Z + W2RcLU(W; Z + b1) + b2,
(B.2)
where Wi Bexd and W2 Rdxr are weight matrices, and R", and b2 Rd are bias.
Definition B.1 (Transformer Block). We define a transformer block h-head, s-hidden dimension, r-MLP dimension, and with positional encoding E E RdxL as
Fl.s (Z) = ((+)) (J(SA) (Z (2+ +E) IRdx RdXL
Now, we define the transformer networks as compositions of transformer blocks.
Definition B.2 (Transformer Network Function Class). Let Th,sir denote the transformer network function class where each function f E Thhes" is a composition of transformer blocks Ph.s.r i.e.,
Thes. (ft _ + Rd fT = = ..... Fh....)
Flow Matching Transformer. Following architecture of diffusion transformers (DiT') in [Hu et al., 2025c, 2024, Peebles and Xie, 2023], we adopt the reshape layer R that converts vector input x E RRd into the sequential matrix input format Z E R for transformer with d₂ = d L.
Definition B.3 (Reshape Layer). The reshape layer R(-) Rd= + RdxL transforms d, d.-dimmesionnl input into d x L matrix. For any d, representation with feature dimension d == p² (p 2 2) and sequence length == (i/p)². Further, We define the reverse reshape layer R- R-'(.) dx + d the inverse of R(-).
image input, R(-) converts it into sequence
Finally, we define the following transformer network function class with the reshape layer. To simplify, define WKQ = (Wk) We and Wov = WoWr.
Definition B.4 (Transformer Network Function Class with Reshape Layer T Th...rr). The transformer network class with reshape layer T h.. (CT, Cko Ckq, G2 Cov, CF, 2,00 Cr, LT) satisfies:
Th.s.r (R-1 fr R. Rdd - dd fT This.r] Transformer network output bound: supz Il (Z) & Cr;
17

---

Parameter bound in JUH); max (I[W) 00 |122l2,0) S c2 max ||22ll2 1 Ozi
Parameter bound in F(SA); |[xxll2 CkQ. WW ll Cov, [W C
||Wov 12,00 I Cov || | 112.0 1 CE. where 2, co-norm follows 12,00 = ma 2z
Lipschitz of fr Their: (21) (Z2))l S Lrll Zell for any Zz. Zz RdxL
We remark that these norm bounds are critical to quantify the complexity of the network class.
B.2 Lipschitzness of Transformer Network
In this section, show the Lipschitzness for transformer network class Definition B.2. We begin with helper lemma and the Lipschitzness of softmax function under inverse-lemperature scaling:
Lemma B.1 (Lipschitzness from Bounded Jacobian, Lemma A.6 of [Edelman et al., 2022]). Let A Ad-1 == {I E 22 0, |||||| 1} and ey > be some constant. Suppose that f RR A d-1 a differentiable function satisfying J (z) 1,1 S ef for all æ. Then, for any T1, R holds
[[((z)) f(r2) s ey||m x2loa
We then give the Lipschitz property the softmax function:
Lemma B.2 (Lipschitzness of Softmax with Inverse Temperature, Modified from Corollary A.7 of [Edelman et al., 2022]). Let y, E Rd. Denote the softmax function with inverse temperature 0 by Softmax, [x) = exp(Axli)) >x exp(Ax for E E k E [d]. Then, it holds
|| Softmax (y) - Softmax(=)II - 2\| - 2||oc.
A
Proof. Our proof follows [Edelman et al., 2022] and incorporates inverse temperature scaling 3 >> 0. Define Softmax,(1) == p; for i e [d]. Jacobian has Jr = Apı(1 - p.) and Jt,, = -Xp,P, for i j.. Then, this yields {(diga(p) - -pp") and [|][|r.1 S 2A. Then we apply Lemma B.I.
This completes the proof.
Therefore, Lemma B.2 implies the Lipschitzness of a self-attention layer given bounded weight matrices WKQ and Wov. Furthermore, with bounded W1, W2 in the feed-forward layer and [ReUU(z) - ReLU(y)) S Jx - yy for any x, E R, we have Lipschitzness a transformer block.
18

---

Define the class of C Proofs in Section 3
In this section, we formalize the high-order flow matching. Appendix C.1 establishes the felo-vel cquivalence through an ordinary differential equation argument (Theorem 3.1). Appendix C.2 ensures the mass conservation in high-order flows (Theorem 3.2). Appendix C.3 derives the marginalization property (Theorem 3.3). Appendix C.4 shows the gradient equivalence between the flow matching and conditional flow matching objectives (Theorem 3.4). Finally, Appendix C.5 unifies the framework by proving that K -order flow matching collapses to the standard first-order case (Proposition 3.
C.1 Proof of Theorem 3.1
In this section, we present the main proof of Theorem 3.1.
Theorem C.1 (Theorem 3.1 Restated: Flow-Velociry Equivalence via ODE). structured k-order velocity fields as those of the form:
[v())) col(u' (x(() (0) (),.............. .,.,,,,,,..................... "k (I (0) .1) E RR V1 - col (0)
(K Jt (K-11) E B
where w² Rdd x [0, 1] + locally lipschitz in y, and continues in for any l: E [K]. Suppose the velocity fields u' (ax² (0) .)..... u (z,)) .) satisfy total derivative constraints (3.3). Then, for any
initial condition yo Rd the ODE de Ut f(()) exists unique local solution yt, which defines
dez (x) _ for all l E [K]. Conversely, any K K-times differentiable flow th I2 I defines velocity field fa via (3.1).
K Kimimes differentiable flow (x(x) == z; and satisfy d
Proof. We prove both directions:
From velocity field fe to flow the Let you (x (0)
E IR kd be any initial condition.
Then, the system (3.1)
de Vt fe(e), with initial condition V/>
is standard autonomous first-order ODE on RKd with a Lipschitz right-hand side. By the Pi-
Lard-Lindelöf theorem, there exists unique local solution Vt- Let define the flow (x) x;
and since y is differentiable, a/a is differentiable. By repeatedly applying the total derivative con-
straint (3.3), establish that dt dee (x) I
(k) for all le e [K]. Specifically, for any k E [K].
we have:
x
(k)
u (x!") (0) ,t)
(y definition of ODE)
d u
(By (3.3)
dt
d (k-1)
dt Lt
(By definition of the ODE)
d
de (z)).
(By induction)
This confirms that the k-th order velocity field corresponds exactly to the k-th time derivative of the flow dt.
From flow di to velocity field f.: Suppose there is K Kttimes differentiable flow vt. Define
kk-
y: (x), (r(,),
dek (x
dk d v((z))
(v.) coll (r()),... dt
19

---

Then, by direct differentiation:
de -ル= S((()).
This completes the proof of the bidirectional equivalence.
C.2 Proof of Theorem 3.2
In this section, we provide the proof of Theorem 3.2.
Theorem C.2 (Theorem 3.2 Restated: Mass Conservation of High-Order Flow). Let 1/1
(x) (0) ....., 2/ (K (( E Kd Let velocity field f((M) (u' (z (0) f). .................... u (z, (0) t))² kkd
where w² (xt
()) .t) is locally Lipschitz and integrable for all E [K]. Let Pt : R + be time- vary-ig probability density over the extended state E Kd follows Definition 3.1. Then the
following statements are equivalent:
1. The pair (fr, Pe) satisfies the Liouville's equation on the extended space:
@
( V. (pe())((v)) 0, for all t E [0, 1).
2. Following Definition 3.1, the probability law of Y, evolves under the flow:
d Y, = f((V), with Yo ~ Po: Y2 ~ Pt-
(C.I)
dt
For some arbitrary probability path pr. we define fa generates if (C.1) holds.
Proof. We prove both directions:
From ODE (C.1) to Liouville's Equation: Let á : Rkdd - IR be any smooth function with compact support (i.e., test function). We first compute the time derivative of following quantity
Elø(Y)) = (uviarlddy.
(C.2)
Since the Y, satisfy the ODE I). the derivative of the expectation becomes:
E d dt dt (())
d
E(o(Y))
(By swiching the expectation and derivation)
d
,(V, J6 Yd]
(By the chain rule)
E(VD((Y)) [((Y))
(By the ODE (C.1))
voo(v) f(())c(()dy
/ ( V (f(()))(o))dd
(By the integration by parts)
Therefore, for any test function dr. it holds
f at (w))(())++ o (f(()))(v))dd ( O.
which leads to Liouville's equation
dt pcly) + V. (p(())f(()) = O.
20

---

From Liouville's Equation to ODE (C.1); According to the equivalence between the flow th and its associated velocity field fr (Theorem 3. 1), the ODE (C.1) admits unique local solution yt, which defines a unique flow th. By the pushforward formula and the definition in Definition 3.1, this flow induces the distribution Y ~ Pt. Moreover, 7r satisfies the Liouville equation associated with the velocity field fe.
Since the Liouville equation admits a unique solution in the space of probability densities starting from the same initial distribution Po. and both Pt and Pe solve the same continuity equation with initial condition Pu. we conclude that Pt = Pt- This completes the proof.
C.3 Proof of Theorem 3.3
This section presents the proof of Theorem 3.3.
Theorem C.3 (Theorem 3.3 Restated: Marginalization). Recall that for some arbitrary probability path Pt. ft generates Ps if Y ~ Pe for all E 1). Let Z be a random variable, fo (=(2) is conditionally integrable and generates the conditional probability path pr(-12), then the marginal velocity ft = (px dz generates the marginal probability path pt-
Proof. Applying the mass conservation follows Theorem 3.2, we only need to verify that the ft and Pe satisfy high-order continuity equation, i.e. Liouville's Equation:
d dt (((()) / dt Puzz(ul=)zzll
(By the law of total prohability)
/ (((00=))x((=)))p<(222)
(By Liouville's equation)
= V / f(lll=)p(ull))pp()))d)
(By switching differentiation and integration)
V / [ILWw)))o(WW))))((///()/ pr(y)dz
V (f(())).v)))
(By the definition of ft(y) and the Bayes' rule)
This completes the proof.
C.4 Proof of Theorem 3,4
In this section, we prove Theorem 3.4.
Theorem C.4 (Theorem 3.4 Restated: Gradient Equivalence of Losses). Let the Flow Matching loss K be defined as in Definition 3.2, and the Conditional Flow Matching loss Lk be defined as in (3.5). Then, when D(.,)) is Bregman divergence, the gradients of the two losses coincide:
PM (0) V.CK
Proof. Similar to the Theorem 4 of [Lipman et al., 2024], the result follows from the Marginalization Trick (Theorem 3.3) and the expectation-swapping property of Bregman divergences (3.6). A direct computation then shows that:
V ( Vo f((I))
(y the definition of Flow Matching Loss)
L,Yoppi
E t.Yeupt
T.D()((Y).f f(,))
(By swaping the expectation and the gradient computation)
E
(By the chain rule)
t.Yepp
E
V.D( E Zmpeia(ta))
L.Yopp
(By the marginalization trick follows Theorem 3.3)
21

---

E
E
V (f
tpmp 2mpaa((-1))
(By the property of Bregman divergence follows (3.6)
E
(f
(By the chain rule)
Vo
E
(By the Bayes' rule)
(12
VoLCFM (0).
This completes the proof.
C.5 Proof of Proposition 3.1
This section gives the main proof of Proposition 3.1.
Proposition C.1 (Proposition 3.1 Restated: Reduction to Standard First-Order Flow Matching). When K = 1, the entire Koorddr flow matching framework, including the governing ODE, the probability path definition via the continuity equation, and the K-order flow matching objective, becomes precisely equivalent the standard first-order Flow Matching framework detailed in [Lipman al., 2022, 2024].
Proof. The equivalence follows by setting K in the definitions of our K-order framework.
1. State Variable and ODE:
From Definition 3.1, when K Y, x X,, The ODE system _ Yi fr(Y) simplifies to d X; u (X;), which is the governing ODE for standard flow models ([Lipman et al., 2022, 2024]). The K-order velocity field fa becomes w!.
2. Probability Path and Continuity Equation: The K-order mass conservation formula (Theo- rem 3.2) for = reduces to the standard Mass Conservation Formula (Theorem in [Lipman et al., 2024]).
3. Loss Objective: The K Koorder flow matching loss (Definition 3.2), which targets matching /l to
fi simplifies matching only the u component: X [D(u) (X,), u, 1.0 (X,))] This is the standard Flow Matching objective (Eq. (5) in [Lipman et al., 2022]). The conditional formulation via Theorem 3.3 similarly simplities to the conditional Flow Matching loss used for standard FM.
Thus, all core components the Koorder framework align with standard Flow Matching.
22

---

D Proof of Theorem 4.1
In this section, prove Theorem 4.1 following steps similar to the velocity approximation in Appendix (i) applying the universal approximation of transformers (ii) leveraging the uub-Gaussian property of the target distribution to bound the approximation emror of the K order velocity field.
Organizations.
Appendix D.I introduces helper lemmas. Appendix D.2 presents the main proof.
D.1 Auxiliary Lemmas
In this section, introduce four auxiliary lemmas. In Lemma D.1, give the lower-bound and upper-bounds on (z)).. In Lemma D.2, we state the classical Gaussian tail bounds. In Lemma D.3, we approximate the k-th order velocity field over bounded domain. To control the error in unbounded regions, we exploit the sub-gaussian assumption of the target distribution g(x1) in Lemma D.4.
We begin with the bounds on (2r).
Lemma D.1 (Bounds on the Density Function, Lemma A.9 of [Fu al., 2024]). Recall that pa(x)= = Sus. pe(a)aa ))(()))(())) and pa(aaaa)) of (2x))a/77 exp(-1|z p401 Assumption 4.1. Then, there exist positive constant C₁ such that
| / 22))) Assume
C4 exp J. tt
(- | of - S pi(a)
C211213 (nî C207)da/2 exp (- 2(n + C2a
G
Then, we apply standard results for Gaussian tail bounds. We remark that the main purpose of stating Lemma D.2 to streamline the main proof of Theorem 4.1 in Appendix D.2.
Lemma D.2 (Gaussian Tail Bounds). Consider random vector X (Xy....,Xa.) Xd.) N(0, Let wa, = 2n 4 [ de ). Then, the following two inequalities hold:
X||>D J/XISD exp
| 2
D
dx wa, of D expp
20
20
1 ||x11>D
(-
|| ||
D2
| exp
dx Wd, (0; + d 2 exp
2o?
200
Proof. We first express the integral in spherical coordinates for X
exp(-1x11/203)) = wdz / dr exp
y2
dr.
2007
Let Jn = 50 -- -exp ( , --2o? dr. Setting = pdz -2 and du = rexp
-
20
dr, we have
,2
du (dg - 3rr and v v= - exp
20
Then,
p2 1
10 n exp
p2
J(D) - exp
(ds - -2)0² [
dr
(D.1)
20 r=D
20
1)2
2)0? (00 1 - expl
2
= of do exp
+
(ds
dr
20
20
(By integration by parts)
D
- D0 exp
(By dropping the second term)
20
23

---

We obtain the final bound
JIXIIS
| 20
D2
dx wa, D2 exp
exp
20
This completes the proof of the first inequality. For the second inequality, we have
J IIXIDDD
X
exp |
)dX
207
22
wds
22 exp
dr
2
1²
wils
dxt exp
dr.
20
Let K(D) = SD ,d exp
2² dr, u = pd and de = rexp
dr. Then,
20
2a²
p²
du drrd dr. and == -0² exp exp
200
Therefore, the integration by parts gives
K(D)
مs o exp
,2
20
+ 1 of exp
00
p2
d dz
d
dr
2007
20
r=0
D
D 2077
10
12
of Ddo exp
+ doo
exp
20
7
dr.
Recalling (D.1)
JD 10 de-1 exp
D2
dr,
and
JD S of
0d exp
200
200
have
K(D)
12
of dds exp
+ doojJD
207
D
D2
< Dd exp
+ dooj
(0, exp
(By the bound on Jp)
20
201
1
Do + d
exp
207
Then we obtain the final bound
JIXISD |
(- | 20?
D2 JdX wa, (0, D exp
exp
200
This completes the proof of the second inequality.
Applying the universal approximation of transformers (Theorem H.2), we first approximate the k-th order velocity field * over a bounded domain with transformers k.0
24

---

Lemma D.3 (Approximate k-th Order Flow with Transformers). Assume Assumption 4.1. Let D be an absolute positive constant. Then, for any 1, [0, 1] and E (0, 1), there exist transformer u (x, 1) Thiis such that
So
Pe(a) 11 (x, u² (x, (e,,t) 122ddd S e²,
for all & E [K]. Furthermore, the parameter bounds in the transformer network class satisfy
Ckq. 22000 = 100 14d+2 -ddd22) Cov. Cov. Cov = O(c); KQ
Cr, - O Cn O(); ((Lx1)
where A- = ((1/e))d 4d+3 the inverse-temperature scaling in the softmax function and and (-) hides all polynomial factors depending d,, d, L, C1, C2-
Proof. By specifying the target function as = u and the transformer-based estimator as g u in Theorem H.2, and applying the bound p=(2)) < 1, the proof follows Theorem H.2 since the reshape layer (Definition B.3) does not harm the uniform continuity. Further, by the Lipschitzness of the k-th order flow, we have ||u* (x, l)ll2 S Lk-1- Then, the parameter bounds in transformer network follow Lemma H.5, where we the model output bound Cy O(Lx-1). This completes the proof. C
To control the approximation probability flow pa (x) and the weighted squared norms of the w² given by ||"((, 0) px()).
unbounded domain, introduce tail bounds for the
Lemma D.4 (Truncation of x, Modified from Lemma A.1 of [Fu al., 2024]). Assume Assump- tion 4.1. Suppose the k-th order velocity field u² (x, t) is Lipschitz continuous for all k = O..... ., K.
Let La denote the Lipschitz constant of u² and then the velocity fields are uniformly bounded as x (z, t) s L for any k E [K]. Then, for any R1,t > 0 and k E [K]. the following hold
1 |1 =n,
C2
pe (x)dr S nT -2 exp - C202)
C2 R2 2(np C20
I 1Iu (z, 1)1 pe(x)de E 22 12700 exp
Proof. For the first inequality, it follows
1 |101 RR
Pc(x)dx
C211=:11
cxp
dx
(By Lemma D.1)
211 + C2a?)
21
C2112113 C2af)
exp
dr
(By ||rl|a 2 ||a]]lo))
|1l122211 Ri
2(k2
C2
$ R -2 exp
(By Lemma D.2)
20p] + C207)
For the second inequality, it follows
J
|/ (x, 1)11 pr(x)dr
|1 R2
1
C211=13
|| |1 21
t)112 exp
dx
(By Lemma D.1)
2(p + C2a7)
/
12 dr
L exp
(By the Lipchitzness of the k-th order flow)
/1 Ri
2(u + C2af)
25

---

C2 2(1 + C20
I 11 R exp
(By Lemma D.2)
This completes the proof.
D.2 Main Proof of Theorem 4.1
We now present the formal proof of Theorem 4.1.
Theorem D.1 (Theorem 4.1 Restated: K-order Velocity Approximation with Transformers). Assume Assumption 4.1. Suppose the k-th order velocity field (I, Lg-Lipschitz for all & E O,.. O.....K K 1 in ly-distance. Let (0,1) be the precision parameter satisfying S O(N-9) for some N E N and smoothness parameter / >> 0. Then, there exists transformers u (1, .)...,................ ( E The such that for any Rd and t [0, it holds:
// to
K
Σ
(x, u² (z. pr(x)dadt = O(N (log N) 1)
JR
k=1
Further, for all E [K]. the parameter bounds transformer network class satisfy
CKQ, Cko = O -1 (28(224+11 (log 2)24+1); Cov, Cgr O(N CF, = O Vlo N Lx-1); CE = 0(1); CT = 0(Lx-)),
where A- O(N log Ny4443 is the inverse-temperature scaling in the softmax function and O-)) hides all polynomial factors depending on d2 d, L, B, C₁2 C2.
Proof Theorem 4.1. For u (z, t), .... u K.0 (x, t) E T we the transformer output bound = 0(Lx)) for the k-th network and let R3 and flow be two positive numbers to be chosen.
First, decompose the target into three components and bound each of them
T
Jaue (I,t:((x, ua, t)1 (o()ddddt
K
[,,t((,,)))) - u (a,, 1)1 pa (x)dddt
11
1=1
(T.)
I ||ut (x,, nn (x, 1) pe(x))ddd.
k=1
(1a))
Bound on (Tr). It holdd
(Ti)
K
Σ
(1,
(x( ()x. pe (x)ddrdd
k=1
to
RRs
T
I
2
2
|nt 1)
pe (x( (x))d
k=1
t0
(By expanding (2-norm)
26

---

K
K (x, pr(x)drdf
de(d) +
Σ
k=1
Cr = =
K
T
(o(dddxdt
(By the Lipschitzness of the k-th order flow)
k-
10lls0>Rs
kml
C2R3
R4-- exp
Σ Li 10
dt.
By Lemma D.4)
22( + C20
C2 R
$ Rg. -2 exp
(By to T E (0,
2( C207)
R=1
Bound on (T2). For any E (0, 1), it holds
(T2) = Σ J
(x, (x, 1)12 pe(r)drdt s Ke?.
(By Lemma D.3)
k=
By the upper-bound (Tr) and (T2), we have
K Σ
(1,)) - (x, Pe (x)dddt
k=1
= =(T)) + +T22
C2
K
S R -2 exp
Σ
Li + Ke
2(u + C20
1=1
R2 2(14 C20?)
)
max
Rg. exp
Finally, for some E N and B > 0, we set
4.(mf + C20,)
log N
R3
and 6= N-8
C2
This gives
E11 Σ Jgds
J (x, u² (x,,t))) (x()drrdt O(N 22- -llo 2
44-1)
The transformer parameter bounds follow Lemma D.3 with - O0.00(1oll N) and E = N 0:
CкQ; KQ -0(A-IN22(2d=1)1
c2 = O(N 9): CF,C2a O(N O
(D.2)
where X O log 4d43 is the inverse scaling in the softmax function.
This completes the proof.
27

---

E Proof of Theorem 4.2
In this section, we derive the estimation rate of the order flow matching using transformers. We decompose the proof of Theorem 4.2 into the following three parts due to its complexity.
Step O: Preliminaries. We introduce several essential definitions, including the order condi- tional flow matching loss, K order empirical risk and their domain truncation. These definitions are the extensions from the velocity estimation analysis (see Appendix 1.3 and Appendix L).
Step 1: Controlling Error from Loss Function outside of the Truncated Domain. By leveraging the sub-Gaussian tail bound and the Lipschitz continuity the k-th order velocity field, we derive an upper bound on the loss function outside of the truncated domain in Lemma F.1.
Step 2: Upper Bound on the Covering Number. We present unified upper bound on the covering number that holds across K transformer networks wt.0 ,..., KS Lemma E.2.
Step 3: Generalization Error. We apply the covering number technique to bound the deviation between the K order empirical risk and the K order true risk in Lemma E.3.
Organizations. Appendix E.1 includes preliminaries the framework of estimators' quality evaluation. Appendix E.2 introduces auxiliary lemmas. Appendix E.3 presents the main proof.
E.1 Preliminaries
In this section, we consider affine conditional (x(xx(])))= = ,x + ag following Section Given k-th order velocity estimator u we aim bound the flow matching risk R& (9):
K
RK (9)
Σ T - to
E xmpp
- (x, t)l)ddr
where the density function Pt and the k--th order flow are induced by the flow th (Definition 3.1). In practice, use the order conditional flow matching loss to train u 1.0 ..... K.O TTh.s...
Definition E.1 (High-Order Conditional Flow Matching Loss). Let q be the ground truth distribution and the normal distribution N (0, be the source distribution p. Considering affine conditional flows (x) = X, + o,t, we define the K order conditional flow matching loss:
C
Σ
E
I1(
xx( x + + x - n (x( .
T L0 to
x
x
up
Further, we define the K order loss function
K
(x(x;ut 1.0
K.0
E (()
[1(
(R) + t x(()) n (x(! dr)
T
to
k=
Given set of i.d.d sample (x:) obtain transformers 1. conditional flow matching loss:
K. by optimizing the empirical
K
KK
1
E 2, (0 x(()) n (xx
[11
(0) dl
T to
tt
Xo (0.1)
Then, we define the K-order empirical risk:
28

---

Definition E.2 (High-Order Empirical Risk). Let wh be the estimator of the k-th order velocity field w², Further, consider i.idd training samples (2i)i=1 and empirical conditional flow matching loss CCFM = - L (xg: Then, we define the K order empirical risk:
Rx( = 1 Σ Cx(onu
1 (x(agu ).
K.0)
Remark E.1. Let R (ff be the ground truth inputs of the high-order risk; that is, w u² for any & [ Then, by the definition of high-order velocity field in Definition 3.1, Rx (ft) = since (3)) (u ..... uth is the collection of K order ground truth velocity fields. Further, the gradient equivalence Theorem 3.4 implies that Rx (+) = Rx (+) Rx = ( LCMM (f+)-
Remark E.2. We use CK and R'x to denote the conditional flow matching loss and empirical risk with training samples { Then, by the i.id assumption the training sample, we have (x)() (O)] (O), and therefore (x)) [R'x (+)] (+).
To obtain finite covering number, we introduce the K truncated loss and truncated risk.
Definition E.3 (Domain Truncation of High-Order Loss and Risk), Let D > 0 be constant. Given K order conditional flow matching loss & (x; w².0 ,..,, u,,,,, defined in Definition E.I, we define truncated counterparts on a bounded domain D = [- [-D, D]d by
ptrune
(x; u 1.e
u K.9) = (x 1.8
K.0)1 f11=11 S D)-
K
Given the K order conditional flow matching risk and the K order empirical risk, we define
Rxm (0) = Rx (o)1 S D),
Rummn(() = Rx( 3 D).
Rx
E.2 Auxiliary Lemmas
We follow the proof of velocity estimation Appendix I.2 and Appendix L.3 bound the order flow matching estimation error. Since direct computation of risk is infeasible, we first decompose the K order flow matching risk Rg into four terms. Then, we leverage the sub-Gaussian property (Assumption L.I) and the Lipschitzness of transformer network class (Definition B.2) to bound each term. Specifically, we introduce three lemmas to bound
1. the error from the domain truncation of loss function class (Lemma E.
2. the log covering number of loss function class (Lemma E.2), and
3. the generalization error bound (Lemma E.3).
Risk Decompasition. For simplicity, we shorthand R Rx(u" (u uu K.0) different set of i..d samples independent of the training sample {x, J Then we decompose:
with Rg. Let {-/]== be
[K
E
R
R Sruunn
{xr
(1)
E
rerune
R
Rtrune
(x 1-m1
(II)
+
R
trune
Rx
Rx
{n;
1
(111)
(IV)
29

---

where we use the fact that [o,))_ [+)] = (+) (Remark E.1). This decomposition follows standard statistical learning theory technique, formulated in [Hu et al., 2025c, Fu et al., 2024].
High-Order Truncation Loss. We begin with the bounds term (1) and term (III).
..... u K.0 The be Lemma E.1 (Upper Bound on the High-Order Order Truncation Error). Let ul. u!......
transformers in Theorem 4.1. Then, for any t E [to. T] it holds
1 - C2D2 max ( (T
E (x(2; u K.0) k (1; (x; u ..... n k1 & KD exp
Proof. By Theorem 4.1, have transformers output bounds CT ((Tx-1) for all k.
For all [K], we define
l (ar = 1
E
(x ,1) (16 (k)
+ of )1]dd
(k)
T to to XomN((,,1)
ex grunn (x; = T to J. XomN(u.1)
Xo. (xx (t) -(46 x + o (1) ) 11 $ D).
Then, holds
[l]ex<1;; (2;; uh:0) exrun (x; (x: []
(E.1)
(x; u F.0 ( [[|[=]] 2 DIl1
(y Definition E.3)
T T
D Now XowN
E
I|l (x, (1) - (1150) to ( (1))
(By Definition E.
T T- to X Xow.o.n 11
(x 1)11 + (k) x + (hh)) x 12 (x)dd
E Xo
(x
(By expanding the f2-norm)
/ to 1o 11i112 (1,,)
X E
,0)l2 + || (k) x + of
(M)
x
0) [ exp (-1 1 Calle1 dxdt
(-2
(By Assumption L.)
T la S S
11112
No~N(0.1) [max{L]] + 1114(0) + or (R) x (0) 1 exp
E
- C2|ar11 drdt
(By OT O (I+)))
T
(max{[ + (o. + (4. (110)1111111) exp
C2 C2l12113 drdt
T to 1 ||x112
2
(5O N(O.
//' T Ddd expl -222D22
dd 1
(max (LX) + (2. at
(n) (
T-to to
T to
BB Lemma D.2)
S SD exp (-10 1 [a)
BBy Assumption 1.2)
Therefore,
El lx (x; 2.0
k.0) gruuc (ax; 2.0 K.0)||
30

---

| ellex (2) uh:0) - (x; 01)
(By triangle incquality)
1 (3DD max{L S KD exp
(By (E.1)
This completes the proof.
Covering Number of High-Order Loss Function Class with Transformers. The next lemma extends Lemma L.5 to its higher-order counterpart. Please see Definition L.4 for precise definition.
Lemma E.2 (Covering Number Bounds for S(D), Lemma K.2 of [Hu et al., 2025c]. Theorem A.17 of [Edelman et al., 2022]). Let E > O. We define the loss function class by S(D) = {xx(2; w ...., u xk.0): D + R 1,0 ...., K.O E Th.""". Further, we define the norm of loss D...ellDDDddd |ex| Then, under transformer parameter configuration in Theorem 4.1 the --voerinn number of S(D) with respect to Ill satisfies:
functions by ||ox
(( s 00 log (nI//ee D2 No (log N))ct8)
e
Proof. We first derive the log covering number of transformers 1.0 ..... K.O in Theorem 4.1. Then, we extend the results K order loss function class.
Log-Covering Number of Transformers Network Class. From (D.2), for all [K], we have
CKQ: C KQ = 0(x-11442 - Cov. Cgp = =(e);
CF, 02.00 = (e-e--Le--))) O(
where X = O(N log N) N) is the inverse-lemperature scaling in the softmax function, I OVvlog N) and c = N-8 > some E N and 3 > 0.
By Lemma L.5, the bounds on log-covering number follow
log.N(ce: Thesk (11112))
a² log (nL/cc) & a² log (nI//cc) 2A
(C2200)
Cov + ((C))))))))C))))CCC))))))))(
4/3 +
4/3 2/3 8/3- + 4/3 2/3 )²
E
(C200)3
(Cr)4/8
(Cov)a/s
(AC
(Cr) (C2
a² log (nlb/te)
(T(8648)/3
-8d/3-233
2
a² log (nL//Ec)
18448 -
By Lemma L.5, we have
a CorC)
< Pe- e - (D--2.(D++ Cg) (Cp)² (Cov) (Cxo)
(By the definition of a)
= D
31

---

Altogether, for all w E ETR Therr we have
log (nL//ee 16
log.N(ee; Tha .11-112) S
C
Further, by |1 11.0 - 112, we have
log (nL//e) 1
log N(e: Th Theer .1l 1lo0) S
(E.2)
e
for all k.0 TTh.s.
Log-Covering Number of Loss Function Class. Let 8 > 0. Let fu ..... u k.9) and = [u ..... _.K.) be two sets of transformers network satisfying ||u T0 co s 6 on domain x E [- Djd, for all w E u and u E u. Further, let wh denote the ground truth k-th order conditional velocity field (Definition E.1):
V == 14 (k)
of (K) X
Then, the distance between two K order conditional loss functions Lx,1 (2; 2.0 ...., ,...,u K.)) and ....,kK. H(.6) follows:
(x; The ....,
[K, (2; 10 u K.0)
(x,2(1;
(E.3)
E Xom-((0.1)
Il
Vi, 11]dd Σ
[|
t 1 |
to
No,N(
(By Definition E.1)
K
E
[uut + t - 2)), (u =b
(By triangle incquality)
k=1
T
to
N1.
K
T
E
11 + thh 24/
(By n. 10.0 8)
T
10
XanN((0.1)
K
Illuki + T + 21
(By Jensen's 's inequality)
T to
Xo~N(0,1)
k=1
K
V mmxx + 211017 11
(By Or Ommxx ( (L+)))
T
8 maxx (Lx]
1² dt
(By the Lipschitzness le-th order flow)
T to
6 maxx(Le)
Finally, we extend the log covering number the loss function class S(D) by setting
C = (e max(L
k
This gives
((D) < log N(Ee T Ther .II oloo).
(By (E.3))
32

---

Therefore,
log, (e,, (1.loo))) log.N(<c: Thi :11 110))
log (nL/Ca) D2 166--10-116-112 e = =0( log (nL//oo - max (g))
(By (E.2))
(By the definition of c
(co)²
Finally, we substitute = 1..O(vloog N) and = N- > 0. This completes the proof.
Generalization Bound. Based on covering number bounds results in Lemma E.2, now analyze the upper bound of generalization error E (xa) [Ri (+) - -R [+)]
Lemma E.3 (Generalization Bound on K Order Flow Matching Risk). For E. > 0, let V := N(c,, S(D), || ))>) be the covering number of function class of loss S(D) following Lemma E.2. Let θ be the collection of parameters of transformers trained by optimizing LCrM(O) following Definition E.1 with i.i.d training samples {a: |i Then we bound the generalization error:
E (x.)
RK
trtune
(7)
R
Strune (6) S R
Rtrune (θ) + O n
log N + tc).
Proof. Let Th. Th.s.. be the approximator of the k-th velocity field n obtained from minimizing the high-order empirical conditional flow matching loss:
CK =
1
E T to to No.~((0.1)
2; + o ( x()) (x) [ [11
k=
Further, we define
R trunc (rrunc(ah.0) = T to to
E
(x, = () {l1olls S D)dt,
pmpp
and
truune (T)k.0)
R
1
E
[1
(k)
+
(())
(x(") . 1f110//bo I D}
1=1
T to
ta
X m
1 12
(k) (11
Hz + 0z (k) ( - n (x. (0) 1 a $ D)
71
T - to
X (0)
1=1
Since every network configurations and log covering number are identical across all K order velocity fields from Theorem 4.1 and Lemma F.2, for any E [K], Lemma L.8 extends to
R trume (at:0) - Rt (ak (Tk.0)]
Strune
<
E [ (arrnneakk11 (at.0) + O log + +cc).
2 (
Therefore,
E
Rtrunc ② Rrunn
θ
33

---

E
R
Rtrane (10)0 - R (1
(By the triangle incquality)
(xi)
E {mr)
R (Tthe + 0 log +c)
By Lemma L.8)
1=1
trune (θ)] + 0 1
R
log. +
2 {a
n
This implies
(+) - 2- R (6) - or log.V V + co).
E
[
{1r Pim
Finally, we conclude that
E fa
R
trune (7) (etrun((@)) S Re (o) + 0 n
log + te).
This completes the proof.
E.3 Main Proof of Theorem 4.2
We now present the main proof of Theorem 4.2.
Theorem E.1 (Theorem 4.2 Restated: High-Order Velocity Estimation with Transformer). Assume Assumption 4.1 and Assumption 4.2. Let T E T be the estimator of the k-th order velocity
field u² trained by minimizing the high-order empirical conditional flow matching loss (3.9). Let θ be the collection of parameters of kt.0 for k E [K] Suppose the k-th order velocity field uz( t) is Le Lipschitz for all k O..... K 1. Suppose choose the transformers in Theorem 4.1, then
RR (²)] 0(n Tα (log 10d.)
E
{c. P
where d is the feature dimension.
Proof of Theorem 4.2. Let (xt) be a different set of i.i.d samples independent of the training sample { Further, R' to denote the empirical risk with samples {>/)=-
Then, decompose B(xs)r [Rx(O]] as:
E {
Rx Rx)
E
E (a
(x)
R trune [)]]
E
R
reruunc (0)]
R
Btrune (θ)
(x:)a
(m
(11)
R K
Strune
( Rx
(+)
E
Rx
[m:).
(III)
(IV)
Then, bound each term and incorporate them to obtain the bound on the estimation error.
34

---

Bound on (I) and III. By Lemma E.1, (I) and (III) are upper bounded by
1 -2D2)) max ( [
(TI) & KD exp
Bound on (II). By the generalization error bound (Lemma E.3), we have
(II)
E
E
[
[+]]
R trmme [9)]
{x.)
E
RR
R ( Rtruno (+)
By Remark E.2)
{xt)
<
E
RK
Strune (θ) + 0 n1
log.N + +
By Lemma E.3)
(x:)
S (IV) - Dd exp (22D
max m
+ 0 O log +
(By Lemma E.1)
Bound (IV). Recall Remark E.1, Remark E.2. We have Rx(0)= (0) (CFM)) (fF) where the collection of parameters of K transformers trained by optimizing (O).
Therefore, it holds
Rx (+) S (OF (fr (O).
Then, for any velocity estimator θ it holds
E (mr)
[xx(
E [xx( Rx (0).
(E.4)
[
This implies
(IV) < S
N 28 -log N) -
(By Theorem 4.1)
Altogether, the estimation error upper bounded by
[+)]
(E.S)
(x:)__
(1) + (II) + (III) + (IV)
S d exp
C2D2
+ +0 - log + + +
(log N) 4
2
(Ts)
(T1)
(Ta)
where
log (nL/Ee) D 2-1661-))12)) (log N))d--). log.V= 0
BB Lemma E.2)
Let y 16d + 12. Then, we n111(188)) E6 n 12 and D V log n)/C2, where + 2772
Mi 7/, 1/3 are constants satisfying 0 S 7/1
1.3 This gives
(T d exp
- C2D2
1
S n (log n) # #
The constraint y7. + 2np < 1 is imposed in order ensure (T2) converges
35

---

Further, we have
log = O(n (log n Sdd +11),
implying
(T2) = O log.N + Ec) = +2 (log n 11)dd+1111111 + 12 12).
Further,
(Tx) = n
2m
(log m) 3/2 -1
Then, (E.5) becomes
E
RRx(e))
(T1) + (T2) (Ts)
0 n min 1-(mn (m +222), n2. 2 } ()ogm)&ks t1l.
For any 77 and 1/2 satisfying
0 m + 2112 < 1,
we consider solving
min {1 (171 + 277)), 12
2m }.
The linear programming problem has simple solution
27/1
(m + 27/2) 172
This gives
2 7 6'
7 6' and
and mh + 27/2 E (0, 1)is satisfied for any m, 122 > 0.
Finally, by 7 16d + 12, these free parameters achieves balance and gives
E K
R (+)
(n- - (log n) (logn)&d++1]) (d1+ 0(" (n_ अ6क (log n) 8d2+
This completes the proof.
36

---

Proof of Theorem 4.3
We now present the main proof of Theorem 4.3.
Theorem F.1 (Theorem 4.3 Restated: High-Order Distribution Estimation under 2- 2-
Distance). Assume Assumption 4.1 and Assumption 4.2. Let P/ be the estimated distribution time T. Then, it holds
[Ws(PF PK) = )े. (log (ogn)) )02) [m:)
where d the feature dimension.
Proof of Theorem 4.3. We first consider two general ODE functions that describe the ground truth velocity field and estimated velocity field respectively:
u ( 1) u (xt (0)
(0)
(x, (0)
t)
.t)
u 2 (x, (0)
.t)
dt 3/t
= f(v,t),
dt Ut
= (u,
k (xt (0) .t)
u K. (x) (0) .t]
where the first rows of ya. E RKd construct ot E Rdd.
According to the existence uniqueness theorem of ODEs, these two functions can induce another following two corresponding flows ((-) Rd and (²) TR. Kd defined for / that satisfy:
d
di o( s)) - s() o(y,s,)) =
d ((o (v.,tt)) s). y.
We define the first d rows of o(.) construct the flow function (x)) and the first d rows of (.) construct the flow function vgg (x). By applying Lemma M.2, holds that
(v. 1o 7) <ly, lo, T)
lo, 8),8) D (őlu, to 8), T)C (o
s(bl, to, 8), s))
We extract the first d rows of left-hand-side and it holds:
(x,lo, - (a, lo, 1. D& (bly, to 8), 8, ()]1
f(oly, los 8), s))ds,
where D (oly, to, s), s;)(( d) denotes the first d rows of the Jacobian matrix.
We then bound w (x, to. T) - w0 to. T) by using similar techniques in proof of Theorem L.4. shows that
Do to s,t) dl2
0t
0
s), d1l2
(By triangle inequality)
01
Du (v) (1) to. s), (0(v, to. 8), s))(; d)
(By chain rule)
< LT DD to s, d)ll2.
(By Lipschitz constant of transformer)
37

---

Therefore,
[ to, s)),, d)ll x expl I [ydu] s expl/ // [ydu] = M.
(By Lemma M.I)
Now we have
1 (z, to, (1,,, to, T)
I' (19(0,) s), 8) ( s(o(y. lo, 8), d))
S M
By Lemma M.
T
M²(
(o) - s), s)(l))dsss
(By definition of f*)
k=1
K
2
(1b2,
nt (v(2,, to, 8), ,13ddd.
(By Cauchy Schwarz inequality)
Then, we take expectation with æ ph, on both sides
E
(x, to T) (a to, T)
xmpph
E 1f 2rwp?
M
(w<(,, to, s), (v (), s)BBds)
ta
= M2 to) Rx
(By definition of higher order risk in Definition 4.2)
Finally, we bound the 2-ssseerteein distance between the estimated and true distributions following Appendix M. By using the definition of the 2- Wasseerstei metric, it follows that
W2(P% PK)
E sr-Pip
[1]i (x, T) (x, to. T) x VRx(O)
/
Then,
E W2(P P S V Rx (
We apply the high-order velocity estimation results in Theorem 4.2
[x] = =0 whth (log n) 8d, (logg)dd.4o 61).
{
This implies
E (
W2
PK PK
)
E [VK((0)]= = 0 -1 16
(log n) 4dg 4dz+6) +6
This completes the proof.
38

---

G
Proof of Theorem 4.4
Recall the Hölder density function class and its minimax optimal rate under 2- 2-Wasserstein distance:
Lemma G.1 (Lemma N.2 Restated: Modified from Theorem 3 of [Niles-Weed Weed and Berthet, 2022]).
Consider the task of estimating a probability distribution P(r1) with density function belonging to the space
{q(x1) ((x1) E ([-1)= B), ( 2 C).
Then, for any 1, > and dz > have
inf sup E [W, P a(Z))P (w.)im
2+l
where {-i} is a set of i.i.d samples drawn from distribution P, and P runs over all possible estimators constructed from the data.
now give the formal proof of Theorem 4.4.
Theorem G.1 (Theorem 4.4 Restated: Minimax Optimality of High-Order Flow Matching Transform- ers). Assume that the target density function satisfies ((x1) (1-1,,
1] B) and g( C for some constant C. Then, under the setting of 18d(3 + 1) = dv + 25, the distribution estimation rate of flow matching transformers presented in Theorem 4.3 matches the minimax lower bound of Hölder distribution class in 2- 2-Sasserstein distance up log and Lipchitz constants factors.
Proof of Theorem 4.4. Since the bounded support [-1, 1] guarantees the sub-Gaussian property in Assumption I.1, the distribution estimation Theorem 4.3 holds under ((x1) E H ([-1, (/,,B)) B):
E {as
(W2(Pr,PP)
$ n 1ेx (log n) 6dz
Then, by Lemma N.2, the distribution rates matches the minimax lower bound up to a log n and Lipschitz constant factors under the setting
18d(3 + 1) d, + 2,3.
This completes the proof.
Remark G.1. Since d, = d L, the condition 18d(3 1) d, + 2,9 implies
d(18/3 + 18 L) = 2,3 and 3(18d 2) = d(L ([L- - -18),
the transformesr achieve minimax optimal rate with reshape layer such that 18 < 183 + 18.
39

---

H Preliminaries: Universal Approximation of Transformers
Prior works [Hu et al., 2025b, 2024, Kajitsuka and Sato, 2023, Yun et al., 2019] develop the universal approximation transformers for continuous functions. Here we revisit these methods to establish foundation for analysis of k-th order flow matching transformers. Specifically, the ability of the transformer function class (defined in Appendix B) to approximate any continuous function compact domain with arbitrary error, (ii) the parameter norm bounds required to achieve the universal approximation. Notably, controlling the magnitude of these norm bounds is essential for subsequent analysis on the velocity estimation error and distribution error.
revisit (i)
Background: Contextual Mapping. Recall the reshape layer Definition B.3. Let 7 RR represent input embeddings. where Z,, E F. denotes the k-th token (column) of each Z sequence. Further, given M embeddings z Z(M) E R say Z() is the i-th sequence for i E [M]. Then, we define the vocabulary corresponding to the i-th sequence at the k-th index in Definition H.I.
Definition H.1 (Vocabulary). We define the i-th vocabulary set for E [M] by V =
UMELL 2 C IR, and the whole vocabulary set V is defined by V USENN v(0) C Rd.
In line with prior works [Hu et al., 2025b, 2024, Kajitsuka and Sato, 2023, Kim et al., 2022, Yun et al., 2019], we assume the embeddings separateness to be (7miin 7mmax, -)-sparatedd
Definition H.2 (Tokenwise Separateness). Let ((1), z())..., 7(M) E R be embeddings. Then, we say (11) (()).... ,..., (M4) are tokenwise (Tmin: Tmax 6) -epparttd if the following three conditions hold.
1. For any i E [M] and & E [n). | > Ymin holds.
2. For any E [M] and & E [nx], || | < 7max holds.
3. For any j.j [M] and I [n] if 2 + 2 then || z,4² holds.
Further, we say z ,.... (MM) is (7, 8) -ssparrtd when only conditions (ii) and (iii) hold. Also, only condition (iii) holds, we denote it as (5) seeparattessss..
Building the token separateness, introduce the contextual mapping, that characterizes the ability of transformers' self-attention to capture the relationships among tokens across different sequences. This allows transformers utilize self- for full context representation.
Definition H.3 (Contextual Mapping). Let Z(),..,, Z(M) RdxL be embeddings. Then, say a map T' d + RdxL is a (7, d) ccconteetll mapping if the following two conditions hold:
.....
1. For any i [M] and & E [4], it holds
(()( ). <7.
2. For any i,j E [M] and k, I [Z] such that v(() * vii or 2 + z it holds
IT(Z()) (())) )..1 >
We introduce results from [Hu et al., 2025b] in Theorem H. which shows that a one-layer single-head attention mechanism is a contextual mapping.
Helper Lemmas. Before presenting Theorem H.I, we restate several helper lemmas from [Hu al., 2025b, Kajitsuka and Sato, 2023] simplify the proof.
Lemma H.1 (Boltz Preserves Distance, Lemma I of [Kajitsuka and Sato, 2023]). Given (7, 6)- tokenwise separated vectors (1)) (WV)
=(N) E RR with no duplicate entries in each vector:
(6)
2
7 2
40

---

where i E [N] and 8, E [T], + t. Also, let
8 2 4 In
Then, the outputs of the Boltzmann operator has the following properties:
Boltz
(2(1) S 7
Boltz (2())
Boltz
2011 > 8' = =In²(n) -e" -22
for all i.j E [N], + j-
We remark that Lemma H.1 gives the key step for constructing a contextual mapping via the self- attention layer Lemma H.4. We extend the results the case with aivers-fempeerttur scaling.
Lemma H.2 (Boltz Preserves Distance with Finite Inverse Divers--Temperatrue Scaling in Softmax). Let (1), (M) E R= be (7, totkenenes separated vectors with no duplicate entries in each vector;
(4) f #t
where n 2 2.5 2 4/nn for some À > 0, i E [M] and 8, E [L], + I. Further, we define
Boltza(2(9) == (1) Softmaxx for all i e [M],
where Softmax,, (=)0)(<)))) (x)()=(0())/ E, exp (xz(A>(())) softmax function with inverse-
temperature scaling À > Then, the outputs of the Boltzmann operator has the following properties:
Boluza (z(0) s:
(H.1)
In Inn((n) c 227
Boltzz (210) - BoltzA (250) > 8 =
(H.2)
for all i,j E [M], j
Proof. We first scale the input vectors by (0) = 200 À for all i [M] and all j E [n]. This gives (R, A)-tokewwise separated vectors y (1), ,.... (M) E R. where R = Xr and A A6.
Next, notice that softmaxx(1) (.) = softmax(A-) and (.) Boltz(A.). (A-tz((A///. Therefore, since the condition 8 > 4 In n/A ensures A 2 4lm Lemma H.1 implies both (H.1) and (H.2).
This completes the proof.
Lemma H.3 (Lemma 13 [Park et al., 2021]). unit vector IRd such that
For any finite subset X' Rd, there exists at least
8 V rd
1
x'1l
S lu (x x¹) S || 2'1l.
12
for any x, x' X.
Lemma H.3 shows the existence of a unit vector E Rd that bounds the inner product of the difference between points in finite subset X' C Rd. Next, we restate the construction of rank-p weight matrices in self-attention layer following [Hu al., 2025b] in Lemma H.4.
Lemma H.4 (Construction of Weight Matrices, Lemma D.2 of [Hu et al., 2025b]). Given (Niii Tmax c) sepparaee input embeddings ((),.... (M) E Rdxt with finite vocabulary set
41

---

V C R There exists rank-p weight matrices WK, We Raxd such that
(W (WQue) (Wxu6) (Wque) 6,
for any 8 >>0 0, any min (d, p and any Va: Ub, Ua with va # vb. In addition, the matrices are constructed as
WK Σ pagl E ER'd We Σnt E ERad
j=1
where qs of E IRd are unit vectors that satisfy Lemma H.3 for at least one i, and pr p/ E IR satisfies
P. (1V 1)a
CYmin
Any-Rank Attention Contextual Mapping. The next lemma shows that the any rank self- attention mechanisms of transformers serve as contextual mappings (Definition H.3).
Theorem H.1 (Any-Rank Attention is (7, --Cnotextual Mapping, Lemma 2.2 of [Hu al., 2025b]). Consider (7min: Timax: --toeenwis separated embeddings Z ..... Z Z(M) Rd and vocabulary set V UsENN] (03) Rd. Let (11), ((N) d be embedding sequences with no duplicate
word token in each sequence; that is, " * - for any i E [M] and k, E [4]. Then, there exists 1-layer single head attention with weight matrices Wo E R and Wv, WK. We E Red that is (7, ccontxxcll mapping for embeddings ( ..... ((M) with
Tmax e/4, 8 = exp(-5 - " dx log L)/X,
where 7/max/ 7min and is the inverse-temperature scaling in the column-wise Softmax function.
We restate the proof Theorem H.1 since it crucial for subsequent analysis.
Proof. For completeness, we restate the proof from Lemma 2.2 [Hu et al., 2025c].
The proof consists of two steps:
Construct the Softmax Attention. We ensure that different input tokens are mapped unique contextual embeddings by configuring the weight matrices in Lemma H.4.
Handle Identical Tokens in Different Contexts. We show that the construction from Lemma H.4 are able to handle identical tokens different contexts by applying Lemma H.2.
We proceed the proof with these two steps. Step Attention Construction. We show the construction of matrices: WK, Wo, Wo and Wv.
Weight Matrices Wx and Wq. First, we construct WH and Wq by:
WK = Ena E axd We = EnSe E Raxd
==1
j=1
where Ps, p/, E R and g. of E Rd. In addition, 8 = Inn and pI,pl E R2 be an arbitrary vector pair that satisfies
8
P P = ([V) + 1)*
(H.3)
6Ymin
42

---

Weight Matrices Wy and Wo. Next, we construct Wo E Rdxs and Wy E Rexd by:
Wv = Σ P
Th ERa
(H.4)
where 9" E Rd, 20.."' and po E R is some nonzero vector that satisfies
WWr''l =
(H.5)
4pymax
This can be accomplished, e.g., Wo = Ei= p for any vector p!"" which satisfies |! 1144pmmmaxlpppp"" for any i E [p].
For simplicity, we define s Softmax (Wx2 T Then, we combine the above weights construction and obtain
( (waz?))
lW (Wvz(1)
Softmax (w>zZ") (
(H.6)
$ Wo (Wrz(0)
k
(By the definition of s
1=1
||st Wo (Wrz(0)
(By triangle inequality)
max EEELL
lloo
WV (00
(By Eie $ 1)
max E)
4
7.19)
2:,00
(By (H.4) and (H.5)
4mmax
max K
(2(1) 7,
(By Lemma H.3)
47max
(By the (Ymmna Ymaxx, é) separateness)
4
for E [M] and E [Z].
Step 2: The Case of Identical Tokens in Different Contexts. For the second part, show that with the constructed weight matrices Wo, Wv, Wx, WQ. the attention layer distinguishes duplicate input tokens with different context, z,60 z/6) with different vocabulary sets V(A) # vG)
We define (1) a as
a (WaZ"") (Wzz!!) ER", a0)) a=)= -w==(Zz (oll))(www2 (Waz!!) ER",
where (9) and aن)) are tokenwise (y, 0) sesparaad Specifically, the following inequality holds
a < ((V) + 1)* d 8
e7min
Since V(0) # vti and there is no duplicate token in (+) and Z(1, we use Lemma H.1 and obtain
Boltz (a()) - Boltz [aci)]
(a())) T
Softmax [a(0] - (ali)) Softmax [a00]
(H.7)
43

---

>8
scaling and shifting the transformer's parameters in J(FF) and ((F)), Hu et al. [2025b], Kajitsuka and Sato [2023] prove Theorem H.2 assuming that the target function f is normalized on domain nlun) e 22
Additionally, using Lemma H.4 and (H.3), and assuming z.t 2 we have
(att) T
Softmax [alt)] (aai)) Softmax [aco]
(H.8)
Tmax (11I+ 1)a rd 8
(11 2(0) Softmax [a[o]] -(1I Softmax [avi]]
C/min
By combining (H.7) and (H.8), we have
Σ 4 2(1)
Softmax [a(0] - (uT2u0) Softmax [a(o]]
8' E7mmin
(H.9)
(MV
1)²
dō dmax
Using (H.5) and (H.9). we derive the lower bound of the difference between the self-attention outputs of 7()) 76) follows:
(SA) (Z
-F(SA) (Z6)
E
8'
ETmin
8
mmax (IV) + 1) dō
where 5 4n L and 6' = In (L)e 27 with y= (DV++) 1)a dō max/ (C Finally, extend the bound to softmax function with inverse-temperature scaling À > 0 using Lemma H.2.
This completes the proof.
Notably, Theorem H.1 shows that, for identical embeddings 2 z with distinct vocabularies
V(0) * yG), any-rank self-attention is able to distinguish two identical tokens distinct contexts.
Universal Approximation of Transformer. We introduce the universal approximation result for transformers with a single self-attention layer from [Hu et al., 2025b, Kajitsuka and Sato, 2023].
Theorem H.2 (Transformer Universal Approximation, Theorem B.1 of [Hu et al., 2025b] and Proposition 1 of [Kajitsuka and Sato, 2023]). Let E (0, and [1, oo). Let ((FF) _F(F) be two feed-forward layers and (SA)) be single-head self-attention layer with softmax function defined in (B.1) and (B.2). Then, for any permutation equivariant continuous function f on a compact domain and any e, there exists (Z) (FFF)) (SAA) ((PF) Th such that dz(( where d, [J (2 (d2)2/)) and
|| is the element-wise fn-norm.
Proof. Since the universal approximation of transformer over any bounded domain differs only by
[0, 1]dxL L for simplicity. To support subsequent derivations of transformer parameter bounds required for e-precision (Lemma H.5), we provide the proof on more general bounded domains.
The proof consists of three steps: (i) Quantization by the First Feed-Forward Layer (ii) Contextual Mapping by the Self Aeff-Attetioon Layer (iii) Memorization by the Second Feed-Forward Layer
Let Ω == [-1, be the domain of f. Without loss of generality, we consider / E N.
First Step Quantization. First, define grid Gp:
Gp = {c Ω Ct - +
St.k D
Se, I,... 21D for all tE E A 1413.
(H.10)
where D N is the granularity of Gp.
44

---

Then, for Z E ΩN we construct piece-wise constant function approximator:
91 91(2) = E -[CIL(ECC+(1//D00
(H.11)
CEGp
By the uniform continuity of for any > there exist such that
91) </33.
(H.
We use a transformer to approximate 91 (Z) using two feed forward layers and self-attention layer. First, we introduce the quantization function that discretizes the input into Gp:
Quantization Function. We define the quantization function quant R - R:
2<-1 - I 1/D <I
-
pu))t
T-1/D
Within Q, quant (x) outputs regions identical to + [-1/D, d)xx defined by the indicator
L function in (H. We extend the quantization quant (Z). RRdxL - RdxL where quantp(2
applied coordinate-wist wise. Then, approximate quant p (2) with the following network*:
ID-1 ReLU [z/8 f/6D] ReLU [2/5 - t/6D
f1(2) = +I Σ
quant (2)) (H.13)
D
where > 0 determines the steepness the change from one quantized level to the next. For E RR [-,,1]. add and subtract the first and last step functions scaled by / to obtain zero output:
==) fz) - (ReLU [2/6 1/5] ReLU [2/6 1
+ [ReI.U[-2// 1/6] ReT.U --26 1-16)))
(H.14)
where (F)(() approximately quamitzes [-1, into
1} and project R |- to
by taking sufficiently small 8. Since every clement shares identical weights, we realize (H. using W'ReLL W. + b;] b'g by constructing W/ e (R--(44D+4)x0 and b2 E (BL((4)44)) as:
((2) w(2))
(
b
+ +
W1
b
where 6
for all i [L], (H.15)
1
(())
(L)
"Tii is the shifting and stacking step functions from (Yun al., 2019].
45

---

and W(O)
W (R( take the form:
1/6
0
0
1/8
0
0
1/6
1/8
1/8
1/8
:
1000
w
1/6
0
WW(21
0
1/8
0
W
0 0
1/6
1/8
O
1/8
1/8
1/6
0
0
0
1/8
0
0 0
1/6
1/8
0
1/8
1/8
1/5
0
0
0
1/8
0
-1/0
1/8
1/8
-1/6)
The first 4ID rows of W1 and by implement the ReLU terms in (H.13), and the last four rows implement the ReLU terms in (H.14). Then, construct W/2 RdxL-(UDD+)) and by IRd by
1/D 1/D
1/D 1ID -T T
I
-7
I/D
1/D
1/D
1/D
-1
I
-1
W
8/ =
(H.16)
1/D 1/D
I/D 1/D -I I
I
Besides the quantization, we include an additional penalty function to signify the case where the inputs are not the target domain [-1, 1J
Penalty Function. We define the penalty function penalty
R +
-21
2 << -1
penalty(z) =
0
2 [-1,]]
(H.17)
211
>> I.
Again, we extend the penalty function to penalty 4x (z) _d + gdx where penalty(z) is applied wossdinate-wise. By taking sufficiently small approximate penalty(z) by
FF (=) ss penalty(z),
where
PFF = 21 ReLU - 11/66) ReLU [(2 - 1)/6 - -1]
(H.18)
Approximate stop from to -27 =
21 ((-LU[(-z-A1) 1)/6 + ReLU [[-2 1)/6 - -11
Approximate step from to 27 at
Let RdXL be matrix with all entries equal to T. Altogether, we define g2 RdxL R
g2(Z) quant dxL ( + I+++d dL -eenally dx (2)
(H.19)
(A)
(B)
Term (A) first quantizes input [| into Gp and scales the grid [0, dx denoted by Gb-
Term (B) ensures non-positive outputs for any Z E _dx \==,,
Together, we approximate (H.19) using the first feed-forward block W2ReLU Z bi] + b2 with W Wi E gL-(41D+8)xd bi E g (+)) W2 L and by E IRd where stack 4L rows to W/, &y implement the penalty function, scale weights in Wz, and set by as a zero vector.
46

---

Second Step Contextual Mapping, Let GD S GD denote the sub grid on [0, 21]ax
ē G CD G. + G. for all k, K
(H.20)
By the construction of Gb. the sub-grid GD is a collection of grids with pairwise distinct tokens, and every G E GD is a token- wise (1/D, /I,d,
sequences.
From the construction of ((SA) in (H.6), it holds:
(F(S Z,, <
max SI WELL]
||
Recall that the magnitude of every entry in (Z) is at most 21dL by the construction of the penalty function. Therefore, for all k E [L], it holds:
|[FF 2TdL Vd.
ZERUXL
This implies
dL 4D
([F(SA) (Z) O FFF (Z) FFF(() F.F
(Z),aal
Taking sufficiently large D, every element of output of Rd |[ [-I, 1-,11 11xx. is upper-bounded by:
F(SA) o FFF (Z),,
for all
[d]d [L].
4D
Also, for input 7 [0, 11aXL have lower bound:
F(SA) o FFP(Z)), >
for all
1 [d], E [Z].
4D
Then, it remains to map sequences on (3/4D, oo) to their corresponding target value, and map sequences on on (-oo, 1/4D)dxL to zero.
Third Step Memorization, For fixed C E Gp and a network input Z, we define = F(SA)( and S = (SA) FFF The goal is to use FFF to implement RRxx + Rdx where
g3( f(C D),
1[u
Sc,,] for all [d] E [L].
and ʃ is the target function. We use the bump function with three-piece ReLU to achieve this:
(S)p
(H.21)
ReLU [Rpp ue 1] 2ReLU [RFF(S ur,x:] + ReLU [RFF(S ue,a) 1],
where inputs with deviation from the correct grid point ur,A greater than 1/RFF are mapped to zero Therefore, by taking sufficiently large Rer, (H.21) implements [[4,e = St,x] exactly.
Suppose that there exist a feed-forward block FFF such that (H.21) holds for all t E [d], k E [L]. By choosing the granularity D sufficiently large such that CD CD negligible, holds
dy do(FFF o F(SA) o JFF FFF .91)
2e/3
(H.22)
47

---

Combining (H.12), we have
dp (FFP F((A) o FFF, .f <e.
It remains to show that there exists a F2 that implements (H.21) for all t E [d] E [L]. This is possi- ble since Theorem H. ensures that (7mins "mmax ? -ttkeeeewise separated sequences are least 8' apart from one another after the self-attention mapping, where 8' exp( ( mmax log L/ Tuin. is the inverse-temperature constant and [V] is the size of the vocabulary set.
Specifically, for fixed C and u JSAA (C,, we construct W(") E R33LXx and b(() E R3dL:
RpF
11
Rpr RpF
R1pu1111 +
0
RFF
O
RRF U221
RFF
RpF
REF
0
RFFU22,1 +
RFF
RFFFU3.1
0
0
ReF
0
RFF 1
REF
REF U3,1 +
RFF
RFF
RFruu,
ReF
ReF Ud,1 +
w(O
(H.23)
RpF
1000
RRFF1, L
RpF
RFFU1,
RPF
RFFUILL
Ryr
-Rppu2,1
RFF
RFFFU2,
0
RFF
0
RPEUU2, +1
RFF
-Rprug,
0
0
ReF
-RpFus, 1
RFF
RRpuu,, +1
1000
RFF
RRpFud,
RFF
Ryyud,L
ReF
Rppua. +1
where the first 3L rows of ( are repeated d times and S = FSSA) o FFF (Z) is mapped to 33
w + 119
48

---

S11, V1.1)
RFr(S1,2 (1,1
R5r(S1, (1,1))
(11 u1
R1r(S1.2 u1,1
RFr(S1.1 u1,1
(S1 (1,1 +1
RFr(S1.2 (1,1 +1
RF (1,1 +1
[Sr112,1 W2.1)
RFr(S2.2 - (2211)
RPr - 12,1)
Rer S2.1 u2,1)
RFr(S2,2
u2,1
Rer (S2.1. u221
RFr(S2,1 S2,1 w2,1) +1
Rpr(S2.2
- u2,1
REF S2,1. u2, +
RFF(Sd: ud,1)
(Sd.2 (dd11)
R (.d,1)
RpF (Sa.1 1
Ud,1)
RFF (.S1)
RFF (S ud,1
ReF (Sa.1 ud,1) +1
(Sa,2 ud
ReF (Sa.r. ud +1
R1r( - 11.
RFr(S1.2 - v1
RFr(S1,1. - v1.I.)
(11(1 U1
RFF(S1,2 L1
RFF(S1,L 11
RFF(S111 U1 ) +1
RpF(S1.2 - L1
+ L
RFF(S1, - 11 + +1
RgF(S2,1 U2
RFF(S2,2 12
RFF(S2,L 122L)
(Sr(S2,1 V2.L
R2r(S2.2 - U2
-1
Rer - 12 -
(Sr(S2,1 W2,L +1
RFr(S2,2 12 +1
Rer(S2,1 122 +1
RFF Ud,L)
REF (Sa,2 - Ud,.)
(SrrL - ud,.))
RFF (Sa,1 ud,
HFF(Sat Ld
RFF (Sa, Ud,
Apr (Sa,1 Ud,L) ( +1
Byr(Sug ud +1
Rpr (Sa.L - ud.L) +
Then, we construct the second matrix by W( = W2 ((1)
W2 (1)
gaxsal with W (()
gdlx33L
and W (()) E RRdxLL, For W
(t)
2f(C 1),1 S(C Dua J(C-I)) I)1
w (()
-2f(C D)a f(C D)a f(ō Daa
which maps the output of the first layer after ReLU operation RdLXL,
w, " S b0o
J(C T),, bu.buupp(S.. -1,1)
I(C I),,1 bump(S -1,11
J(C I)1,1 bump u1,1)
f(C T) bump(S2,1 U2
f(C T)2, bump(S22 U2
S(C T)2 bump U2
J(C a, bump(Sa, ud,1)
J(C I) bump (S,.2 Ud,1
J(C D) bump(Sa,r ud,1)
f(C I) 1,2 bubbuu(s,,, v1.2)
f(C hh.zbump((5) U1
f(C bump(S,,,. U1.2)
J(C ()2abunp(S2 22
S(C T)2.bumpp(S2. 22
J(C T)2abump(SS 2222)
S(C - a) auuup((Ss (42))
S(C I) 2)aabuuppSSass 4422)
S(C I Mazbmmp(SSa. ud.2)
11C I),, bump -1,z)
J(C I),, bump (S1, u1,1)
-1,11
S(C I) Cumpp(S1, u1,1)
f(C I)2, bump u2
f(C
42
f(C I)2, bump (S2,1 W2
J(C Dumpbunp(S -a
f(C =T),bbump(Saz ud
J(C DampbumpSSu. ua.)) (H.24)
49

---

Recall that we have tokenwise (1/D, 21 // -separated sequence after the quantization of the target domain Ω and [V] < (2TD) Therefore, it suffices to set the scale of the bump function as
RFF = ( ((ex((E00X(L) ad+2 d log L)).
(H.24) is a partition of L vertical blocks, each dimension L. Then, is permutation of u, single column within each block is comprised of correct target output since their bump function all evaluates to one, and all other columns containing zeros. Also, The position of this all-ones column distinct for every block. Conversely, if S is not permutation of u. (H.24) becomes the zero matrix.
Lastly, we construct W22 (() by (La, La, where the d x d identity matrix is concatenated L times.
La) sum over every column in each block matrix,
Up to permutation equivariance, there are total Q2 (2I possible u, stack (") 66(0) and ((()) with weights corresponding to each distinct u using the identical construction.
2
In sum, there exist feed-forward block FFP with Wi RidLgxdd a bj E RadLez and W2 pdx3dd- that implements the bump function (H.21) for all f [d] and le E [L].
This completes the proof.
Remark H.1. Note that Theorem H.2 uses 2 FFN layers and g E Till", where hidden dimension r - O(dL(ID)dL /L!). By Definition B.2, T1.11 belongs to our transformer network class.
To eliminate the permutation equivariance required for the target function, we incorporate the positional encoding to Theorem H.2 to break the symmetry following [Yun et al., 2019].
Corollary H.2.1 (Universal Approximation of Transformers with Positional Encoding, [Kajitsuka and Sato, 2023]). Let e E (0,1) and p E [1, oo). Let F(FF)
(FFF) {(( be two feed-forward layers and (SSA) be a single-head self-attention layer with softmax function defined in (B.1) and (B.2). Then, for any continuous function ff compact domain and any there exists a positional encoding
E E [ L and a (Z) ((FF) (SSA) F (FFF) (Z + E) Thiss such that do h) < where d = ( (1( - gf (Idd)) L/P and || !|p is the element-wint wise fo-norm.
Proof. It suffices show that the universal approximation remains valid with the inclusion of positional encoding E E RdXL to the weight matrices constructed in Theorem H.2.
Recall the (H.14) and (H.20). We have token-wise (L/D, 21 /d, 1/D)-separated sequences on [0, 24,27 after quantization. Then, we add the positional encoding to the quantized sequences:
21
41
21L
E
21
4I
2IL
giving token-wise (2IVd, 2(L
sequences, where the first column is in [0, [0,47]4, d the second is in J0, 67) and so on. For each row, entries are monotonically increasing.
The second step contextual mapping remains valid by scaling the penalty function (H.18) to 2I(L+1). By Theorem H, columns after the self-attention mapping are at least 8' apart (in (z-distance) from each other, where 8' exp -
log L/7min) and | =((ID)d)
Recall the bump function (H.21) and (H.24). The construction of ((()) ((0) and W follows by setting the scale parameter RpF = (1/8)) Because of the lack of permutation equivariance here, necessitates to stack these matrices and biases for all 42 (2TD)aL possible values
This completes the proof.
Parameter Norm Bounds for Transformer Approximation. Next lemma provides matrices norm bounds required to achieve the universal approximation of transformer with any error e.
50

---

Lemma H.5 (Transformer Matrices Bounds, Modified from Lemma F.4 and Lemma F.5 of [Hu et al., 2025cl). Let (0, 1). Let Z [-1, 1 be an input sequence, where I is an absolute positive constant and L 2 2. Let f(Z) [|1, - I be any Lipchitz continuous function with respect to norm dy. Then, for g E TR.,. that approximates f within precision, i.c. dy (f.9) < e, the parameter bounds the transformer network class follow:
CкQ; k = 10(A-11dd+) ) Cov.Con = 0()); Cr,
22 Cr. = O max (1 CE = 0()),
where -- ([//e)dd+3 is the inverse-temperature scaling in the softmax function and O(.) hides polynomial and logarithmic factors depending on and L. Further, for all feed-forward layers,
max bl [|22|oc] O)
Proof. Hu et al. [2025c] provide parameter bounds for the universal approximation of transformers
on domain [0, 1]=x2, We specify these bounds for approximation on domain [-1, 1]
Recall the construction of weight matrices in the proof of Theorem H.2. We achieve the universal approximation by choosing "sufficiently large" granularity D and "sufficiently small" (H.13).
To prove Lemma H.5, we first identify the order of 6, D and ReF in terms e. Then, we derive norm bounds matrices in two feed-forward layers J F.FF and the self attention layer FSA
Bound on Recall the approximation of quantization function in (H.13). each step function, we have extra partition L1/D,1/D 8). Therefore, suffices to take ō = o(1/D).
Bound on the Granularity D. Recall the contextual mapping step in the proof of Theorem H.2, The total omitted duplicated points in the grid Gb are GD GD D- (2ID)d4), where GD S G3 the sub grid consisting of sequences with non-duplicated tokens. Further, by the extreme value theorem, || l S By for constant By 0. Then, the difference between the target function f and the piece-wise constant approximator J1 with granularity D is bounded by
dr(F.91)
= (/ -1 -2(2)))))))
- By (1)D)
= O(D 1d
Forp E [1, oo), we have that O(D 4/p )d This implies D O(e-rid.1-L/p) Without loss of generality, we drop T-L/p E (0,1) 1) and drop the constant p. Then, we have that D (1-d/)).
Next, recall the picce-wise constant approximation (H.10), (H.1l) and (H.
For Lipchitz continuous target function f. there exist a grid Gp on domain [-I, n such that
dp(f(Z). Z Z' L Z'Ile VdLL,/D,
where Z' E GD and L is the Lipchitz constant with respect to the matrix 2-norm. Therefore, it suffices to take VdLL,/ D. Altogether, we take 1) O such that Theorem H.2 holds. Next, we derive the norm bounds on transformer weight matrices.
Bounds on We and Wx FSA, For the self-attention layer, we denote the separatedness the input tokens by (Tmin Tiaxx ex) and the separatedness of the output tokens by (7, 5.).
51

---

Recall Theorem H.1. We construct rank matrix We and WK in the self-attention layer by
WK = Σ Piq. E E Wq - Σ Pig E ER
1=1
==1
with the identity p. (IV) (5s (es/mi))) Then, the bounds
WKQ follows
[[ - WxxllF II(Wx) ( WollF = -0 0. V
(, Vinin
| Wolk) O 0 [6=121*
EssTiin
We identify the order of each terms. Recall the first step quantization (H.13). We have total (DI)dr. input that are token-wise (1/D, 2d -/epaarat
Further, since there are at most DI possible values that each entry can take, we have vocabulary [V] = ((DT)") and Ymin: C, (2DI)- Further, from the proof of the second step contextual mapping in Theorem H.1, we construct the self-attention such that ó, A = 4 L, where we specify the choice of À in (H.25). Finally, by D O(e=!) the bounds on WRQ& follows
[[xxlll & CkQ = -(A - 1d-22, rdd+2);
[WKoll2,0 S C = 0(A o(x-tc-Ad-2 ( rld+2).
Bounds on Wo and Wy in FSA, From the proof of contextual mapping Theorem H.1, we have,
Wv = End "T E e Wo = Σ m E Rddx
with the identity ||r S e (4p(4pmaxlllpp 1I from (H.5), and p. E R is any nonzero vector.
With the (7min = 1/D, 7max = vd, E, 1/ separateness and D O) we have
Wwlll = sup |[Vv S = = (Vi).
||||| ==1
[Will2.. max [[ cz = 0 =
isish
[Woll2 = sup |ooll & Co = 0 (Vp p mmxx (6)) = 0 0 (d
||
[oolla.s max l((o)) s cgx0 = 0 p 7max (G) 0
15i5t
Therefore,
[[ov =
| WoWvllk s Cov = 0(e);
||oovl2, = [oWWvll2.0 S C2 = O(e).
Bounds on W1 and W2 in FFF Recall(H. By 6 = o(1/D) and D = O(E )) we have
max ( ( | [| s C2 =
max [Wx[|WWl22.... W2 |2,00 < 02200 = - 10 IDL).
Bounds W2 and W2 in FFF. Recall the construction of bump function (H.24). We take
RFF 0(exp(640X(1D) 4422 &l L)),
52

---

where (Ymin Ymmax: c) -2 Recall that D = O Then, we take
(ID) logg
x=
(H.25)
640d llog L
This gives
max [| S - max [I](
max Will,,, [2 I CF Ole max ff(
where Ω [-1,1jdx4 is the domain of the target function f.
Bounds on Positional Encoding Matrix E. By Corollary H.2.1, we have:
2T 41
2TL
=
".
21 41
21L
Therefore, we have CF = O(T).
Finally, recall that (i) = F(A) (C) from (H.23) (ii) Cov Occ) and (iii) ReF c- by (H.25).
Then, the bound on the bias holds by the construction (H.15), (H.16) and (H.23).
This completes the proof.
53

---

I Statistical Rates of Flow Matching Transformers (FMTs)
In this section, we present statistical rates for the first order flow, i.e., the velocity field, ue (z)..
Specifically, we consider the target density function (a)) in the Hölder space (Definition I.1 with sub-Gaussian property. Then, we bound the approximation and estimation error for ut (z()) Further, extend these results to derive distribution estimation rates under the 2- 2-Wasserstein distance. Compared to high-order flow matching statistical rates Section 4, we remove the requirement of Lipschitz continuousness of the velocity field uy (z).
Organizations. Appendix I.1 presents velocity approximation under a generic Hölder smoothness assumption. Appendix I.2 adopts stronger Hölder smoothness assumption; this yields tighter approximation error bounds toward minimax optimality in velocity estimation. Appendix I.3 utilizes these approximation results to develop velocity estimation bounds and distribution estimation rates. Finally, Appendix I.4 establishes the nearly minimax optimality of flow matching transformers.
1.1 Velocity Approximation: Generic Hölder Smooth Data Distributions
Establishing our statistical theory begins with upproximating the velocity using transformers. We present the corresponding velocity approximation theory under the Hölder smoothness assumption on the initial data [Fu et al., 2024]. This theory ensures our approximation rate adaptive to the initial data's smoothness. First, we restate the definition of Hölder space and Hölder ball.
Definition L. (Definition 4.1 Restated: Hölder Space). Let E E and let 3 = k1 + 7 denote
the smoothness parameter, where ką = [B] and y [0. 1). For a function f R + R. the Hölder space H (IR.) is defined as the set of a-differentiable functions satisfying: 7 (R2) = {s Rd IR Il (1 < }o), where the Hölder norm || (((())))))) satisfies:
10 f(x) - s0)(())
(R)))) max sup 18t [(x)+ +
max sup axI[oil b1 xttt
| x' 113
Also, we define the Hölder ball of radius B by H (Rd B) = ( Rd + R Illl < B}.
Before presenting the main result of velocity approximation, we state our two assumptions: (i) the Generic Hölder Smooth assumption on the target distribution g(=1). (ii) the regularity assumption on the first derivative of path coefficients. In particular, (i) and (ii) are the counterparts of Assumption 4.1 and Assumption 4.2 in the K order flow matching framework (Section 4) respectively. Notably, remove the Lipschitzness assumption via a more fine-grained analysis on the velocity field u, uz()).
Assumption I.1 (Generic Hölder Smooth Data). The density function ((x1) belongs to Hölder ball of radius B with Hölder index 3 > 0 (Definition 4.1), denoted by q(x1) H. (12d, BB. Also, there exist constant C1, C2 > 0 such that g(x1) C1 exp -C2ll=a11//2)
Assumption I.2 (Path Regularity). Consider the affine conditional flow ()((((1)) = u + oza. The first-derivative of path coefficients o; and é are continuous on [4o, 7), where to, T E (0. 1).
We remark that such path assumption is general and applies to number of common scenarios. For instance, Lipman et al. [2024] present: (i) the conditional optimal transport schedule: (x(21)) tæ + (1-t)æ, (ii) the polynomial schedule: v (2|211 t" x1 + (1, (iii) the linear variance preserving schedule: (x(()) frı + v1 fr.. These cases satisfy Assumption I.2.
Remark I.1.
We now present the velocity approximation for flow matching transformers.
Theorem L.1 (Velocity Approximation with Transformers under Generic Hölder Smoothness). As- sume Assumption I.1 and Assumption I.2. For any precision parameter 0 < < 1 and smoothness parameter > 0, let e s O(N for some N E N. Then, for all E [tg. T] with ty, T E (0, 1).
54

---

there exists transformer uz(2, E Thh. such that
uo (z( drdt O (BBN~~ (log Ny)a+ (untt)
T
Let d be the feature dimension and L be the sequence length defined by the flow matching reshape layer in Definition B.3. Then, the parameter bounds in transformer network Th: satisfy
CкQ. k O(> N (log N) +d==+) Cov. Cor = O(
CF, 2200 = ( (log N) +1); Cg O(1); Cr O Ooooo N).
where A-1 = O(N9, log N the inverse-temperatuue scaling in the softmax function and (-))
hides all polynomial factors depending on d d, L,, B, C1. C2-
Proof Sketch. We adopt the following strategy:
Step 1: Approximation on Compact Domain via Transformer Universality. To reflect the Hölder smoothness of the target density g(21), we begin by applying multivariate Taylor expansion to construct compactly supported approximation of velocity field uz(x). We then approximate this function on compact domain using the universal approximation of transformers.
Step 2: Extension to the Full Domain via Sub-Gaussian Tails. We exploit the Gau-Gaa tail behavior of the target distribution control the approximation error outside the compact region. Combining the errors from both regions yields the final approximation rate for the velocity field.
Please see Appendix for detailed proof.
1.2 Velocity Approximation: Stronger Hölder Smooth Data Distributions
We obtain tighter velocity approximation rates than Appendix I.1 by imposing stronger Hölder smoothness assumption on the target distribution q(x,).
Assumption 1.3 (Stronger Hölder Smooth Data).
Let C, C₁ and C2 be positive constants. The density function satisfics q(x1) = exp exp (-21 l/2/2 f(x1), where f belongs to Hölder space f(x1) E 2 (Rd4,, B) (Definition 4.1) and satisfies C 2 f(11 2 C for all æ1.
The density lower bound prevents f(z) from taking small values, ensuring well-conditioned ap- proximation. Without this bound, small values of f(x) require chosen threshold to maintain uniform approximation. positive lower bound eliminates the need for such adjustments, keeping the approximation error controlled across the domain and enabling efficient convergence.
Assuming Assumption I.3, derive the velocity approximation for flow matching transformers.
Theorem I.2 (Velocity Approximation with Transformers under Stronger Hölder Smoothness). Assume Assumption I.3 and Assumption 1.2. For any precision parameter 0 < < 1 and smoothness parameter 0, let e S O(N-5) for some E Then, for E [to, T] with 4, (0, 1), there exists a transformer ug(x, t) E T
Th.ss" such that
[/) //
us (z 1) (x()ddz O (B²N 2 (log N) JR [|(((z) [11())
d.)++)).
Further, the parameter bounds in the transformer network class follows Theorem I.1.
Proof Sketch. The proof strategy closely follows Theorem I.I:
55

---

Step O: Velocity Decompasition. We invoke Assumption I.3 to decompose the velocity field into target function that lower bounded. This step mitigates the influence low-density regions and enables a more refined approximation analysis, in contrast to the setting under Assumption 1.1.
Step I: Approximation with Transformer Universality on Compact Domain. To capture the Hölder regularity of the target density ( we construct compactly supported function as an intermediary to approximate the velocity field u, (z) using multivariate Taylor expansion. We then apply the universal approximation of transformers to approximate the constructed function.
Step 2:
Full Domain Approximation. We extend the approximation to the full space by lever- aging the sub-Gaussian tail behavior, ensuring that the error outside the compact region remains controlled. Then, we incorporate all errors terms to achieve the final approximation rates for (x:)..
Please see Appendix K for a detailed proof.
I.3 Velocity Estimation and Distribution Estimation
In this section, we study the statistical estimation problems and develop sample complexity results based the established approximation results in Appendix 1.1 and Appendíx 1.2. Specifically, present the estimation error bound of flow matching transformers in Theorem I.3. Applying the velocity estimation rates, we further study the distribution estimation in Theorem I.4.
Velocity Estimation Building on the transformer-basee velocity approximation, we evaluate the performance of the velocity estimator uo trained with i.i.d. data points { empirical loss (2.12). To quantify this, we define flow matching risk:
J by optimizing the
Definition 1.2 (Flow Matching Risk). Let be the target distribution and X1 Given velocity estimator ug, we define the flow matching risk R(us) as the expectation of the mean squared difference between the us and the ground truth uy:
1
(x),,)) (x:)))1 dt, R(us) == T to to z(-pi
E
where marginal probability path pt and marginal velocity field u, are induced by affine conditional flow (x([2)) = 1 + ogæ follows (2.2), (2.3), (2.5) and (2.6).
Let ûg be the trained velocity estimator with i.i.d samples {-:)/-1- Then the following theorem presents upper bounds on the expectation of R(ss) w.r.t training samples {z, 1 where ~
Theorem I.3 (Velocity Estimation with Transformer). Let d be the feature dimension. Suppose we choose the transformers as in Theorem I.1 and Theorem I.2 correspondingly. then we have
Assume Assumption I.I and Assumption I.2. Then,
E [R(T)) = O(n TodtTs (log 20d. + +4,4220)
Assume Assumption 1.2 and Assumption I.3, Then,
E {w.)
[R(a))] O(n 5x45 (log 20d 202d 204,+484+20 +48420).
Proof Sketch. Recall (2.12) from Section 2. We obtain the velocity estimator (1( t) E T Thi by minimizing the empirical conditional flow matching loss:
{'
(cm((u0) Σ [=1 3/ T to 11.)(0.1)
E
[ + oo. (JizZ + d.
56

---

To derive the estimation error, we adopt a standard strategy in empirical process theory. This involves bounding the generalization gap between empirical and true risk using covering number techniques:
Step I:
Domain Truncation for Risk Control. We truncate the domain of the flow matching risk and the flow matching loss to ensure the transformer network has a finite covering number. We then control the error outside of the truncated domain by using the sub-Gaussian tail bound.
Step 2: Analysis on the Complexity of the Transformer Network Class via Covering Number. Using the norm bounds on transformer parameters from Appendix L.2, we derive upper bound on the covering number of the transformer networkfunction class. This captures the model complexity required achieve a desired approximation rate on the compact domain.
Step 3: Final True Risk Upper Bound, We apply the covering number bound to control the deviation between the empirical risk and the true risk. Lastly, we incorporate all sources of error from previous steps to derive the final estimation rate for the learned velocity field üg (I,, 1) E T har via the minimization of the empirical conditional flow matching loss (ug) (2.12).
Please see Appendix L for a detailed proof.
Distribution Estimation. Next, we analyze the distribution estimation rate for the velocity estimator do through the 2- 2-Wasserstein distance between estimated and true distributions. Based the velocity estimation results in Appendix I.3, the next theorem presents upper bounds on the 2- Wasserstein distance between the target distribution and the estimated distribution induced by the velocity estimator Ug trained from optimizing the empirical conditional loss (2.12).
Theorem L44 (Distribution Estimation under 2- Wasseesttein Distance). distribution time T. Let d be the feature dimension.
Let Pr. denote the estimated
Assume Assumption 1.1 and Assumption 1.2. holds
E (x.)
[2((Pr P+)] O(n 528450 (log l0d, +28+10).
Assume Assumption I.2 and Assumption I.3. holds
E {
[Wr(Pr, Pr)] O(n re (log n) 10d, 424,+22110)
Proof Sketch. We derive the distribution estimation rate under the 2- 2-Wasserstein distance by relating the velocity estimation error through the flow dynamics. Our proof follows three steps:
Step 1: Flow Deviation via Globseey-Gröbner Lemma. We apply the Alekseey-Gröbner lemma (Lemma M.2) bound the deviation between the learned flow the and the true flow in terms of the difference between the estimated velocity 0g (z, 1) and true velocity fields (xxx)
Step 2: Bounding the Jacobian via Grönwall's Inequality. The flow deviation bound given by the Bockseev-Gröbner lemma involves the Jacobian matrix Drbg. To ensure the deviation remains controlled over time, use Grönwall's 's inequality (Lemma M. 1) along with the Lipschitz continuity of the network to upper bound the Jacobian norm by an exponential function.
Step 3: From Velocity Error to Wasserstein Distance. We integrate the velocity error over time and apply the definition of the 2- Wasseerstei metric relate the flow deviation to W2(Pr. Pr). Substituting the velocity estimation error from Theorem I.3 then gives the final convergence rate.
Please see Appendix M for a detailed proof.
57

---

1.4 Minimax Optimal Estimation
In Theorem 1.4, we present fine-grained analysis of distribution estimation. In this section, further show that the derived estimation rates match the minimax lower bounds in Hölder space under the 2- 2-Wasserstein metric in specific setting. We begin by recalling the minimax optimal rate for distribution estimation over Hölder smooth function classes
Lemma I.1 (Modified from Theorem of [Niles-Weed and Berthet, 2022]). Consider the task of estimating a probability distribution P(x1) with density belonging to the space
P = (1(x1))a(11) E H (1-1, 1) g(r)) 2 C}.
Then, for any r 2 1, > and d, > 2, we have
inf sup E q( (w.
[W,
where {-;) is a set of i.i.d samples drawn from distribution P. and runs over all possible estimators constructed from the data.
Proof. Please see Appendix for detailed proof.
We show flow matching transformers match the minimax optimal rate under specific conditions.
Theorem I.5 (Minimax Optimality of Flow Matching Transformers). Under the setting of (16d - (B + 1) de + 23, the distribution estimation rate of flow matching transformers (Theorem L.4) matches the minimax lower bound of Hölder distribution class in 2- 2-Wasserstein distance up to a log n and Lipschitz constants factors.
Proof. Please see Appendix for a detailed proof.
58

---

Proof of Theorem I.1
In this section, transformers approximate velocity and give upper bound of the velocity approximation error. We prove Theorem [.1 following the three steps shown the proof sketch.
Organizations. Appendix J.1 introduces auxiliary lemmas. Appendix J.2 establishes bound on the velocity approximation error over bounded domain by applying the universal approxima- tion of transformers. Appendix J.3 presents the main proof by incorporating the bounded- approximation error and controlling the unbounded region using the Gab-Gans assumption.
J.1 Auxiliary Lemmas
In this section, we introduce auxiliary lemmas for velocity approximation. Specifically, we decompose the velocity field (x) into three components in Lemma based on the setting of affine conditional flows (Section 2). To approximate each component, we clip the integral domain of æ in the integrals defining @1 (ar, t), (2, 1), and (3 1) closed and bounded region Lemma J.2. This step allows us to perform the approximation by restricting the integral. Furthermore, we revisit the bounds on the density function pe (z) in fx-- -istatneee, and extend these bounds to the velocity field (x) in Lemma J.3 and Lemma J.4.
bounded domain while controlling the error introduced
Decomposition of Velocity Field. We present the next lemma decompose the velocity field (x))).
Constructing an approximator for (x( is difficult due to its complex structure. This decompo- sition splits the velocity into three functions, each satisfying properties that make approximation feasible. These components allow the use of uub-Gaussian assumptions on the target distribution (Assumption I.1) and provide better control over the approximation error (Lemma J.9).
Lemma J.1 (Decomposition of Velocity Field). Under the flow matching setting (Section 2), the velocity field follows a decomposition:
( (2(2,)) + (0r
N400 )0 t)).
ue (2) =
#4
where
@ (I, [
||3: 14 x1 112
oft² (2m)da/22 cxp
gg dri,
20
#2(x, ffam de (2T))
| Me
exp
q((1)) dæ
20
/
I 14c-11
|| Me
03(2r 1)
JRds
at.
ot dd.2
de
exp
20
g(x1) dri.
Proof. By (2.5), the density function pr(2) has the form
pr(x)= / p((x)z1) q(x11) dr
/ exp
|| - x1|²
(((x)) dxy-
0, (2x)a./12
207
Therefore, we have pr(x:) = + (ar,)).
Then, rewrite the velocity field time by
Mx(xx)
(x)) .
( dr
59

---

[ 14 (11)
+ th (11 p1(x)
(By (2.6) and (2.8))
pr(z) Ja
Sxmn
Ō2(xx H4 xI1)
M4 (x pe 1..21 x1) He _ p((r)a1))(()1)d
pr(z)
ot
Nr
ue
io0
$z(2, + i (2(1,t)
(I,,
Ō2 (x,
Me
(By the definition @1, 22 and 2x)
( 12(2,
(ōr 1 ) (z)
This completes the proof.
Based on decomposition, we construct separate approximators for & (xx 1). 22 (z; (), and (a(2, (). Then, we approximate (x) by combining these approximations in Appendix J.2.
Clipping Integral Domain. Next lemma handles unbounded integral domain of 4 1 t), 22 t). and (2, t). Lemma J.2 ensures that for any small error >> 0 and any fixed x E Rdd, bounded domain By dependent on and x exists, where the integral outside B3 remains bounded by c.
Lemma J.2 (Clipping the Multi-Index Gaussian Integral, Lemma A.8 of [Fu et al., 2024] and Lemma F99 of [Oko et al., 2023]). Assume Assumption I.I. Let de be the dimension of the target data 21 and n N. Then, for any x E 24 with ||<||| n, æ1 E Rd and S 1/e, there exists constant C(n, dz) 2 1 such that
J J \B,
(
g(x1) (2 dd//2
|1 x²/2
exp
des Se,
01
20²
where ( (( = (( ( ل )
perde dd ا yrldd.) is multi-inde vector and
x + o, o,C'(n de de) /lo (1/e) OnC,n, dz) log (1/e)
B₂ =
Ur
Mt
n[c dz) ( log (1/c). C(n, de) log (1/c) de
(J.1)
Remark J.1. The rationale behind this error choice follows from the need to control the clipping error, when we construct a polynomial-like approximator for the components of the decomposed velocity 11, 22 and $ on the bounded domain B, Specifically, these approximations capture the smoothness of the density function in Hölder space and leads an error of order N-8 up to logarithmic factor. Therefore, the clipping error is set to match this order.
Bounds on Density Function and Velocity. We introduce two lemmas that provide bounds on the density function Pt and the velocity field (Ir). These bounds are crucial because the maximum output of the transformer network class plays a key role in analyzing the capacity of the loss function class in estimation error analysis (Appendix I.3), We start with the bounds on pt (z) and log Pt (z)).
Lemma J.3 (Bounds on the Density Function, Lemma A.9 and Lemma A.10 10 of [Fu et al., 2024I).
Recall that (x:) Sge. (x (2()))(()))))) )dx; and pt (xr (n) o,- "2n)2dd,77 exp(- |xx p.01 |1 20
Assume Assumption I.1. There exist a C7 such that
C7 d.
exp (- |111 +
S p(( =
Ci
C211=11 2(m + C20
exp (-
(N) C20
Moreover, there exist positive constant C₁ such that
C2 S
| logg(()))l
([01|2 1).
of
60

---

By Lemma J.1, the velocity field (x:) follows the decomposition
(
14 )0s(1; (xx
u<(x) = (x, 1)-
$2(x1,) + (Ør
14t
14
With this expression, we apply Lemma J.3 to obtain bound on the velocity ue( in distaacce..
Lemma J.4 -Bounds on the Velocity Field), positive constant Cs such that
Assume Assumption I.1. Then, there exists
litel
ie δz Me
(z))
S
|| + C3s
(1|=] 2 + 1).
Me
de
Proof. Recalling from Lemma J.1, we have the velocity decomposition
(#i #4
ju (2)( t)),
(x(x) Φ1 (x,
(2(x,))
+ (dr
where
(x, {
| 14 221 112
Jzh 0,400 (2x))=/2 exp
q((i) dæ
20
Φ2(111) 1) =x =x/ "SSr o.t0 (27))2222
|| pe x' 11
exp
(((x1) dr
20
| Me x1 21112
I u4 211
(2)xtt)= JA
of (2nr) exp
{((2)) dxy.
Ot
20
First, we rewrite the expression of (2 and & (2(x; (). Then, derive the bound on (x)).
Step 1. Rewrite Φ2 1) and E3(x,, t). By the definition of 22(2,, () and a2 (), it holds
(2(11)) = /
1
| - 14 x1 11
udo o. (2x)d
exp
207
(((1)) der = x $ (x. ().
Therefore, for all i E [dz], it holds
it
(2 t)(6]
T (x,
(J.2)
14
Me
Next, since the gradient of (xr) has the expression
Vp((x)= = / x u1 21
(-11x- - Mr x
oo
o (2T)).//2 exp
20?
g(z)) dri,
we have 1z(2, t) <pi(z) oc.
Therefore, for all i [dz]] it holds
(
= =((0x
#2 t)[i]
ar Vp((.)
(J.3)
1t
Mr
61

---

Step 2. Bound Velocity Field. Based on Step I, the following holds for all i E [dz]
[e 12(2, t)(i] - (0+
12 (2, t)(81)
Φ (2, t)
2 (x, 1) -1
( 22) t)(8))
(2(2, ()-,t)-1 (((2;))-((6))) -(00
Heot)
Me
(By triangle inequality)
[ 01(111)) + $1 (x), (meo - Vp(())(i))
$ (xx
14
14
(By (J.2) and (1.3))
V Vl
(By Vlog Vpt/Pe)
14g
jo
1 81000
z|i]
+
C
(1|2|12 + 1)
(By Lemma J.3)
He
o
Therefore, by symmetry,
Litl.
[(ee())
1oc
5
|001000 + C3
([|=]12 + 1).
Mr
This completes the proof.
J.2 Velocity Approximation on Bounded Domain
In this section, we approximate the velocity field (z) on a bounded domain through two-step approach. Specifically, the first step constructs three compactly supported continuous functions V1 (2. t). (2, t) and (2, t) as approximators for 1 (I, t), $22x1, f). and P3 t) in Lemma J.5, Lemma J.6, and Lemma J.7 respectively. Then, the second step applies the universal approximation to approximate V (z, t), 2 t) and (2, t) with transformers in Lemma J.8. Bu incorporating these steps, we derive the velocity approximation on a bounded domain Lemma J.9.
Before proceeding, we reiterate on the velocity expression. By Lemma J.1, wx(x) has the form
( (2 (ōr #e
peot )z)(z.))
ue (I) 41 (x,t)
Ht
where
[
1
| 14 12112
(x, Igt dz (2x))d=/2
exp
q(xz)) dæ
20
(2(x11 SS oh (2T))a./
|| P4 Ir 11?
exp
(((1)) dx1,
207
(x(x,))= / (I- 21
| - #c x1ll" 11
122=
Tt
du (2.r)
exp
20
(((x) da
Approximation of (2r 1). This step builds on [Hu et al., 2025c, Fu et al., 2024].
By the expression of Φ; (x.
(x = / 0² (27r)da/2
||4221 xx|
x((x1) dæ
exp
20
62

---

we approximate ((21) and exp
| 1+-,,=111 with kr-order Taylor polynomial and k2-order Taylor 2d
polynomial on bounded domain N2 introduced in the integral clipping (Lemma J.2). Altogether, we approximate i with the local polynomial Vi (z, 1) on B2,N with the expression:
R nz!
allalll
0 11
V (o,, = E
91 (z, Day U, (),
(I.4)
Ox
Sky
x= C=
where 2d is multi-index, Ra > 0 is constant depending on the Hölder ball radius B,
91 ne v,t) = II Exsep 92 nu [i], e/d), 12). and
g2(2[1], n, [1], v/1). = G.V2T
Z1 Ka
+
nz(d)
(- 1e2
2121111 2
12
dæy
Hu et al. [2025c]. Fu et al. [2024] consider the setting of conditional diffusion transformer with classifier-free guidance. contrast, we apply (J.4) by removing the condition y E R
Since v (z, t) is an approximator of & (z, t), we need to ensure that is lower bounded away from zero that the denominator of velocity (x) in Lemma J.I does not blow up.
Therefore, we introduce an additional definition.
Definition J.1 (Truncated Density Approximator). Let flow a positive real number, and let V1 (z, () be scalar-valued function defined in (J.4). Then, we define
(x f) == max (xr, (), flow
We specify the choice of clow in Lemma J.9. For now, we approximate & (2, with V (z t).
Lemma J.5 (Local Polynomial Approximation of 4 Lemma A.4 of [Fu et al., 2024]). Assume Assumption I.1. Let V (zx t) be the approximator it holds
(z, t). Then, for any E [0, 1] and x E Rd,,
(i(x, (x, $ BN (log N) "
Next, we approximate $2(21, t). Approximation of Φ2 t). By Lemma J.1, the following identity holds
|| - Me Fz 112
22(x, = Sea. of (2T)d./2 exp
g(x1) dri = x (x, (I,
(J.5)
20
Building upon the local polynomial i (x, t), we use x V I f) as the approximator of 22 t).
Next lemma gives the approximation error rate of P2(1, 1) using V2 1) = x 1 (xr
Lemma J.6 (Local Polynomial Approximation of (22). Assume Assumption I.1. Let 1z(2,, t) be the local polynomial and (z t) = z (z, t). Let Cz B, C1, C2) be a positive constant. Then, for any & E [0, 1] and x E C. Tog N, C. Vlg Njd, holds for all i E [dzz]
dutki
1 |i] 22 t)
BN # (log N)
Proof. Since V2(xr, t) = N (z, and (2,,t) = T (z. t), for all E [dz], it holds
V2[1 T22 = 1 x
< |x[|]|| | 21
(By (J.5))
< 2|i] BN (log N) datha
(By Lemma J.5)
63

---

< BN- (log N) dethea1
(By æ E (-Coylog N, C. M.y
This completes the proof.
Approximation (3(2, t). Similarly, we have approximation results for L2(2, t).
Lemma J.7 (Local Polynomial Approximation of 4b Lemma A.6 of [Fu et al., 2024]). Assume Assumption 1.1. Let (d 8, C1, C2) be positive constant. There exists local polynomial Vz(x, such that for all t 0, E [d.] and E [-Cx Vlog N, C² Vlog NJd, it holds
BN (log N) d=
33(2, t) or Vp. |+]|
Remark J.2. We clarify that Lemma J.7 gives the approximation of Φz () using V 1). First, the density at time t has the form:
(x(x) = J oi (2m)d./2 exp
|x - Mr x111²)
(((x)) dx
dz)
207
Then, the gradient of (x) with respect to x has the form:
Vp((x)
/ - (r
Rd
-
o,
#4 21
of de
(2/)d2/22 exp
|| p4 21 22112)
203
(((2)) dx1.
By Lemma J.1, we have Pz(2,)) = lo (p) Therefore,
(a O]] - P2 M|I S B N)
(By Lemma J.7)
Velocity Approximation with Transformers on Bounded Domain. We first approximate the veloc- ity approximator constructed with v (z, t), (2 t) and (2 t). We reiterate that transformers take input d L matrices, where x L dz. Then, the next lemma specifies the network configuration for the approximating the velocity approximator with arbitrarily small error.
Lemma J.8 (Approximate Velocity Approximator with Transformers). Assume Assumption I.1. Let (d 3, C1, C2) be positive constant. Further, let (2x, [-Czvlog N, Caylog Nj
[0, Rd= be the target function:
(a 1) jo, 1)/pz + (ò, iiro /1/ (z(x1))
(x(
Then, for any / E [o, 1] and any E (0, 1), there exist transformer g(x, 1) This such that
J Un
/og lg((,,t) (a ((xtt)l/drdt < 22
Furthermore, the parameter bounds in the transformer network class The satisfy
Cxq, C2 OOx~-((1og N) a -d-2 2); Cov, 22.0 O()): Cr, Sp,02.00 O1v O(Vloo N - max | 12); Cr = 0(1):
where -1 = (log N/e) 4d+3 is the inverse-temperattue scaling in the softmax function and O(-) hides all polynomial factors depending on d,, d, L, B, C₁ Cz.
Proof. Since the path coefficients are smooth and the first-step approximators V, (), V2(xr, 1), and V2(2, t) integrate polynomials, the target function is Lipschitz continuous compact domain.
64

---

Further, the reshape layer Definition B.4 does not harm the continuity of the element-wii b2- norm. This continuity ensures that the function satisfies the conditions for applying the universal approximation of transformers. Also, we concatenate as additional sequence. Then, we apply Theorem H.2 with 2 and Z E El-Cavlog N, C2 /log N)dx(L+1), For any E (0, 1), it holds
d2(a. n ( (1((,,)) - v(x,, ()ldddd)
1/2 S.
(By Theorem H.2)
The parameter bounds in the transformer network class follow Lemma H.5. This completes the proof.
Remark J.3. Lemma J.8 modifies Lemma 1.6 of [Hu et al., 2025c] by adapting the transformer approximation to decomposed velocity components (Lemma J.1), whereas their work focuses on approximating V (x). Our flow matching framework eliminates the label y and reduces the number of hidden dimensions to one.
Then, by analyzing the error accumulation from both the transformer approximation (Lemma J.8) and the local polynomial approximations (Lemma J.5, Lemma J.6, and Lemma J.7), we establish a bound on the velocity approximation error over a bounded domain.
Lemma J.9 (Velocity Approximation with Transformers on Bounded Domain). Assume Assump- tion I.1 and Assumption 1.3. Let to. T E (0, Let (3,3, C2) and C3 be two positive constants. Let foo C3N- 3 (log N) (d2 k Then, there exist transformer (z(x,t) t) Thh..." such that for all [-CvVlog N, Co Vlo Nid,, E [to, I'] and pe(x) 2 clow, holds
uo(z t) [. -ule(t))uo(t,
|ivel Me
je òi ) B²N 28 (log N) Sá. +hi+1 +1
(p) dzdt 2
+
Mr
ot
Furthermore, the transformer parameter bounds satisfy
CKQ, CQQ 0 -1 - (log N) t2);; Cov,
Cov,Cov Cbr O
Cr,C7 (N (log N) 1x +1); CE = O(T); Cr 0( log N).
where A 1 = O(N3 log NY4d+3 the inverse-temperatur scaling in the softmax function and (-))
hides all polynomial factors depending on d,, d, L, C1, C2.
Proof. We use the notation "S" our derivation when an inequality holds up to a constant factor. We prove Lemma J.9 with following two steps.
Step A: Approximate velocity with constructed function. We approximate the components P1 (zx t), (2 t), and (2(2,)) t) using local polynomials [1 t), 22 t), and (z(2,, t), respec- tively. Based on the velocity decomposition given in Lemma J.1, construct approximation V (xx f) by combining these polynomial components to approximate the full velocity field uy (r).
Step B: Approximate with Transformers. We leverage the universal approximation of transform- ers (Appendix H) to approximate the constructed function V. Based on this approximation, we derive the final velocity approximation rates with the required bounds model parameters.
By Lemma J.1, the velocity field un (xr) takes the form
12(2, t) + (or (
1u &d ( Mh
u (x) = (x,
#4
'Please Appendix H for detailed proof.
65

---

where
(x,, {
1
||-- 14 21 11
Jada of (2x) exp
q(x1)) dæ
20
(b2(x,1) -SSa o² (2T))a2²²
||0 M4 x11 11
exp
(g(21) dr1;
20
(2
/
1
21
of (2
exp
|| M4 x1 11
20
(z1)) da
Moreover, by Lemma J.4, the bound on the velocity field -distane follows
|/u(())looo
(J.6)
AAel
iie
oz
|1011100
+
(1|
l2 1)
Lemma J.4)
H4
14
AAel
ju
oz
24
Vlg
N
(v log N + 1).
(By x [-Oz N, C2 Coylog ())
F4
Pt
dt
Set the transformer network output bound GT equal to the right-hand side of the expression. Then we are now ready to present the proof of Lemma J.9.
Step A: Approximation via Local Polynomial.
We construct the approximator for ut (x) based on Lemma J.5, Lemma J.6, and Lemma J.7. Specifically, we define (x, t) E Rd with each element given by
(z, t) min { j [iVVao]/m (òr i4,, p
(J.7)
Vi
where U the upper-bound of the ground truth velocity un (x) under the sub-Gaussian assumption (Assumption 1.1) and
Nill Vlog N + C3
(Vlog N + 1). ju 8z He or
U =
Pe
(By Lemma J.4 and æ
N. C. vlog N)
Notice that, for all i E [dz], the difference between (xx [)] and un (x) follows
1 (x) [i] (xx +||l
22 [1]1111 He + (ā, ino, Me Poli
bc42200/p + (o. - ino Vs[l]
+
(By the definition of u, and (. t))
Mi 22/2200/pu Mr + (ō, on 2ald
2 μr + (ā. io., (00e/11)000 Va
DC
(Ti)
ju D2(il//1 + (ō, - pu (ee P
irc t2 [t]/ Me + (ōr ino (
1
vrc
(Ta)
(By triangle inequality)
Next, bound (Tr) and (T2).
66

---

Step A.1: Bound term (T₁). Recall Definition J.1. By the definition of clow we set
(i, max (i((,, C3 N (log N) dx k1
2
By Lemma J.5, we have
(I( -p(())) BN (log N) dath1
(J.8)
and (I.8) implies
(r(x) - KBN- (log N)
12 t),
for some positive constant K. Next, recall that we consider
CaN (log N) toth
pp((.).
By setting C3 = 2K B, holds
KBN (log N) dethy
C3 N-8 (log.N (p((1)/2.
dxthy
2
leading to
doths
Pr(x) px()//2 (x(2) KBN [) S (x,
As result, (x))/2 v < Vi holds.
This allows us to replace the approximator Vi with (z) by dropping constant 1/2. Then,
(T1)
(J.9)
ju De [i] 14 + (ō,
/ (40 Psli
iu V2/0// de + (ò. i
4/
2 it
[22
Walil)
+
(Pa(i)
Vslil)
(By vi p2
Pt P4
Mt
2 |ial
2
iu0
[1]
22
|
Va(
(y triangle incquality)
Pt
1t
Pt.
1
Pi
dxthi+1
6
BNN
(log N)
(By Lemma J.6 and Lemma J.7)
Pt
1
[pg
δt
BN -lo- N) dCtti-
(By or E [0,
Pt
[4g
Pr.
BN -N /
(log N)
de
11
(By Assumption 1.2)
Pt
Next, we bound (T2).
Step A.2: Bound term (T2). By Lemma J.4 and || & Vlog N. it holds
Iu, (x)
lite
ō
jt
V/ogg1 N
+
(V (Vlog
N + 1)
14
0t
1e
livel
itt
))
14z
+
Vlog N
0e
67

---

for all i [dz]. Next, by the decomposition of velocity in Lemma J.1, it holds
B4 22 +
02 j
1ite
āt
jut
j4
1
+
log N
+1
f
o
0x
14t
(J.10)
Therefore,
(T2)
(I.11)
Me 22[
1
+
23
14
Ft
V
2
1
litel
j4
llg N
(By (J.10)
14
0z
10
14t
4
V
((
libel
+
d.
14
Vlog
N
17
|
1
1
11
(By factoring out 1/41 1//1)
1
|Aee
log N + i4
+
|
vil
(By 1 > De (2(
Pt
[4
0t
00
14
1
[Aee
+
Vogg
N
+
BN (log.N N)
d=tte
[4g
14
By Lemma J.5)
Combining (J.9) and (J.11), we have
|ur
[[]]
(J.12)
(Tı): Pr
+ (T2) Pt
Ael
ju te )
9 #(log N) de 4h141
(By (I.9) and (I.11))
He
Hr
oe
for all i [dz]. Therefore,
pi [1)((x) (xr, 1)12
(J.13)
S p de dellu(()) (z, 1)
(By || ||2 S dall lloo)
$
jig
+
ju
δe 2
B² N (log tydithit1
(By (I.12))
#4
σe
Step B: Approximation with Transformer.
By Lemma J.8, there exists a transformer wo(x, 1) E Th.na such that
/ |uee(z (x d)l s ?
(J.14)
By setting == N-9, it holds
/ pi || nz( 1)
< $ 211le(()) (xr, 12dddd + / De (1,(),)) uo( d)
(By triangle inequality)
68

---

(2, () 12ddd / [1((,,)) us u2 (2x ()lld (By0 s 1)
pt
[lux(x))
14
σz
B N - 23 (log N) N) / dodl + / [V((2,)) (((1,,))uush us(o, 1) [2ddodl
+
14
14
(By (J.13))
|pte
+
σz
B² N
23 22 (log N) 8dz k1 +
(x, () uz 1)
14t
1tt
0+
(By 1alloo S C2V N and t [0,1])
it
ōz
(log N) 2 -1+11 ) B²N (1og
+
(By (I.14) and N=
Pt
He
By (J.6) and x E [-Cx Vog N, Co Vog N de we have
U U- O(Vlog N).
and by (I.12) we have
1e = ONN N) dottitt
This implies
(xr, t)l1 = O0 N + N- N) detheti
We take looser bound on v (z, t) such that it holds for all dg:
[z, t)ll S d (z, le O(log N) 1e
Then, the parameter bounds follow Lemma J.8 with N-8
This completes the proof.
J.3 Main Proof of Theorem 1.1
We establish the velocity approximation with transformers in Lemma J.9. However, it is valid under two settings: (i) the bounded domain E 1-Czvlog N, C2 Vog Njd= with some constant (3, C2) (ii) the mild and high density region (x(x) flow- To obtain general approximation results, we introduce two additional lemmas to tackle the uncontrolled region.
Lemma J.10 (Truncation of x. Modified from Lemma A.I of [Fu et al., 2024]). Assume Assump- tion I.I. Then, for any R4 > 1, t > O, the following hold
S (r(x)dr S R 2 exp
C2R2
2(p + CCoof))
C2
J [ur(2)) (x))dr Rt exp
2( + C2a
Proof. For the first inequality, it follows
/ || a R4
(e(x)
(- C2 |0:
exp
dx
(By Lemma J.3)
2(m
C20?)
69

---

C211
exp
dx
(By ||rl| 2 ||all
2( + C20
1
C2
S 22 exp
(y Lemma D.2 and dropping constant terms)
2( C20
For the second inequality, it follows
|| (x) (e()dd
|1 2n14
C2 C211211
[lur(x)]12 exp
dr
By Lemma J.3)
2 + C2a?)
|1 R4
litel
1| +
dz
(112112 + +1)
exp
C211=111
dz
|| RR
Me
of
2(p C20
(By Lemma J.4)
C2110118
110:12 exp
dx
(By Assumption 1.2)
2( + C207)
|1 284
/
C2 C21121112
||a1|[exp 11
exp
da
(By ||a||| 2 ||a]|lo))
2(1 + C2a7)
C2R3
2 R exp
(By Lemma D.2)
2(p C20
This completes the proof.
Lemma J.11 (Bound Low-Density Region, Modified from Lemma A.2 of [Fu et al., 2024]). Assume Assumption I.1. Then, for any Rs, Clow > 0, the following two inequalities hold
J 1(pp(x)) <low> (r(x) I R
Elow
J ((pr((x) < <low) 11,((=))12 pe(x)do S Rd,+2 flow-
Proof. The proof for the first inequality is identical to [Fu et al., 2024].
For the second inequality, it follows,
/ SR H(pp(()) < <low> - pe(x)dx
2 / Iellosss
[itel f4
[ie ōt) dt|_
(11=11 =))² p((x)drr
1(lp((x)) < dow3
|| +
Me orl
By Lemma J.4)
< flow / S
litel
jix
ā:
([ær12 + 1)) dr
||o||lo
+
Me
< Rde+2 Clow-
(By Assumption 1,2)
This completes the proof.
Next, we present the formal proof of Theorem I.1.
Theorem J.1 (Theorem I.1 Restated: Velocity Approximation with Transformers under Generic Hölder Smoothness). Assume Assumption 1.1 and Assumption I.2. For any precision parameter 0 < e < and smoothness parameter > 0, let e < ONN~)) for some N. Then, for all
70

---

t (tu T] with to. T (0, 1). there exists a transformer uo(x, t) This such that
T J//
00 Rd= J' [lu((t)) u0(z, 1)1 (r()drrd O (B2N~ (log N)da
Furthermore, the parameter bounds in transformer network This. satisfy
CKQ. C OX (log + Cov, Cov,Cor ONN C2,C7 = O (N (log N) +1: CE = 0 Cf = 0 (lo N).
where -- O(N9 log N) y is the inverse-temperauure scaling in the softmax function and (-) hides all polynomial factors depending on d2, d, I., 3, C₁,C2
Proof of Theorem 1.1. Let Rg C2 Czvlog N and C2 v48(Hz C2al)/C2. Further, have
CT O0Vlog Élow C3N (log N) (dxtks)/2
(By Lemma J.9)
First, we decompose the target into three components and bound each them
/ |lue url13 -P(())dddt
to
1lu0 - ue (1ll f- I IAlIISS 1 (()((a) < <low <low Hwll vell26(x))da
|1 Ro
(T1)
(T2)
T /
+
(xp((x)= 2 flow Hlue - uo))pe())dxctt.
30
SRo
(T's
Bound on (Tz). It holds
f
llue - uel22 pe (a))dr
to RR
|nol12 px(x) dx + 2 f 11=1l0>>10 =16
|lueli (e()ddx
(By expanding f2-norm)
|| >Ro
2dg
| (x) + 2
1lull (r(x)d
(By 11-112 s d, 11
|
11
/
log px(x)dr+
1uel2 (x(ddd..
|a Ro
(T1.1)
(T1.2
(By CT = O(vlog N) from Lemma J.9)
We bound (TI.) by
(Tr.1)
" | Ro
=lg N
Pe(x)dz
C2R
S log N: Rg. 1dd--2 exp
(By Lemma J.10)
2(2 + C207)
71

---

x log N. (log N) N Nー
(By the choice of Covlog and C. /20(1 C207)/C2)
We bound (Ti.2) by
(T).2)
l pc(x)) dr
||=||=
> Rg
114e
2 R exp
C2
(By Lemma J.10)
2(0 + C207)
Hit
lite
dz
2
(log N) 2 N
P4
0y
(By the choice of Rs Cr v log N and Cr = /2 + C20f)/C3)
Therefore,
[ell
ie
oe ))
8e -
) .llgg N) " N
(T.)) S
+
Pt
Bound on (T2). It holds
1 1 S1o
(pp(x) <low) llue - wel12 pe (e())d
(pp( < [ow) (1lu0l12 + [lux pe()dd
(By expanding {2-norm)
SRo
2
1(p((z) < clow (dz |luel + 11 pe (x)
(By || |1 < dzll |1
SRa
1iv < (ow) ||uall3 Pe(x)dx + /
2
1 (P: < [low 1|u.112 Pe dx.
SR
(T2.1)
(T2.2)
We bound (T2.) by
(T2.1)
=1 |1l1oos So
(x) < (low) [1)0(x)113 (r(d)dr
I log.N. N.
(p((zz) < Cow (x
(By CT = = N) from Lemma J.9)
SRo
log N Clow Rd.
(B Lemma J.11)
log N (log N) +
(By the choice of Rs C. and C. = /28(kz + C2ot)/Co)
flow
x log N N) 4 N (log N) tr
(By the choice of stow N (log N) dotty
We bound (T2.2 by
(T2.2)
-/ - SRo
fpp(xx)
(low) -1luelB pc (x)dx
72

---

Clow R2=+²
By Lemma J.11)
E 9ow (log N) 2x²²
(By the choice of Rg Ca Vogy N and C2 /2 + C2ai)/Co)
S N (log N)
(log N) N) dut2
(By the choice of clow C&N -P (log N) dmtthe
Therefore,
(T2) S (T21) + (T22) S ~ Ny
Bound on (Ta). We bound term (Tg) by
(T3)
T
1(pe(x) [ow) |l10 -uell: px(x)rrdt
| SRg
- [(p(()) 2 Elow dzllus u. 11 (dr)
(By multiplying pt /P:)
|| SRo Pt
1 1 (p((2)) 2 [ow) - ddeluu - uell3 (p.(p)())) drddt
(By 1/p S 1/Flow)
1/pc
/|=1|>0 SRo Clow
dn
juc
)² N- (log N) 1 +h
+
By Lemma J.9)
Clow
14
0t
livel +
i
òi )² N22 23 (log N) 24 +h++1
I N flog N)
11
14
ot
(By the choice of flow)
| +
jtt
δz
2 )²²
BB2N
(log N) +
=
H4
By the upper-bound on (Tı), (T2) and (Ta), we have
Iue uo (x, t)ll3
pt drdt
$
(T₂)
(T2) + (Ts)
lo (0,
[i4e
14
2
0 122 8. -(log N))++
+
14
|it
+
i4
0 (B'N- (log N)
(By hs S B)
M4
0
B2 N-..
(log N f1))++++1)
(By Assumption 1.2)
Furthermore, the transformer parameter bounds follow Lemma J.9.
This completes the proof.
73

---

K Proof of Theorem 1.2
In this section, we derives tighter error bound for velocity approximation using transformers.
Organizations. Appendix K.1 introduces auxiliary lemmas. Appendix K.2 establishes a bound on the velocity approximation error over a bounded domain by applying the universal approximation of transformers. Appendix K.3 presents the main proof by incorporating the bounded-domain approximation error and controlling the unbounded region using the sub-Gaussian assumption.
K.1 Auxiliary Lemmas
In this section, we introduce auxiliary lemmas for velocity field approximation. Specifically, Lemma K.I applies a stronger Hölder assumption to decompose the density function (r()). Lemma K.2 further decomposes the velocity into two components, differing from the decomposition under generic Hölder assumption. Then, Lemma K.3 and Lemma K.4 establish upper and lower bounds for the decomposed components and the velocity in eo -distannce respectively.
We begin with the density function decomposition.
Lemma K.1 (Density Function Decomposition, Lemma B.1 of [Fu et al., 2024]). Assume Assump- tion I.3. Then, the density function ((()) and V (o)))(>)) follow the decomposition:
C2111111
pe(x) (Mî + C2 o2
exp
2(4 + C2o,) ((,,t),
-C2x Vh(o,t)
V (logp(()) 11 + +2o?
1(x,1))
(2 (- (-1mi-neanli] dx,, ōe == (u?+C20tyUs and Tu = (uZ+C207)
where 1(x, 1) =
f(x1) exp
pr
20.
Then, give the velocity field decomposition.
Lemma K.2 (Velocity Decomposition under Stronger Hölder Smoothness Assumption). Assumption I.3. Then, the velocity field u. (z) follows the decomposition:
Assume
ür (b,0, ft
( -C2x Pt (1) Czo? h(z,t)
+ Vh(z,1)
u4 (z)
Remark K.1. The key aspect of Lemma K.2 is the velocity field uz(x) having denominator bounded away from zero. Specifically, we apply f(x1) 2 C' to derive the lower bound on h(z, t) (Lemma K.3). This removes the need to impose additional lower threshold on the density function approximator. In contrast, under Assumption 1,1, the approximator is constrained to stay above the threshold Clow prevent explosion, and therefore leads slower approximation rate.
Proof. Our proof builds on [Fu et al., 2024].
By Lemma J.I, the velocity field u, (xr) has the form
E2 t) + (ō. - Mor )Z t))
(x(x)= = + 1 t)
where
(x, = [
|| - 14 x 1|
Juto 0fa 12 exp
q(()) da
20
|| P4 F
02(x,t) t) = x
gds 0² (2
exp
20
(((z)) dr1,
74

---

23(x, /
I ue 21
|| Me 2111²)
R2z
ot
ds (2 exp
20
(((z)) dx
Furthermore, we have
ot Ve)((
/
x M4 21
|| H4 x11l
of
0f2² (2 d
exp
20
q( dæ
/
Me #1
x Me x1 2111
exp
((xi) dæ
Ot
- (2 dda/2 (2n)da/12
20
03
(2, 1).
Therefore,
vx(xx)
in 1 )rbz)
$2 + (ō:
14
14
ju
)o V log
(By 92 200, @3 5r Vpc))
Pt
ju x (0,0,
i107)
C2x
Vh(z,t)
(By Lemma K.1)
Pt
Hr
11
+ C2of
h(z,t)
This completes the proof.
The next lemma bounds h(r, t).
Lemma K.3 (Lemma B.8 of [Fu et al., 2024]).
Assume Assumption I.3. Then, it holds
sV 2
C h(,, B,
Vh
S
Lemma K.3 ensures that ((,,, 1) remains bounded above and below by a constant. As a result, we (x) stays finite for all x. This eliminates the need for additional threshold étow (Definition J.1) in the constructed approximator to prevent divergence, leading to a faster approximation rate.
Bound on Velocity Field. We give the foo -bund on (z) under stronger Hölder assumption.
Lemma K.4 (Bounds Velocity Field). Assume Assumption I.3. Then, there exist positive constant Cg such that
[l))()) S P + (ō
no0 fr Pi
C2 C2o?
|101 Co i.
(
+
Proof. Recalling from Lemma K.2 and Lemma K.3, the velocity field has the expression
ür fr (āro, in0
-C2x Czo?
Vh(z,() () h(,, t)
ue (x)
+
14:
1t
m
where o; = -0/ 1 + C2o7, De 1e/(1 + C20,) and
1(x,1) / [(x1) (2T) 22 0d.
| eo 2112
exp
dr
27.
75

---

By Lemma K.3 and Assumption 1.2, it holds
v)(x, h(x,t)
1l I Th
V= BCi =0
By Lemma K.3)
Te
06
Therefore,
[1, (1)
At + (ōr0t
n.o ) C2
)
Vh(x, ()
110:100 +
He
Mi + C20?
h(r, t)
(By triangle inequality)
jt (ōrgz
nio Ht 11 + C2o?
C2
+
"
||<0||0 Co
(By (By Lemma K.3)
Me
for some positive constant Cg.
This completes the proof.
K.2 Velocity Approximation on Bounded Domain
In this section, we approximate the velocity field uy (x))) using transformers in two steps. The first step constructs two compactly supported continuous functions, @1( t) and Q2 t). as approximations of h(x, t) and Vh(z, (Lemma K.5 and Lemma K.6). The second step applies the universal approximation of transformers to approximate (z, t) and Q2(z, t) (Lemma K.7). Combining these steps, we present the velocity approximation on bounded domain Lemma K.8.
Before proceeding, we reiterate the expression decomposed velocity under Assumption I.3.
Ar 1t:
-C2x
+ Vh(a,t) t) Mxxt))
We (x)
(āco,
C2o7
14t
Then, we construct two local polynomials as the approximators for hax, t), and Vh(x, t). Approximation of fxx, t) and Vh(I, t). The differences between
1
||21 - The x x11
1(2, t) = f(x1)) (2r)da/2 .2
ad, exp
dæ
20,
and
pe(x) ((xx)) (2 d2a) 12
exp
|||| Me
x
dx
of
20:
lie in (i) the target function f(x and g(x1) (ii) the path coefficients G. Pe and oe, Pr- We define local polynomial V (2r, 1) the approximator for pe(x) in (J.4), Given the differences between h and pt. the construction of an approximator for h(x, t) follows the formulation of V Formally, we approximate h(æ, t) around with:
Il lla gms
Q1(x,))= Σ Σ
T1 (x. nz, f),
(K.1)
nz!
Dr
vE[N]de [|,, Sk
u==
where n, Zdz is multi-index, Ra 0 constant depending on the Hölder ball radius B,
91 ne v, = 1 Exse 02(2 n. [1] v/8], Az). and
76

---

n=
[1]
A2
g2(2[i], nz olil, =
21 Rg
dr1.
R2
20r
Remark K.2. Given the differences between h(r, and (t(2),, replace (i) 8m +1 /dx" with @== f/x=/0x== (ii) oN and lt with o; and īhe respectively. Then, the formulation of Q1 (I. t) follows constructions identical the density function approximator 1 (x,,t)
Remark K.3. When the context is clear, refer Qi (2, () local polynomial and distinguish it from 1 (r, The generic Hölder assumption (Assumption I.I) applies to V (2, t), while the stronger Hölder assumption (Assumption I.3) applies to (x,
Then, we approximate h(z, using (z,
Lemma K.5 (Approximate of h(x, t), Lemma B.4 of [Fu et al., 2024]). Assume Assumption I.3.
Let Q (x, 1) be the approximator of h(,, (), and Cr B, C1,C2) positive constant. Then, for
any E [0, 1] and [-Cavlog N, C2 /log NJd, it holds
1Q - h(x, t) $ BN (log N) N)
Based on the approximation of huxr, L) using local polynomial (1 t), we construct a approximator of Vh(x, t) following similar formulation
Definition K.1 (Approximator of Vh(a, t)). We define Q2(x, t) as the approximator Vh(x, t), with each component Q2[i] following the form of local polynomial presented in (K.1).
Then, we approximate i (x: t) and Vh(r,t) t) with Q2(x, t).
Lemma K.6 (Approximate (z(z, t). Lemma B.6 of [Fu al., 2024]). Assume Assumption 1.3. Let (d2, S, C1, C2) be positive constant. Then, for all [-C, Vlg N, C. Vlog N [dz]
and > 0, holds
ã. Tto
h++1
Q2(2, Vh(z, t) z BN (log.N)
Approximate Velocity Approximator with Transformers Before deriving the velocity approx- imation with transformers on bounded domain, we first approximate the velocity approximator constructed with Qı(z, t) and Q2(z, t) using transformers.
Lemma K.7 (Approximate Velocity Approximators with Transformers Network). Assume sumption L.3. Let C2 be positive constant dependent on dz, B, C1 and C2. Then, for any C-Covvlog N, Czvlog and t [0, 1), there exist transformer T Thii. such that,
[. [T((,,))
ft x (ō.a, - ino? (--22r be
-Oz
The VQ2li) [Bdrde se?
14 Czo7
6,Q1
Further, the parameter bounds in the transformer network class follows Lemma J.8.
Proof. The proof closely follows Lemma J.8.
Approximate Velocity with Transformers on Bounded Domain. We incorporate the approxima- tions with Q1, Q2 and T T(x, t) to derive the velocity approximation on bounded domain.
Lemma K.8 (Velocity Approximation with Transformers Bounded Domain). Assume Assump- tion I.3, Then, for any & [-Ox Vlog Cavlog djd= and E [fo, 2] with positive constant C2 (dz, 8, Cr, C2) and lo, T (0, 1), there exist a uo uz( 1) E TTh..s such that
1." Jime /
[x(x)= - (2 (x) S B2N 28 "log (o fttd
SCo
77

---

Further, the parameter bounds in the transformer network class follows Lemma J.9.
Proof. Building upon [Hu al., 2025c, Fu et al., 2024], we prove Lemma K.8 with two steps.
Approximate velocity with constructed function. We approximate the decomposed velocity field (Lemma K.2) and components with approximator Qi (zz t) and Q2(2, t).
Step 1:
Step 2: Approximate with transformers. We apply the universal approximation of transformers presented in Appendix H to approximate the constructed function in Step 1.
Before proceeding, we recall some previous lemmas to prepare our proof.
By Lemma K.2, the velocity follows the decomposition under Assumption I.3;
ux(x)= f -broo, (u02) (
C2x O2o?
Vh(x, t) h(z,t)
+
fr
1t
where o; = 01/(n/ C20 Tio = Ne/(1/ + C20 and
M(x,) = / ((x1) (2T) 0/
[1 pr x112 dr
exp
26.
Furthermore, by Lemma K.4, the bound on ux(x) in loo -distance follows
pi )
C2
)
|| + -o ú0
[1))( s
+ (ō.0, Pt + C2o?
Hr
First, we apply ||>||2 I Vlg to Lemma K.4. Next, we apply Lemma K.4 and Lemma K.6 to construct the first-step approximator Q(x,t) Rd=, with each element defined by:
ju x (0,0r 1e
Do0 14
-C2x
Th. VQ2ll)))
| min
(K.2)
1 C207
0,Q1
The first element consists of approximators for h(x, () and Vh(x, (). The second element ensures that (z, 1) does not output value larger than the maximum of un (x) in lxo
Step A: Approximation via Local Polynomial.
By symmetry, for all E [dz]. the difference between Q( [i] and (x) follows
1 [0]
Q[i]
Vh(i [1
(ō.0,
14g
h
0,Q1
Vh[i
Q212
0,0+
1t
T
o
Vh(1
Vh[i]
!or?
Vh[i
P.Q2
+
μ
h
Q1
Q1
の.Q1
(Tτ)
(Ta)
(B triangle inequality)
Next, we bound (Tr) and (T2).
78

---

Step A.1: Bound (T1). By Lemma K.3, we have C1 s h S
B and
v
2
S -
14
Moreover, by Lemma K.5, it holds
1Q((x)1)-(a)) - BN- (log N)
It implies that for some positive constant K'. This gives
M(z,t) K'BN (ogg)** (I(,t)).
1Qi(z, BN BN
(K.3)
Therefore,
Vh[i
V/jij
(T))
h
Q
Q1
hQ1
PL
Q1
B
(By Lemma K.3)
hQ1
N
9
(Log
N)
Step 1.2: Bound (T2). It holds
Vhli
22
(T2)
Q1
0iQ1
Q2
2 Vh
74
By factoring out Pa/ p..Gat)
Q1
B
A (log N) +1
<
N
(By (K.3) and Lemma K.6)
at
Combining bounds on (T₃) and (T2), holds
m [i]
Q(1]
(K.4)
?or?
0,0,
((T,) + (T2)
14e
BN (0--8(1pp)
Ploog N)ggN)" N) tt4'
14
for all [dz].
Therefore, by symmetry, it holds
[[u((x)) - ((x 1)11
(K.5)
(z))))()) - Q(x, 13
(By 1ol1-lle 5 dall 1loe)
79

---

$
0,0
B (log N
23
(By (K.4)
Ne
Step B: Approximation with Transformers.
By Lemma K.7, there exists a transformer us(x,)) t) E Thi such that
iu x (010, / [Iuo(z;t)
i40i )( C2of
-C2x
Pe [2
+
Ibgddt S c.
H 1i
0,Q1
By setting = N- the velocity approximation using transformers follows
ʃ (ellu((r) - us(2 12dddt
$ / pellue(z) - Qxx. t) dxdd + / pa (I((,t us t)l
(By triangle incquality)
< / [1(((x) - Q(x, 1) + / [1Q(z,t) - us(a, d)ldd
pi(x) - 1)
B (log (log.N NM / doat+ / [1Q(z,t) u0( 112ddtt
1/ 2A
110
(By (K.5))
?
N2N-20 23 (log N) Nhtttes (1Q(2,)) - uo(x dll
0111
(By t (0, 1) and || S Ca Cavlog N)
23
110
B²N 23 (log Nytttdd
(By Lemma K.7)
x
B 29 (log td.
(By Assumption 1.2)
By Lemma K.4, it holds
lu (x)
jt (ō,0,-
io07
)(
C2 11 + C2o?
1|01100 + Co
(By Lemma K.4)
Mr
Me
3 0 OVolog
(By Assumption 1.2)
Therefore, we set transformer output bound Cy == (1)) Then, the parameter bounds in the transformer network follow Lemma J.9.
This completes the proof.
K.3 Main Proof of Theorem 1.2
In Lemma J.9, give the velocity field approximation using transformer on bounded domain E [-Caylog N, C. /lo N]d, under stronger Hölder assumption. To obtain general approximation result, we introduce the next lemma that bounds the uncontrolled region.
Lemma K.9 (Truncation of Modified from Lemma B.2 of [Fu al., 2024]). Assume Assump- tion I.3, Then, for any R7 > and t > 0, the following hold
J |1 2R7
C2R2
pr(x)dx $ R4= exp
2( + C2a7)
80

---

J ||e||>> 2R 2R;
C2R
[ p((2)dx R exp 2(p + C20
Proof. The first part of the proof is identical to Lemma J.10 Recall Lemma J.3. The density function at time é is upper bounded by
C1
C211=11 2(14 C207)
Pt S (ng + C207)d./22 exp
(By dropping constant term)
C211211
2 exp (- 2(p C207)
Furthermore, by Lemma K.4 we have
( (x)
(K.6)
f4 (ōror
1407 C2
||00|100 + Co
Me
11
+ C2
14
||
(By Assumption 1.2)
||1||2-
(By | Iloo S 1| lb)
Therefore, the second inequality follows
[Ipe(=)(3p<<<)dd (x)de
d
[un(=r)l33PP (x)dr
(By || 1-11a s doll Iloo)
217
1 (x))d
(By (K.6)
2R
Cz |0:1 dx
1 exp
(By Lemma J.3)
2( + C207)
||
C211=113
||0112 exp
2nu + C20 dr
(By || 2 ||=||oo)
|1
C2R2 2(0} + C20
R exp
BB Lemma D.2)
This completes the proof.
Next, we present the main proof of Theorem I.2
Theorem K.1 (Theorem 1.2 Restated: Velocity Approximation with Transformers under Stronger
Hölder Smoothness). Assume Assumption 1.3 and Assumption I.2. For any precision parameter O < < 1 and smoothness parameter 3 > 0, let < O(N- for some N N. Then, for all t [tu, T] with to. T (0, 1), there exists transformer uo( t) Thh... such that
[1x(()) [=(x,t)13 (x(ddd = 0 N 28 (log ( 12
N)).+
Further, the parameter bounds in the transformer network class follows Theorem I.1.
81

---

Proof of Theorem 1.2. Recall Lemma K.8, Lemma K.9. We have Cy OVvlog N) and we set
43(14 + C2o log log
Rg
(K.7)
C
Then, holds
[ /
[1u0(x)) u4 (2)))
I [|u((x) (2) (x) L. I Mell [|u((x) - ve(x)) (2p(
SRs
T
/
(110(2))12 (z))) p) (r(rddadl+ f S [1ua(<) u. 12p (x)drd
11
(By expanding || l8)
1 111low
(log N + [[(((1)1B) pe(x)dodt + J'J SR [1u((x) ue (1)) (x)
(By CT = OVVoo N))
R2 2(12 + C20f)
+ T
2 (log.N N. R/.-2 Rgy exp
+ Sto I
[u0(x)) us (2))
(By Lemma K.9)
$ (log N) de N 28 + 1-
[|u((x) - (,(=)
(By (K.7)
to SRll
< (log N) - N 23 + B²N N 23 (log Nytt
(By Lemma K.8)
(B 23 (log N) tdd)
(By S B)
Furthermore, the parameter bounds in transformer network follow Lemma K.8.
This completes the proof.
82

---

L Proof of Theorem 1.3
In this section, we prove Theorem I.3 following the three steps presented in the proof sketch.
Organizations. Appendix L.1 provides fundamental definitions of flow matching and discusses key properties of the flow matching loss. Appendix L.2 introduces several auxiliary lemmas that support our proof. Finally, Appendix L.3 presents the main proof of Theorem I.3.
L.1 Preliminaries
In this section, we consider affine conditional flows (1)())(1) μ,z1 + o follows Section Given velocity approximator ug we aim to bound the following flow matching risk R(ug):
T R(uo)= {'" T
E X.rps
(X un (X))) di,
(L.1)
where marginal probability path pt and marginal velocity field un are induced by affine conditional flow dx(x)[{)) = 14_11 + oyz follows (2.2), (2.3), (2.5) and (2.6).
In practice,
conditional flow matching loss to train velocity estimator ug:
Definition L.1 (Conditional Flow Matching). Let q be the ground truth distribution and the normal distribution N0, 1) be the source distribution Considering affine conditional flows (arlar) = 14X1 + ogs, we define the loss function and the conditional flow matching loss:
E Xo (0.1) {(z; = T to La
uooe + o.Xo.t) (pe + oxXo) 12)dt
(CFM(UE) = T lo to X1 X1~0,Xo~N(011)
Ill (1ll + to, (Ji.Xi + or do)
Remark L.1. Holderrieth al. [2025] prove that the gradients of the flow matching loss (risk) and the conditional flow matching loss coincide. Therefore, minimizing the flow matching loss (risk) Ru,)) is equivalent to minimizing the conditional flow matching loss (ug).
To better evaluate the estimator ug, now introduce the empirical flow matching risk R(ua).
Definition I.2 (Empirical Risk). Consider velocity estimator uo E Th.ssrr and i.i.d training samples { } the empirical conditional flow matching loss (crM(us) := Em= ((x; uo). Let u* 2 be the ground truth velocity field, define empirical flow matching risk:
R(us): = ECFM((uO) EoFM(u*)=
(xos; us)
1 Σ (xxz"").
1=1
Remark L.2. Notice that R(u*) ) = O since u² is the ground truth velocity field. Furthermore, the fact that the gradients of the flow matching loss (risk) and the conditional flow matching loss coincide implies that R(ug)= R(ua) - R(u*) = (ug) LCFM("*),
Remark L.3 (Unbiased Property). We use ECEM and R' to denote the conditional flow matching loss and empirical risk with training samples In Then for any velocity estimator ug, the i.i.d. assumption implics that EE [z [Cerm (uæ)] = LCFM (u)), leading to E(r R (up)] ) = R(ug).
Next, introduce the truncated version of (i) loss function {(x; uo). (ii) conditional flow matching loss LCFM(Ug) (iii) the conditional flow matching risk, R(up) (iv) the empirical risk R(up).
83

---

Definition L.3 (Domain Truncation Loss and Risk). Given uo), LCFM (uo). R(uo) and
R(uo), we define their truncated counterparts bounded domain D == [-D, Dj by
etrnne (z; up) = [(; up
1 |0||00 D}, CCPM [(ue)[ { D).
Rtrune (ua) = R(;; up 1 |2 S D}. Rtrunc (ux) = (((u)) 2 D),
where 0 a constant.
With Definition L.3, we refer to gtrune (2; uo). CFM (uo), (trun((uo) and Rtrune (ug) as truncated loss, truncated CFM loss, truncated risk and truncated empirical risk respectively.
L.2 Auxiliary lemmas
Since the target distribution ((x)) is unknown, direct computation of the risk is infeasible. Therefore, we first decompose the estimation eπror into four components and present supporting lemmas to bound each of them. Then, we incorporate these results in the main proof in Appendix L.3.
Estimation Error Decomposition. Let ügg be the optimizer of the empirical conditional flow matching loss CCFM(U)) (ug) using i.i.d samples {-:] Next, we introduce a different set of i.i samples {</}== independent of the training sample {ar J Then, we decompose (n,) [R (R(u))]:
(R(Ta))
R'(TT)
R trunc (Up)
(m,) (m
[x
E
E
R'
trunc trunc(uo)
R trunc (uo)
(1r Pimi
(1))
E (mt)
Rtrune (Ug) R((u))
E {x))
R(
(L.2)
(III)
(IV)
We refer to terms (T) and (IIT) as truncation error, and we control these errors by leveraging the sub-Gaussian assumption in Lemma L.1. Then, derive the generalization bound control term (II) using covering number in Lemma L.5 and Lemma L.6. Finally, we apply the approximation error using transformers bound term (IV) in Lemma L.8.
Truncation Error. We apply the Lub-Gaussian assumption to bound the truncation error.
Lemma L.1 (Upper Bound on the Truncation Error). Assume Assumption L.1. Let lo. T E (0, and ug (z, t) be the velocity approximators in Theorem I.I and Theorem I.2. Then, it holds
B (x; ug) - gtruno (z; uo) ] S Dd exp - - C2D2) log N for any t ee [to T].
Proof. Our proof builds Section D.2 of [Fu et al., 2024]. By Theorem I.I and Theorem I.2, the transformers output bound Cf O(Vlog N). Then, for all approximator uy Theer it holds
E| ((; up) gtrune (x; ]
= E [(x; up [[]x]] 2 D]I
By Definition L.3)
S' T
T Jto
||x||/ 1 N(o.)
E [l|uo (jier + de +1
By Definition L.
E
[| 1 || + dyzo (2
(By expanding the {--norm)
T 3o
((0,))
84

---

T to S
D2 D (0,)) I 2 + |1 + 6xz0 d220 exp
C2112112 dædt
(By Assumption 1.1)
log N + | + demoll2) exp ( C2 C2112r112 drdt
E
T to 20 || FO
(By CT O(vlog N))
1.0 T
1
(log N + + exp C2 dadl
(By N(O,
T to /1/1/22
Dd= eexp( (xp(=)(2D²)
D expleepppp exp((3/2203) /r (log N ood)dt+
f j
T -
T- to
to
(By Lemma D.2)
Ddr exp -2DD2)
expl
log N.
By Assumption 1.2)
This completes the proof.
Covering Number of Loss Function Class with Transformer Estimator, Recall (II) in (L.2):
ITT)
E Fime
E {x
R Retrune (To) ( - (
To derive an upper bound on (II,, we introduce (i) the covering number technique in Lemma L.5 and Lemma L.6 (ii the generalization error bound to bound in Lemma L.8.
We begin with the definition of covering number.
Definition L.4 (Covering Number). Let Ω be compact domain and {r,}," be data drawn from distribution P. Denote the joint distribution {-:)=_ ~ Pm == P @ P. Given a function class F, a / Ω norm || and c 0, the fe-covering number N(e, F, [ ) x 12 []) is the smallest size of a collection {f, 1 C F such that for any f E F, there exists a j E [N] satisfying
sup max [[ t) fa f)ll S.
(L.3)
Also, we define the covering number with respect the data distribution P and size
(rah Ω -1). Further, for Q = 0. we denote the covering number by Nie, F,
N(c,F, x P" -1
sup (PP P
Nfc,F,
III and N, I.D..
Remark L.4 (Covering Number of Transformer Network Class), We define the covering number over domain {x:}= } x Q to align with the flow matching loss formulation in Definition L. where temporal dependence in transformers introduces no additional statistical error. Specifically, the loss averages over the time component, unlike the n i.i.d. data points sampled from target distribution g".
Next, we derive an upper bound on the covering number of transformer networks. Our proof builds on [Edelman et al., 2022] and studies the class in Definition B.4, with self-attention layer that applics softmax under inverse-temperature scaling and a feed-forward layer with ReLU activation.
Covering Number of Linear Function Class. Our norm-based upper bound on the covering number of the transformer network class extends the classical norm-based bound for the linear function class:
Lemma L.2 (Covering Number of Linear Function Class, Lemma 4.6 of [Edelman et al., 2022] and [Zhang, 2002]). Let 21. 2n E Rd be sample points satisfying ||2;|| S Bx for all E [n]]. Then,
85

---

for linear function class F { Wz W Rd'xd || 2,11
Bw, Bw > 0}. it holds:
BB 62
le, (2) -
log (d'n) for any
e >> 0.
Remark L.5 (Covering Number Equivalence). We remark that Lemma L.2 applies to our feed- forward layer for two reasons. First, the bias terms b1 and b2 admit augmented form. By appending a bottom row of ones to and set W1 == (W1, b1), it holds that WiZ + b W where Wi E RRxd by E IR", and W1 E [R*x(++1), Z E R(d+1)xL This two forms define the same function class. Second, the matrices operator. Since our S and Oc-) notation hides polynomial and logarithmic factors in d and L, Lemma L.2 gives the covering number for the linear class our feed-forward layer with biases.
bound in Lemma H.5 keeps the biases the same order as the
Following [Edelman et al., 2022], we extend Lemma L.2 transformer block. We view the block as a composition of linear function classes. We construct covers for each linear function class, balance the emoos across components, and minimize the size of the concatenated cover using the next lemmas:
Lemma L.3 (Lemma A.8 of [Edelman al., 2022]). Consider the following optimization problem:
min
02 2
subject to
Σ t=1
wis = C
(L.44
for some wi and constant C. Then, (L.4) has solution
EE a. 3
when
C
Wi
Equipped with Lemma L.3, we have the upper bound for the covering number of transformer block:
Lemma L.4 (Covering Number of Transformer Block, Modified from Corollary 4.5 of [Edelman et al., 2022]). Let [ be sample points satisfying max, ||< llo < D for some constant D > 0 and R(.): 1R > Rddx be the reshape layer (Definition B.3). Let { == {R(,))); E E IR be the positional encoding and T 1.r (Z) denote a two-layer transformer class (Definition B.1) with single-head self-attention and s-hidden dimension and r-MLP dimension. Then, holds:
Ther log.N(E, Th (2.)=ill1 11xo)
log(nL) ((CC e
(orC)) ((C)))22)))
where a = CC
Proof. Our proof builds [Edelman al., 2022] by incorporating the scaling À 0 for the column-win wise softmax function. For Z E [-D, Djdx and WK, Wq E Raxd we define
FKQ = (s: (Wx2) (WoZ) WKQ WTW [WRolla s CkQ),
for some constant > Similarly, for Wy x and Wo pdxx we define
Fov {f Z + Wo WvZ Wov WoWy, |Wovl < Cov}.
Recall Remark L.5. For fced-forward layer, we define:
((FF) = { Z - W5ReLU|W, ||, Well s Cr).
86

---

For simplicity, we denote the k-th column of Z4 by 2; (k) for all i [n)] First observe that
max iE|m
( = max se|m],ke[L]
-((
(L.5)
for any distinct f.f in FoQ or F(FF), With this, we consider { as nL samples in [-D, Dj and apply Lemma L.2 to construct covers CRQ. Cov and CFP for J FRQQ Fov and F(FF),
Covering Number for FKQ and Fov. Since Z, E [--D, DJdxL, for any i, [n]] and E [4]. we (k)
have that [l(()) -f()) M13 - dD² |[(W K W2 211. Thus, by Lemma L.2 and (L.5), it holds
D²Ckq log (nI.),
log leg (Z))=ill (1~0)
(L.66)
where $ hides polynomial factors dependent on d and Similarly, for Fov, it holds
DºCov
log log.N(rov, ( 11s0) S
log (nL),
(L.77
co
That is, we have cover CKQ. Cov for FKQ. Fov whose sizes are upper-bounded by (L.6) and (L.7).
By triangle inequality and by Lemma B.2, for any WK, We E FxQ and any Wo, Wv E Fov, there exist some WK. WQ E CKQ and Wo: Wy Cov such that for all E [n] and all & [L], it holds
(Wo=!*)) TWoo= (W( (8) Softmax ([ ((x=2
- WoWWvz (k) Softmax (Wx= (WQ= (kl
(L.8)
< [Wo(Wzz (x)) Softmax (W ) > Softmax ( (W (1)
(1)
(W(Wvz Softmax (W<x((t) ( WoV (x) Softmax
(10 ( (WQ=
(II
By triangle incquality)
WWoW - WoWl)e/" WWWWy? (1)) Softmax (Wxz (M) T( [n) Softmax Softmax (8)) ( )]]
(11)
(By |Sootmax dL)
[WoWy 2oW (1) + Cov DII SSftmax((Wi (Wx2) (8) ( (k)
Doftmax[[Wisr (k) (W (4)
2
(I)
[Wovl] Cov and ||= 100 D)
Seov + Cov DJI Soltmax Soltmax (Wxz!k)) T(Wo= (+))
(Wftma ( (
(1)
(By (L.6)
(Wxz)1)) (Wo= (k) Seov + 2XCov D[][W (
(Wo==
(k)
(By Lemma B.2)
(1I)
<ov + DeKQ
(By (L.7)
Then, we have cover for the self-attention layer
CSA = {f: Z Wo Softmax (Wx (W>Z)) WoWy e Cov, WFWO e Ckq).
87

---

Covering Number for F(FF), Similarly, we apply Lemma L.2 twice for the inner and outer linear function function class operated by Wi and 22. Specifically, we construct cover C₁ and C2 for
(FF = (f: z WiZ [ill2 - CF} and J(FF) = (f: W22 ||2ll2 I CF}
respectively. Then, by Lemma L.2, we have cover Ci and C2 for
D2C2
log leg. ((FF), , (Z0)=iill 1100 2
F
log (nL).
(I.9)
F.1
and
D²
log.N 1og,N(eF,2 F(FF), (Z: 1xx) x
log (nL).
(L.10)
2
F22
That is, we have cover Ci, C2 for F(FF), 1
(FF) whose sizes are upper-bounded by (L.9) and (L.
Let (Z) denote the self-attention layer with weights chosen from cover CSA. For any Wi.
WK, Wo Wo, Wv, there exist some W1 E C1 and ((SA) such that for E [m]. h: E [7], holds
[ ((SA) (x)) W o ((A)) (1))
(L.11)
$ | o F(SA) (z(1)) - W1 o F(A)) (2,
+ 1| (=SA) (1)) - W1 o (z, || )1
(By traingle inequality)
< (C -
F(SA (k)
(z(
+ 1| o ((A (2. - W1 (SA |1) |||
(2
(By norm bound on Wi
[CE|
-F(SA) (k)) -
Covll( (=
(By the norm hound on Wo Wv)
CF((Evv + 2ACoy + COVEF11
(By (L.8) and (L.9)
Building on (L.11), we have the cover for F(FF) o F(A))
W ((SA) - WaReLU[W, (=!4))]
(L.12)
W2W, ((SA) (1) F(SA)
(x( (x)
I ((SA) -W,,, )2
(1)
By triangle inequality and Lipschitzness of ReLU)
x CrCovl(W2 (2)2((1) (k)
[WW,, ((A) (2! (3)) W2W, (SA) (z! )
(1)
(1)
By norm bound on W12 and Wov)
24 C=Covl( (2) +CF ((A)) (10 W (SA) (z(*))]
(By norm bund on W2)
(1)
(1)
S CpCovera+ + Cz + 2XCov DEXQ) + CrCoverj
(By (I..10) and (L..11)
The overall size of the c-cover of a transformer block Flasr = F(FF), ((SA) is
Cirans = (F(FF) o ((A)) WoWy E Cov, WWW E CKQ. W,, W. Crr).
This gives
log CCrauel = log [xqql log Covv + log |CFE],
88

---

satisfying CpCov(eR1 + tp2) + (e + 2x DeKQ) I Let
a1 DOCk log
02 ==:=0' Dv log (nLi); 0z 0z == log (nL);
W1 == CoC
w2 = C D; w3 XCrCov.
Finally, we apply Lemma L.3 and obtain the optimal size of the the cover Cirans:
lo (C COVD 1/3 + C D2/3 + 22 )/3)
22
We extend the argument to two blocks by invoking the composition step in [Edelman et al., 2022]. For inputs with positional encoding E, we replace the bound D with D + Cg, where ||]]| I Cg.
This completes the proof.
Remark L.6 (Looseness of the Covering Number Bound). The bound in Lemma L.4 is loose yet sufficient for our two layer transformer class with inverse-temperature scaling and one attention head (recall Appendix H). The extension from one block to two follows from the composition step in Theorem A.17 of [Edelman et al., 2022], where it provides the complete induction argument.
Lemma L.5 (Covering Number Bounds for Transformer Network Class, Modified from Theorem A.17 of [Edelman et al., 2022]). Let T T.. (CT, CKQ CKQ. Cov Cov, Ce. C2 CF, LT) be the class of functions of one transformer block satisfying the norm bound for matrix and the Lipschitz property for
layers. Then, for all data points satisfying ||æ,||2,0c I D, it holds
log.N(ee; Th P x [0, 11,11 1.l1-11. l12)
(L.13)
log(nL/ee) a² (C (N(Cr) ((C)))
where a = (CCCCCCCC((((
Proof. Lemma L.4 shows Lemma L.5 with the absence of domain [0, 1]. That is.
log.N(Ec: Theo ,m,,p,, l1e)
(L.14)
log(nL), a² (( (C2:00) + (oCC))CovC ((C)))22))))
S e
holds with data points drawn from PD_ To extend it to pm x [0, 1]. we discretize [0, 1] into a 6-grid g = {4 = = 0.l. [1/8], 6 E (0, })}. For simplicity, we denote |g] == m O(155).
Let Ther be the transformer network class on domain [- [-D, DJd x G. We first suppose that
N(ce Thssr pm 1] I12) S (Ne(e Th..rr pd x 1.11-112)
(L..1)
holds when 8 == (ALT), where LT is the Lipschitz constant of the transformer block. Then, since domain pm x G a set sample points with size at most nmn, (L.14) gives
N(ce/2,The" Ther pm G. 112)
(L.
log(nmL)) a² (C)) ( 2200 ( $)² (ce/2)2
Therefore, (L.15) and (L. implies (L.13):
((ee, This pm x 10, 1) l12)
89

---

M(e/22, (Rgg Pm x G. -..1-112)
(By (L.15))
log(umL) a
(c ((C)
(X(Cr) 2000 ((CC))COr))
(By (1.16))
(ce/2)2
log(nL/Ee) a (C2200) (oVCF)
22000 KQ
((Cp) ((r))
(By 6 = Ee/ l. and dropping lower order terms)
Lastly, it suffices to prove that (L.15) holds when 8 == <c/(4Tr)) We show this by utilizing the Lipchítz property the transformer networks (Definition B.4.) Specifically, (f) i. be a e cover of R.9
sh.s. Then, for any f E This and & [0, 1], holds:
[[(a;, 1) (1(,,,))12 < [[f(x;))) f(xn, twll2 faa,tt)l2 [[f (x) (1 fa (A)
(B)
(By triangle inequality)
We then show that the RHS is bounded by c and this implies (I.11)). For (A) and (C), it holds:
(A). (C) S LT t txl S LT 8 S 4
where the first inequality is by the Lipchitzness of the transformer network, the second inequality is by the definition of the 8-grid G, and the last inequality is by taking = ce/(4L)).
Further, (B) is bounded by 2 by the definition of {ff M Altogether, we have that
cc + Ec. 4
[lf (zs, <
(L.17)
Since (L.17) holds for all 1, (L.3) in Definition L.4 holds after taking the supremum over L. This completes the proof.
Equipped with Lemma L.5, now derive the the covering number bounds of loss function class under transformer weights configuration in Theorem I.1 and Theorem I.2.
Lemma L.6 (Covering Number Bounds for S(D)). Let E. O. We define the loss function class by
S(D) {f(x; up) D R ug E T Th....", ). where D = [-D, DJd= for some D 0. Given fixed set of i.i.d. sample {a; 1 drawn from the target distribution we define the norm of loss functions by [|{(:; 1g sup... gerune (æ,; ug (z:)).. Then, under parameter configuration Theorem I.1 and Theorem 1.2, the e, covering number of S(D) with respect to ||-Il satisfies:
||- 1lor)) 0( log (nL/Ee)
D N 1165d-122 129 (log N) 20ddd, +48+17).
2
For all f(x, E Th Ther E [o. 1] and {a.)i ~ q", we equip the transformers with the norm:
Ilf(,, f)g = [| f) Dill.
Then, the co-oovenngg number of the transformer network class satisfies:
log: Thes d" x [0, 13,11 (l), S =0( (nL/Ec) D N1 (log N) 20)00,+44++16
2
Proof. First, we apply transformers parameter bounds in Theorem I.1 and Theorem 1.2. Then, we extend the covering number bound to loss function calss S(D).
90

---

Log-Covering Number of Transformers Network Class. From Theorem 1.1, we have
CKQ, Cke 0AA-1 (log N) +2); Cov, Cov, Cov O(N-));
Cp, Cp.Ck.0 ( N) " +++)): +); CF 0()); CT O(v 1oylog N).
By Lemma L.5, the bounds on log-covering number follow
-4" x [0, 11, l12) log log.N(ce Theer a² logn///e (12200) + (X(C)) + + (()))()))) e. a log(nL/ee) (ov 2200)
(By dropping lower order terms)
where
(Cv) 02.00
KQ
O (log N) 122dd+2244 N-28 N88d443
(log N) N)8d+44
(Cr)
(Cov)²
(ACx))²
O(N854469 1010d= 10d22+22+8)
Therefore,
a² (log(nLT)
NggN(ess The "00 [0,1],11-12 S
( (N884469 (log N) 10da 12d442348)
By Lemma L.5, have
S Cor) Ce)
N (log N22=+042 N N194428 (log (2) Cx)
(By the definition of a))
(Cs)²
(Cov)
(AOxq)
O(DN (log N) ()dd+++41)
Altogether, we have
log (NLT)
log BogN((o: This .dt x [0, 1],11 112) S
D2 166d4412 (log 27200,,444
2
Further, by || 11lo S ||. 1|2, we have that
logN (ce; Ther ddth x (0, 1].ll lloo) log(nITT) D2 16694412 (log N722d (L.
Log-Covering Number of Loss Function Class. Recall the of loss function Definition L. and its truncated counterpart Definition L.3. Let 8 > 0 and un (Ir, (), ua (2( () E Thire be any two transformers satisfying max, (2,, t) u2 u2(2, (z,, ol & for all i [m].
First, we derive the upper bound on the expectation of (x|x1))
E
(x|x)()())))
(L.
Xa-N((0.1)
E Xo>N((..1)
[l] + x.ll
(By Definition L.1)
91

---

[1 + 0x
(By Jensen's 's inequality)
Xo~N(0,1)
E
[0 + o7 xoo1x
(By expanding the & norm)
V ((..)
E
| + +
(By Xo N(O, n)
(0-)(
(2
(By x E ()),
Then, the distance between loss function (2; un) and (2( follows:
(2; u1)
(2; u2)l
(L.20)
10 (o,) (x, (x(x<))) [122(x,)) (2)
T to
(By Definition L.1)
(x, u2(2, t) (x)(m)))) (n (zr, (2 t)]]dt
T to
No,NN(..1)
t0
E [1 t) u2 t) 2
(By I|u1 wall 8)
T
to
(0,)
[[]uu +
2u, )
(By Jensen's 's inequality)
T to to
V Xo~No.11
12
2
XamN(a.)) [1, u2 1) + (<(22)
T 30
(B expanding the & norm)
T
E XomN1o,11
llog N 21/x(((
(By CT Ol O(vlog N))
T to
log + D2 + 457dt
(By (L.19))
T to to
288log log D
(By Assumption 1.2)
Finally, we extend the log covering number the loss function class S(D) by setting
< = a(c, Vlg No N
D2).
This gives
log (()) & ((c, Thes -11)))
(By (I.20))
Therefore,
((D),
log.N(ee Th.s." ,11 11,0)
log (nL). D2 1169d4112 (log N) +42001
(By (L.)8))
log (nL) D 166dd+12 (log N) 20d 1000d_44444 +43+177). (c))
(By the definition of e
92

---

This completes the proof.
Generalization Bound. Based covering number bounds results in Lemma L.6, we analyze the upper bound of generalization error E [2.)%_ [Rtrunn (ūp) - ( Note that the following
distinction separates generalization bound for the flow-machhing loss from classical learning theory.
The empirical risk (Definition L.2) takes the form {(2; wg) f(z; u²), where u" denotes the ground truth velocity. While most standard losses stay non-negative almost everywhere, the flow matching loss may take negative values. We use the next lemma in (L.25), which bounds the second moment of the flow matching loss in terms of its first moment. Without the sign issue breaks the derivation.
Lemma L.7 (Bounds on Second Moment of Flow Matching Loss, Modified from Lemma C. of [Yakovlev and Puchkin, 2025] ) Assume Assumption I.1 and Assumption 1.3. Then, holds
E z~g
(x; uo) etruno u ) E K. E 2ng
gerune uo) gerune (1; u )
where K = D2 + Vlog N.
Proof. Recall Definition L.1 and Definition L.2. We have
grrnn uo) = up 10) I D} and R(us) = > E(x uo) - > ((a,; w*),
where u² (xx pe(x]
Se ue (ark// (() (2r )((()) dæi is the ground truth velocity and
1 - uor; T to Xo~N(o.1) ,o,, Ill Illlo o, t) (pip + lo
For any Te the flow matching loss takes the form ((x;; uo) Mx, w"). To simplify notation, omit the indicator 1 { [||=]]oc S D} when expanding grrune with the understanding that we focus only on the bounded domain where the flow matching loss is defined. Then, we compute
ptrune (x) gtrune (I; u²)
1
E T to Xo~N(o,1)
[llug (1i dxXo) - ||u- (مio + oeXo)l1)d
1
E T to Xo~N(o.1)
[(uo u²)
(et)(uup (uo + - (1iv3 + dtto
T
E [lluo - u w111)de) (f T
1
ug + (1ir dxxo))11)
T to
10
tg
(A)
(L.21)
where we apply the Cauchy-Schward inequality for the last inequality. Next, we bound (A) using previous results for the bounds on the true velocity, conditional velocity and transformer network. Recall Lemma J.4. holds
litel 1e
f Me
de
||0||
+
Os
(||=)|1
+ 1),
and by Assumption I.3 we have
113 < || and here consider bounded domain || S D.
93

---

Further, under the transformer network configuration in either Theorem 1.1 or Theorem 1.2, we have the transformer output bounds CT O Ooylog N). Lastly, for pم, + ox holds:
E [llik + oxXoll S (o.1)(o.1)
E Nom))
[
where we invoke Assumption 1.3 and | < de D2 in the last inequality. Altogether, have
(A) T -n
[ug + ||u 112- (pear + dx D² ++ VlogN. N.
tu
Therefore, (I..2)) becomes:
(1; (rrunc(x; (1; (./0
E [llug w* w1]]att) (D² + Vlog N).
T to
Then, conclude that
E zm-g
ptrunec (I;; uo) - grrune (x;2²)) )1²
$ S(D² + Vlog.)
EE T ang
IR Xo-Ni(o.1)
[|v0 u=' u=112)1 d/
= (D² + vlog.N.
[uo - ut 12
(By tower property)
T zerups
(BR)
=(2² + N)o
E zm1
grrune (z: us) - etrune (x; u²))
(By Remark L.2
We remark that (B) is the conditional flow matching risk R(un) defined in (L.
This completes the proof.
Lemma L.8 (Generalization Bound, Modified from the Theorem C.4 [Oko et al., 2023]). Let Úe be the velocity estimator trained by optimizing CCFM(ug) following Definition L.1 with i.i.d training samples { For c, > O, let N := N(cc, S(D), " q" [1-llxo) be the covering number of function class of loss S(D) following Lemma L.6. Then we bound the generalization error:
Rtrune (āo) Rerunc (2n) S
E {
R (ūn)] + t loggN+c)) + te.).
(x:)
Proof. We use EErM and R' to denote the conditional flow matching loss and empirical risk with ghost training samples { Further, let u* denote the ground truth velocity field.
Then, following Remark L.3, we rewrite the generalization error:
Rtrunc (wp) Rerunc (ūe) (ao)
(L.22)
E
B
R trunc (ūo) - R truuc (Up)
(By Remark L.3
[
(
E
R' R'trune (Te) (Trnn((Ti)=
(Rtrumc(Tt))
(By the independence hetween of and R R(ūo))
94

---

I3 n (ar Dim1
grune (1/1 ūo)
graane (1 uu"))
grune (rz úp)
grene (x);;
(By Definition L.2)
LNr be a c--overing of the loss function class S(D) with the minimum cardinality in the Loe metric. Note that f fy have domain D= = [-D, Djd= by Definition L.3 and Definition L.4. Further, let be the random variable such that [|e(: (i) Es(s ul)
For cc 0 be chosen later, let J := {11 (2..,......N
Ec. Moreover, we introduce following definitions for simplicity:
w(x) = grrunc (1; Up) prrue ur;;"), wr (xr) = Ez(1;; Uz) - gerune (x; u
hy == max A,
L8 (2; - gurunc (z; u
wy(xg) W/ (x:)
Ω = max ISJSN
hg
where Z q is independent of {2,, of
Then we can further bound (L.22) as follows:
E n a Jim
etrune (x/z ūo)
grane (2)1 u()u())
gerune (x:; ūi))
greune
(Izs) u =())
(L.23)
E
1
(w)((+) - wx((=)]]]
+ +ce
(By the definitions of wa and covering number)
12 (ar Dim1
2m1
E
(w)())) - -w>(x:))
2cc
(By the property of expectation)
n ( 1111
1=1
1
E
[h,s]] + 2
(By definitions of by and (?)
n ( im1
1
E
(h3)
I
[S2]] 20c
(By Cauchy-Schwarz inequality)
n
<
1
7
E
[h]]+ + 2n (m. m
(S291) + 2E,
(By AM-GM Inequality)
n
2
1
1
E
[13]
+
[S27] + 2c
2 1
(
2n
(
Now we bound
E
[h3]
and E Ba af )r [22) separately. For (e, [k3).. we have
E
[13]
+ < A²
(Z(G(z;))) -
gtrune (trunc(2; [2;u")]]
(By the definition of hg)
(
(
2
E
[B[etrunc (z; uo)) .trunc (2; ( + 2cc
(By the definition of ex)
(ar
< A
[ (ūo) + 2
(By Remark L.3)
Then we start to bound E ( (xx )a and (=()a we have
[9P] By the definition of (z) and the independence between
Wy (21)),s)))1))
E
n2
1
E
[ws((x)]]
E (x()) ai!
(By the independence between by and (2s, 2f
n.
95

---

1
L way (w)()))))
(By the independence between wy and (2), 2/
h
2 0.
(L.24)
To use Bernstein's Inequality, for any j. we bound the following expectation as
Wyy
- u)(o)))
E
{ 13
hy
W (2x))
(2)(z()
)9
21
[w (2; (1)) (
E
E
fej
by
20 2
h2
Wi (y((t)
2
(W)(r')g 27
E ai
+
(By (L.24))
h
Recall that for any j E [M] (x)x)
= Eg 2z(2;u, u,) grrune (2; w*). For any l E S(D), assume
|etrune (;; u0)l S K, then for any E [n].j E [W], we have Bus (x:)) = Eu,, [x)( which
leads to
E [1,(+:)]] 2i
E [x)(
E IC, (d); - (a's )*]]
(By the definition of wy (x))
[lc,(2 uz) - Eeruuc(2;; (2;u*))
s E [h3]. 24,7!
(By the definition of hj)
Then, it holds
1 wy(w:) w)((()) )21
( x
hnj
)2 + W/ ( )21
E
hj
Wi (3rc)
+ (2)((()
< 2k
E
)
By Lemma L.7
h3
h13
1=1
< 4nk.
(L.25)
(s)(=))=u((r) ( h2
Since
x < á and Bl
0. by Bernstein's Inequality, we have
for any j E [W], h >
n
Wi (x:)
W3 w3(x()
W/ (x;) (x+
2 vil
Pr
2 l
= Pr
hi
hj
1=1
h h/2
I 2exp
x(4n +
shh
3A
Thus, we have
NV
(y(xx) wy(a!)
wy
Pr[s2 2 h| <
Σ
Pr
Σ
)2 h
(By union bound.)
h;
96

---

h/2
2N exp
k(4n + y/h
Thus, for any ho > 0, we bound E (x [127] as
[02a]
ho
/ Pr[s 2 hl
Pr[s 2 hldh
ho
h/2
< ho + h 2Vexp
dh
(By tail-sum formula)
m(4n +
v/h 3A
{ [exp
h
3A
I ho + 2N
+ exp
]dh
ho
16kn
4x
ho
SKyho
ho
32k
3Avho ho
I ho + 2N [16nnes
+
exp
16xM
3A
9.22
4x
Taking A who and hg = 16mm log NC, we have 12n
E
s9 log.g.N.
(
Combining above, we bound the generalization error
E {
R petrune(Ae
(ûg)
Rrrmn (ūe)
I
[13]
+ 2n (,r,-=1)1=
E [S2 + 2
(By (L.22))
2 (1r,,ar Piz
<
A
E
R trtrue (Ro)] + -60) + 2n12
Onn log
2
<
[ [2o)] 00 log.N NVg.o + cc).
2
{as 1iz
This implies
E [x
Rtrune (2o) $2. E
Rirune (ūn) + 0 n
log. + co).
[x:)7
Therefore,
E (trune((g)) Rerunc (āo) S E
R Rtrunc
+ 0
log. + +
This completes the proof.
L.3 Main Proof of Theorem 1.3
We now give the formal proof of Theorem L.3.
Theorem L.1 (Theorem I.3 Restated: Velocity Estimation with Transformer). Let the feature dimension. Suppose we choose the transformers as in Theorem I.I and Theorem I.2 correspondingly. then we have
97

---

Assume Assumption 1.1 and Assumption 1.2, Then,
E [R(ūa]] On rod11s (log n) 20d,, + +13+20).
(x:)--1
Assume Assumption 1.2 and Assumption I.3, Then,
E [R(ūe]] = O(n 5d43 (log n 200 444-4434 (a:)
Proof of Theorem 1.3, Let {<'}== be different set of i.idd samples independent of the training sample {r.} Further, use R' to denote the empirical risk with samples {2/]==1-
Then, following (L.2), we decompose (2,))_ [R(ūs)] as:
E R((aa)]
E P
E (a: 2
R' (ūs) R. rrunc (wa)
(x.)
(1)
R (ūo))
Rtrume (uo)
far
(1I)
E
Rtrune (ug) R((ūs)
E
R(ūa)
(x))
(III)
(IV)
Then, we bound each term and incorporate them to obtain the upper bound on the estimation error.
Bound (I) and (III). By Lemma L.1, term (I) and term (III) are upper bounded by
1 -20222 log N.
(L), (III) S dd= exp
Boundd (II). By the generalization error bound (Lemma L.8), we have
(II)
E (m
RR trunc (Go) - Rtrunc (āo)
(L.26)
E
[R pntrune (To) Rtrunc (To))
(By (o [ trune
R trunc
Rttune + O(_ log.N +ec)
E
By Lemma L.8)
[x
< (IV) + dd exp 1-DDD log + +o n
log.N + cc)-
(By Lemma L.1
where No.c, S(D), 1c))) the covering number (Definition L.4) of loss function class.
Bound (IV). Recall that R(ū,)) (ū)) (urM(u") and ug is trained by optimizing ECFM (ug) following Definition L.2. Therefore, for any velocity estimator ug, holds
R(Ua) S (CF (crM("*) R(uu).
98

---

Then, for any velocity estimator ug. it holds
E
[R(Ta)]
E [R(ue]] R(ua).
(L.27)
{mi)
{as)--1
Altogether, the estimation enror is upper bounded by
(R((g) [1)
(L.28)
{ 1
= (1) (II) + (III) + (IV)
< D exp( exp(-C22D) (2C2D²) log N 0 log.N V + <c) +
3 OON-22 (log 11)x//2111 + 0 log N + +.)) + 2(IV).
(By setting D = /23 log N/C²)
Furthermore, the log covering number upper bounded by
log (eg.Nee:s(D)))) lloo))
(L.29)
log(nIn/ee) D N 1160d+129 123 (log N) 20) 20d2++)++77
(yy Lemma L.6)
0 log(nL/Ee) N26dd4128 (log 2) +494299)
(By D v 28 log N/C₂)
Next, we bound the velocity field estimation error.
Estimation Rates under Generic Hölder Smoothness. By Theorem I.1, it holds
(IV) & R(us(1, t))
(By (L.27))
/ T to O 9, (log N
/ 1/1
J [lue(x) us u0(x, (x)
(By Theorem I.1)
Then, (L.28) becomes
E R((a)] (m:)
< O(N 28 (log N)d+/2+1) + +o n
log NogN+cah + NOBB (log N
< O(N 28 (log N)d./2+1) + 0 log (nL/Ec) No 20d, +48+19 + (e) O(B"NN-BNNNO & (By (I.29))
ne?
where v 16,3d + 12/3.
Let 71, 72 E (0, 1) be two arbitrary numbers. We take N = n2//v and 62 = n 722 Then,
E [R(ūg)]
< 00nn
28m
(log n) 4/2 +1 ) + O(n -1+m (log N 20d +49420 logn + ">)) +)+ + O(B2n - En
(logn)d++
< O(n min 8r1 .1-7 22) (log n)) +48422
For any 71, 72 E (0, 1) satisfying
11 + 272 < 1,
99

---

we consider
871
min
1 - 71 -2 p2).
To simplify, we set
371
1 71
-2 72
giving
V
B
71
38"
+ 33
Therefore,
E { 1
[R(Go)] On 150913 (log n) 20d, +19221)
Estimation Rates under Stronger Hölder Smoothness. By Theorem I.2, holds
(IV) S R(us(x, f))
(By (L.27))
/ / [2x((x) - uo(x, t) Pt (x))drd
= O(B2N N1IB2N-2910 29 N)
Then, (L.28) becomes
E [R(a)] (
< O(N 28 (log N) +1) + 0
log N + ce) + - (log N)d+t
< ON .)
) +1 + O
log n
N (log N) N)200,+13+11 + + + O(BPN (log N) No)=+
ne²
(By (L.29))
where v = 163d + 126.
Let 73, 7/ (0, 1) be two arbitrary numbers. We take N n2/v and 6 n 74 74, Then,
E [R(ū))]
(x.
< O(n (log n) dx +1) O(n 27 (log n) ( +0d, 18++20 -= O(B"n 28th (log n) dott
< O(n min 2813 1-79 27 (log n) 200d
For any 73, 74 E (0,1) satisfying
7/s + 27/4 1,
consider
minf 2,873
1 73 -yya, 7a)-
100

---

To simplify, we set
2
1 1-3
73 - 2 = 7/1
1
giving
V
2/3
78
7/1
+ 6
63
Therefore,
E {mx)
[R( = O(n 5u13 (log n) 200d + +48+20)
This completes the proof.
101

---

M Proof of Theorem 1.4
In this section, apply the Grönwall's inequality and the Alekseev-Gröbner lemma to extend the velocity estimation to distribution estimation under 2-Wasserstein distance.
Organizations. Appendix M.1 introduces auxiliary lemmas. Appendix M.2 presents the main proof.
M.1
Auxiliary Lemmas
In this section, introduce auxiliary lemmas for extending the velocity estimation distribution estimation in 2- Wasseerstei distance. Specifically, we state the Grönwall's inequality in Lemma M.I. Furthermore, we introduce the Alekseev deeseev-Grröber lemma that quantifies the deviation between solutions of two distinct ODEs in terms the discrepancy between their velocity in Lemma M.2.
We begin with the Grönwall's inequality.
Lemma M.1 (Grönwall's ' Inequality, [Gronwall, 1919]). Let a, E R with Let g(l) and y(t) be two real-valued continuous functions defined on (a, bj. Then, if v(t) is differentiable on (a, bj and satisfies:
d de v1 s v(t)(((), LE (a
holds
y(t) s v(a) exp (f o(s)ds)
Next, introduce the Alekseev-Gröbbnrr lemma.
Lemma M.2 (Alekseev-Gröbner Lemma, Lemma 16 of [Fukumizu et al., 2024], Proposition 2 of [Benton et al., 2023], Theorem 14.5 of [Hairer et al., 1993]). Let u(x, () and uo(a, 4) be smooth vector fields and (1x s, () and who (1r, 8, L) be the respective flows defined for 2 that satisfy
d
dt (x,,, (, (18,8,8) =
d
dt s, s,,,,) so(o,
Then,
v0la, lo 7) (2 1o, T) = 1' Drio(vb(a, lo 8), (uo (o, 4), s). 8) u(bbæ, to s))ds,
where the partial derivatives in the Jacobian matrix Debo(v(a, to s), T) is with respect its first argument.
M.2 Main Proof of Theorem L.4
now present the main proof of Theorem I.4.
Theorem M.1 (Theorem I.4 Restated: Distribution Estimation under Wasserstein Distance). Let
Pr denote the estimated distribution at time T. Further, we define a constant // = 16(L+ + 1) + 12/d.
Assume Assumption 1.1 and Assumption I.2. It holds
E [
W2(Pr. Pr)] = O(n 3 (log n) +20+000
102

---

Assume Assumption 1.2 and Assumption I.3, holds
E (x.)_-
[W((Pr, Pr)] O(n tod (log n) l0d, +23+ +23+10).
Proof of Theorem 1.4. We bound the 2- Wasserstein distance between the estimated and true distri- butions with the & difference of the velocity field network and the true velocity field. Our proof structure follows [Fukumizu et al., 2024, Theorem 3] and [Benton et al., 2023, Theorem 1].
The distributions Py and Pr are the pushforwards of Pto by to, and sh to. T). Thus, using the definition of the 2- Wasserstein metric, it follows that
W2(PT, Pr) S
E xempio
[llb,a,, to, (0 (1)
We use Lemma M.2 to bound the lg difference of the flows. To that end, let us first bound the Jacobian matrix (v tu s), s,t). We have
Dibe to s), t)l1
0t
a
Dúaa (vo(1, to s,
Dt
Duo (tho (v,( to s). s)D (1b 8), 1)ll
S LT Dibo(o to, 5), 1,)l2,
where the first inequality follows from triangle inequality of the ||- 1|2-norm, and the second equality follows from the flow ODE in the assumption of Lemma M.2, and the third inequality follows from the Lipschitzness of transformer network (Definition B.2). Therefore,
to, 1)12 2 eexp expp J. LTduJ s expof LTdu } M. (By Lemma M.1)
Now have
| to. T) vxx, to 7)11
< 2
|luo(V(I, s), 8) u()((x, s), slsss
< 2 /" [|uo(H(z, u(b((,, lo. s), 1)1,ss
where in the first line we apply Lemma M.2 and in the second line we apply the Hölder's inequality. Then, we take expectation with respect Pto on both sides of the above inequality
J." [ to s), u(bb(r, to 8), s)
E [1l to T) (a, to. T) S M gooPto
E poupee
M
E [1]u0(z, 8) - u(x s)
to
wapp
where the last equality follows since the samples (zx to 8) with x ~ Pro are the same as the samples Ps by construction of the flow.
Therefore, we have
Wo(Pr.P Pr), S M M./..
'// E
(z] s) u(æ, ss) 1 xmp,
103

---

where
T
E (1) - u(x, s)l = (T - to) xmops
(y Definition 1.2)
Then, by Assumption 1.2, we have
E
(T)( M -T-14 10) (=:)
IVR(To)
S M
E IVR(Ta))
(x:)
(x:)
Finally, we apply the flow estimation results in Theorem I.3 and get
E
R((x))] On IaFF (log n) 2nd 20d,, +48440)..
E R(wx)] = O(n 56क5 (log n) 20d, 20dd-434+.0) 4,3220).
under Assumption I.1 and Assumption 1.3 respectively. These imply
M Pr)l S
E [VR(ūo)) = O(n midiom (log n) 10dz +23+113+
{ mm
{x
[W2(Pr,PT) M
E [v((ū)) = O ied:es (logn) 10d +29+10).
This completes the proof.
104

---

N Proof of Theorem 1.5
In this section, prove the nearly minimax optimality results of flow matching transformers under specified settings (Theorem I.5).
We begin with the definition of modulus of smoothness following [Oko et 2023].
Definition N.1 (Modulus of Smoothness). Let f be domain in Rd and f E I (S2) be function for some p' E (0, ooj. We define the r-th modulus of smoothness of f by:
wry (f.t)
sup |4 |Al1sSe
where (1)) is the difference operator defined by
(1) 1(x jh), if + ju E Ω for all j.
(T))
==
otherwise.
Next, we define the Besov space.
Definition N.2 (Besov Space ) Let 0 < p',d' S 0o, 8 and r == [s]
The Besov
norm of function T.P (St) is defined by [lfll 115ll + 151 where
1.00 0,
Wt (f.t)) dt +)
+
q oc,
| By
g 00.
Given m, we have the Besov space B
b (L, m) = U (2)
I1 < L, 2 m).
The next lemma provides the minimax optimal rate for density in the Besov space Bg
Lemma N.1 (Theorem 3 of [Niles-Wei Weed and Berthet, 2022]). Let Ω = [-1, 1jdt be the domain of density ((21) in Besov space B m). Then, for any r.p'.q' 2 and 0,
inf sup qEB (L.)) (x,)2
[W,
P)] =
where {ri) is a set of i.i.d samples drawn from distribution P, and P runs over all possible estimators constructed from the data.
Then, we revisit the definition of Wasserstein distance:
Definition N.3 (2- Wasserstein Distance). Let X and Y be two random variables with marginal densities H, and by respectively. We define the 2- Wassersttii distance by:
/ |1= dl v)
inf W2(Mer Py) (EM((H))H))
where M(ps Pny) denotes the set of joint measures with marginals and py.
We then give the minimax optimal rate in the Hölder density function spaces.
Lemma N.2 (Modified from Theorem of [Niles- Weed and Berthet, 2022]). Consider the task of estimating probability distribution P(z1) with density function belonging the space
{g(x)))9(x1) HD ([-1, 11d=, B), ())) 2 C).
105

---

Then, for any r 2 1, B and de > we have
inf sup E (Zz))PP (as
[W,
25-25
where {-,) is a set of ii.d samples drawn from distribution P, and runs over all possible estimators constructed from the data.
Proof. Let Ω be domains. Since o0 directly implies Lemma N.2. This completes the proof.
(2)
(I2) for any R4 24, Lemma N.1
Next, we present the proof of Theorem I.5.
Theorem N.1 (Theorem 1.5 Restated: Minimax Optimality of Flow Matching Transformers). Under
the setting of (16d (8 1) dz 28, the distribution estimation rate of flow matching transformers (Theorem I.4) matches the minimax lower bound Hölder distribution class in Wasserstein distance up to log and Lipschitz constants factors.
Proof of Theorem I.5. By Theorem I.4, we have the distribution estimation rate in W-sasserttein distance under Assumption 1.2 and Assumption I.3:
(W2(Pr,Pr) = O( - rediis (log n2) 10d, +28+10)) {x
E 1n
Then, by Lemma N.2, the distribution rates matches the minimax lower bound up to log n and Lipschitz constant factors under the setting
(16d + 18)(3 + + = d. + 2,3
This completes the proof.
Experimental Validation
To provide empirical support for the proposed High-Order Flow Matching (HOFM) framework, we conduct series of synthetic experiments designed to evaluate the practical benefits of incorporating higher-order dynamics. We compare the performance of standard first-order flow matching (cquivalent to our framework with K 1) against second-order flow matching (K = 2).
O.1 Experimental Setup
Task and Datasets.
We evaluate the models on 2D density matching tasks, transitioning a standard multivariate Gaussian distribution, Wo, three complex target distributions, Ti. Following the experimental setting in [Chen et al, 2025], we use target distributions shaped as: ) a square, (2) two intertwined spirals, and (3) three intertwined spirals. These datasets chosen to test the models' ability to learn distributions with sharp corners and high-curvature manifolds.
Evaluation Metric. To quantify the quality of the generated samples, we measure the 2- Wassersstii distance between the generated distribution and the target distribution. A lower Wasserstein distance indicates better match and, therefore, superior performance.
0.2 Results and Discussion
The results of our comparison are summarized Appendix O.2. The findings demonstrate the advantages of using second-order dynamics.
Across all three target distributions and for every sampling step count (10, 50, and 100), the second- order model achieves a lower Wasserstein distance than the first-order model. This suggests that incorporating higher-order information allows the model to learn more accurate and stable generation paths, which aligns with the motivations discussed in Section 1.
106

---

Distribution
Sampling Steps
First Order (K'
Second Order (K' 2)
Square
8.51
7.09
50
6.45
6.08
100
5,48
2.82
Two Spirals
10
114.39
74.57
50
73.37
68.47
100
66.15
46.71
Three Spirals
10
192.19
109.93
50
123.53
87.70
100
93.26
68.81
Table 1: Comparison of first-order and second-order flow matching on synthetic 2D datasets.
Furthermore, these results highlight notable improvement sampling efficiincy. For instance, in the Three Spirals task, the second-order model with only 50 sampling steps (Wasserstein distance of 87.70) outperforms the first-order model with 100 steps (93.26). This empirical evidence supports the theoretical premise that HOFM lead to more efficient sampling strategies (Section 5).
107

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section presents the K-order flow matching framework.
Section 4 presents the statistical rates of K-order flow matching using transformers. Appendix I provides the statistical rates of the first order flow matching using transformers.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long as it clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation and potential future works in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model sell-specificatinn, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-ted system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
108

---

Justification: Assumptions are stated before every results, and we either give the complete proof directly or refer it to the corresponding section in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided or not)?
Answer: [NA]
Justification: This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is a novel architecture, describing the architecture fully might specific model and empirical evaluation, may be necessary
suffice, if the contribution to either make it possible for others to replicate the model with the same dataset, provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed instructions for how replicate the
While NeurIPS does not require releasing code. the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
This paper does not include experiments.
109

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines htttps://nips..ccpput Seaidee/ooeSSubee minissionn onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: httpppppllcgiddee/p/pppppp
/
s onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLS to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: This paper does not includes experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI. if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
110

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [NA]
Justification: This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, every respect, with the NeurIPS Code of Ethics https:/ /ttp://///eeupspeeee cc/publcc/ttiiicccideeellii
Answer: [Yes]
Justification: All aspects of this work comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work dedicated to advancing the fundamental understanding of dow-matching generative models; we do not anticipate any significant positive negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed. the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
111

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g,, gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cwm/ /ataasatss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/codemmodel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
112

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper involves neither crowdsourcing nor research with human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements your country or institution) were obtained?
Answer: [NA]
Justification: This paper involves neither crowdsourcing nor research with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The primary methodological contributions of this study do not depend on large language models in any meaningful, novel, non-standard capacity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
what Please refer to our LLM policy (https: htt.eerrrp....eooooeeeeeee/////e should or should not be described.
113


Reviews:
[{"review_id": "dJ0JsxmkWb", "reviewer": "Reviewer_hems", "summary": "The paper introduces a unified theoretical framework for high-order flow matching (HOFM), extending the standard FM approach to incorporate higher-order trajectory derivatives. The key contributions include generalized formulation and providing convergence rates for transformer-based HOFM, showing near-minimax optimality in distribution estimation.", "questions": "- Does higher $K$ always help to improve performance? Are there trade-offs (e.g., computational cost vs. accuracy)? How should we choose $K$ in practice? \n- How does the general method scale with dimension $d$ or order $K$? The rates provided depend on $d$, but no empirical scaling analysis is provided \n- Can we extend the framework to incorporate noise (e.g., for diffusion based sampling)?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Overall the paper is well organized despite its length \n- It provides a comprehensive mathematical foundation for high-order flow matching within a unified framework.The proofs are detailed and technically sound\n- The provided analysis aligns with modern architectures (transformers) and applications, making it relevant to real-world generative modeling\n\nWeaknesses:\n- The main weakness lies in a lack of empirical validation. While the theory is robust and sound, the paper does not include experiments to validate the claimed benefits of high-order dynamics (e.g., improved sampling efficiency or training stability). A carefully designed experiments (even on toy datasets) to demonstrate the theory would significantly strengthen the paper \n-  Limited Insight into $K$-dependence is provided: The framework does not theoretically explain why using higher $K$ should improve performance, leaving concerns on the practical advantages of high-order terms\n- There is a lack of discussion on whether the provided framework generalizes popular probability paths, such as optimal transport paths and diffusion based paths, making its relationship to established FM variants unclear. A deeper discussion of design choices and compatibility would strengthen the paper’s impact \n- Complexity of implementation is not studied. The high-order ODE system and transformer architecture may be computationally costly, but this is not discussed in depth \n- The sub-Gaussian and Lipschitz assumptions may not hold for real-world datasets, limiting the applicability of the framework", "was_revised": true, "final_justification": "My final decision is a borderline accept. I believe that even though the theoretical contributions might seem incremental, the overall presentation is solid and provides a unified framework to understand flow matching. This could be useful for the NeurIPS community. However, as discussed with the authors, inclusion of some empirical demonstrations could be useful to strengthen the theoretical claims and provide intuitions. Therefore, I am slightly inclined to accept the paper provided that the empirical demonstrations are included in the final version.", "rebuttal": "Thanks for your detailed review. We have addressed all concerns below, and made revisions accordingly in our latest draft.\n\n---\n### **W1.** The main weakness lies in a lack of empirical validation. While the theory is robust and sound, the paper does not include experiments to validate the claimed benefits of high-order dynamics (e.g., improved sampling efficiency or training stability). A carefully designed experiment (even on toy datasets) to demonstrate the theory would significantly strengthen the paper.\n\n---\n\n**Response:** Thank you for your comments. To clarify, this work is a **theoretical paper** that aims to provide a unified framework for any order flow matching and establish its statistical properties.\n\nOur framework naturally subsumes several empirically successful high-order models, such as HOMO ($K=2$) and ForM ($K=2$) [1,2], which have already shown significant improvements over standard flow matching.\n\nHowever, in response to your comments, we conduct new synthetic experiments of 3 sets of distribution transitions from a multivariate gaussian distribution $\\pi_0$ to the target distributions $\\pi_1$. In particular, we consider 3 different target distributions:\n1. a shape of square, \n2. two round spirals and \n3. three round spirals \n\nfollowing the setting in [1]. \n\nWe use the wasserstein distance [3] as the metrics (lower is better) to evaluate the distance between target and generated distributions.  Here is the results of the first order flow matching and second order flow matching:\n\n| Distribution             | Sampling Steps | First Order | Second Order |\n|-----------------------------|----------------|-------------|--------------|\n| **Square**               | 10             | 8.51        | **7.09**         |\n|                                  | 50             | 6.45        | **6.08**         |\n|                                  | 100            | 5.48        | **2.82**       |\n| **Two round spirals**    | 10             | 114.39      | **74.57**       |\n|                                       | 50             | 73.37       | **68.47**        |\n|                                       | 100            | 66.15       | **46.71**        |\n| **Three round spirals**     | 10             | 192.19      | **109.93**     |\n|                                           | 50             | 123.53    |  **87.70**     |\n|                                           | 100            | 93.26      | **68.81**       |\n\n\n\n**Discussion: Uniformly Positive Results.** We can observe that the second order method outperforms the first order flow matching under the same sampling steps across all the three different target distribution settings which shows the higher order flow matching can improve the performance.\nBesides, the second order method can use fewer steps to achieve the similar performance of the first order method.\n\n[1] Bo Chen, et al. High-order matching for one-step shortcut diffusion models. arXiv:2502.00688, 2025.\n\n[2] Yang Cao, et al. Force matching with relativistic constraints: A physics-inspired approach to stable and efficient generative modeling. arXiv:2502.08150, 2025.\n\n[3] Peyré, Gabriel, and Marco Cuturi. \"Computational optimal transport: With applications to data science.\" Foundations and Trends® in Machine Learning 11.5-6 (2019): 355-607.\n\n---\n### **W2.** Limited insight into $K$-dependence is provided: The framework does not theoretically explain why using higher $K$ should improve performance, leaving concerns on the practical advantages of high-order terms.\n\n---\n\n**Response:** Thank you for your insightful comment. Sorry for any confusion caused, but we believe there is a slight oversight. In our Section 5 “Discussion, Limitation, and Open Question”, we have identified this as an open problem and provided two sketched resolutions. Specifically:\n* The framework enables a $K$-th order Taylor expansion sampler that achieves a local truncation error of $O(h^{K+1})$ per step. This is a significant improvement over the error typically associated with first-order methods. Furthermore, all $K$ velocity components can be evaluated in parallel, which could lead to more efficient sampling.\n* Our $K$-order approach solves the ODE without the feedback loop present in standard methods like Runge-Kutta. This non-sequential nature may result in more stable error propagation, as approximation errors are less likely to propagate and amplify within a single step.\n\n---\n### **W3.** There is a lack of discussion on whether the provided framework generalizes popular probability paths, such as optimal transport paths and diffusion based paths, making its relationship to established FM variants unclear. A deeper discussion of design choices and compatibility would strengthen the paper’s impact.\n\n---\n\n**Response:** Thank you for your comment. Our higher-order framework, detailed in Section 3, does not restrict the choice of probability path or the source and target distributions. Therefore, the framework **naturally generalizes and subsumes standard flow matching** with any arbitrary probability path, including optimal transport paths and diffusion-based paths. The use of affine Conditional Flows in our paper is a design choice specifically made for the purpose of calculating statistical rates, as it is a popular and well-understood choice for this type of analysis [1,2].\n\n[1] Yuling Jiao, et al. Convergence analysis of flow matching in latent space with transformers. arXiv:2404.02538, 2024.\n\n[2] Kenji Fukumizu, et al. Flow matching achieves almost minimax optimal convergence. arXiv:2405.20879, 2024.\n\n---\n### **W4.** Complexity of implementation is not studied. The high-order ODE system and transformer architecture may be computationally costly, but this is not discussed in depth.\n\n---\n\n**Response:** Thank you for pointing this out. Our work focuses on introducing a unified framework to unify existing higher-order flow matching models with statistical guarantees. We believe the time complexity would not increase significantly because the training and inference both could be **implemented in parallel**, as most current dominating FMs are already transformer-based. We leave a more in-depth study on computational complexity for future work.\n\n---\n### **W5.** The sub-Gaussian and Lipschitz assumptions may not hold for real-world datasets, limiting the applicability of the framework.\n\n---\n\n**Response:** Thank you for raising this point. We acknowledge that our sub-Gaussian and Lipschitz assumptions are primarily for the purpose of deriving statistical rates. These assumptions are **standard in prior theoretical work** [1,2] on flow matching. We are enthusiastic about exploring weaker assumptions or releasing these conditions in future work.\n\n[1] Yuling Jiao, et al. Convergence analysis of flow matching in latent space with transformers. arXiv:2404.02538, 2024.\n\n[2] Kenji Fukumizu, et al. Flow matching achieves almost minimax optimal convergence. arXiv:2405.20879, 2024.\n\n---\n### **Q1.** Does higher $K$ always help to improve performance? Are there trade-offs (e.g., computational cost vs. accuracy)? How should we choose $K$ in practice?\n\n---\n\n**Response:** Thank you for your questions. To the best of our knowledge, a larger $K$ does not always guarantee improved performance. Instead, the choice of $K$ is a **practical trade-off** that depends on the specific use case, data, and available computational resources. There is no single optimal value of $K$ in a general setting.\n\nThe trade-offs involve balancing the increase of memory use of learning multiple velocity fields with the potential for improved accuracy and sampling efficiency. A higher-order $K$ allows for a $K$-th order Taylor expansion sampler, which has a local truncation error of $O(h^{K+1})$ per step, a significant improvement over lower-order methods.\n\nIn practice, the choice of $K$ has been explored empirically in models that our framework unifies, such as HOMO for $K=2$ and $3$. However, formally exploring the empirical question of an optimal $K$ and its trade-offs is an intriguing open question that we have left for future work.\n\n---\n### **Q2.** How does the general method scale with dimension $d$ or order $K$? The rates provided depend on $d$, but no empirical scaling analysis is provided.\n\n---\n\n**Response:** Thank you for your valuable feedback. Regarding the scaling of our method with dimension $d$ and order $K$:\n* Our formal statistical analysis already quantifies how performance scales with the data dimension $d$. In particular, we prove that the estimation error decays on the order of $O(n^{-\\Theta(1/d)})$. This means a higher dimension $d$ yields slower convergence rates (a manifestation of the curse of dimensionality), and our rate **matches known minimax lower bounds** up to log factors. In other words, the dependence on $d$ in our rates is fundamental rather than an artifact of a specific algorithm.\n* From a computational perspective, the cost of sampling may not scale linearly with $K$ because the $K$-th order Taylor expansion sampler is designed so that all $K$ velocity components can be **evaluated in parallel**.\n\nBecause our work is a theoretical study of fundamental capabilities, we focused on formal results rather than empirical experiments. We believe the question of “scaling laws” is addressed analytically through our rates, which is appropriate for a paper centered on nearly-optimal statistical guarantees.\n\n---\n### **Q3.** Can we extend the framework to incorporate noise (e.g., for diffusion based sampling)?\n\n---\n\n**Response:** Thanks for the question, but we believe there is a slight oversight. This paper focuses only on the **statistical rates of distribution estimation**, not on the sampling process itself. Could you kindly clarify the question? Thanks!"}, {"review_id": "8XWrd8VFYp", "reviewer": "Reviewer_7E7x", "summary": "The paper presents a unified framework for flow matching by extending it to high-order flow matching, which incorporates trajectory derivatives with respect to the time variable up to an arbitrary order $K$. The authors analyze several statistical properties of this general framework, including its universal approximation property (Sec. 4.1), the corresponding estimation error rate (Sec. 4.2), the estimation of the 2-Wasserstein distance between the empirical and ground-truth solutions (Sec. 4.3), and its minimax property (Sec. 4.4). The paper does not include any experimental results.", "questions": "The paper includes a helpful summary of its contributions in lines 42–59, which makes it easier to follow. However, it remains unclear what exactly the authors improve upon relative to prior work. In particular, I could not find a clear explanation of why one should use high-order flow matching, or what statistical advantage it offers compared to standard flow matching. Overall, Section 4 reads more like a collection of general statements rather than providing concrete justification for the proposed extension. Could the authors please clarify these points and better articulate the practical or theoretical benefits of the high-order formulation?", "limitations": "Please refer to Strength and Weakness section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "W1. I believe the paper should more clearly emphasize how its contributions differ from previous works such as standard flow matching and [1]. At present, I do not see a meaningful distinction between the proposed method and these prior approaches. In particular, the paper should better explain the role and significance of each theorem, and clarify how each result specifically relates to the extension to higher-order flow matching.\n\n- In Section 3, the authors derive the high-order flow matching objective. However, this seems rather straightforward: by definition, $\\frac{d^k}{dt^k} \\psi$ is simply the velocity field of $\\frac{d^{k-1}}{dt^{k-1}} \\psi$. Given this, the resulting formulation appears to be a direct and obvious extension of standard flow matching, without clear additional insights.\n\n- In Section 4, the paper presents several statistical properties. For example, Section 4.1 shows a universal approximation result, but this appears to be a direct application of the findings in [1]. In Section 4.2, Theorem 4.2 states a $K$-order velocity estimation rate, but the result does not actually depend on $K$ in any meaningful way—it simply reflects a general property of flow matching. I do not see why this theorem is significant in the context of higher-order estimation. Similarly, the estimation result for the 2-Wasserstein distance in Section 4.3 is a standard statistical bound that does not reveal any unique property tied to the higher-order framework. Overall, these theoretical results feel like a straightforward collection of known properties rather than new insights specific to the proposed extension.\n\nW2. The main claim of the paper is that it extends flow matching to KK-th order derivatives. If this is the central contribution, then the paper should demonstrate—either theoretically or empirically—why this extension is meaningful. In its current form, there are no examples, experiments, or concrete settings showing how or why using higher-order flow matching improves performance over standard flow matching. I strongly encourage the authors to include empirical evidence or clearer theoretical justification that illustrates the practical or conceptual benefits of the proposed higher-order estimation.\n\nReferences\n[1] On statistical rates of conditional diffusion transformers: Approximation, estimation, and minimax optimality, ICLR, 2025", "was_revised": true, "final_justification": "The main weakness was a lack of empirical validation and clarity of each component of the paper. Authors clarified through the rebuttal.", "rebuttal": "Thanks for your detailed review. We have addressed all concerns below, and made revisions accordingly in our latest draft.\n\n---\n### **W1.**  I believe the paper should more clearly emphasize how its contributions differ from previous works such as standard flow matching and [1]. At present, I do not see a meaningful distinction between the proposed method and these prior approaches. In particular, the paper should better explain the role and significance of each theorem, and clarify how each result specifically relates to the extension to higher-order flow matching......\n\n---\n\n**Response:** Thank you for your insightful comments. We apologize for any confusion caused regarding the contributions of our work and appreciate the opportunity to clarify.\n\nOur paper offers **two primary contributions** that distinguish it from prior work [1].\n\n**First, we introduce a unified framework for High-Order Flow Matching** that subsumes existing variants, including standard flow matching. While the extension may appear straightforward, its theoretical foundation is non-trivial. The key technical innovation is our **high-order marginalization technique (Theorem 3.3)**, which converts an intractable $K$-order loss into a simple, conditional regression problem with exact gradients. We also formally identify the **total derivative constraints (Remark 3.1)**, which are crucial for ensuring the consistency and uniqueness of the high-order velocity fields and their corresponding flows (Theorem 3.1).\n\n**Second, we provide the first rigorous statistical analysis** of this unified framework, specifically when implemented with modern transformer architectures. The statistical results are significant because they are tailored to the unique properties of our high-order framework, rather than being a straightforward application of prior results:\n* **Velocity Approximation (Theorem 4.1):** While inspired by general approximation theories for transformers, our analysis includes a **fine-grained error analysis** that specifically accounts for the Gaussian tail bounds and the decomposition of the high-order velocity field. Our key technical innovation is **omitting the unnecessary local diffusion polynomial** in approximation theory compared to [1].\n* **Velocity and Distribution Estimation (Theorems 4.2 & 4.3):** We admit that our theoretical rates do not yet show a significant improvement with increasing order $K$. However, this is an important and open question for future work, as noted in Section 5. The primary significance of these theorems is that they provide the **first rigorous statistical guarantees** for a general class of high-order models, establishing that they achieve **near-minimax optimal rates** on par with their first-order counterparts. Furthermore, our derivation for the 2-Wasserstein distance (Theorem 4.3) is novel in this context, applying the **Alekseev-Gröbner Lemma** to the high-order ODE system to bound the distributional error.\n\nIn summary, the core contribution lies in the theoretical unification of high-order methods and the establishment of a rigorous statistical foundation. Our framework provides a basis for understanding practical benefits such as improved sampling efficiency and stability, as discussed in Section 5.\n\n[1] On statistical rates of conditional diffusion transformers: Approximation, estimation, and minimax optimality, ICLR, 2025\n\n---\n### **W2.** The main claim of the paper is that it extends flow matching to $K$-th order derivatives. If this is the central contribution, then the paper should demonstrate—either theoretically or empirically—why this extension is meaningful. In its current form, there are no examples, experiments, or concrete settings showing how or why using higher-order flow matching improves performance over standard flow matching. I strongly encourage the authors to include empirical evidence or clearer theoretical justification that illustrates the practical or conceptual benefits of the proposed higher-order estimation.\n\n---\n\n**Response:** Thank you for your comments. We appreciate the opportunity to clarify our contributions and the significance of our work. Our paper presents a unified theoretical framework for flow-based generative modeling that incorporates trajectory derivatives up to an arbitrary order K. \n\nThis work provides a rigorous theoretical foundation for a line of research that has already shown significant empirical success . While the initial submission focused on theory, we have, in response to reviewer feedback, **conducted new experiments** on synthetic data that directly address this point. We kindly invite you to view the results of these experiments, which are detailed in our **response to reviewer hems W1** (we can not show results here due to the character limits.\n\nThe new results demonstrate that the second-order method consistently outperforms the first-order method on all tested distributions, often achieving better performance with significantly fewer sampling steps. This directly supports the theoretical justifications for practical benefits discussed in our paper. For instance, our conclusion discusses how a K-th order Taylor expansion for sampling leads to a local truncation error of  $O(h^{K+1})$ per step, a significant advantage over standard first-order methods. We believe these findings, combined with our rigorous statistical analysis, bridge the gap between empirical success and the lack of a comprehensive theoretical explanation for this class of models.\n\n\n---\n### **Q1.** The paper includes a helpful summary of its contributions in lines 42–59, which makes it easier to follow. However, it remains unclear what exactly the authors improve upon relative to prior work. In particular, I could not find a clear explanation of why one should use high-order flow matching, or what statistical advantage it offers compared to standard flow matching. Overall, Section 4 reads more like a collection of general statements rather than providing concrete justification for the proposed extension. Could the authors please clarify these points and better articulate the practical or theoretical benefits of the high-order formulation?\n\n---\n\n**Response:** Thank you for your questions. We appreciate the opportunity to clarify our contributions and the benefits of our framework.\n\nFirst, to address the core justification for our framework, its value lies in three key areas:\n1.  **Unification and Theoretical Foundation:** Our work introduces a generalized theoretical framework for flow-based generative modeling that incorporates trajectory derivatives up to an arbitrary order $K$. This framework provides a unified perspective and formal grounding for emerging high-order models that have demonstrated empirical success, such as FlowMP ($K=3$) and HOMO ($K=2$).\n2.  **Sampling Efficiency:** As discussed in Section 5, our framework enables the use of a $K$-th order Taylor expansion sampler. This sampler achieves a local truncation error of $O(h^{K+1})$ per step, a significant improvement over the $O(h^2)$ error of standard first-order methods.\n3.  **Error Stability:** The $K$-order flow matching approach solves the ODE without a feedback loop, which might lead to more stable error propagation compared to the sequential nature of standard methods like Runge-Kutta, where errors can propagate and amplify within a single step.\n\nSecond, we acknowledge your point regarding Section 4 and apologize for any confusion caused. The statistical analysis presented is not a simple application of prior work but rather a **rigorous, tailored analysis** for our high-order framework. For instance:\n* **Velocity Approximation (Theorem 4.1):** Our analysis involves a **fine-grained error analysis** that accounts for the specific high-order velocity decomposition and the Gaussian tail bounds of the data distribution. Our key technical innovation is **omitting the unnecessary local diffusion polynomial** in approximation theory compared to [1].\n* **Estimation and Near-Minimax (Theorems 4.2, 4.3, 4.4):** Our derivation for the 2-Wasserstein distance is a novel application for this context. We apply the **Alekseev-Gröbner Lemma** to the high-order ODE system to bound the distributional error, which is distinct from methods used in prior work. Furthermore, our minimax optimality theorem is also novel, as it establishes **near-minimax optimal rates** on Hölder densities, using a distinct minimax lemma (Lemma N.2).\n\nWe will add a dedicated section in the revised manuscript to more clearly emphasize these distinctions and the significance of our contributions."}, {"review_id": "igOWBhHcis", "reviewer": "Reviewer_icvX", "summary": "The paper introduces High-Order Flow Matching (HOFM), a generalized theoretical framework that extends standard flow matching to incorporate trajectory derivatives up to arbitrary order $K$. To this end, the authors work with a $K$-order velocity field that captures higher-order temporal dynamics and develop a high-order marginalization technique to convert the intractable $K$-order loss into a tractable conditional regression. They claim to establish statistical rates for K-order flow matching with transformer networks, asserting $O(n^{-\\Theta(1/d)})$ convergence rates that match minimax lower bounds up to logarithmic factors. The framework allegedly subsumes standard first-order flow matching ($K=1$) and claims to provide theoretical grounding for recent high-order approaches like FlowMP and HOMO. However, several of these claims are questionable due to unrealizable assumptions and missing critical implementation details.", "questions": "- How do you ensure the total derivative constraints (Equation 3.3) are satisfied during training? \n- Is there a regularization term or constraint optimization approach?\n- Can you provide a corrected version of Assumption 4.1 that doesn't contain contradictory bounds?\n- Why are there no experiments showing the practical benefits of higher-order matching compared to standard flow matching?\n- How does the computational cost scale with order $K$, and when does the higher accuracy justify the increased complexity?\n- What happens when the learned velocity fields violate the consistency constraints. Does the method still converge? Does it introduces biases?", "limitations": "- The framework requires K-times differentiable flows, which may be restrictive in practice\n- There is no discussion of computational complexity scaling with order $K$\n- The statistical rates do n0t show significant improvement with increasing $K$, questioning the practical value\n- Missing analysis of stability and error propagation in high-order numerical integration schemes\n- The paper does n0t address how to choose the optimal order $K$ for a given problem\n- No comparison with existing high-order methods in terms of computational efficiency", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Potential Strengths** (if correct):\n\n- Claims to provide a unified theoretical framework that generalizes standard flow matching to arbitrary order K\n- Attempts to establish rigorous statistical analysis with sharp approximation, estimation, and distribution learning rates\n- Claims to show near-minimax optimality of the proposed estimators\n- Attempts to connect various existing high-order flow methods under a single theoretical umbrella.\n\nHowever, the validity of these contributions is questionable given the reliance on unrealizable assumptions (e.g., Assumption 4.1) and the extremely long appendix that makes verification of proofs difficult.\n\n**Weaknesses:**\n\n*Critical consistency issue:* The higher-order velocity fields must satisfy total derivative constraints (Equation 3.3), but the paper fails to explain how this consistency is enforced in practice or during training\n\n*Flawed assumptions:* Assumption 4.1 contains impossible bounds where $C < q(x_1) ≤ C_1\\exp(-C_2|x_1|^2/2)$, which would require the lower bound $C$ to be less than the upper bound for all $x_1$, creating mathematical inconsistencies.\n\n*Lack of empirical validation:* No numerical experiments are provided to validate the theoretical claims or demonstrate practical benefits\n\n*Missing credit:* The paper fails to acknowledge stochastic interpolants alongside flow matching and rectified flows, despite these being essentially equivalent approaches proposed simultaneously.\n\n*Unclear practical implementation:* The paper doesn't address how to ensure the learned velocity fields maintain the required derivative relationships during optimization", "was_revised": true, "final_justification": "The paper mostly focuses on theoretical results and it would benefit from experiments: without them, it is difficult to assess the practical impact of these results. In particular, I remain concerned that  the total derivative constraints (Equation 3.3) is not necessarily satisfied during training. As a result, the theoretical benefit  of the method may not be achieved in practice, since the method is not $K$-order accurate if the aforementioned constraint is not enforced. This paper would benefit from a resubmission.", "rebuttal": "Thanks for your detailed review. We have addressed all concerns below, and made revisions accordingly in our latest draft.\n\n---\n### **W1.** *Critical consistency issue...*\n---\n\n**Response:** Thanks for the comment. We agree that a thorough explanation of how to enforce the total derivative constraints in practice would be a valuable addition. However, we’d like to remind the reviewer that our work is a **theoretical paper**. It’s about:\n1.  A unified framework for a family of empirically strong high-order FM models (as cited and discussed in our main text) and,\n2.  Under this framework, establishing **nearly‑optimal statistical rates** of these models assuming Eqn (3.3),\n\nwhich Theorem 3.1 shows is both necessary and sufficient for a $K$‑times–differentiable flow, as it guarantees the equivalence between flows and $K$-order velocity fields.\n\nBecause the paper proposes no new training algorithm or model, we do not prescribe a particular enforcement mechanism. Nevertheless, here we sketch existing practice:\n- **Soft penalties** (e.g. HOMO, Chen et al. 2025) add a regularization loss term $|D_t v - u_2|^2$ to enforce chain-rule consistency between the predicted velocity and acceleration.\n- **Shared‑network differentiation** (e.g. similar to Consistency Flow Matching [2]) enforces Eq. 3.3 exactly by automatic differentiation.\n\nOur theory formally identifies why such mechanisms are needed and quantifies what happens if Eq. 3.3 fails (we will add a new Remark 4.2 to strengthen this). Given the length and density of the current submission, we believe detailing engineering choices is therefore outside this paper’s scope, but our results provide the rigorous foundation on which future practical work can build.\n\n[1]  arXiv:2502.00688, 2025.\n\n[2]  arXiv:2407.02398, 2024.\n\n---\n### **W2.** *Flawed assumptions...*\n---\n\n**Response:** Thank you for your careful review and for identifying this critical typo in Assumption 4.1. We apologize for the confusion caused. You are correct that the lower bound $C$ creates a mathematical inconsistency. This was a typo. **The lower bound is not required for our analysis.**  We have corrected this typo in the latest revision.\n\n---\n### **W3.** *Lack of empirical validation...*\n\n---\n\n**Response:** Thank you for your valuable feedback. We appreciate your suggestion to include empirical validation.\n\nIn response to this invaluable suggestion, we have **conducted new experiments** on synthetic data that directly address this point (Please see our **response to W1 of reviewer hems**. We found that the second-order method **consistently outperforms** the first-order method across three different synthetic distributions. For example, our results show that the second-order method can achieve the performance of a first-order method in significantly fewer steps, empirically validating our theoretical discussions on improved sampling efficiency.\n\nThese findings, combined with our rigorous statistical analysis, bridge the gap between empirical success and the lack of a comprehensive theoretical explanation for this class of models. We will include these new experimental results in the revised manuscript to strengthen our paper's overall argument.\n\n---\n### **W4.** *Missing credit...*\n---\n\n**Response:** Thank you for your valuable feedback. We appreciate you pointing this out. In response, we have revised the introduction and extended related work sections to include these prior works as follows:\n\n> Flow Matching (FM) learns the drift of a continuous normalizing flow by regressing it to the probability current of a deterministic interpolant between a Gaussian base and the data, achieving strong likelihoods and fast ODE sampling [1]. Rectified Flow (RF) independently arrives at an equivalent quadratic objective but chooses the straight-line displacement path, enabling one-step generation with near–state-of-the-art FID [2]. The stochastic-interpolant (SI) programme constructs a finite-time stochastic bridge whose drift minimises the same quadratic loss: an initial two-author version introduces SI for normalizing flows [3], and a subsequent, expanded formulation unifies flows and diffusions and proves that the deterministic, zero-noise limit recovers RF exactly [4]. These works therefore constitute simultaneous, conceptually equivalent instantiations of the same continuity-equation objective, differing only in the choice (deterministic vs. stochastic, straight-line vs. diffusion) of the interpolant path. Empirically, FM, RF and SI each report competitive or superior ImageNet/CIFAR-10 sample quality while requiring orders-of-magnitude fewer ODE steps than diffusion baselines, reinforcing their practical equivalence.\n\n[1]  arXiv:2210.02747, 2022.\n\n[2] Xingchao Liu, et al. “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.” ICLR 2023.\n\n[3]  arXiv:2209.15571, 2022.\n\n[4] arXiv:2303.08797, 2023.\n\n---\n### **W5.** *Unclear practical implementation...*\n---\n**Response:** Thank you for your comments. In practice, these constraints can be enforced through various methods. For instance, in models like High-Order Matching for One-Step Shortcut Diffusion (HOMO), which our framework subsumes, a regularization term is included to enforce chain-rule consistency between predicted velocity and acceleration. This regularization provides a practical way to align with our total derivative constraint. **One of our contributions is to provide theoretical results to understand why such a constraint is necessary, but we do not focus on a specific method to enforce it.**\n\n---\n### **Q1 & Q2.** *How do you ensure the total derivative constraints (Equation 3.3) are satisfied during training? Is there a regularization term or constraint optimization approach?*\n---\n**Response:** Thanks for the question. Please see our response to **W1** above.\n\n---\n### **Q3.** *Can you provide a corrected version of Assumption 4.1...*\n---\n**Response:** Yes, we have removed the lower bound assumption in the revised version. Here is the corrected version:\n\n**Assumption (Sub-Gaussian Property and Hölder Smoothness of Target Distribution)**\n\nThe target distribution $q(x_1) \\in \\mathcal{H}^\\beta(\\mathbb{R}^{d_x}, B)$. \nFurther, there exist two positive constants $C_1$ and $C_2$ such that\n$q(x_1) \\leq C_1 \\exp( -C_{2}\\| x_1 \\|_2^2/2 )$\n\n---\n### **Q4.** *Why are there no experiments showing...*\n\n---\n**Response:** *Thanks for the question. Please see our response to **W3** above.*\n\n---\n### **Q5.** *How does the computational cost scale...*\n---\n**Response:** Thank you for your insightful questions. We believe the time complexity would not necessarily scale linearly with $K$ because the training and inference can be **implemented in parallel**. For example, the $K$-th order Taylor expansion sampler allows all $K$ velocity components to be evaluated simultaneously.\n\n---\n### **Q6.** *What happens if violating (3.3)?*\n---\n\n**Response:** Thank you for your question. The total derivative constraints (Equation 3.3) are crucial for ensuring the statistical consistency (i.e., learning the right distribution) of our framework. According to Theorem 3.1, these constraints are **necessary for the existence of a unique local solution** to the governing ODE. Therefore, violating these constraints would lead to inconsistent estimation. In practice, this would likely prevent the method from converging to the correct target distribution or introduce a significant bias in the estimated distribution.\n\n---\n### **L1-6.** *The framework requires $K$-times differentiable flows, which may be restrictive...*\n---\n**Response:** Thank you for your insightful comments. \n\nWe kindly remind reviewers that **we do not introduce any new method or model**. Our framework is built to provide a **rigorous theoretical understanding** for existing high-order flow models like HOMO and ForM [1,2], which have already shown significant empirical performance over first-order FMs. Although a discussion about how to get $K$-times differentiable flows in practice is beyond the scope of the current paper, we have addressed this in the response to W1.\n\nSimilarly, our goal is not to conduct empirical comparisons or analyze computational complexity, but to provide the theoretical tools that enable such studies. For instance, we provide a **preliminary discussion** on the potential for improved sampling efficiency through parallel evaluation of velocity components and more stable error propagation, which are important directions for future work. Our work establishes the theoretical foundation for these models, and the empirical validation of these properties is an important direction for future work.\n\n[1]  arXiv:2502.00688, 2025.\n\n[2]  arXiv:2502.08150, 2025.\n\n---\n### **L3.** *The statistical rates do not show significant improvement...*\n---\n**Response:** Thanks for the question, but we believe there is a potential oversight. As we discussed in Section 5, we acknowledge that the estimation rates do not show a significant improvement compared to standard flow matching. However, our paper provides **theoretical justifications for the practical benefits** of higher-order dynamics beyond statistical rates. As discussed in our conclusion, using a $K$-th order Taylor expansion for sampling leads to a local truncation error of $O(h^{K+1})$ per step, a significant advantage over standard first-order methods. We also present that our method might lead to more stable error propagation compared to sequential methods like Runge-Kutta.\n\n---\n### **L5.** *The paper does not address how to choose the optimal ...*\n---\n\n**Response:** To the best of our knowledge, the design choice of $K$ really depends on the use case. There is no optimal value of $K$ in generic settings. It is possible to make strong assumptions on data (e.g., a Mixture of Gaussians) and derive the corresponding optimal choice of $K$. However, we believe this is beyond the scope of the current paper."}, {"review_id": "M8NXhtDMhE", "reviewer": "Reviewer_MiuU", "summary": "The paper introduces a theoretical framework for high-order flow matching: models that extend standard flow matching by incorporating higher-order trajectory information. Given recent empirical work on more expressive continuous-time flows, the authors propose a general method that models derivatives of the flow up to an arbitrary order. They formulate a marginalisation technique that transforms the otherwise intractable higher order loss into a tractable conditional regression objective with exact gradients, generalising the flow matching marginalisation technique to higher velocity fields. They also derive sharp statistical convergence rates for high-order flow matching when implemented with transformer architectures by generalising flow matching bounds to an arbitrary order velocity. In addition to the technical contributions, the paper and supplementary material serve as a comprehensive reference on the theory of flow matching. It consolidates existing results on convergence and extends them into a high-order setting, making it a valuable resource for researchers.", "questions": "Could other architectures (e.g., MLPs, CNNs) could potentially achieve similar rates? to what extent your results depend on the specific architecture class?\n\nThe theoretical rates themselves do not currently improve with increasing order. Are there any particular settings where you suspect the theoretical gap might close? besides errors in numerical methods.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality:\nThe paper is technically strong and builds on foundational work in flow-based generative modelling. The extension to high-order flow matching addresses practical needs in current generative models for greater expressiveness and stability. The high-order marginalisation trick is an elegant formulation which follows the flow matching version in the first order. The statistical analysis is rigorous and thorough. It will be interesting (in further research) to see empirical studies on how increasing the order affects performance or training stability.\n\nClarity:\nThe paper is clearly written and structured. The progression from standard flow matching to its high-order generalisation is easy to follow. Some parts of the statistical results in Section 4 are quite dense and could be more accessible with illustrations, like plots visualising bounds in with respect to order or feature dimension.\n\nSignificance:\nThis work can serve as a central theoretical reference in the flow matching literature. Even without experimental validation, the theoretical results are crucial for researchers developing or analysing continuous-time generative models. It provides a clear direction for further work on scalable training and evaluation of high-order flow models. The focus on transformer architectures and to provide detailed statistical rates for this specific architecture is significant.\n\nOriginality:\nThe paper brings together existing ideas in a novel way. While at least two high-order flow models have been proposed recently, this is the first work to systematically formalise the theory behind them and relate them to a general framework for higher order flow matching. The high-order marginalisation technique is an original contribution that improves the applicability of these models in the future.", "was_revised": true, "final_justification": "I maintain that the paper presents a technically sound and original approach that is well grounded in prior work. Thus, I believe my current score remains appropriate.", "rebuttal": "Thanks for your detailed review. We have addressed all concerns below, and made revisions accordingly in our latest draft.\n\n---\n### **W1.** The paper is clearly written, but some parts of the statistical results in Section 4 are quite dense and could be more accessible with illustrations, like plots visualizing bounds with respect to order or feature dimension.\n\n---\n\n**Response:** Thank you for your careful review and for your feedback on the clarity of our statistical results. We agree that due to page limitations, the content in Section 4 may appear dense. We appreciate your suggestions for making it more accessible. In the revised version, we are glad to **add figures to visualize the statistical rates** with respect to order and feature dimension. Furthermore, we will **rewrite Section 4 to include more detailed explanations** of each theorem and to better articulate the relationships between our results, thereby improving readability.\n\n---\n### **Q1.** Could other architectures (e.g., MLPs, CNNs) potentially achieve similar rates? To what extent do your results depend on the specific architecture class?\n\n---\n\n**Response:** Thank you for your questions. We believe that **similar statistical rates could be achieved with other architectures**, such as MLPs or CNNs, provided that they have a sufficiently powerful universal approximation theory. Our results are grounded in the specific **universal approximation theory for transformers**, as detailed in Appendix H.\n\nThe specific rates and dependencies would change based on the particular **approximation capabilities of the chosen architecture**, but the general principle of deriving statistical rates from such a foundation remains the same.\n\n---\n### **Q2.** The theoretical rates themselves do not currently improve with increasing order. Are there any particular settings where you suspect the theoretical gap might close, besides errors in numerical methods?\n\n---\n\n**Response:** Thank you for this insightful question. You have correctly identified a **key limitation** of our current framework, as discussed in Section 5.\n\nThe question of how to close this theoretical gap in statistical rates is an **intriguing open question** that we are actively exploring and have left for future work. We believe the **main advantage of high-order flow matching is decreasing the numerical errors**."}, {"review_id": "8lcPKT1HR5", "reviewer": "Reviewer_23Kf", "summary": "This paper construct a theoretically unified k-order flow matching framework which has several high-order FMs and original FM as its special cases. This paper further establish the sample-complexity bound of k-order FM under the usage of transformers.", "questions": "-\tCan the analysis of the transformer in the paper be directly applied to the U-Net and Dit? These two models are commonly employed in current SOTA models like FLUX or SD-3.5.\n-\tThe other one of my concerns is about the understanding of the choice of k. To illustrate the practical usage or limit of the high-order FMs proposed in the paper, it may be beneficial to train several high-order FMs with different k on synthetic data like a 2d chessboard to see the ability of high-order FMs in various choices of k. What will be the best choice of k in practice, the larger the better, or is there a best one?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n \n- The theoretical derivations and the formulation of this paper are rigorous from a mathematical perspective.\n- The paper is well written, with a clear and logical structure that makes it accessible and easy for readers to follow.\n- It represents a significant milestone by being the first to construct a unifying framework for high-order Flow Matching (FM) methods. This framework encompasses several previous methods, such as the HOMO approach proposed in reference [1], as special cases. Such a unifying structure is expected to substantially deepen our comprehension of high-order generative models.\n \nWeaknesses:\n \n- The analysis of transformers appears to be somewhat restricted. The paper directly incorporates the complexity results of transformers from prior studies into its own analysis without further in-depth exploration. Conducting a more in-depth PAC learning theory analysis specifically related to high-order FM would greatly enhance the theoretical depth and comprehensiveness of the study.\n- While this paper makes important contributions to the theoretical understanding of FMs, its practical implications are somewhat limited. The stability of the model when the order k exceeds 3 cannot be guaranteed. As pointed out in reference [2], a severe oscillation phenomenon will emerge when higher-order flow matching models are employed.\n \n[1] Chen B, Gong C, Li X, et al. High-Order Matching for One-Step Shortcut Diffusion Models[J]. arXiv preprint arXiv:2502.00688, 2025.\n\n[2] Barrett J H. Oscillation theory of ordinary linear differential equations[J]. Advances in Mathematics, 1969, 3(4): 415 - 509.", "was_revised": true, "final_justification": "after the rebuttal, my opinion is not changed.", "rebuttal": "Thanks for your detailed review. We have addressed all concerns below, and made revisions accordingly in our latest draft.\n\n---\n### **W1.** The analysis of transformers appears to be somewhat restricted. The paper directly incorporates the complexity results of transformers from prior studies without further in-depth exploration. Conducting a more in-depth PAC learning theory analysis specifically related to high-order FM would greatly enhance the theoretical depth and comprehensiveness of the study.\n\n---\n\n**Response:** Thank you for the suggestion. Thank you for the suggestion. However, we believe it’s a bit beyond the scope of the current paper. To be concrete, can you kindly elaborate more on what you mean by “a more in-depth PAC learning theory analysis”?\n\nOur paper already presents a **unified theoretical framework** with established statistical optimality, including theoretical results on:\n1.  Transformer approximation\n2.  Flow function estimation\n3.  Distribution estimation\n\nIn particular, our analysis for (2) and (3) is already connected to **standard (covering number, norm-based) generalization bound learning theory analysis**. Given the length and density of this submission, we leave a more specific PAC-style investigation for future work.\n\n---\n### **W2.** While this paper makes important contributions to the theoretical understanding of FMs, its practical implications are somewhat limited. The stability of the model when the order $k$ exceeds 3 cannot be guaranteed. As pointed out in reference [2], a severe oscillation phenomenon will emerge when higher-order flow matching models are employed.\n\n---\n\n**Response:** We appreciate the reviewer’s observation. **Classical oscillation theory** for differential equations shows that higher-order systems often exhibit oscillatory behavior, which can manifest as instabilities in our high-order flow matching (FM) models. In line with this, we acknowledge that simply increasing the order beyond $k=3$ can lead to **diminishing returns and even severe oscillations** in practice. The immediate practical utility of very high-order ($k>3$) FM models is indeed limited by these stability challenges.\n\nImportantly, **our theoretical framework and results hold for arbitrary order $k$**, independent of these practical stability limitations. All derivations and performance guarantees in the paper are formulated in general terms, so the core theory remains valid regardless of whether $k=4$ or higher models can be stably trained. This **does not constrain our theoretical contributions**.\n\n[1] Chen B, et al. High-Order Matching for One-Step Shortcut Diffusion Models. arXiv:2502.00688, 2025.\n\n[2] Barrett J H. Oscillation theory of ordinary linear differential equations. Advances in Mathematics, 1969, 3(4): 415-509.\n\n---\n### **Q1.** Can the analysis of the transformer in the paper be directly applied to the U-Net and DiT? These two models are commonly employed in current SOTA models like FLUX or SD-3.5.\n\n---\n\n**Response:** Thank you for your question. Yes, the analysis of the transformer in our work **can be directly applied** to models like DiT and those based on the U-Net architecture. Specifically, our transformer architecture, defined in Appendix B, **follows the common architecture of Diffusion Transformers (DiTs)**.\n\n---\n### **Q2.** The other one of my concerns is about the understanding of the choice of $k$. To illustrate the practical usage or limit of the high-order FMs proposed in the paper, it may be beneficial to train several high-order FMs with different $k$ on synthetic data like a 2d chessboard to see the ability of high-order FMs in various choices of $k$. What will be the best choice of $k$ in practice, the larger the better, or is there a best one?\n\n---\n\n**Response:** Thank you for your question. To the best of our knowledge, there is **no single optimal value of $K$** in a general setting. The choice of $K$ depends on the specific use case, data, and computational resources available. Our framework, which handles an arbitrary order $K$, provides a **theoretical foundation** for models that have demonstrated empirical success with specific orders, such as **HOMO ($K=2$) and ForM ($K=2$)** [3,4].\n\nWe agree that training on synthetic data with varying $K$ would be beneficial for understanding the practical trade-offs. However, exploring the empirical question of an optimal $K$ is an intriguing open question that we have left for future work, as our paper is **primarily focused on establishing the theoretical and statistical properties** of the generalized framework.\n\n[3] Bo Chen, et al. High-order matching for one-step shortcut diffusion models. arXiv:2502.00688, 2025.\n\n[4] Yang Cao, et al. Force matching with relativistic constraints... arXiv:2502.08150, 2025."}]

Author rebuttal:

Meta-review: This paper presents an interesting theoretical generalization of flow matching to incorporate higher order derivatives of the flow field.  It includes an analysis of the statistical efficiency of the method but did not originally include any practical implementation. During discussion, the authors reported new experimental demonstrations, validating that the new methods have practical potential.

The authors should be sure to include the newly reported experimental results and other clarifications in the final version.
Decision: Accept (poster)