Stream-level Flow Matching with Gaussian Processes
Authors: Ganchao Wei
Li Ma
Venue: ICML 2025 poster (2025)
Keywords: Generative Model
Normalizing Flows
Flow Matching
Gaussian process
Primary area: deep_learning->algorithms

Abstract: Flow matching (FM) is a family of training algorithms for fitting continuous normalizing flows (CNFs). Conditional flow matching (CFM) exploits the fact that the marginal vector field of a CNF can be learned by fitting least-squares regression to the conditional vector field specified given one or both ends of the flow path. In this paper, we extend the CFM algorithm by defining conditional probability paths along "streams'', instances of latent stochastic paths that connect data pairs of source and target, which are modeled with Gaussian process (GP) distributions. The unique distributional properties of GPs help preserve the ``simulation-free'' nature of CFM training. We show that this generalization of the CFM can effectively reduce the variance in the estimated marginal vector field at a moderate computational cost, 
thereby improving the quality of the generated samples under common metrics. Additionally, adopting the GP on the streams allows for flexibly linking multiple correlated training data points (e.g., time series). We empirically validate our claim through both simulations and applications to image and neural time series data.

Stream-level Flow Matching with Gaussian Processes
Ganchao Wei Li Ma'
Abstract
objective and proposed the flow matching (FM) algorithm. FM exploits the fact that the marginal vector field inducing desired CNF can be learned through a regression formu- lation, approximating per-sample conditional vector fields using smoother such deep neural network (Lipman et al., 2023). In the original FM approach, the training objective is conditioned on samples from the target distri- bution, and the source distribution has to be Gaussian. This limitation was later relaxed, allowing the target distribution to be supported on manifolds (Chen Lipman, 2024) and the source distribution be Gau-Gan (Pooladian et al., 2023). Tong al. (2024) provided unifying framework with arbitrary transport maps by conditioning both ends. While their framework is general, its application requires the induced conditional probability paths to be readily sam- pled from, and as such, they considered several Gaussian probability paths. Moreover, most existing FM methods only consider the inclusion of endpoints, and hence cannot accommodate data involving multiple correlated ob- servations, such time series and data with grouping structure. Notably, Albergo et al. (2024) recently proposed multimarginal stochastic interpolants, which aim to learn a multivariate distribution based on correlated observations.
Flow matching (FM) is family of training algo- rithms for fitting continuous normalizing flows (CNFs). Conditional flow matching (CFM) ex- ploits the fact that the marginal vector field of a CNF can be learned by fitting least-squares re- gression to the conditional vector field specified given one or both ends of the flow path. In this paper, we extend the CFM algorithm by defining conditional probability paths along "streams", in- stances of latent stochastic paths that connect data pairs of source and target, which are modeled with Gaussian process (GP) distributions. The unique distributional properties of GPs help pre- serve the "simulation-frre" nature of CFM train- ing. We show that this generalization of the CFM can effectively reduce the variance in the esti- mated marginal vector field at a moderate compu- tational cost, thereby improving the quality of the generated samples under common metrics. Addi- tionally, adopting the GP the streams allows for flexibly linking multiple correlated training data points (c.g., time series). We empirically validate our claim through both simulations and applications to image and neural time series data.
In this paper, we go one level deeper in Bayesian hierar- chical modeling FM algorithm and specify distributional assumptions streams, which are latent stochastic paths connecting the two endpoints. Our approach extends the stochastic interpolant framework proposed by Albergo & Vanden-Eijnder (2023); Albergo et al. (2023), and it leads to a class of CFM algorithms that condition at the "stream" " level, which broadens the range conditional probability paths allowed in CFM training. By endowing the streams with Gaussian process (GP) distributions, these GP-CFM al- gorithms provide a smoother marginal vector field and wider sampling coverage over its support. Furthermore, condition- ing on GP streams allows for flexible integration of corre- lated observations through placing them along the streams between two endpoints and for incorporating additional prior information, while maintaining the analytical tractabil- ity and computational efficiency of CFM algorithms.
1. Introduction
Deep generative models aim to estimate and sample from unknown probability distribution. Continuous normalizing flows (CNFs, Chen et al. (2018)) construct invertible and differrentablec mapping, using neural ordinary differential equations (ODEs), between a source and the target distri- bution. However, traditionally, has been difficult to scale CNF training large datasets (Chen et al., 2018; Grath- wohl et al., 2019; Onken al., 2021). Recently, Lipman et al. (2023); Albergo & Vanden- Mijnnen (2023); Liu et al. (2023) showed that CNFs can be trained via a regression
'Department of Statistical Science, Duke University, Durham, Ganchao Wei <gan-
NC 27708, USA. Correspondence chao. wei wwi@@uke.edu>> Li Ma <li.ma@dukc.eduu
In summary, the main contributions of this paper are as follows.
Proceedings of the n International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We generalize CFM training by augmenting the spec-

---

Stream-level Flow Matching with Gaussian Processes
ification of conditional probability paths through la- tent variable modeling on the streams. We show that streams endowed with GP distributions lead to sim- ple stream-level CFM algorithm that preserves the "simulation-free" training.
conditional distribution of (x) given z. Here, a is the conditioning latent variable, which can be the target sample 21 (e.g., Ho et al. (2020); Song al. (2021b); Lipman et al. (2023),) or pair of (1) on source and target distribu- tion (e.g., Liu et al. (2023); Tong et al. (2024)). Specifically, Tong et al. (2024), generalizing the result from Lipman et al. (2023), showed that
2. We demonstrate that appropriately specified GP streams can lead to smoother marginal vector fields and reduced variance in marginal vector estimation, and thereby generate higher quality samples.
= = f (z] Pr p((a)2)q(2) dz = E (uG(=)2)2z = =x), (c(2)
ux(x)
3. We show that the GP-based stream-level FM can read- ily accommodate correlated observations. This allows FM training to borrow information across training sam- ples, thereby improving the marginal vector field es- timation and enhancing the quality of the generated samples. Our approach offers additional flexibility to accommodate designs that are challenging for existing approaches, such as allowing correlated observations to be observed along irregularly spaced time points.
where the expectation is taken over z, which one can rec- ognize is the conditional expectation of un (a(2) conditional on the event that x, = z. The integral is with respect to the conditional distribution of 2 given I2 = I.
The FM algorithm is motivated by the goal of approximat- ing the marginal vector field (x) through a smoother of (typically a neural network), via the objective
(F) = (... (z) (1())
4. These benefits are illustrated by simulations and appli- cations to image (CIFAR-10 10, MNIST and HWD+) and neural time series data (LFP), with code for Python im- plementation available at ht ht //ggggtttiss. com/ Baiigedsb/GP-CFV
In the following, we follow carlier works and assume t ~ (o, 1) though the algorithms discussed are valid for other sampling distributions of l as well. The FM objective not identifiable due to the non-unique non-uniqueness of the marginal vector fields that satisfy the boundary conditions without further constraints. FM algorithms address this by fitting of to the conditional vector field we (1rlz) after further specifying the distribution of g(z) along with the conditional probability path z). through minimizing the finite-sample version of the marginal squared error loss. The corresponding loss is referred to as the conditional flow matching (CFM) objective
2. Background and Notation
We start by reviewing the necessary background and defin- ing the notation for flow matching (FM). At the end of this section, we briefly present Bayesian geeision-theored perspective on FM training, providing an additional justifi- cation for FM algorithms beyond gradient matching (more details in Appendix A).
CCPM (0)
Consider i.i.d. training observations from an unknown pop- ulation distribution Q1 over Rd A CNF is time-dependent diffeomorphic map &t that transforms a random variable zo E R2d from a source distribution go into a random variable from q1- The CNF induces a distribution of 3y = =0 (2)) at each time I, which is denoted by Pt. thereby forming a probability path (p: 0 0 t S 1). This probabil- ity path should (at least approximately) satisfy the boundary conditions Po go and p1 q1. is related to the flow map through the offange-f--vriablle formula or the fosh-forward equation
Traditionally, optimizing the CFM objective is justified be- cause it has the same gradients w.r.t. 0 to the corresponding FM loss (Lipman et al., 2023; Tong et al., 2024). In Ap- pendix A we detail another justification for CFM without involving the gradient-matching argument. In particular, we view this algorithm from a Bayesian estimation perspective and show that approximating the conditional vector field by minimizing the marginal squared error loss (CFM) can be interpreted as computing the "posterior expectation" of uy (z|2) under a like-likel setup. This is the Bayes rule under square error loss and exactly equal to un (1)..
Pt- [dx] [.0]
FM aims at learning the corresponding vector field u, (2:), which induces the probability path over time by satisfying the continuity equation (Villani, 2008).
This Bayesian estimation justification holds for any coher- ently specified probability model q(2). So long as the condi- tional probability path (<(a=])) is tractable, a suitable CFM algorithm can be designed. Therefore, one can enrich the specification of g(2) using Bayesian latent variable model- ing strategies. This motivates us to generalize CFM training to the stream level, which we describe in the next section.
The key observation underlying FM algorithms is that the vector field u, (z) can be written as a conditional expectation involving a conditional vector field u, (2]2), which induces a conditional probability path pe (-12) corresponding to the
2

---

Stream-level Flow Matching with Gaussian Processes
3. Stream-level Flow Matching
A general stream-level CFM loss for learning iy (3)) is then
3.1. A Per-stream Perspective on Flow Matching
L.CrM (0) = Rr,, (l.. (st) - ur (ar [|] Ee (s+)
&e]
A stream 8 is a stochastic process 8 = {st 0 0 1}, where each 5y a random variable in the sample space of the training data. We focus on streams connecting one end To in the source to the other xi in the target. From here on, will take the place of the latent quantity z.
where the integration over s is with respect to the marginal distribution of 8 induced by x (1o, 21) and pa (-12r, 2 As in previous research such as Lipman et al. (2023); Tong et al. (2024), we can show that the gradient of LaCTM equals that
of LFM with details of the proof in Appendix J.2. However, stream-level CFM can be justified from a Bayesian decision- theoretic perspective without gradient matching, For more details, see Appendix A.
Instead of defining a conditional probability path and vec- tor field given one endpoint at t = 1 (Lipman et al., 2023) or two endpoints at & 0 and (Tong et al., 2024), we shall consider defining it given the whole stream connect- ing the two ends. To achieve this, we need specify a probability model for This can be separated into two parts-thee the marginal model on the endpoints π T< v1) and the conditional model for s given the two ends. That is
3.2. Choice of the Stream Model
Next, we specify the conditional model for the stream given the endpoints (=(- 21). This model should emit streams differentiable with respect to time, with readily available velocity (either analytically or easily computable). Previous methods such as optimal transport (OT) conditional path/ linear interpolant (Liu et al., 2023; Lipman et al., 2023; Tong et al., 2024) achieve high sampling efficiency but can provide rather poor coverage of the (t, z) space, resulting in extensive extrapolation of the estimated vector field uf (a:). Furthermore, Albergo et al. (2023) demonstrated that the stochastic interpolant suppresses spurious intermediate fea- tures, thereby smoothing both the path and the vector field, Consequently, introducing stochasticity to the path simpli- fies the estimation and accelerates sample generation. Thus, it is desirable to consider stochastic models for streams that ensure smoothness while allowing streams to diverge and provide more spread-out coverage of the (t,z) space.
(20,21) and
s So = To. $1 I1 Pa 21).
(1)
Our model and algorithm will generally apply any choice of coupling distribution that satisfies the boundary con- dition, including, for example, the popular OT-CFM and Schrödinger bridge renutopy-rggullriiedd-CCCCCCC CFMM, considered in Tong et al. (2024). We defer the description of the spe- cific choices of (-)(ar,,2) .1) to the next section and for now focus on describing the general framework.
Given stream 8, the "per-stream" vector field (x(sss) rep- resents the "velloity" (or derivative) of the stream at time 1, conditional on the event that 8 = x, i.e., the stream 8 passes through x at time t. Assuming that the stream is differentiable within time, the per-stream vector field is
To preserve the simulation-fra free nature of CFM, we con- sider models where the joint distribution of the stream and its velocity is available in closed form. In particular, we model the streams using Gaussian processes (GPs). A de- sirable property of GP is that its velocity is also GP, with mean and covariance directly derived from the original GP (Rasmussen & Williams, 2005). This enables efficient joint sampling of (st, &t) given observations from a GP in stream-level CFM training. By adjusting covariance kemels for the joint GP, one can fine-tune the variance level to con- trol the level of regularization, thereby further improving the estimation of the marginal vector field (xx) (Section 4.1). The prior path constraints can also be incorporated into the kernel design. Additionally, a GP conditioning on the event that the stream passes through a finite number of intermediate locations between two endpoints again leads to a GP with analytic mean and covariance kernel (Section 4.2). This is particularly useful for incorporating multiple correlated observations.
uy (1) s)= == 8 = dss/dt,
which is defined only on all pairs of (1) that satisfy = x. See Appendix B for a more detailed discussion on how the per-stream perspective relates to the per-sample perspective on FM.
While the endpoint of the stream 81 = 2i is an actual observation in the training data, for the task of learning the marginal vector field un (zx), one can think of our "data" as the event that a stream 8 passes through a point x at time t, that is s x. Under the squared error loss, the Bayes estimate for the per-stream conditional vector field ux (x)())) will be the "postcrior" expectation given the "data", which exactly the marginal vector field
(x) = E(ur(c))))s = x) = -E(Se/s = r).
(2)
Following Theorem 3.1 in Tong et al. (2024), we can show that the marginal vector (x) indeed generates the proba- bility path Pr (x). (See the proof in the Appendix J.1.) The essence of the proof is to check the continuity equation for the (degenerate) conditional probability path Pt (a|s).
Specifically, given M time points t = (t,, 12, (M) with t1 = 0 and tu = 1, we let st = (St, Sty..... Stn ) and consider a more general conditional model for pa St =
3

---

Stream-level Flow Matching with Gaussian Processes
Algorithm I Gaussian Process Conditional Flow Matching (GP-CFM)
the following, we set E(1) = 0 and use squared exponential (SE) kernel for c11 for each dimension (may be with addi- tional terms such as in Figure 2). The details of SE kernel can be found in Appendix E.
Input: observation distribution π (Ltab)) initial net- work v", and GP defining the conditional distribution (5t,8) | |st obss ~ ( Ex), for t E [0, 1].
Probability paths with time-varying variance, such as those considered in Song & Ermon (2019); Ho et al. (2020); Lipman et al. (2023), also motivate the adoption of non- stationary GPs whose covariance kernel could vary over t. For example, to encourage samples that display larger deviation from those in the training set (and hence more regularization), one could consider using a kernel producing larger variance as t approaches the end with finite training samples (Figures 2 and 8). Moreover, because the GP model for 8 is specified given the two endpoints, both its mean and covariance kernel can be specified as functions of (ro, ). For example, if 21 is outlier of the training data, e.g., from a tail region of (1, then one may incorporate a more variable covariance kernel for Pa (= x1) to account for the uncertainty in the optimal transport path from xo to 2
Output: fitted vector field vg (a))
while Training do
Bobs ~ T (Zoba) ~ (0,1)
(st, 8c) 8t- = Bohs ~ (ir En) L.CFM (0) + [1u - scll2 0 s update (0, (OL.CFM(O)
end while
J. where Tobs = (Itt Pty...., ) are a set of "ob- served values" that we require the statistic process 8 to pass through at time (t1, .t,,t,,,.t... Note that this con- tains the special case of conditioning on two endpoints (i.e., M- 2) described in Section 3.1. We consider more general construction for M 2 2 because later we will use this to incorporate multiple correlated observations (such as time series or other measurements from the same subject).
4. Numerical Experiments
In this section, we demonstrate the benefits of GP stream models through several simulation examples. Specifically, we show that using GP stream models can improve the generated sample quality at a moderate cost of training time, by appropriately specifying the GP prior variance to reduce the sampling variance of the estimated vector field. Moreover, the GP stream model makes easy to integrate multiple correlated observations along the time scale.
We construct a conditional GP for & that satisfies the bound- ary conditions, with a differentiable mean function m and covariance kernel k11- Since the derivative of GP is also GP, the joint distribution of s and the corresponding velocity process & := (& f E [0, 1}} given 8 is also a GP, with the mean function for á being mn()= = dm(t)/ /tt and ker- nels defined by derivatives of 11 (Rasmussen & Williams, 2005). To facilitate the construction of this GP, we consider an auxiliary GP on s with a differentiable mean function E and covariance kernel C11- Using the property that the conditional distribution of Gaussian remains Gaussian, we can obtain joint GP model on (s, $) St, which satisfies the boundary conditions. For computational efficiency and ease of implementation, we assume independence of the GP across dimensions of 8. Notably, while we are modeling streams conditionally given s as a GP, the marginal (i.e., unconditional) distribution of at all time points is allowed to be non- non-Gaasi,nn, which is necessary for satisfying the boundary condition and for the needed flexibility to model complex distributions. The detailed derivation can be found in Appendix C, and the training algorithm for GP-CFM is summarized in Algorithm 1.
4.1. Adjusting GP Variance Improve Sample Quality
We first show that one can improve the estimation of un (x) by incorporating additional stochasticity in the sampling of streams with appropriate GP kernels. A similar observation has been made by Albergo et al. (2023), who also explained that this is because the stochastic interpolant can smooth the conditional probability path and suppress spurious in- termediate modes. This applies to our GP-stream algorithm as well. See Appendix for an illustration of the smoother conditional probability paths produced by our approach.
We have found it helpful to understand this phenomenon from the perspective of variance reduction in the estimation of the marginal vector field. As illustrated in Figure IA, for estimating 2-Gaussian mixtures from standard Gaussian noise, the straight conditional stream used I-CFM covers a relatively narrow region (gray). For points outside the searching region, there are no "data" and the neural network of (z) must be extrapolated during sample generation. This is form of overfitting, which causes highly variable esti- mates of the vector field in the extrapolated regions and can lead to potential "leaky" or outlying samples that are far
Several conditional probability paths considered in previous works are special cases of the general GP representation. For example, if we set mtt)= = f.² + (1 t)æo (therefore, (i((() 21 - zo) and kn (1) = o² Ta. the path reduces to the OT conditional path used in I-CFM with constant variance (Tong et al., 2024). The I-CFM path can also be induced by conditional GP construction (Appendix C) using a linear kernel for C11, with more details in Appendix D. In
4

---

Stream-level Flow Matching with Gaussian Processes
from the training observations.
Table I. Comparison of linear and GP streams Consider gener- ating 2- Gaussiian target from standard Gaussian. Here, we train models with I-CFM, GP-I-CFM. OT-CFM and GP-OT-CFM 100 times and calculate 2- E-Wassesttiin (W2) distance between gener- ated and test samples. Results of 100 seeds are summarized by mean and standard error.
In constructing the GP streams, we condition on the end- points but expand the coverage region (red) by tweaking the kernel function (e.g., decrease the SE bandwidth in this case). This provides a layer of regularization that protects against extrapolation errors. As illustrated in Equation I, the OT strategy for endpoints coupling (Tong et al., 2024) can be complementary to our GP-stream method enhance per- formance. Therefore, we train the CNFs via four algorithms, i.e., I-CFM, GP-I-CFM, OT-CFM and GP-OT-CFM (OT for endpoints coupling and GP for stream model), 100 times using a 2-hidden layer multi-layer perceptron (MLP) with 100 training samples at t = 1, and calculate 2- Wasserstein (W2) distance between generated and test samples. For fair comparisons, we set a 0 for linear interpolant (I-CFM and OT-CFM), and use noise-free GP streams (GP-I-CFM and GP-OT-CFM). The results are summarized in Table 1. Empirically, the models trained by GP- CFM have smaller W2 distance than the corresponding linear inter- polant (i.e., GP-I-CFM vs. I-CFM and GP-OT-CEM vs. OT- CFM), and the model trained by combining two strategies (GP-OT-CFM, OT for coupling and GP for stream) performs best. We further generate 1000 samples and streams for I- CFM and GP-I-CFM with the largest W2 distance in Figure IB, starting with the same points from standard Gaussian. In this example, several outliers are generated from I-CFM.
MODELS
MEAN
SE
I-CFM
1.54
0.08
GP-I-CFM
1.51
0.08
OT-CFM
1.43
0.05
GP-OT-CFM
1.35
0.05
product kernel to the SE kernel. Specifically, denote the kernel for auxiliary GP on s in dimension as cu, for + all! for increas- ing variance and (t, c ci (a(tt-)))) decreasing variance, where (t,t") [0, and (t,
== I....,,, d. Let (t, = (I,
for
o² exp tails.) Some examples of the streams connecting two end- points under different variance schemes are shown in Fig- ure 2. We train the models 100 times and calculate the 2- Wasserstein (W2) distance between generated and test samples, and the results are summarized in Table 2. In this example, with infinite samples at t = 0 but 100 samples at
(t-t")² (See Appendix C for additional de- 21d
= L, injecting noise at f = 0 worsens estimation. However, when approaching the target distribution (t = 1), adding noise can improve estimation with small samples (100). The noise perturbs the limited data, encouraging broader explo- ration and adding regularization reduce estimation error. In addition to using standard Gaussian source, we further consider the transformation between two 2-Gaussian mix- tures with finite samples (100) at both ends. Results are shown in Appendix G. In this scenario. injecting noise near either endpoint improves estimation.
A
B
I-CFM trainingi generated
GP--CFM
potential "losky" sample
stream
1.FM region
GP-LGFM
region
Figure 1. GP streams reduce extrapolation by expanding cov- erage area. Generate samples of 2-Gaussian mixture from the standard Gaussian. Training observations are shown in red, gen- crated samples in orange, and noise source samples black. A. FM with straight conditional stream (e.g., I-CFM) may generate "leaky" or outlier samples due to extrapolation errors. The FM method with GP conditional stream has broader coverage area. B. We train models with I-CFM and GP-I-CFM 100 times and calculate 2- Wasserstein (W2) distance. Among these 100 trained models, generate 1000 samples (orange) and streams (blue) for I-CFM and GP-I-CFM with largest W2 distance (worst case).
No Noise
Const. Noise
Decr. Noise
Incr. Noise
1.5
0.5
-0.5
0.0
0.5
1.0
Figure 2. Change variance over time by tweaking the covari- ance kernel. Examples of conditional stream between two points, under different variance change scheme.
We can further modify the GP variance function over time to efficiently improve sample quality. Here, we consider the task of estimating and sampling a 2-Gaussian mix- ture from the standard Gaussian, with 100 training sam- ples at = 1. For constant noise, diagonal white noise is added to perturb stream locations while retaining the SE kernel. For varying noise, we add a non-stationary dot
4.2. Incorporating Multiple Correlated Training Observations
Besides generally improving FM-based fitting of a marginal vector field, an additional benefit of GP streams is that they
5

---

Stream-level Flow Matching with Gaussian Processes
easy solution is to further condition the neural net (1r covariate (subject label) c, so that the optimization ob- jective is LeCFM where q(s c) represents the distribution of s given The covariate-dependeent FM (guided-FM) algorithm has been proposed in Isobe et al. (2024); Zheng et al. (2023), and the validity of approximating the covariate-dependent vector field using the above optimization objective in our stream- level CFM is shown in the Appendix J.3. In this example, similar subjects have close starting points at / = 0, and we let c 20. By conditioning on (covariate model), the neural net is separated for different subjects, and hence the training of the vector field will not be confused (Figure 4C).
Table Comparison of different variance schemes of GP-1- CFM Reconsider the generation of 2-Gaussian target from stan- dard Gaussian. We train models under each variance scheme 100 times and calculate 2-Wasserstein (W2) distance for each. The results of 100 seeds are summarized by mean and standard error.
(...)) (l c) 3rll².
VARIANCE SCHEME
MEAN
SE
NO NOISE
0.269
0,013
CONSTANT NOISE
0.305
0,011
DECREASING NOISE
0.329
0.013
INCREASING NOISE
0.243
0.012
enable the flexible inclusion of multiple correlated observa- tions the training data, such as in time series. Correlations between training observations allow information sharing and can improve estimation at each time point.
A
B
Data
GP-I-CFM (no cov)
GP-I-CMM (cox.
We first illustrate the main idea through toy example. Con- sider 100 paired observations and place the two observations in each pair at t = 0.5 and t = 1, respectively (Figure 3 A) while t = O corresponds to the standard Gaussian source. Here, we show the generated samples (at / = 0.5 and 1) and the corresponding streams for GP-I-CFM and I-CFM. Again, 2-hidden layer MLP is used in this case. The CFM strategy employs two separate models with I-CFM algorithms (Figure 3B), whereas GP-I-CFM offers a single unifying model for all observations, resulting in a smooth stream across all time points (Figure 3C).
Iso Siter
10
0
-5
5
Figure 4. Further conditioning the starting points helps with stream generation. A. Paired data with observations on three time points: = 0 (black), = 0.5 (red) and 1 (orange). The two stream cross regions are marked with light blue square. B. The generated samples and streams for GP-I-CEM (without covariate), where the initial points at f = 0 are generated from noise using a separate I-CFM. C. The generated samples and streams for GP- I-CFM with covariate using the same starting points, where the neural network is further conditioning on data at 0.
A
B
C
Data
I-CFM
GP-I-CFM
25
15
5. Applications
t stream
1-05
We apply our GP-based CFM methods two hand-written image datasets (MNIST and HWD+), CIFAR-10 dataset, and mouse brain local field potential (LFP) dataset to illustrate how GP-hased algorithms 1) reduce sampling vari- ance (MNIST and CIFAR-I0) and 2) flexibly incorporate multiple correlated observations and generate smooth trans- formations across different time points (HWD+ and L.P dataset). The reported running times for the experiments are obtained on server configured with 2 CPUs, 24 GB RAM, and 2 RTXA5000 GPUs.
181
-5
-10
0
Figure 3. GP streams accommodate correlated points flexibly. A. Paired data with observations on t = 0.5 (red) and t= 1 (orange). B. The generated samples (red for 1 = 0.5 and orange for t = 1) and streams (blue) for I-CFMs. The I-CFMs contain two separate models trained by I-CFM, = 0 (standard Gaussian noise) to 0.5 and t= 0.5 to 1. C. The generated samples for GP-I-CFM.
In some cases, the GP streams may not be well separated, and thus may confuse the training of the vector field at crossing points. In Figure 4, we show a time series dataset over 3 time points, where training data at & = 0 and t are on one horizontal side while points at L = 0.5 are on the opposite side (Figure 4A). Therefore, these streams have two crossing regions (marked with blue boxes in Fig- ure 4A), where the training of the vector field is deterio- rated when simply using the GP-I-CFM (Figure 4B). An
5.1. Variance Reduction
We explore the empirical benefits of variance reduction using FM with GP conditional streams MNIST (Deng, 2012) and CIFAR-10 (Krizhevsky, 2009) databases. Here, we consider four algorithms in the MNIST application: two linear stream models (I-CFM, OT-CFM) and two GP stream models (GP-I-CMM, GP-OT-CEM). The I-CFM and GP-I- CFM are implemented for the CIFAR-10 example.
6

---

Stream-level Flow Matching with Gaussian Processes
In the MNIST application, we set 0 for linear stream models and use noise-free GP stream models for fair compar- isons. U-Nets (Ronneberger et al., 2015; Nichol & Dhariwal, 2021) with 32 channels and I residual block are used for all models. It takes around 50s, 51s, 52s, and 53s for I-CFM, OT-I-CFM, GP-I-CFM, and GP-OT-CFM to pass through all training dataset once for model training. To evaluate how much the GP stream-level CFM can further improve the estimation, we train each algorithm 100 times, and calculate the kernel inception distance (KID) (Bińkkwski et al., 2018) and Fréchet inception distance (FID) (Heusel et al., 2017) for each. The histograms in Figure 5 show the distribution of these 100 KIDs and FIDs, with results summarized in Table According to KID and FID, the independent sampling al- gorithms (l-alorrthmms) are comparable optimal transport sampling algorithms (OT-algorithms). However, algorithms using GP conditional stream exhibit lower standard error and fewer extreme values for KII) and FID, thereby reducing the occurrence of outlier samples (as in Figure 1).
and exhibit more details compared to those generated by the linear interpolant (I-CFM). The mean (with standard error) Fréchet inception distance (FID) (Heusel et al., 2017), calculated by the clean-fid -id library (Parmar et al., 2022) with 50,000 samples and the running time to generate 10 images 20 times using the DOPRIS solver, is reported in Table 4.
In terms of FID, GP-I-CFM is the best and significantly improves the I-CFM. In this case, the OT-CFM is com- parable to I-CFM, as observed in the MNIST application (Figure 5 and Table 3). It may suggest that the benefit of OT-CFM is less significant with increasing dimension, since in the high-dimensional case, minibatch OT approximation is poor for true OT and further adopting the GP stream strat- egy (GP-OT-CFM) does not remedy the issue. terms of sample generation time, using GP stream or OT coupling strategy leads to more computationally efficient model. However, combining these two strategies does not improve the efficiency of sample generation in this case.
5.2. Multiple Training Observations
KID
FID
81
Finally, we demonstrate how our GP stream-level CFM can flexibly incorporate correlated observations (between two endpoints at t = 0 and = 1) into single model and provide a smooth transformation across different time points, using the HWD+ dataset (Beaulac & Rosenthal, 2022) and LFP dataset (Steinmetz et al., 2019). The HWD+ example concerns transformations on artificial time, and the LFP dataset is time series data from the mouse brain. Here, we show results for the HWD+ dataset; refer to Appendix I for the LFP application.
-CFM
10
OT-CFM
6
GP4-GFM
QP.OT-OPM
4
2
0
0.005
0.015
42.5
47.5
Figure 5. Application to MNIST dataset. We compare the per- formance of four algorithms (I-CFM, OT-CFM, GP-I-CFM and GP-OT-CFM) on fitting MNIST datasct. We fit the models 100 times for each, and evaluate the quality of the samples by KID and FID. The figures above show the historams of KID and FID.
The HWD+ dataset contains images of handwritten digits along with writer IDs and characteristics, which are not available in the MNIST dataset used in Section 5.1. Here, we consider the task of transforming from "0" (at 0) to "8" (at t = 0.5), and then to "6" (at tt = 1). The inter- mediate image, "8", is placed at t = 0.5 (artificial time) for "symmetric" transformations. All three images have the same number of samples, totaling 1,358 samples (1,086 for training and 272 for testing) from 97 subjects. The U-Nets with 32 channels and I residual block are used. Both mod- els with and without covariates (using starting images, as in Figure 4C) are considered. Each model is trained both by I-CFM and GP-M-CFM. The I-CFM transformation contains two separate models trained by I-CFM ("0" to "8" and "g" to "6"). Noise-free GP-I-CFM and I-CFM with a = 0 are used for fair comparisons. In each training iteration, we randomly select samples within each writer to preserve the grouping structure of data. The runtime for all algorithms (I-CFM, GP-1-CFM and corresponding labeled versions) is similar, which takes 0.74s for passing all training data once. However, since I-CFMs fit two separate models, the running time is doubled.
Besides the MNIST dataset, to evaluate the performance in the high-dimensional image setting, we perform an experi- ment on unconditional CIFAR-10 generation (Krizhevsky, 2009) from a standard Gaussian source. We use a similar setup to that of Tong et al. (2024), such as time-dependent U-Net (Ronneberger et al., 2015; Nichol & Dhariwal, 2021) with 128 channels, learning rate of 2 10-1, clipping gradient norm to 1.0 and exponential moving average with decay of 0.9999. Again, four algorithms (I-CFM, OT- CFM. GP-I-CFM, and GP-OT-CFM) are implemented. We add diagonal white noise 10 into GP-stream models to prevent a potential singular GP covariance matrix, and set 10-3 in linear interpolations for fair comparisons. The models are trained for 400,000 epochs, with a batch size of 128. The linear interpolation (I-models) runs around 3.6 iterations per second, while GP-stream (GP-models) runs around 3,0 iterations per second. Figure 9A in Appendix H shows 64 generated images from four trained models, using a DOPRIS adaptive solver. Visually, images gener- ated by GP-stream (e.g., GP-I-CFM) are generally sharper
7

---

Stream-level Flow Matching with Gaussian Processes
Table Comparison of different algorithms for MNIST dataset Train models times using different algorithms. Calculate the mean and standard emror for KID and FID.
I-CFM
OT-CFM
GP-I-CFM
GP-OT-CFM
KID
MEAN
0.0040
0.0036
0.0032
0.0031
SE
0.0002
0.0002
0.0001
0.0001
FID
MEAN
44.50
44.21
43.55
42.99
SE
0.18
0.17
0.13
0.14
explicitly accounting for the grouping effect by conditioning on starting images (red) further improves performance.
Table 4. Comparison of different algorithms for CIFAR-10 dataset We fit models by I-CFM, OT-CFM, GP-I-CFM and GP- OT-CFM. For each trained model, we 1) calculate FID using 50000 samples 20 times and 2) generate 10 images 20 times. The means and standard errors of each summarized as follows.
FID to"0"
FID to"8
FID to"6"
MODELS
FID
SAMPLE GEN. TIME (S)
GPA
GP-+-DFM-o
300
MEAN
SE
MEAN
SE
200
200
200
I-CFM
3.75
0,006
1.30
0.01
160
OT-CFM
3.74
0.009
1.07
0.02
100
100
100
GP-I-CFM
3.62
0.008
1.27
0.02
GP-OT-CFM
3.75
0.009
1.20
0.01
0.0
Figure 6. Application to HWD+ dataset. We models for trans- forming "0" to "8" and then to "6", Both covariate and non- covariate (on starting images) models are considered, and cach model is fitted by both I-CFM and GP-I-CFM. The I-CFM trans- formation consists of two separate models trained by I-CFM ("0" to "8" and "8" to "6"). The figures above show corresponding FID to "0", "8" and "6" for these four trained models over time.
The traces for 10 generated samples from each algorithm are shown in Figure 9B in Appendix H, where the starting im- ages ("O" in the first rows) are generated by an I-CFM from standard Gaussian noise. Visually, the GP-based algorithms generate higher quality images and smoother transforma- tions compared to algorithms using linear conditional stream (I-CFM), highlighting the benefit of including correlations across different time points. Additionally, the transforma- tion generally looks smoother when the CFM training is further conditioned on the starting images.
6. Conclusion
We then quantify the performance of different algorithms by calculating the FID for "0", "8" and "6", and plot them over time for each (Figure 6). For FIDs, the GP-based algorithms (green& red) outperform their straight connec- tion (I-) counterparts (blue & orange). especially for the FID for "8" at t 0.5 and the FID to "6" at é = 1. This also holds for the FID for "0", as the GP-based algorithms are unified and the information is shared across all time points. This aligns with the observation by Albergo et al. (2024) that jointly learning multiple distributions better pre- serves the original image's characteristics during translation. However, for the I-algorithms, the conditional version (or- ange) performs worse than the unconditional one (blue), conditioning on the starting images makes the stream more separated, requiring more data to achieve comparable perfor- mance. In contrast, the data in GP- based algorithms is more efficiently utilized, as correlations across time points for the same subject are integrated into one model. Therefore,
We extend CFM algorithms using latent variable modeling. In particular, we adopt GP models on the latent streams and propose a class of CFM algorithms based on sampling along the streams. Our GP-stream algorithm preserves the simulation-free feature of CFM training by exploiting dis- tributional properties of GPs. Not only can our GP-based stream-level CFM reduce the variance in the estimated vec- tor field thereby improving the sample quality, but al- lows easy integration of multiple correlated observations to achieve borrowing of strength. The GP-CFM complemen- tary to and can be combined with modeling on the coupling of endpoints (e.g., OT-CFM).
A potential drawback of GP-CFM is the Monte Carlo error induced in the vector field estimation. This error can be mitigated through techniques such as importance sampling over time t, as in Song et al. (2021a) and antithetic sampling of GP trajectories (Botev & Ridder, 2017).
8

---

Stream-level Flow Matching with Gaussian Processes
Acknowledgments
Botev, Z. and Ridder, A. Variance Reduction, pp. 1-6, John
Wiley & Sons, Ltd, 2017. ISBN 978111844511 doi:
This research was supported by NIGMS grant ROI- GM135440 and NSF grant EEC-2133504.
URL httoss///oollineeiiierayyy. www.p//wwwww...............e..w. wtt.///oooccccco/////o// abs/10. abtp://...000/771111144522222251
Impact Statement
Chen, T. Q. and Lipman, Y. Flow matching on general geometries. In The Twelfth International Conference
This paper presents work whose goal is to advance the field of deep generative models. The proposed method can po- tentially help generate samples more efficiently with higher quality many (scientific) fields. There may be some po- tential societal consequences of our work, none of which feel must be specifically highlighted here.
on Learning Representations, 2024. URL https:// openreview .nt
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equa- tions. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URL https://procedinng neurips. cc o//saper_iileeeppprr////ff/i 6 1588fffbllofedd55522224228888883 pdf.
References
Albergo, M. S. and Vanden-Eijnden, Building normaliz- ing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL http:p::///peeeeeeeeeeeeeeeeee u id=1i7qeRbbR11t
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions, 2023. URL abs/2333. 08797.
Deng. L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, (996))1411-42 2012.
Grathwohl, W., Chen, R. T. Q. Bettencourt, J., and Du- venaud, D. Scalable reversible generative models with free-form continuous dynamics. In International Confer- ence Learning Representations, 2019. URL https: / //openreview. net
Albergo, M. S., Boffi, N. M., Lindsey, M., and Vanden-Eijnden, E. Multimarginal generative model- ing with stochastic interpolants. In The Twelsth In- ternational Conference on Learning Representations, http://wwwwwee///uuuutee..
2024. URL id=FHqAzW12ww.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31 International Conference on Neural Informa- Processing Systems, NIPS' 17, pp. 6629- 09946660, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Amaral, D. G., Scharfman, H. E.,, and Lavenex, The dentate gyrus: fundamental neuroanatomical organization (dentate gyrus for dummies). Progress in brain research, 163:3 22, 2007. ISSN 0079-6123 (Print). doi: 10.1016/ (0079-6123(00))6300
Beaulac, C. and Rosenthal, J. S. Introducing a New High-Resolution Handwritten Digits Data Set with Writer Characteristics. SN Computer Science, 4 (1):66, 2022. ISSN 2661-8907.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
doi: 10.1007/ 042979-022-01494-22 URL http:::/ddoiooooo 1007 020-9777--22---1--1 -94--2.
Isobe, N., Koyama, M., Zhang, J., Hayashi, K., and Fuku- mizu, K. Extended flow matching: method of con- ditional generation with generalized continuity equa- tion, 2024. URL https: 18839.
Berger, Statistical Decision Theory and Bayesian Analy- sis. Springer Series in Statistics. Springer, 1985. ISBN 9780387960982. URL https:/ /bwoks.o...//////o http://ooooso.l....... google. AC.
pettts://arxxee //txxvxxiv././///////aaa/aaaaava org/
Binkkwski, M., Sutherland, D. J., Arbel, M., and Gretton, A. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. URL https:// openrevieve wtt.eeeeeeeerttteeeeeeeee d=r1 1=r110020c10 100zWCW.
Krizhevsky, A. Learning multiple layers of fea- tures from tiny images. pp. 32-33, 2009. URL https: htt::///www.ccm www.ws.ttronto...oo..o toronto. www.ett..ooooooooootoooooeee eato:ddid//iir fearm

---

Stream-level Flow Matching with Gaussian Processes
Lipman, Y., Chen, R. T. Q.. Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference Learning Represen- tations, 2023. URL https: http://pppneeeeeeeeeeeee net/ forum? wwwpdpppaaaaat...
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021a. URL https: openreview nett/ forum? Peeiiidd=--i A.ddkkttttxnxss
Liu, Q. Rectified flow: A marginal preserving approach to optimal transport, 2022. wwtttttss//axiiii
Song, Y., Sohl-Dickstein, J.. Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score- generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021b. /ttp////tttttttttgg//e/eeeeeooie http///ppeeeeeeeeeeeeeeeeeeeeeep nwwp:/rruttttr? forum? id=PxTIG12RRES.
abs/2209, 14577.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023, URL https: BKKLhtts//pppereeeiii http///ppppee.e.eeeeeee/e..e..ep =V3TTInw5z.
Steinmetz, N. A., Zatka-Haas, P., Carandini, M., and Har- ris, K. D. Distributed coding of choice, action and en- gagement across the mouse brain. Nature, 576(7786);
Nichol, A. Q and Dhariwal, P. Improved denoising dif- fusion probabilistic models. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. 18-24 Jul 2021. URL htatttpss//prroceddigggo mlr.. press/v139 nichecccchla html.
266-273, 2019. ISSN 1476-4687. doi: 10.1038/ s41586-019-1787-x URL https: /tt::://doooooo org/ -19--11111111117-------88---11--
Tong, A., Fatras, K., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Wolf, G., and Bengio, Y. Improving and generalizing flow-based generative models with mini- batch optimal transport. Transactions Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: // openreview net / forum decDdddccD3naammmm Ex- pert Certification.
Onken, D., Fung, S. W. Li, X., and Ruthotto, L. OT-Flow: Fast and accurate continuous normalizing flows via optimal transport. In AAAI Conference on Artificial Intelligence, volume 35, pp. 9223-922 9232, May 2021. URL.htts:://ojst wwaiiorgge index. Chp/AAai AAAI/AAAAAAAA////// http//AAA/artilleevieee////
Villani, C. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008. ISBN 9783540710509. https://books: httpb/oooo.oog.l..o.o.ooooo.o... httpss//boooo.oogoolo.c/mooooooo com.bookside hv805R7_5tkk
Parmar, G., Zhang, R., and Zhu, J.-Y. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
URL
Pooladian, A.-A., Ben-Hamu, H., Domingo-Enrich, C., Amos, B., Lipman, Y., and Chen, R. T. Q. Multisample flow matching: straightening flows with minibatch cou- plings. In Proceedings of the 40th International Confer- ence on Machine Learning, ICML'23. JMLR.org, 2023.
Zheng, Q., Le, M., Shaul, N., Lipman, Y., Grover, A., and Chen, R. T. Q. Guided flows for generative modeling and decision making, 2023. URtps httpp::s///xxx...//////o// 0bs/2311.13443
Rasmussen, C. E. and Williams, C. K. I. Gaussian Pro- cesses for Machine Learning. The MIT Press, 11 2005. ISBN 9780262256834.
doi: 10. 00.755//mitpres//3206 001.0001. URL https: /dwi. org/10, 10. 7551/
mi mitpress /200/2226622 000100000011
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. F. (eds.), Medical Image Computing and Computer-Assisted
Intervention - MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319- 24574-4.
Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Curran Associates Inc.,
Red Hook, NY, USA, 2019.
10

---

Stream-level Flow Matching with Gaussian Processes
A. Bayesian Decision Theoretic Perspective on (stream-level) Flow Matching Training
is well-known in Bayesian decision theory (Berger, 1985) that under squared error loss, the Bayesian estimator, which minimizes both the posterior expected loss (which conditions on the data and integrates out the parameters) and the marginal loss (which integrates out both the parameters and the data), is exactly the posterior expectation of that parameter. This implies immediately that if one considers the conditional vector field uy (z/22) the target of "estimation", and the corresponding "data" being the event that i; zz i.e., that the path goes through at time I, then the corresponding Bayes estimate for (z(zz) will be exactly the marginal vector field uy(2), as it is now the "posterior mean" of uy (z)2z. We emphasize again that here the "data" differs from the actual training and the generated noise observations, which in fact help form the "prior" distribution. Therefore, the FM objective (CFM) defined in Section 2 provides a reasonable approximation to (x)).
In stream-level FM algorithm, because the (population-leve)) minimizer for the sCFM loss is ue minimizing the sCFM loss provides reasonable estimate for the marginal vector field uu (x)). To see this, rewrite the sCFM loss by the law of iterated expectation as
(0) = E, (Ilef (st) 1t)
The inner expectation can be further written in terms of another iterated expectation:
Ex (11)(())) 1t) = E,, E. (1le) 3, 1, 8t).
t, ( = z). whose minimizer is the conditional expectation of à, given 8t = x, which is exactly u4 (x:). Hence, one can estimate us (x) by minimizing (0). This justifies training (x) through the sCFM loss without regard to any specific optimization strategy.
For any z. Es (Ile( (s:) - -, = 2) = E. (11)(()) $e11
B. Discussion on Per-stream Perspective on Flow Matching
It is helpful to recognize the relationship between the per-stream vector field and the conditional vector field given one or both endpoints introduced previously in the literature. Specifically, the per-sample vector field in Lipman et al. (2023) corresponds to marginalizing out given the end point x1, that is, ur(x x1) = E (ur( 18) st = x, 81 x Similarly, the conditional vector field of Tong et al. (2024) corresponds to marginalizing out given both zo and ø1. that is (x( xo, ) = E (we (a: |s) 8t = J 80 = Jo, 81 = .1). Furthermore, when Ps(- ra, 21) is simply unit-point mass (Dirac) concentrated on the optimal transport (OT) path, i.e., a straight line that connects two endpoints To and 21, then (e( 8) (x mn) = (x( 21) for all (x) tuples that satisfy so = 2o, 81 , 8, = x. Intuitively, when the stream connecting two ends is unique, conditioning on the two ends is equivalent to conditioning on the corresponding stream s. In this case, our stream-leve FM algorithm (Section 3.2) coincides with those previous algorithms. More generally, however, this equivalence does not hold when (- (00, 21) is non-degenerate.
The per-stream view affords additional modeling flexibility and alleviates the practitioners from the burden of directly sampling from the conditional probability paths given one (Lipman et al., 2023) or both endpoints (Tong et al., 2024). While the per-stream vector field induces a degenerate unit-point mass conditional probability path, we will attain non- degenerate marginal and conditional probability paths that satisfy the boundary conditions after marginalizing out the streams. Sampling the streams in essence provides data-augmented Monte Carlo alternative to sampling directly from the conditional probability paths, which can then allow estimation of the marginal vector field in (z) when direct sampling from the conditional probability path is challenging. Additionally, as we will demonstrate later, by approaching FM at the stream level, one could more readily incorporate prior knowledge or other external features into the design of the stream distribution ( ro, 2.).
C. Derivation of joint conditional mean and covariance
For computational efficiency and ease of implementation, we assume independent GPs across dimensions and present the derivation dimension-ww wise throughout the Appendices. We use si to denote the location of stream s at time in dimension i, for = .....,d, d. Suppose each dimension of stream s follows a Gaussian process with a differentiable mean function g and covariance kernel d/1. Then, the joint distribution of si...... = (si,,..., y and $) $i....., = (s.,,..., st, y at g time points is

---

Stream-level Flow Matching with Gaussian Processes
(a.m.d ~ NV
((5) (Stammod E1
Ei En
(3)
Ej
T
where < (1), = dgg/dt, 5t Sti.... =
Sty Str
and covariance Ej is determined by
kernel c/p. The kernel function for the covariance between 8 and & in dimension i is d12 t') Bcj, (t.2") and the kernel
defining covariance of á is cº2 = Pei,(t,t" (Rasmussen & Williams (2005) Chapter 9.4). The conditional distribution of (s, s) in dimension given M observations s zbos also a (bivariate) Gaussian process. In particular, for E [0. 1]. let ii = (ci, ' and pobs = (Ei,,,..,, St,), the joint distribution is
Ster
(5 6/ trtaa) )' ~N[((
(
E DET t,obs
Loo )):
(orss)
Vobs
where 2j = (ov(s;, 5t) and Vins = Cov(arata) Accordingly, the conditional distribution (st, 8f) zo ~ Natt Et), where jij pt + 27 2o Voke -1 "xx Maba) and En 2 20,
-'ynT
f,obs
D. Optimal transport path from Conditional GP Construction
In this section, we show how to derive the path in I-CFM (Tong et al., 2024) from the conditional GP construction (Appendix C) using a linear kernel. Without loss of generality, we present the derivation of "noise-free" path with o² = - (i.e., the rectified flow, Liu et al. (2023); Liu (2022)).
Let zbbs
(#6 2tt, ş
+: and ci (t, = o of(t 1)(t' 1). such that
o + of
1)²
ob((-11))
oī 1))
oz
1)
0
2
2 L,obs
ot 1)
2
0
0 + oz a
1
1
-1
2%.
abs
2
-1
+
o
Therefore,
p Σi L.abs obx
52
1
1 - ( -1
1)
t
-(
t)aa
+ tz
x
x - 2
Σ Σ rhos obs
DiT r,obs
= O
E. Covariance under Squared Exponential kernel
Throughout this paper, we adopted the squared exponential (SE) kemel,, with the same hyper- syper-parammters for each dimension. The kernel defining block covariance for 8, (s, 8) and & in dimension i from Equation 3 are as follows:
(', exep (- [t(t))² t¹)2
(- (t-))²
(t-
de(l,)))
p (t- -1- (()exp
2/2
212
(- (1(--)) 17)2
(1(l,,t)= -c)2((,(()
(2(1,)) 74 [ - -(t - -t)- exp
212
F. GP stream produces a smoother probability path
The proposed GP-stream model can be considered as a more general framework than the stochastic interpolant (Albergo et al., 2023), which can produce a smoother path and suppress spurious intermediate modes. This also holds for our GP-stream model, and the smooth path makes it easier for vector field estimation and more computationally efficient numerical integration for sample generations. The running time for generating high-dimensional images, i.e., CIFAR-10. significantly shows the computational benefit (Table 4 in Section 5.1).
Here, we consider the transformation from a 2-Gaussian mixture to 3-Gaussian mixture. The models trained by four algorithms are compared: I-CFM, OT-CFM, GP-I-CFM, GP-OT-CFM. The generated paths and samples are shown in Figure Generally, either I-CFM or OT-CFM produces spurious modes, and the path not smooth.
12

---

Stream-level Flow Matching with Gaussian Processes
0.0
0.25
0.5
0.75
t= 1.0
I-CFM
50
OT-CFM
GP4-CFM
50
GP-OT-CFM
50
-2.5
0.0 2.5
-2.5 0.0 2.5
-2.5
0.0 2.5
-2.5 0.0
2.5
-2.5
0.0
2.5
Figure 7. GP streams provide smoother probability paths. Here, consider the transformation from 2-Gaussian mixture 3 3-Gaussian mixture. We fit models by four algorithms: I-CFM, OT-I-CFM, GP-1-CFM and GP-OT-CFM. The generated samples at t 0.25, 0.5, 0.75 and for each model visualize by histograms.
G. A Supplementary Example for Variance Changing over Time
Here, instead generating data from standard Gaussian noise, consider 100 training (unpaired) samples from Gaussian another 2-Gaussian (Figure 8A). The example streams connecting two points under different variance schemes shown in Figure 8B, again using additional nugget noise for constant noise, and dot product kernel for decreasing and increasing noise, described in Section 4.1. We then fit 100 independent models and calculate the W2 distance between generated and test samples at = 1. The results are summarized in Table 5. Now, since both ends have finite samples, injecting noise (a.k.a. adding regularization) both ends helps.
A
Training Data
B
No Noise
Const. Noise
Dec. Noise
Inc. Noise
11.2
4
-1.4
t=
2
t=
-1.6
-188
0
-2
0
2
0.0
0.5
1.0
Figure 8. Supplementary Example for Variance Change over Aime.A. The 100 observations in training data at = 0 and Examples of streams between two points, different variance change scheme.
1. B.
13

---

Stream-level Flow Matching with Gaussian Processes
Table Comparison of different variance schemes GP-1-CFM for 2-Gaussian to 2-Gaussian Train models 100 times and calculate 2- Wasserstein (W2) distance between generated and test samples for each. The results of these 100 seeds are summarized by mean and standard enror.
NO NOISE
CONSTANT NOISE
DECREASING NOISE
INCREASING NOISE
MEAN
0.681 0.044
0.350
0.413
0.411
SE
0.036
0.032
0.032
H. Image Generations
In this section, we show generated sample images for the CIFAR-10 and HWD+ dataset.
A
I-CFM
GP-I-CFM
B
I-CFM
GP-I-CFM
"O"
0.0
0
0
0
0
o
O
0
D
9
0
0
0
e
no COV.
4
P
8 d
9
"8"
0.5
8
8
3
r 8 8
8
8
8
8
6.
6
6
"6"
1.0
6
6
3
&
G
OT-CFM
GP-OT-CFM
"0"
0.0
0.25
8
COV.
"8"
8
8
t
0.75
8
g
6
8
8
E
6
"6"
C
6
Figure 9. Image Generation. A. 64 generated CIFAR-10 samples for I-CFM, GP-1-CFM, OT-CFM and GP-OT-CFM, starting from the same standard Gaussian samples. B. 10 HWD+ sample traces for four trained models. The starting images ("0's the first row) are generated by I-CFM from standard Gaussian noise, and all four trained models use the same starting images.
I. Application to LFP dataset
In this section, to illustrate the usage of proposed GP-CFM for time series data, we apply the labeled-GP-I-CFM to a session of local field potential (LFP) data from mouse brain. In the LFP dataset, the neural activity across multiple brain regions is recorded when the mice perform task choosing the side with the highest contrast for visual gratings. The data contains 39 sessions from 10 mice, and each session contains multiple trials. Time bins for all measurements are 10 ms, starting 500 ms before stimulus onset. Here, we study LFP from stimulus onset to 500ms after stimulus, and hence each trial contains data from 50 time points. See Steinmetz. et al. (2019) for more details on the LFP dataset.
Here, we choose recordings from a mouse in one session, where the trial repeated 214 times. For each single trial, the data contains a time series from brain regions. To illustrate the temporal smoothness over time in a visually significant way, we subset the data so that there are evenly-spaced time points. In summary, the training data have 214 observations and the dimension for each observation is 5 x 7. The observation time is scaled to [0, 1). Here, fit the data by covariate GP-I-CFM, using the starting point as covariates, and generate 1000 LFP time series for each region (the starting L.PP is generated from an I-CFM). For each second, the algorithm can run around 100 iterations per second (it runs around 2.5 iterations per second and takes longer time to converge when using all 50 time points). The results are shown in Figure 10. The generated time series can further be used to study neural activity in different brain regions. For example, the mean trajectories in Figure 10A suggest that the LFPs in Cornu Ammonis region 3 (CA3) and dentate gyrus (DG) are highly
14

---

Stream-level Flow Matching with Gaussian Processes
correlated, which is consistent with the experiment fact that the rat DG does not project any brain region other than the CA3 field of the hippocampus (Amaral et al., 2007). Besides this, and insightful conclusions. But this is beyond the scope of this paper.
the generated samples make more scientific
A
B
Mean
CA3
DG
100
ACA
MDs
DG
0
CA3
VISp
-5
-10
-100
0.00
0.01
0.02
0.03
0.04
0.00
0.01
0.02
0.03
0.04
Figure 10. Application to LFP data. We apply the GP-I-CEM with covariate (on starting point) to session of local field potential (LFP) data from 7 regions of mouse brain. In the training datasct, there 214 observations (repeated trials), For cach observation, it is time series data of 5 time points from brain regions. Here, we generate 1000 LFP time series for each region, where the starting LFP generated from I-CFM. A. The trajectories over 1000 samples. B. The generated 1000 time series for CA3 and DG.
J. Proof of propositions
In this section, we provide proofs for several propositions the main text. All these proofs are adapted from (Lipman et al., 2023; Tong et al., 2024).
J.1. Proof for conditional FM on stream
Proposition J.1. The marginal vector field over stream uxxx) generates the marginal probability path pe(x) from initial condition po(r).
Proof. Denote probability over stream as ((s) = Pa(s No. 21 (T) 2 21)d(au. x1) and p: |s) = 5(x - -s,), then
ti (x) = di f pi(x s)g(s)ds
d
d
Assume the regularity condition holds, such that we can exchange limit and integral (and differentiation and integral) by dominated convergence theorem (DCT). Therefore,
= =ʃ - (s)g(s)ss
pela
To handle the derivative on zero measure, define sy-centered Gaussian conditional path and corresponding flow map as
Pa.(21 8) == M(x s2. 2
Vo,( s) := a2 + St,
for 2 ~ No,, I), such that lim Po. ) = (x s). Then by Theorem of Lipman et al. (2023), the unique vector field defining vooa(2 (2 s) (and hence generating (a s)) is ug (z(s) = da, dt = u, (s; s), for all (t, z). Note that un (x 8) extends (x s) by defining on all æ, and they are equivalent when s; = æ. Since u, (- 8) generates Pa.l s),
15

---

Stream-level Flow Matching with Gaussian Processes
by continuity equation,
d di (i()) ff d lim Poa(# s)
= /
lim div(u) s)poa( ())((s)ds a-30
Then by DCT,
= lim div (ʃ uu(( s)pa,( 8))(s)ds
o-
= =diiv div (f ui((x) (1 8) lim Po,c(1 ( {){())ds
= -div (IE (ue( 8) ss p:)(()))
By definition in equation
= div (ur())p
which shows that p-()) and u.(-) satisfy the continuity equation, and hence ux(x) generates Pe (2:).
J.2. Proof for gradient equivalence stream
Recall
(FM = Et,ello (x) ue(2)]1 (C B. u (st) - u1(x s)1??
where x ~ pr(x). s q(s) and g(s) = J pa(s 21
Proposition J.2. VoLrm(() (oLocru(O).
Proof. To ensure existence of all integrals and to allow the changes of integral (Fubini's Theorem), we assume that g(s) are decreasing to zero at sufficient speed || > oc and that ue, Us Vou, are bounded. To facilitate proof writing, let pr(x 8) 0(x 8t).
The L-2 error in the expectation ca be re-written
|| (x) (=)]1 = (z)) 12 ((c](a),, ux(x)) || (s:) - (z s)(l = (>))2 + [1,(1 8) ux 8))
Thus, it's sufficient to prove the result by showing the expectations of terms including 0 are equivalent.
First,
Eg 10 (2) = / (=)
= - 110 (1)(()))Pp)(x s)g(s)dads
E, / [n((())))(( st)da
= (s))ll
16

---

Stream-level Flow Matching with Gaussian Processes
Second,
Ex (x), (x), ux(x))
ua(x))
/ (u)())) Sui( s)p((r ()q(s)ds
pp (x)dd
Pr(x)
/ (Vb(O), / ui(a s)p((< dss
/ (vg(()) (n(x 8))8((r (t)(())d)
E. (8)), ue(ar 8))
These two holds for all i, and hence VoLFM(() = (CC.
J.3. Proof for gradient equivalence conditioning on covariates
Let be response, be covariates, and be the stream connecting two endpoints (æo. Ø1). Given covariate c. denote the conditional distribution of s as ((s c) = ['px( 2o,41, T1, () x1 )d( and marginal conditional probability path as Pe(x c). Further,
LaM(()) (0) = Ex,el1y (z, ux(x e)]l? (O) Er,a 110 (st: ux |)]]
where x ~ (x((e and q(8
Proposition J.3. (oLeru(A) VoL.crm
Proof. To ensure existence of all integrals and to allow the changes of integral (Fubini's Theorem), assume that g(: ee) decreases to zero at sufficient speed as ||||| + and that ef. ue, Vorg are bounded. To facilitate proof writing, let (a(x s) = 6(z st).
The 1.-2 error in the expectations can be re-written as
(,, c) - ux(x c) = || (z, e) + [ 2))l2 c), ae c))
(s,, (x s) [1, e) + ||ua( s)] 2(0/ (st, c), ue(ar s))
Thus, it's sufficient to prove the result by showing the expectations of terms including 0 are equivalent.
First,
/ 111 (), cedd
(z. (2
= / / (o, c)llpp( s)q(s) c)dads
E. / [1, ()1,))))((( -t)ddr
= -E, (st, 2)12
17

---

Stream-level Flow Matching with Gaussian Processes
Second,
/ ()) c))p((( c)ddr
Ex (x,, ue(2 c))
/ (Vfle,c), [u(( s)p((z s)g(s c)ds (e c)dr
pr(2 lc)
I (V((ar))) f uo(or) s)pp(ar s)q($ | c))s)dx
/ / (o8(a,,), ux(x s))5(( st)((s} c)dsdr Eg (s) c), u1 s)) These two holds for all t, and hence (o VoL.crM(
18


Reviews:
[{"review_id": "JXhEobRRrK", "reviewer": "Reviewer_3C93", "summary": "The paper introduces stream-level flow matching with Gaussian processes (GP-CFM), which extends conditional flow matching to matching streams, i.e. latent stochastic paths that connect the source and target end points using Gaussian processes. The proposed framework naturally allows to include correlated observations (e.g. time series date) while remaining simulation-free, as the position and velocity can be readily sampled from the GP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot to the reviewer for their positive comments. Here, we clarified some specific points…\n\n1. > However, the authors do not directly access estimator variance and I don't see how the improvement in average Wasserstein-2 distance (or FID) can be directly attributed to a lower (per stream) estimator variance. Can you please clarify how the improvement in Wasserstein-2 distance (or FID, KID) relates to a reduction in estimator variance opposed to a possible reduction in bias?\n\nWe thank the reviewer for pointing this out, which has helped us clarify the sources of bias and variance in our algorithms.\n\nThere are two sources of variance: 1) systematic variance from extrapolation of the neural network, 2) Monte Carlo variance of vector field estimation. The GP-stream reduces the systematic variance by expanding the search region, but the stochastic interpolation introduces more Monte Carlo error. To reduce the Monte Carlo error, instead of sampling one GP path, we can sample multiple GP paths at the same time.\n\nTo validate the argument above, we did a quick experiment. We considered the target to be a 1D 2-Gaussian mixture, and drew 200 samples for training. We tried I-CFM, GP-I-CFM with 1 GP path (GP(1)-I-CFM) and 10 GP paths (GP(10)-I-CFM). We repeated the training 30 times and generated 10,000 samples for each. We smoothed the generated samples by Gaussian kernel density estimation (KDE). The means of the KDE match the true density well for all three algorithms. However, compared to I-CFM, the standard deviation of the KDE lines is lower in GP(1)-I-CFM (oscillates less around the true density), and GP(5)-I-CFM further reduces the standard deviation. To summarize results, we calculate mean of $|\\overline{kde}(x) - f(x)|$ and $s_{kde}(x)$, where $x$ are 1000 evenly spaced points over a $(-6, 6)$. Denote $\\overline{\\lvert \\overline{kde}(x_i) - f(x_i) \\rvert} = \\frac{1}{1000}\\sum_{i=1}^{1000} \\lvert \\overline{kde}(x_i) - f(x_i) \\rvert$ and $\\overline{s_{kde}(x_i)} = \\frac{1}{1000}\\sum_{i=1}^{1000} s_{kde}(x_i)$\n\n|             | $\\overline{\\lvert \\overline{kde}(x_i) - f(x_i) \\rvert}$ | $\\overline{s_{kde}(x_i)}$ |\n|-------------|-----|-----|\n| I-CFM       |   0.0035  |  0.0154   |\n| GP(1)-I-CFM    |   0.0036  |  0.0146   |\n| GP(5)-I-CFM|   0.0036  |  0.0136   |\n\nSince we are not allowed to update the manuscript or upload figures in the rebuttal, we will include more detailed experiments and discussions in the updated manuscript.\n\n2. > Did you derive the true posterior probability to report the results for the synthesis 2D 2-Gaussian mixture?\n\nNo, we didn't. For GP-I-CFM of 2D 2-Gaussian mixture, we can derive the marginal probability path. However, here we are considering the quality of generated samples, and checking the bias and variance of generated samples at $t=1$ plays the same role as checking the whole sample path from $t=0$ to $t=1$.\n\n3. > While broadening the coverage region reduces problems associated with extrapolation it intuitively also seems to increase the data demands to robustly learn a model covering the broader coverage region. Is this a potential concern for real application? Are there practical guidelines on how practitioners should choose/tune their covariance kernels to achieve a good amount of coverage while extending it too much.\n\nIf we sample 1 GP path for each iteration, it's a trade-off between systematic variance and Monte Carlo error (as mentioned above). When implementing GP-CFM, we currently manually choose the GP parameter so that the GP conditional path covers a slightly wider region than linear interpolation (by checking several paired samples from the target to the source).\n\nTo reduce the Monte Carlo error while preserving the reduction in systematic variance, instead of sampling one GP path, we can sample multiple GP paths or resort to importance sampling over $t$. We will add a discussion on tuning parameters in our camera-ready version if the paper is accepted.", "claims_and_evidence": "The claims and evidence, experiments, and arguments provided to validate the claims are generally sound (more details below). The authors demonstrate the workings of their method on synthetic data, which adds to the overall exposition, and evaluate its utility on multiple standard datasets. While none of the considered datasets are inherently high dimensional or used to compare state-of-the-art image generation with flows, it sufficiently demonstrates the utility of the proposed framework in multiple settings.\n\n**Variance reduction in the estimation of the marginal vector field leads to improved sample quality.**\n\nThe authors show that using the GP-stream variants of CFM results in improved sample quality in terms of lower average Wasserstein-2 distance (synthetic example), FID (MNIST, CIFAR10), and KID (MNIST). However, the authors do not directly access estimator variance and I don't see how the improvement in average Wasserstein-2 distance (or FID) can be directly attributed to a lower (per stream) estimator variance. Without further assumptions it could also be attributed to lower bias. If the assumption is that the estimator of the marginal vector field is unbiased, I can see how an improvement must be the result of a reduction in estimator variance. However, in practice we don't have access to the true posterior probability path required to estimate the marginal vector field (Equation 2 in the manuscript), but have to resort to the learned approximation, which leads to a biased estimator. \n\nCan the authors please clarify this point (see questions)?\n\n**GP stream variants can naturally accommodate correlated observations.**\n\nThe authors first demonstrate how GP-steam variants can leverage multiple training observations using 2d synthetic examples. They further show the utility of their framework in the time series setting on data from the LFP dataset and synthetic data, corresponding to modifying digits from HWD+. The latter shows significantly better FID scores, w.r.t the correct digit distributions, across time.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have not checked any proofs.", "experimental_designs_or_analyses": "I have read the experiment section and have validated the soundness of the experiments.", "relation_to_broader_scientific_literature": "The authors extend prior work from Lipman et al. [1] and Tong et al. [2], which specify conditional probability paths by defining a reference vector field given one or both endpoints respectively, by instead specifying a stochastic process (specifically a GP) that connects these endpoints.\n\nThe authors appropriately cite relevant work on conditional flow matching and cite Rasmussen & Williams [3] for their fundamental work on GPs. While not essential I think the authors should make an effort to also point out existing work that aims to directly model ODEs using GPs, and to discuss the conceptual differences to these approaches.\n\n[1] Lipman et al. Flow Matching for Generative Modeling. ICLR, 2023.\n[2] Tong et al. Improving and generalizing flow-based generative models with mini-batch optimal transport. TMLR, 2024\n[3] Rasmussen and Williams, Gaussian Processes for Machine Learning. MIT Press, 2005", "essential_references_not_discussed": "All essential references are included.", "other_strengths_and_weaknesses": "### Strengths\n\n- The paper presents a novel extension to Conditional Flow Matching, which is mathematically elegant and offers several advantages:\n- The approach provides a principled way to model the uncertainty in flow paths.\n- The GP formulation remains \"simulation-free\"\n- The ability to incorporate multiple correlated observations into a unified model\n- The framework is complementary to existing methods like OT-CFM.\n\n### Weaknesses\n\nThe computational overhead of the GP calculations and their implication should be discussed in more detail. While the authors mention \"moderate computational cost,\" a more detailed discussion of the computational complexity, especially in the case of high-dimensional data (without independence assumption) and multiple correlated observations, would be valuable.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Regarding the reduction in estimator variance\n- Can you please clarify how the improvement in Wasserstein-2 distance (or FID, KID) relates to a reduction in estimator variance opposed to a possible reduction in bias?\n- Did you derive the true posterior probability to report the results for the synthesis 2D 2-Gaussian mixture?\n2. While broadening the coverage region reduces problems associated with extrapolation it intuitively also seems to increase the data demands to robustly learn a model covering the broader coverage region. Is this a potential concern for real application? Are there practical guidelines on how practitioners should choose/tune their covariance kernels to achieve a good amount of coverage while extending it too much", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q87VRHpTis", "reviewer": "Reviewer_23pK", "summary": "This paper proposes a generalization of conditional flow matching (CFM) models using Gaussian process (GP) streams. While CFM uses two endpoints as condition, GP stream defines a GP over time that connects 2 or more points from $t=0$ to $t=1$, providing more controls over the mean and variance of the path (thus providing stronger regularization) and enabling time-series modeling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot to the reviewer for their positive comments, and thanks for suggestions on writing. We will update our manuscript accordingly. Here, we clarified some specific points…\n\n1. > Some aspects of the experimental setup are unclear.\n\nThe details of GP construction can be found in Appendix C and E.\n\nThe GP interpolant is different from stochastic interpolant referred here (linear interpolant with $\\sigma > 0$). To visualize the difference, please refer to Figure 2 and 8. The $\\sigma$ adds random jitters to the interpolation, but GP stream allows the interpolant oscillate smoothly. We can design $\\sigma(t)$ to further help the GP interpolant, as shown in the Figures.\n\nIn figure 7, we set $\\sigma=10^{-3}$ for all four algorithms.\n\n\n2. > Although the GP stream enables time-series modeling, an alternative approach is to treat it not as a time-series flow but as a joint distribution over multiple frames (e.g., as in video diffusion models). Is it possible to model videos as a time-series flow? Would this offer any actual advantages over the current paradigm? \n\nYes, we can model the time-series via joint distribution over multiple frames, but we may need to model in latent space and factorize the spatial and temporal components of model architecture. Otherwise, the dimension is too high to be feasible for applications such as video generation. Using the GP-CFM for time series modeling explicitly models the correlation over multiple correlated samples into one unifying model, without increasing the dimension linearly with number of time points, in the context of time series.\n\n3. > It seems to me that the additional conditioning (covariate) would introduce extra complexity, making it more similar to auto-regressive modeling. Speaking of auto-regressive modeling, would time-series modeling using a GP suffer from drifting (error accumulation)?\n\nIn our paper, adding additional covariates (labels) is related to AR modeling, but not exactly the same. Here, we use the starting point at $t = 0$ as covariates, which is static along the time. Therefore, there's no drifting issue. It's possible to use past lag-p observations  as covariates to be AR model, but this would make problems more complicated, and as mentioned by the reviewer, we need to concern about drifting issue.", "claims_and_evidence": "Yes. The claimed contributions appear are supported by the theoretical analysis and experiments.", "methods_and_evaluation_criteria": "The method is derived from a theoretical perspective and is a reasonable extension to flow matching. The scale of the experiments is small but reasonable for a theory-focused paper.", "theoretical_claims": "The theoretical claims appear correct. The proofs in the Appendix are adapted from established prior work.", "experimental_designs_or_analyses": "Some aspects of the experimental setup are unclear. In Section 4.1, how is the GP specifically constructed? Please provide an equation or reference the appendix. Is a GP stream connecting two endpoints the same as or different from stochastic interpolant ($\\sigma>0$)? Why would noise-free GP streams still provide stronger regularization (my understanding is that only the mean is altered)?\n\nIn Fig. 7, the comparison does not seem fair because I-CFM and GP-CFM appear noise-free, whereas their GP counterparts appear stochastic.", "relation_to_broader_scientific_literature": "The proposed method is a theoretical extension to CFM, with finer control over the flow paths. The finding that adding noise to the path improves sample quality due to stronger regularization and less over-fitting is already observed in prior work.", "essential_references_not_discussed": "Essential references have been discussed.", "other_strengths_and_weaknesses": "The paper is challenging to read as the writing is also unclear in some parts. For example, in Section 3.2, there is no formal definition of the Gaussian process in equation form in the main text. Although it is presented in Appendix C, it would be better to include a simplified equation in the main text (e.g., assuming a diagonal covariance) or at least provide a direct reference to the appendix. Additionally, several notations are not explained, such as $m(t)$ and $I_d$.", "other_comments_or_suggestions": "The y-axis label in Fig. 2 is missing, which should be $x$ I think.", "questions_for_authors": "Although the GP stream enables time-series modeling, an alternative approach is to treat it not as a time-series flow but as a joint distribution over multiple frames (e.g., as in video diffusion models). Is it possible to model videos as a time-series flow? Would this offer any actual advantages over the current paradigm? It seems to me that the additional conditioning (covariate) would introduce extra complexity, making it more similar to auto-regressive modeling. Speaking of auto-regressive modeling, would time-series modeling using a GP suffer from drifting (error accumulation)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhp4WKtH6I", "reviewer": "Reviewer_QyKi", "summary": "The paper proposes a novel flow matching method that incorporates **stochastic** bridges instead of **deterministic** bridges, which are typically used in the flow matching framework. In the context of generative modeling, flow matching (FM) is used to train neural ODEs with an initial distribution so that the distribution of their solutions matches a target (or terminal) distribution, commonly a data distribution. Specifically, FM enables the ODEs to learn and mimic the path measure (in a weak sense) defined by a collection of deterministic bridges, each connecting two points—one from the initial distribution and the other from the terminal distribution. A deterministic bridge is often chosen as a linear interpolation between two points over time. However, it is also possible to use a nonlinear deterministic bridge as long as it is path-wise continuously (time-)differentiable.\n\nUnlike the typical FM, this paper proposes using stochastic bridges instead of deterministic ones. Here, stochastic bridges mean that for any joint sample pair from the initial and terminal distributions, there can be multiple time-dependent functions—i.e., sample paths (or streams, as termed in the paper)—that connect the two points. In order to generate such stochastic bridges, the paper proposes using Gaussian measures, as their sample paths are path-wise continuously (time-)differentiable, and the time derivative of the sample path has a closed-form solution. In particular, this property of Gaussian measures naturally facilitates their use within the flow matching framework.\n\nIt is important to note that the proposed method differs from Bridge matching, which relies on Brownian bridges and other Itô diffusion-based bridges while still employing a Markovian projection-style training approach similar to FM. In particular, sample paths generated by Itô diffusion-based bridges may not be time-differentiable in the conventional sense, even though they remain continuous. Nevertheless, the paper theoretically demonstrates that incorporating such stochastic bridges into flow matching is effective.\n\nThe authors demonstrate the efficacy of the proposed method on several benchmark datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot to the reviewer for their positive comments, and thanks for suggestions on writing. We will update our manuscript accordingly, and add more details in the experiments whenever possible. Here, we clarified some specific points…\n\n1. > However, in comparison to this widely adopted approach, the motivation for introducing nonlinear bridges remains unconvincing despite its crucial role in the proposed method. Similarly, making the bridges stochastic does not seem to offer a clear advantage, and its justification is not well-supported.\n\nThe stochastic interpolant was previously studied theoretically in [1], where the benefits of stochastic interpolant was discussed in section 4.3 of their paper. In their paper, they demonstrated that the stochastic interpolant suppresses spurious intermediate, which smooth path and vector field (Figure 7 in our paper). Therefore, stochastic interpolant can simplify estimation and accelerate the ODE integration. \n\nIn our paper, we further extend the “stochastic interpolant” in [1], since the Brownian bridge used in [1] is a special case of a Gaussian process (GP), and therefore, conditioning on the entire stream and modeling it using a GP offers greater modeling flexibility. Furthermore, by resorting to properties of GP (conditional distribution and derivative of a GP are still GPs), we preserve the simulation-free property of FM algorithm. Besides, we further provide a bias-variance trade-off perspective for advantages of stochastic interpolation.\n\n\n2. > It could be helpful to explicitly clarify how this method differs from other Bridge matching approaches.\n\nThe connection to bridge (e.g.  Schr\\\"odinger bridge) and other related models are discussed in [1]. Specifically, in section 3.4, they showed that the stochastic interpolants can recover the Schr\\\"odinger bridge (from source to target densities), if we explicitly optimize over the interpolant. We will add more discussions on this in the updated manuscript.\n\n[1] Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, 2023", "claims_and_evidence": "Overall, I find the paper novel and original. However, the motivation for using nonlinear stochastic bridges in the flow matching context is not entirely convincing, both theoretically and empirically.\n\nFor example, the use of a linear map (often referred to as the condOT path) is straightforward, and the discussion relating its linearity to reducing numerical errors provides a reasonable justification—particularly in how it accelerates the solving of ODEs with fewer evaluation steps. However, in comparison to this widely adopted approach, the motivation for introducing nonlinear bridges remains unconvincing despite its crucial role in the proposed method. Similarly, making the bridges stochastic does not seem to offer a clear advantage, and its justification is not well-supported.\n\nIn this regard, the experimental results also appear somewhat limited. While the authors provide several comparative results on popular image generation datasets such as MNIST and CIFAR-10, the performance differences between the proposed method and existing approaches do not appear substantial. The reported improvements, if any, seem marginal, making it difficult to assess the practical advantages of using nonlinear and stochastic bridges. Furthermore, it remains unclear whether the observed gains, if they exist, are due to the proposed modifications or other confounding factors. It would be helpful if the authors could provide stronger empirical evidence or further theoretical insights to clarify the benefits of using nonlinear/stochastic bridges of this framework.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "See “Claims And Evidence”", "experimental_designs_or_analyses": "See “Claims And Evidence”", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See \"Other Comments Or Suggestions\"", "other_comments_or_suggestions": "Once again, I find the paper novel and original. As mentioned earlier, it is also distinctive even when compared to Bridge matching. In addition, I appreciate how this work broadens my perspective on flow matching, and I found it an enjoyable read.\n\nHowever, aside from its novelty and originality, the experimental results did not fully convince me. I believe the chosen tasks may not be well-suited to the proposed method. There could be more relevant applications where the sample paths of the generation process need to be controlled in specific ways, which might better showcase the advantages of the approach and strengthen its motivation.\n\nAdditionally, while it may not be strictly necessary, it could be helpful to explicitly clarify how this method differs from other Bridge matching approaches. At first glance, I initially found the two somewhat confusing due to their use of stochastic interpolation paths (or streams), and I believe other readers might have a similar impression. Addressing this distinction more clearly could enhance the paper’s accessibility and impact.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMIdUp2TmU", "reviewer": "Reviewer_R6jf", "summary": "This paper extends Conditional Flow Matching by introducing Gaussian processes to model latent \"streams\" connecting source and target distributions. Key contributions include: (1) a generalized CFM framework using GP streams while maintaining simulation-free training; (2) demonstrating reduced variance in vector field estimation and improved sample quality; (3) enabling flexible incorporation of correlated observations; and (4) empirical validation on synthetic, image, and time series datasets.\n\n## update after rebuttal\nThe author resolved my confusion, so my current high rating remains unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot to the reviewer for their positive comments, and thanks for suggestions on writing. We will update our manuscript accordingly, and add more details in the experiments (e.g. compare to more methods besides CFM variants) whenever possible. Here, we clarified some specific points…\n\n1. > Potential limitations of the GP approach in very high dimensions, e.g, Imagenet64/256, aren't fully addressed.\n\nFor computational convenience, we use independent GP for each dimension. Therefore, the proposed method should be scalable to high dimensional data. Even for many correlated data (e.g. long time seires), since the GP covariance is independent on observed values, we can calculate the covariance matrix inversion once before training. The current running time is reported with re-calculating matrix inversion for each training iteration. We will update the running time later.\n\n\n2. > In the section of time series modeling, incorporating a discussion of Trajectory Flow Matching (Zhang et al., NeurIPS 2024) would provide useful context and highlight how your GP-based approach extends or differs from their method.\n\nThanks for mentioning this paper. The time series application is one extension of our GP-CFM framework. The trajectory flow matching (Zhang et al., NeurIPS 2024) is based on autoregressive model (using FM to fit the AR functions), while we model the time seires by GP conditional stream. We will add this reference and the discussion in our camera-ready version if the paper is accepted.\n\n3. > How does the computational complexity of GP-CFM scale with data dimensionality compared to standard CFM approaches?\n\nSince we use independent GP for each dimension, the computational bottleneck comes from the GP covariance matrix inversion over (artificial) time. However, the GP covariance and its inversion is independent on observed values, and hence we can calculate once before training. The current running time is reported with re-calculating matrix inversion for each training iteration. We will update the running time in later. \n\n4. > What guidelines would you suggest for choosing appropriate GP kernels and their hyperparameters for different types of data?\n\nThe choice of GP kernels may depend on prior information and constraint of problems. For example, if there’s a periodic pattern, we should chose the periodic kernel.\n\nFor hyperparameter selection, it's a trade-off between systematic variance (from extrapolation error of neural net) and Monte Carlo error. The GP-stream reduces the systematic variance, but will introduce more Monte Carlo error. When implementing GP-CFM, we currently manually choose the GP parameter so that the GP conditional path covers a slightly wider region than linear interpolation (by checking several paired samples from the target to the source).\n\nTo reduce the Monte Carlo error while preserving the reduction in systematic variance, instead of sampling one GP path, we can sample multiple GP paths or resort to importance sampling over t. We will add a discussion on tuning parameters in our camera-ready version if the paper is accepted.", "claims_and_evidence": "The main claims are generally well-supported by both theoretical analysis and empirical evidence. The authors show (1) GP streams produce smoother vector fields with reduced variance; (2) GP-CFM improves sample quality across datasets;(3) The approach effectively incorporates correlated observations.", "methods_and_evaluation_criteria": "The methods are sound and well-motivated. The authors evaluate their approach using standard metrics (FID, KID, Wasserstein distance) on appropriate datasets. Their comparison against baseline methods (I-CFM, OT-CFM) is reasonable, though they could potentially compare against more recent state-of-the-art generative models for broader context.", "theoretical_claims": "The theoretical foundations appear solid. I verified (1) the proof that the marginal vector field generates the probability path (Section J.1), (2) The gradient equivalence proofs (Sections J.2 and J.3), (3) the Bayesian decision-theoretic perspective (Appendix A).", "experimental_designs_or_analyses": "The experimental design is generally sound:\n\n1. Synthetic examples effectively demonstrate the conceptual advantages.\n\n2. Image generation experiments follow standard practices.\n\n3. Time series experiments illustrate the unique capabilities of GP-CFM.\n\nHowever, some experimental choices could be better justified:\n\n1. The choice of hyperparameters for the GP kernels isn't thoroughly explained.\n\n2. The comparison on CIFAR-10 could include more recent generative models beyond just CFM variants.", "relation_to_broader_scientific_literature": "The paper builds appropriately on previous work in flow matching (Lipman et al., 2023; Tong et al., 2024) and stochastic interpolants (Albergo et al., 2023, 2024). It extends CFM by incorporating Gaussian processes. The authors also make connections to Bayesian perspectives on generative modeling and optimally regularized estimation.", "essential_references_not_discussed": "The paper covers most relevant literature.", "other_strengths_and_weaknesses": "**Strengths**:\n1. The paper presents an theoretical framework for extending Conditional Flow Matching with Gaussian process streams.\n\n2. The authors provide a novel perspective on variance reduction in flow matching through appropriate GP regularization.\n\n3. The method shows valuable practical applications for time series and correlated data that standard CFM approaches cannot handle effectively.\n\n4. The empirical results demonstrate clear improvements across multiple datasets as measured by standard quality metrics.\n\n**Weaknesses**:\n1. Computational complexity analysis could be more thorough.\n\n2. Some hyperparameter choices need better justification,such as what principles should guide kernel selection and parameter tuning for different data modalities (images v.s. time series). \n\n3. Potential limitations of the GP approach in very high dimensions, e.g, Imagenet64/256, aren't fully addressed.", "other_comments_or_suggestions": "1. It would be better to include a more detailed comparison of computational requirements in the main text.\n\n2. In the section of time series modeling, incorporating a discussion of Trajectory Flow Matching (Zhang et al., NeurIPS 2024) would provide useful context and highlight how your GP-based approach extends or differs from their method.", "questions_for_authors": "1. How does the computational complexity of GP-CFM scale with data dimensionality compared to standard CFM approaches? \n\n2. What guidelines would you suggest for choosing appropriate GP kernels and their hyperparameters for different types of data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes using Gaussian processes to model the latent stochastic paths between draws from the coupling that arise in flow matching. 
This enables finer control over the regularization of the path and thus provides variance reduction and is more suited for time-series and correlated data.
The reviewers found the work to be original and a valuable theoretical contribution. The proposed approach is readily compatible with other variance
reduction approaches that define the choice of coupling (e.g. minibatch OT) and opens up a new design space regarding the kernel design for the GP.
While the experimental validation of the proposed approach
is somewhat limited and the paper would benefit by benchmarking on higher-dimensional datasets, the positives outweigh the drawbacks.
Decision: Accept (poster)