Variational Flow Matching for Graph Generation
Authors: Floor Eijkelboom
Grigory Bartosh
Christian A. Naesseth
Max Welling
Jan-Willem van de Meent
Venue: NeurIPS 2024 poster (2024)
Keywords: generative modeling
flow matching
variational inference
categorical
discrete
graph generation
molecular generation
Primary area: diffusion_based_models
TL;DR: We propose a variational perspective on flow matching and apply it to graph generation.

Abstract: We present a formulation of flow matching as variational inference, which we refer to as variational flow matching (VFM). We use this formulation to develop CatFlow, a flow matching method for categorical data that is easy to implement, computationally efficient, and achieves strong results on graph generation tasks. In VFM, the objective is to approximate the posterior probability path, which is a distribution over possible end points of a trajectory. VFM admits both the original flow matching objective and the CatFlow objective as special cases. We also relate VFM to score-based models, in which the dynamics are stochastic rather than deterministic, and derive a bound on the model likelihood based on a reweighted VFM objective. We evaluate CatFlow on one abstract graph generation task and two molecular generation tasks. In all cases, CatFlow exceeds or matches performance of the current state-of-the-art models.

Variational Flow Matching for Graph Generation
Floor Eijkelboom" UvA-Bosch Delta Lab University of Amsterdam
Grigory Bartosh
AMLab University of Amsterdam
Christian A. Naesseth UvA-Bosch Delta Lab University of Amsterdam
Max Welling -vA-Bosch Delta Lab University of Amsterdam
Jan- Willem van de Meent UvA-Bosch Delta Lab University of Amsterdam
Abstract
We present formulation of flow matching variational inference, which we refer to as variational flow matching (VFM). Based on this formulation we develop CatFlow, flow matching method for categorical data. CatFlow is easy imple- ment, computationally efficient, and achieves strong results on graph generation tasks. The key observation in VFM that we can parameterize the vector field of the flow in terms of a variational approximation of the posterior probability path, which the distribution over possible end points of a trajectory. We show that this variational interpretation admits both the CatFlow objective and the original flow matching objective as special cases. We also relate VFM to score-based models, which the dynamics are stochastic rather than deterministic, and derive bound on the model likelihood based on a reweighted VFM objective. We evaluate CatFlow on one abstract graph generation task and two molecular generation tasks. In all cases, CatFlow exceeds matches performance of the current ttate-of-the--rt.
Introduction
In recent years, the field of generative modeling has seen notable advancements. In image generation [41, 45], the development and refinement of ceffusiio-bassed approaches - specifically those using denoising
matching [59] have proven effective for generation at scale [15, 54]. However, while training can be done effectively, the constrained space of sampling probability paths a diffusion requires tailored techniques work [52, 64). This is in contrast more flexible approaches such as continuous normalizing flows (CNFs) [8], that are able to learn a more general set of probability paths than diffusion models [53], the expense of being expensive train they require one to solve an ODE during each training step (see e.g. [5, 46, 14]).
Recently, Lipman and collaborators [27] proposed flow matching (FM), an efficient and simulation- free approach to training CNFs. Concretely, they use per-sample interpolation to derive a simpler objective for learning the marginal vector field that generates desired probability path in CNF. This formulation provides equivalent gradients without explicit knowledge of the (generally intractable) marginal vector field. This work has been extended to different geometries [7, 23] and various applications [60, 9, 13, 24]. Similar work has been proposed concurrently in [32, 1].
This paper identifies reformulation of flow matching that we refer to as variational flow matching (VFM). In flow matching, the vector field any point can understood as the expected continuation toward the data distribution. In VFM, we explicitly parameterize the learned vector field as an expectation relative to a variational distribution. The objective of VFM then to minimize the
These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Kullback-Leibe (KL) divergence between the posterior probability path, i.e. the distribution over possible end points (continuations) at particular point in the space, and the variational approximation.
We show that VFM recovers the original flow matching objective when the variational approximation is Gaussian and the conditional vector field is linear in the data. Under the assumption of linearity, a solution to the VFM problem is also exact whenever the variational approximation matches the marginals of the posterior probability path, which means that we can employ a fully-factorized variational approximation without loss of generality.
While VFM provides a general formulation, graph generation, where the data are categorical. This setting leads to a simple method that we refer to as CatFlow, in which the objective reduces to training classifier over end points per-component basis. We apply CatFlow to a set of graph generation tasks, both for abstract graphs [35] and molecular generation [40, 19]. By all metrics, results match substantially exceed those obtained by existing methods.
primary interest in this paper is its application to
2 Background
2.1 Transport Framework for Generative Modeling and CNFs
Common generative modeling approaches such as normalizing flows [42, 38] and diffusion models [15, 54] parameterize a transformation y from some initial tractable probability density Po - -typically a standard Gaussian distribution to the target data density p1. In general, there is a brate-offbetwwen allowing şo to be expressive enough model the complex transformation while ensuring that the determinant term is still tractable. One such transformation is continuous normalizing flow (CNF).
Any dime-dependent' vector field i [0, RD IR gives rise to such transformation called a flow as such a field induces a time-dependent diffeomorphism [0, 1] x R - DR defined by the following ordinary differential equation (ODE):
Pc(x) = (x)( with initial conditions spo (x) = T. di
(I)
In CNFs, this vector field is learned using neural network of Through the change of variables formula (z) can be evaluated (see appendix E) and hence one could try and optimize the empirical divergence between the resulting distribution P1 and target distribution. However, obtaining gradient sample for the loss requires ones to solve the ODE induced during training, making this approach computationally expensive.
2.2 Flow Matching
In flow matching [27], the aim is to regress the underlying vector field of a CNF directly on the interval / 1]. Flow matching leverages the fact that even though do not have access the actual underlying vector field which we denote as and probability path pt, one can construct a per-example formulation by defining conditional flows, i.e. the trajectories towards specific datapoints t,, Concretely, FM sets:
Pc(x #1) (21) dx1: ue (x) = -fula ue(x x1) P
Pe(x)
where (x 1x) is the conditional trajectory. The most common way to define u,(x 11) is as the straight line continuation from x to Iı, implying one can obtain samples pr( xn) simply by interpolating samples zio ~ Po for some Po and xi ~ Pl.
= T1
U (or m)
x=
+ (1 - t)ar is a sample x ~ pe I xT).
(3)
1
Crucially, the flow matching objective
(F) Ee-jo.a, [|l](()) - ue (1))
(4)
'Time dependence is denoted through the subscript & throughout this paper.

---

is equivalent in expectation (up to constant) to the conditional flow matching objective
LCFM
[1 ue(2r x 2))l
(5)
An advantage of flow matching is that this conditional trajectory uy (z: 21) can be chosen to make the problem tractable. The authors show that diffusion models can be instantiated as flow matching with specific conditional trajectory, but also show that assuming simple, straight-line trajectory leads more efficient training. Note that in contrast to likelihood-basee based training of CNFs, flow matching is simulation free, leading to scalable approach to learning CNFs.
3 Variational Flow Matching for Graph Generation
We derive CatFlow through novel, variational view on flow matching we call Variational Flow Matching (VFM). The VFM framework relies on two insights. First, we can define the marginal vector field and its approximation in terms of an expectation with respect to a distribution over end points of the transformation. This implies that we can map a flow matching problem onto variational counterpart. Second, under typical assumptions on the forward process, we can decompose the expected conditional vector field into components for individual variables which can be computed in terms of the marginals of the distribution over end points of the conditional trajectories. This implies variational
that, without loss of generality, we can solve a VFM problem using fully-factorias approximation, providing a tractable approximate vector field. For categorical data, the corresponding vector field can be computed efficiently via direct summation. This results in closed-for objective to train CNFs for categorical data, which we refer to as CatFlow. We develop the theory of VFM in section 3 and we relate VFM to flow matching and score-based diffusion in section 4.
3.1 Flow Matching using Variational Inference
In any flow matching problem, the vector field in eq. (2) can be expressed as an expectation
ue ux(x) = ue( | (1) [x)d Epr w) [ur(x: (ar))
(6)
where (01 xx is the posterior probability path, the distribution over possible end points æn of paths passing through at time
21)
(1 ())
(e x1) Pe xx)) Paata(X)))
Pr(x))
This makes intuitive sense: the velocity in point v is given by all the continuations from x to final points 21, weighted by how likely that final point is given that (x),, one has to evaluate a joint integral over D dimensions.
x. Note that to compute
This observation leads to propose change in parameterization of the learned vector field. Rather than predicting the components the vector field directly, we can define an approximate vector field in terms of an expectation with respect to variational distribution of with parameters 0,
o (x) = ue( ( (1) 98 (x1 [x) dr..
(8)
Clearly, in this construction (x) will be equal to (x( when of (z1 x) and (r ()) are identical. This implies that we can map flow matching problem onto variational inference problem.
Concretely, we can define a variational flow matching problem by minimizing the Kullback-Leible (KL) divergence from Pt to d. which
express as
E, KL (pi(z)p: | x) || (r())q( (z1 =))] = E,,,. [log of (x1 |x] + const,
where Uniform 1) and 11 ~ Pt (x, 21) (see appendix A.1 for derivations). This leads us to propose the variational flow matching (VFM) objective
LVFM (0) = Er,,, [log d (a1 |x]]
(10)
While this variational formulation of flow matching promising, two potential drawbacks emerge in practical applications. First, although is feasible to reformulate any flow matching problem as
3

---

a variational inference task, doing so requires learning an approximation of a potentially complex. high- lighddimensional distribution pr(z) 2) (including any correlations between the different =1). Second, representing of (z) as an expectation may pose intractabilityy challenges. Interestingly, under typical assumptions about the conditional velocity field in flow matching, this objective simplifies significantly, making no more computationally demanding than standard flow matching, a point we will further address in section 3.2.
3.2 Mean-Field Variational Flow Matching
Decomposing the conditional vector field. At first glance, do seem to obtain much from this variational view due to the intractability of (z ()) and uf (a:). Fortunately, we can simplify the objective and the calculation of the marginal vector field under the typical case where the conditional vector field (2 2t ) is linear in I₁, such as in straight line interpolations commonly used in flow matching. This leads to two simplifications. First, we notice that the expected value of linear conditional velocity field simply equals the conditional velocity field towards the expectation/mean of the distribution we are considering. i.e.
(x)x)= = (x.(s) (n) [ue(xr x1)] = uz(2 Ep, [2] æ]) if uz( (1) is linear in ø1.
(11)
This means that as long as our variational distribution has the same mean as pr(21 | x), we will learn the same underlying field. Second, we realize that the expected value of of only depends on (z( Iz), and thus as such as long as of (atg z²) has the same marginal expectations EE [a x we will learn the same field as under pe(21 x)). In other words, it suffices to learn fully-factorized approximation (x1 I²), there is need to fully characterize the covariance of the posterior probability path. Thich allows us to reduce a figg-diimensionl variational problem to a series of low dimension problems.
Formally, the following holds:
Theorem 1. Assume that the conditional vector field u,(x 21) is linear in Z1. Then, for any distribution rz (x1 x) such that the marginal distributions coincide with those of pr(x1 x), the corresponding expectations of (x 21)) are equal, i.e.
(x(()1))) [ur( xx1] = Ep (x1l)))) [u( xx)]-
(12)
We provide proof in appendix A.3.
It follows directly from theorem 1 that without loss of generality we can consider the considerably easier task of a fully-factorized approximation
D ( x) = II
de (14 xz)
(13)
We refer to this special case mean-field variational flow matching (MF-VFM), and the VFM objective reduces to
D Σ
log of (2f 2)
CME-VEM(0) = -EE,, [log qi (x) x=]] Er,
(14)
Even though factorized model, it is worth emphasizing that due to the linearity of the conditional field, a mean-field variational distribution can learn the solution exactly.
Computing the marginal vector field. To calculate the vector field of (z). can simply substitute the factorized distribution of (x1 x)) into eq. (8). However, this still requires evaluation of an expectation. Fortunately, leveraging the lincarity condition significantly simplifies this computation, since as long we have access to the first moment of one-dimensional distributions of (of x)), we can efficiently calculate (x)) by simply considering the conditional field towards it. Note that for many distributions - e.g. Gaussian - learning its parameters equivalent to learning its expected value. Note that the training procedure will differ for two distinct distributions e.g. Gaussian versus Categorical so the form of the distribution of (a) xx) remains practically important, a flexibility provided through the variational view on flow matching.

---

ug (ate)
1 Et
Zt
TD
M4 (xt)
H4 (are)
Figure I: Parameterization of the vector field in CatFlow. Given interpolant = tæz + (1 t to CatFlow predicts a categorical distribution (x1 xx) parameterized by a vector (xxe). The resulting construction for the vector field of (Ire) = (He( xt)/(1 - -) ensures that trajectories converge to point the simplex at t 1.
If we now the standard flow matching case of using conditional vector field based linear interpolation, the approximate vector field can be expressed in terms of the first moment of the variational approximation:
(x) of (m1 (x)
T1
11
A1 := B of (f (mr (a)) [x1].
(15)
Note that this covers both the case of categorical data, which we focus on in this paper, and the case of continuous data, as considered traditional flow matching methods. We provided the general algorithm for training and generation in appendix B.I.
At first glance, the linearity condition the conditional vector field uz(2 x1) in theorem 1 might seem restricting. However, in most saate-of-the-ane generative modeling techniques, this condition is satisfied, e.g. diffusion-based models, such flow matching [27, 2, 32]. diffusion models [15, 54], and models that combine the injection of Gaussian noise with blurring [44, 16], among others [10, 51].
3.3 CatFlow: Mean-Field Variational Flow Matching for Categorical Data
The CatFlow Objective In CatFlow, directly apply the VFM framework to the categorical case. Let our parameterised variational distribution of (2q x) = Cat(rş (x)), and let us denote the parameters of this categorical
thhh (x) of (x = k Φ) of (a) Mx k]].
(16)
Then, the dth component of the learned vector field is
Kd
k]
0.d (x)
(17)
Intuitively, CatFlow learns distribution over the conditional trajectories to all corners of the probability simplices, rather than regressing towards an expected conditional trajectory.
In the categorical setting, the MF-VFM objective can be written out explicitly. Writing out the probability mass function of the categorical distribution, we see that
K'd
log df (af x)
log
(
(4g (z))
Σ k] log 1 dk (z).
(18)
k=1
As such, we find that CatFlow objective is given by standard cross-entropy loss:
kd
CCaFlow (0) - -,,
k] log th (x)
(19)
Note, however, that when actually computing vg this can be done efficiently, since
zdd
n zd
E (xf(le)) [ue(x [x))] = (xf(e(
(20)
1
1-t
5

---

since (²(x) := Bo (=fla)) [4f 2)] and the other terms are not in the expectation. Note that this geometrically corresponds to learning the mapping to point in the probability simplex, and then flowing towards that. This procedure illustrated in fig. Because of this, training CatFlow is no less efficient than flow matching. We provided the algorithm for training and generation for CatFlow and a Gaussian VFM objective in appendix B.2 and appendix B.3 respectively.
More precisely, training CatFlow offers two key benefits over standard flow matching. First, given that CatFllw predicts points in the probability simplex and parametrizes the velocities to point into an inductive bias is introduced ensuring all generative paths align with realistic trajectories, hence avoiding misaligned paths. Second, using a cross-entropy loss instead of a meen-squared error improves gradient behavior during training. Both aspects enhance learning dynamics and speed up convergence, which is evaluated in section 6.3. Lastly, CatFlow's 's ability to learn probability vectors rather than directly choosing classes as is common in discrete approaches allows the model to express uncertainty about variables at specific time. This is especially useful in complex domains like molecular generation, where initial uncertainty about components decreases as more structure is established, leading to more precise predictions.
Permutation Equivariance. Graphs, defined by vertices and edges, lack a natural vertex order unlike other data types. This permutation invariance means any vertex labeling represents the same graph if the connections remain unchanged. Note that even though the following results apply to graphs, an unordered set of categorical variables can be described by graph without edges. Under natural conditions - see appendix we ensure CatFlow adheres to this symmetry (see appendix A.4 for the proof).
Theorem 2. CatFlow generates exchangeable distributions, i.c. CatFlow generates all graph permu- tations with equal probability.
4 Flow Matching and Score Matching: Bridging the Gap
In this section, we relate the VFM framework to existing generative modeling approaches. First, we show that VFM has standard flow matching as a special case when the variational approximation is Gaussian. This implies that VFM provides more general approach to learning CNFss Second, we show that through VFM, we are not only able to compute the target vector field, but also the score function used in score-based diffusion. This has two primary theoretical implications: 1) VFM simultancously learns deterministic and stochastic dynamics as diffusion models rely on stochastic dynamics, and 2) VFM provides variational bound on the model likelihood.
Relationship to Flow Matching. VFM admits FM special case, under certain assumptions on (x x1), when the variational approximation is Gaussian. Formally, the following holds (see theorem 3 in appendix A.2 for the proof):
Theorem 3. Assume the conditional vector field (x(: [m)) is linear in æ1 and is of the form
(x(21))) = (((x)= + b,(x),
(21)
where Axx)): [0,1] + +R x and bx()) [0,1]x== + RD. Moreover, assume that A((x) is an invertible matrix and q. (z1 z)) = =x(x (x( (x)), where Ex(x) = 1 (AT (x)) Then, VFM reduces to flow matching.
Relationship to Score-Based Models. Flow matching [27] is inspired by score-based models [54] and shares strong connections with them. This leads us two observations. The first is that it should be possible to define variational parameterization of the score function in diffusion models that is analogous to the one in VFM. The second is that we can build on existing results that show that many diffusion model objectives, including the standard flow matching objective with linear interpolant, can be expressed as special cases of a general weighted loss function [22, 21J. Here we simililarly define a bound on the log- likelhooo in terms of reweighted VFM objective.
In based models, the objective is to approximate the score function V. (x) with function sf (z). A connection to VFM becomes apparent by observing that the score function can also be

---

expressed as an expectation with respect to pt (z x:) (see appendix A.5 for derivation):
Ve logpi(x) = pt( V) po x æ1)dz = (x1lz)) V po x1)]
(22)
where V log x11) ) is the tractable conditional score function. Similarly, parameterize s (x)) in terms of an expectation with respect to variational approximation qf (21 z),
$ (x) = / 4. (x1 V) log x1) dæ1.
(23)
now clear that (x) = v log Pe(z) when af (x) x) = Pe(2i z). This suggests that there exists variational formulation of score-based models that is entirely analogous to VFM. Indeed, existing work on continuous diffusion for categorical data [11] defines a parameterization of the score function of this form (see section 5 for more detailed discussion).
Following [54, 3], we can construct stochastic generative dynamics de = ii (x)ddt + g:dw to approximate the true dynamics da (x)ddt + gidw (see details in appendix A.6), with
oe ux( an) v log 1m))
(x) = (x) + o 0 (z). (24)
(2) = (x)))
Here 9r : :0, R4 is scalar function, and w is standard Wiener process.
This connection has two important implications. First, it shows that learning variational approxi- mation of (2) x)) can be used to define both deterministic and stochastic dynamics, whereas flow matching typically considers deterministic dynamics only (as flows are viewed through the lens of ODEs). Second, it enables us to show that a reweighted version of the VFM objective provides a bound on the likelioel of the model. This result, inspired by [21], provides another theoretical motivation for learning using the VFM objective.
Theorem 4. Rewrite the Variational Flow Matching objective as follows:
(V) = Er,x [ca (t, (t) where (t, = -Ez, [log (21 x)]
(25)
Then, the following holds:
Ery [log of (x1) < Br [A(x)) + C,
(26)
where (x( is a non-negative function and C' is a constant.
We provide proof in appendix A.7. We further note that that applying the same linearity condition that we discussed in section 3.2 to the conditional score function maintains all the same connections with score-based models.
5 Related Work
Diffusion Models for Discrete Data. Several approaches to diffusion models have been developed for graph generation. In [58], the authors define a Markov process that progressively edits graphs by adding removing edges and altering node or edge categories and is trained using a graph transformer network, that reverses this process to predict the original graph structure from its noisy version. This approach breaks down the complex task of graph distribution learning into simpler node and edge classification tasks. Morcover, [20] proposes score-based generative model for graph generation using system of stochastic differential equations (SDEs). The model effectively captures the complex dependencies between graph nodes and edges by diffusing both node features and adjacency matrices through continuous-ting generation approaches that perform on par with autoregressive ones, such as in [31, 26, 36). Other non-autoregressive approaches worth mentioning [28, 34, 33].
processes. These are non-autoregressive graph
There also work on diffusion-based models for discrete data the form text and other sequential data [4, 33, 61, 18). Though not explicitly formulated in terms of variational perspective, the work on continuous diffusion for categorical data [11] arrives at an approach that closely related that of CatFlow. This work defines diffusion in the embedding space of transformer- based language
7

---

Table 1: Results abstract graph generation.
Ego-small
Community-small
Degree
Clustering +
Orbit 4
Degree 4
Clustering +
Orbit +
Graph VAE [50]
0.130
0.170
0.050
0.350
0.980
0.540
GNF [29]
0.030
0.100
0.001
0.200
0.200
0.110
EDP-GNN [37]
0.052
0.093
0.007
0.053
0.144
0,026
GDSS [20]
0.021
0.024
0.007
0.045
0.086
0.007
CatFlow
0.013
0.024
0.008
0.018
0.086
0.007
model. It defines approximation of the score function expected value of conditional score function as in eq. (23), which leads to an expression for the learned score function in terms of a mean embedding, analogous the other than in that it defines flow in the embedding space of language models, is in how the objective defined. The authors also minimize cross-entropy loss, but employ a time-warped objective, similar to the general weighted objective proposed in [22], where the warping is optimized to ensure that the entropy of predictions decreases linearly when moving from noise to data in uniform time.
obtain in eq. (15). Where this approach differs from CatFlow,
Flow-based methods for Discrete Data. Recently, two flo-bbsee methods for discrete generative modeling have been proposed, which differ both in terms of technical approach and intended use case from the work that we present here.
In [56], a Dirichlet flow framework for DNA sequence design is introduced, utilizing a transport problem defined over the probability simplex, similar diffusion on simplices proposed in [43]. This approach differs from CatFlow in that it represents the conditional probability path pr 21) using a Dirichlet distribution. This implies that points æ are constrained to the simplex, which not the case for CatFlow. Dirichlet Flows have not been evaluated graph generation, but we did carry out preliminary experiments based on the released source code. We compare to this approach in our experiments.
In [6), Discrete Flow Models (DFMs) are introduced. DFMs use Continuous Markov Chains to enable flexible and dynamic sampling in multimodal generative modeling of both continuous and discrete data. Though sharing goal, this approach differs significantly from CatFlow as in the end the resulting model does not learn CNF, but rather generation through sequential sampling from time-dependent categorical distribution. As in the case of Dirichlet flows, no evaluation on graph generation was performed.
The switch to the variational perspective is inspired by [58], showing significant improvement through viewing the dynamics as a classification task over end points. However, CatFlow is still a continuous model, and integrates rather than iteratively samples during generation.
Experiments
We evaluate CatFlow in three sets of experiments. First, we consider an abstract graph generation task proposed in [35], where the goal of this task is to evaluate if CatFlow is able to capture the topological properties of graphs. Second, we consider two common molecular benchmarks, QM9 [40] and ZINC250k [19]. consisting of small and (relatively) large molecules respectively. This task is chosen to see if CatFlow can learn semantic information graph generation, such as molecular properties. Finally, perform an ablation comparing CatFlow to standard flow matching, specifically in terms of generalization. The experimental setup and model choices are provided in appendix D.
Note that we treat graphs as purely categorical/discrete objects and do not consider 'geometric' graphs that are embedded in e.g. Euclidean space. Specifically, for some graph with K2 node classes and K. edge classes, we process the graph as fully-com colyecconeetted graph, where each node is treated as
Fllo matching and diffusion models have also been proposed for geometric graph generation, e.g. in [23, [5] and [17. 57] respectively, but since these approaches are continuous (as they generate coordinates based some conformer) they consider fundamentally different task than the ones we consider here.
8

---

Table Results molecular generation.
QM9
ZINC250K
Valid
Unique f
FCD 4
Valid f
Unique 1
FCD 4
MoFlow [63]
91.36
98.65
4.467
63.11
99.99
20.931
EDP-GNN [37]
47.52
99.25
2.680
82.97
99.79
16.737
GraphEBM [30]
8,22
97.90
6.143
5.29
98,79
35,471
GDSS [20]
95.72
98.46
2.900
97.01
99.64
14.656
Digress [58]
99.00
96.20
Flow Matching [27]
94.10
98.20
5.155
94.01
96.68
18.764
Dirichlet FM [56]
99.10
98.15
0.888
97.52
99.20
14,222
CatFlow
99.81
99.95
0.441
99.21
100.00
13.211
categorical variable of one of K2 classes and each edge of Ke + classes, where the extra class corresponds with being absent.
6.1 Abstract graph generation
We first evaluate CatFlow on abstract graph generation task, including synthetic and real-world graphs. We consider 1) Ego-small (200 graphs), consisting of small ego graphs drawn from larger Citeseer network dataset [48], 2) Community-small (100 graphs). consisting of randomly generated community graphs, 3) Enzymes (587 graphs), consisting of protein graphs representing tertiary structures the enzymes from [47], and Grid (100 graphs). consisting of 2D grid graphs. We follow the standard experimental setup popularized by [62] and hence report the maximum mean discrepancy (MMD) compare the distributions of degree, clustering coefficient, and the number of occurrences of orbits with nodes between generated graphs and test set. Following [20], we also use the Gaussian Earth Mover's Distance kernel to compute the MMDs instead of the total variation.
The results of the Ego-small and Community-smen -smal tasks are summarized table 1, and additional results (and error bars) provided in appendix C. The results indicate that CatFlow is able capture topological properties of graphs, and performs well on abstract graph generation.
6.2 Molecular Generation: QM9 ZINC250K
Molecular generation entails designing novel molecules with specific properties, complex task hindered by the vast chemical space and long-range dependencies in molecular structures. We evaluate CatFlow on two popular molecular generation benchmarks: QM9 and ZINC250K [40, 19].
We follow the standard setup e.g. as [49, 33, 58, 20] of kekulizing the molecules using RDKit [25] and removing the hydrogen atoms. We sample 10,000 molecules and evaluate them on validity, uniqueness, and Fréchet ChemNet Distance (FCD) evaluating the distance between data and generated molecules using the activations of ChemNet [39]. Here, validity is computed without valency correction or edge resampling, hence following [63] rather than [49, 33), as is more reasonable due the existence of formal charges in the data itself. We do not report novelty for QM9 and ZINC250K, as QM9 is an exhaustive list of all small molecules under some chemical constraint and all models obtain (close to) 100% novelty on ZINC250K."
CatfFllo obtains 49% novelty QM9.
gju a
Figure 2: CatFlow samples of QM9 (top) and ZINC250K (bottom).

---

100
Row Vaxciing CXPT1W
Row Vesching Caffire
Scaee 10
23
Narm Layen
70ar1 Lagerra
Thari Lageere
(a) Score 100% of data.
(b) Score 20% of data.
(c) Score 5% of data.
Figure 3: Ablation results. We compare standard flow matching and CatFlow. We visualize perfor- mance degradation terms of score, which is the percentage molecules that is valid and unique, for varying number of layers and percentage of the training data.
The results are summarized in table 2 and samples from the model are shown in fig. 2. CatFlow obtains state-of-the-ar results on both QM9 and ZINC250K, virtually obtaining perfect performance on both datasets. It is worth noting that CatFlow also converges faster than flow matching and is not computationally more expensive than any of the baselines either during training or generation.
6.3 CatFlow Ablations
To understand the difference in performance between CatFlow and standard flow matching formula- tion we perform ablations. we focus on generalization capabilities, and as such consider ablations that the number of parameters in the model and the amount of training data.
In fig. 3 we report a score, which is the percentage of generated molecules that valid and unique. CatFlow not only outperforms regular flow matching in the large model and full data setting, but is also significantly more robust to a decrease in model-size and data. Moreover, we observe significantly faster convergence (curves not shown). We hypothesize this is a consequence of the optimization procedure not exploring 'irrelevant' paths that do not point towards the probability simplex.
7 Conclusion
We have introduced a variational reformulation of flow matching. This formulation in turn informed the design of simple flow matching method for categorical data, which achieves strong performance on graph generation tasks. Variational flow is very general and opens up several lines of inquiry. We see immediate opportunities to apply CatFlow to other discrete data types, including text, source code, and more broadly to the modeling of mixed discrete-continuous data modalities. Additionally, the connections score-based models that we identify this paper, suggest path towards learning both deterministic and stochastic dynamics.
Limitations. While the VFM formulation that we identify in this paper has potential in terms of its generality, we have as yet only considered its application the specific task of categorical graph generation. We leave other use cases of VFM to future work. A limitation of CatFlow, which is shared with related approaches to graph generation, that reasoning about the set of possible edges has cost that is quadratic in the number of nodes. As result CltFlow does not scale well to e.g. large proteins of 10d or more atoms.
Ethics Statement. Graph generation in general, and molecular generation specifically, holds great promise for advancing drug discovery and personalized medicine. However, this technology also poses ethical concerns, such as the potential for misuse in creating harmful substances. terms of technology readiness, this work is not yet at level where we foresee direct benefits or risks.
Acknowledgements.
This project was supported by the Bosch Center for Artificial Intelligence.
10

---

References
[1] Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
[2] Michael Samuel Albergo and Eric Vanden-Fijnden. Building Normalizing Flows with Stochastic Interpolants. In The Eleventh International Conference on Learning Representations, September 2022.
[3] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[4] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 11111799119999 2021.
[5] Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel, Aditya Grover, Ricky T. Q. Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages Jul 2022.
PMLR, 17-23
[6] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Confer- ence Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5453-5512 PMLR, 21-27 Jul 2024.
[7] Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries. In The Twelfth International Conference on Learning Representations, 2024.
[8] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[9] Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. arXiv preprint arXiv:2307.08668, 2023.
[10] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar. Soft diffusion: Score matching with general corruptions. Transactions on Machine Learning Research, 2023.
[11] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categor- ical data, December 2022.
[12] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning Graphs: Methods and Applications, 2021.
[13] Ehsan Ebrahimy and Robert Shimer. Stock-flow matching. Journal of Economic Theory, 13555):3335-1333 2010.
[14] Will Grathwohl. Ricky Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable re- versible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[16] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. The Eleventh International Conference Learning Representations, 2023.

---

[17] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation 3d. In International conference on machine learning, pages 8867-8887. PMLR, 2022.
[18] Naoto Inoue, Kotaro Kikuchi, Edgar Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10167-10176, 2023.
[19] John Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: a free tool to discover chemistry for biology. Journal of chemical information and modeling, 12(7):1757-1768, 2012.
[20] Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations. In Proceedings of the 39th International Conference on Machine Learning, pages 10362-1038) PMLR, June 2022.
[21] Diederik Kingma and Ruiqi Gao. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation. Advances in Neural Information Processing Systems, 36:65484-655 December 2023.
[22] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems, volume 34, pages 21696-21707. Curran Associates, Inc., 2021.
[23] Leon Klein, Andreas Krämer, and Frank Noe. Equivariant flow matching. In Thirty-Seventh Conference on Neural Information Processing Systems, November 2023.
[24] Jonas Kohler, Yaoyi Chen, Andreas Kramer, Cecilia Clementi, and Frank Noé. Flow-matching: Efficient loarse-graining of molecular dynamics without forces. Journal of Chemical Theory and Computation, 19(3):942-955 9522 2023.
[25] Greg Landrum. Rdkit: Open-source cheminformatics software, 2016.
[26] Renjie Liao, Yujia Li, Yang Song, Shenlong Wang, Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. Advances in neural information processing systems, 32, 2019.
[27] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
[28] Phillip Lippe and Efstratios Gavves. Categorical Normalizing Flows via Continuous Transfor- mations. In International Conference on Learning Representations, October 2020.
[29] Jenny Liu, Aviral Kumar, Jimmy Ba, Jamie Kiros, and Kevin Swersky. Graph normalizing flows. Advances in Neural Information Processing Systems, 32, 2019.
[30] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. In Energy Based Models Workshop ICLR 2021, 2021.
[31] Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. Advances in neural information processing systems, 31, 2018
[32] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
[33] Youzhi Luo, Keqiang Yan, and Shuiwang Ji. Graphdf: A discrete flow model for molecular graph generation. In International conference on machine learning, pages 7192-7203. PMLR, 2021.
[34] Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe. Graphnvp: An invertible flow model for generating molecular graphs. arXiv preprint ar arXiv:1905.11600, 2019.
12

---

[35] Karolis Martinkus, Andreas Loukas, Nathanaël Perraudin, and Roger Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In International Conference on Machine Learning, pages 15159-15179. PMLR, 2022.
[36] Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Leurning: Science and Technology, 2(2):025023, 2021.
[37] Chenhao Niu, Yang Song, Jiaming Song. Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference Artificial Intelligence and Statistics, pages 4474 4484. PMLR, 2020.
[38] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayannn. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-644 2021.
[39] Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fréchet chemnet distance: metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, 1839)117361171,, 2018.
[40] Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2:X1v:2204.06//55, 1(2):3 2022.
[42] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning, pages 1530-15383 PMLR, 2015.
[43] Pierre H. Richemond, Sander Dieleman, and Arnaud Doucet. Categorical SDEs with Simplex Diffusion, October 2022.
[44] Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In International Conference on Learning Representations (ICLR), 2023.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[46] Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. Moser flow: Divergence- based generative modeling on manifolds. Advances in Neural Information Processing Systems, 34:1 176666 2021.
[47] Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research, (itssupp__))DD33--AA
DD43, 2004.
[48] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
[49] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2019.
[50] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In Artificial Neural Networks and Machine Searnnng-ICAND 2018: 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part I 27, pages 412-422. Springer, 2018
[51] Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where to diffuse, how to diffuse and how to get back: Automated learning multivariate diffusions. International Conference on Learning Representations, 2023.
13

---

[52] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[53] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021.
[54] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[55] Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant Flow Matching with Hybrid Probability Transport for 3D Molecule Generation. In Seiett-Seventh Conference on Neural Information Processing Systems, November 2023.
[56] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications DNA sequence design. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages PMLR, 21-27 Jul 2024.
[57] Brian L Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2022.
[58] Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. DiGress: Discrete Denoising diffusion for graph generation. In The Eleventh International Conference on Learning Representations, September 2022.
[59] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7)::661-1674 2011.
[60] Jonas Wildberger, Maximilian Dax, Simon Buchholz, Stephen Green, Jakob H Macke, and Bernhard Schölkopf. Flow matching for scalable simulation-based inference. Advances in Neural Information Processing Systems, 36, 2024.
[61] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[62] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708-577 17. PMLR, 2018.
[63] Chengxi Zang and Fei Wang. Moflow; invertible flow model for generating molecular graphs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 617-626, 2020.
[64] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2022.
14

---

A Proofs and Derivations
A.1 Derivation of the Variational Flow Matching Objective
We derive eq. (9) that states the equivalence of the optimization of the VFM objective and mini- mization of the KL divergence between the true endpoint distribution (z x)) and the variational approximation of (Ir xr). Note that
min F (pa | xx) || of (m1 x])] = max Rr, [log [:)]
(27)
where Uniform(O, 1), x ~ (xr) and xı ~ Pama (x1).
First, we rewrite the KL divergence as combination of entropy and cross-entropy KL (p (] H(p.g) 9) H(p):
=
B KL (p
x)
|| (x [))] =,,, (p( Ix), o |x))]
(28)
(H( [))]]
(29)
We observe that the entropy term does not depend on parameters 0. Consequently we disregard when optimising the variational distribution of (#1 x.). Second, we rewrite the cross-entropy term:
Fr, [H(p(< I d [))] = - [log (x1 [+)]
(30)
Therefore, the second part of eq. (27) corresponds to negative cross-entropy and minimisation corresponds to maximisation of negative cross-cntropy.
A.2 Flow Matching Special Case of Variational Flow Matching
Theorem 3. Assume the conditional vector field (e( (1) is linear in æ1 and is the form
ue(x/<11 (x()) + b,(2),
(21)
where Axxx): 1] + = and (zx) : [0, 10. + R.. Moreover, assume that A an invertible matrix and q (x1 r) -\an (x), (x)) where Ex(x) = (z))
Then, VFM reduces to flow matching.
Proof. Let us substitute the assumed form of of (21 x) into the VFM objective:
E, E [log qf (x |x)]
(31)
Ex
log (2m---/2 D/2 [E.(x)) 1/2
exp (- (11A(()))) (01 14 (x) (1)
(1)))
(32)
[A(x (m1 (z())) [1]] + 1 Ben
B1,,,
Dlog (2xx + log Ex(x)]
(33)
(A((2))) bx(z)) (A,(=)) (4) be
+c
(34)
|u/(x 21) of (x) (x) +C,
(35)
which is what we wanted to show.
A.3 Decomposition of the Flow
Theorem 1. Assume that the conditional vector field ue(x | x1) ) is linear in æ1. Then, for any distribution (x x) such that the marginal distributions coincide with those of pe(x1 x)), the corresponding expectations of (z(x)) x1) are equal, i.e.
Ert(zs)e) [ur(x] æ1)] Ep((a1le) [ur x1)].
(12)
15

---

Proof.
Applying the linearity condition, we rewrite the conditional vector field (x(x)11) as such:
Ue ( π1) (x())=1 + (x)),
(36)
where Ax(x): [0, RD + R x RD and (,(x): [0, x RD - R.. Then, we substitute into equation:
(x)( [ur( æ1)] (z)l)) [A(x)) + (x)) = A. (a. [mi] + (z)).
(37)
(38)
Then, we know, that for any distribution r such that the marginal distributions coincide with those of p the following holds:
E, [x] Eppla.
(39)
Applying this fact to eq. (38) obtain:
(p(())))) [u æ1)] = (x)x) (m)l=) [21] (x))) (x (b) [x) + (x))
(40)
(41)
(x (m) (4 x1)),
(42)
which is what we wanted to show.
A.4 CatFlow
Let G = (V,E) be graph with node and edge features given by H, E Rd and H. E RIVI xV|xxd respectively, and æ denote the graphs and its features. Moreover, let E S be permutation and its associated permutation matrix, such that the action of the group is defined
π H : PH
* H. = PH.PT.
To simplify notation, we will simply write x to denote the above operation. Lemma 1. If (z() is permutation equivariant w.r.t Sly, then so is Ut.
Proof. Note that
( x)
(T x))
πI
(.((x) - x
1-t
where the last step follows from permutation equivariance. Moreover, since v acts on æ through permutation matrices, leverage the distributive property of linear operators, i.e. conclude that
pe(x)
π
(po(x) x)
1 vz(x),
finishing the proof.
Theorem Let po be exchangeable distribution - c.g. standard normal distribution and pe (x) be permutation equivariant. Then, all permutations of graphs are generated with equal probability.
Proof. By result we know that pa(x) being permutation equivariant implies that vx(x:) is per- mutation equivariant. Moreover, if we let I(x) = x + So dt(x))dt, the for π S, we have that
I(T: x) = / u2( - rt = π x S π. (xx)dtt = I(x),
where again the last step follows by basic properties of linear operators. Therefore, since Po assigns equal density all permutations æ, the resulting distribution P1 preservers this property, which what wanted to show.
16

---

A.5 Derivation of the Score Function
In this subsection we want derive the equation that allows us express the score function V (r)) in terms of the posterior probability path pr æ1). Notice that
log pi(x) VaP((x)
(43)
pr(x) v / pi(z p1)p(s))d
(44)
V xı)dx
(45)
pc(x)
Pelar
xi (i)p((r)
Vz log an dddd
(46)
pela)
-f
pe(ar |[r(an
V ( an )d
(47)
(pp(l)l)) (a)
V log (e x1)]
(48)
A.6 Stochastic Dynamics with Variational Flow Matching
In this subsection, we discuss how the VFM framework can be applied construct stochastic generative dynamics and how it relates to score-based models.
First, let consider the marginal vector field (z). It provides the deterministic dynamic that can be written as the following ordinary differential equation (ODE):
dx (t)
(49)
For the vector field (x) we know that - starting from the distribution Po(x) - it generates some probability path (x). However, as we know from [54, 3], if we have access to the score function V, log (I) of distribution (ar), we can construct stochastic differential equation (SDE) that, staring from distribution (r(),, generates the same probability path (z)):
gf. v log (x)) dt + g,dw, 2
dx (x(x)
(50)
where g : + Rzo a scalar function, and w is standard Wiener process.
Given that
ue(x)) = E (x(())))) [ur(x x1)] and V logp(())) Epp(e1la) lw) V logpe(1 [
(51)
can rewrite eq. (50) in the following form:
dæ (x)ddtt + g,dw, where
(52)
g v. log x1).
(z) (ay(a) [x m1)] and ú x1) ue (æ 21)
(53)
Importantly, the function Sr does not affect the distribution path pe (z): it only changes the stochasticity of the trajectories. In the extreme case when 9t 0, the SDE in eq. (50) coincides with the ODE in eq. (49).
Moreover, if we construct the stochastic dynamics in this way, we obtain acore-based models as special case. In score-based models [54] the deterministic process that corresponds to probability path (x) has the following form:
da = u (z)dt where uxxx= = f((x) + of v log (x)), 2
(54)
where fe(x) is tractable. Substituting this u, (x) into eq. (50) we obtain:
dr = [f(()) of 2 [z) dt + d.d.
(55)
17

---

The SDE in eq. (55) only depends on the score function, so in wore-based models, the aim is to learn the score function.
Now, let us note that similarly to vector field ue (2), the drift term üe (x) in eq. (52) can be expressed in terms of end point distribution pe (21 x). Consequently, having variational approximation of end point distribution of (x1 2)), allows to construct an approximation of the drift term (x)):
oy (z) of [ū xx1] = (z) gi s (x),
(56)
where (1) = of of (m))a)) [we( xi) and (x) (x)())) V2 log [1)]-
(57)
Thus, we may define the following approximated SDE:
dr = (x)dt + gidw or dr = [1(()) + of s,())) dt + gidw.
(58)
2
Then, eq. (58) is not just simply new dynamic, it is family of dynamics that admits deterministic dynamics as special case when = O. Importantly, the only thing we need to construct the stochas- tic process in eq. (58) is approximation of the end point distributions of (2r x)) Additionally know that when Pe (21 z) qf (x1 z), (x) o. (z), the processes coincide for all functions 9t- Therefore, we can train the model with the same objective as VFM.
A.7 Variational Flow Matching as a Variational Bound on the Log-likelihood
In this subsection, we leverage the connections of VFM with stochastic processes to show that reweighted integral over the point-wise VFM objective defines a bound on the data likelihood the generative model.
Theorem 4. Rewrite the Variational Flow Matching objective as follows:
(V) = E [C (t, where (t,,,) -Ez, [log x)]
(25)
Then, the following holds:
Ez, [log of (2 & Eg, [t))) +
(26)
where Ae(x)) is a non-negative function and C' is a constant.
Proof. Let consider the two stochastic processes, that discussed in appendix A.6:
dx (e(x)dt gedw, where dx vi (x)dt + g.dw.
(59)
Note that they both start from the same prior distribution (z). The first one, by design, generates probability path Pz(z) and ends up the data distribution Pdata (x) pi (z). The second process generates some probability path qf (a:) that depends on the variational distribution (f11/a1)) (z)
We want to find variational bound on KL divergence between P1 (x) and of (z). We start by applying the result from [1] (see Lemma 2.22):
1 [ū(()) o0())]12
KL (p1)al)) < Ee,x
(60)
2g²
1
// (pe I x)) - of x)) (.(x)21) )dz
E,,x
(61)
2g
2
1
( (pe(Er 1x)
< E
- af (I1 x)) بي ( |dz
(62)
2g7
1
(loxler ()) - ( || |l
27
< Er,zx
(63)
207
18

---

Now, let us introduce two auxiliary functions:
4(x) gi
(x(x) sup (-(مم\) and (xx))
(64)
If we utilise (z) to write down the following bound, we see that
(x())) 2
(flento x) - Le (an x)) dey
27
KI.
S B,,x
(65)
Next, we can apply Pinsker's inequality, which states that for two probability distributions and the following holds:
I [o(x) /10(0) ((=))((x)) I 2KL(p)
(66)
Applying it to the inner integral, have:
KL (pI s E, ( (c((11
(p((ar I 1x) (<1 m
(67)
We may rewrite the left part of inequality combination of the data entropy and the model's likelihood, where only the likelihood depends on parameters
KL (pi (I) (xf --H (x1)) Ez, [log (11))]
(68)
The right part of inequality can be rewritten as an expectation of entropy that does not depend on any parameters 0 plus reweighted VFM objective with weighting coefficient Xe(x)):
Er (L pr( | x))l (x1 I [1)]] = Er, ()(z))H(p(<z
xx)]]
(69)
Er [de(z) z)]
(70)
We see that this reweighted version of the VFM objective defines upper bound on the model likelihood, which was what we wanted show.
A.8 Stochastic Dynamics under Linearity Conditions
In this subsection, we discuss the connection between VFM and stochastic dynamics under the condition of linearity in 1 of the conditional vector field ux(2 x1) and conditional score function V, log (x x.).
As discuss in section 3.2, under the linearity condition, may express un (x) in terms of any
distribution of end points (r .r) if it has the same marginals as (r z). In appendix A.5, demonstrate that the
function V, log (I()) also be expressed in terms of end point distributions Pa (r w). Therefore, the score function may also be equally expressed in terms of distribution (2(1) ar) if has the same marginals pt (3) xr). This fact is easy to show in the same way as we present in appendix A.3.
Consequently, under the lincarity conditions, the drift term ü, (x) can also be expressed in terms of Tx( | I)), as it is just a linear combination the vector field ux(x) and score function V, log pr (z). Hence, the transition from the distribution p(Z) I)) to some distribution r( x) does not affect the discussion of connections of stochastic dynamics in appendix A.6.
Furthermore, the transition from distribution Pa(21 .2) to some distribution re(21 2) does not
affect connections of the VFM objective with the model likelihood that we discuss appendix A.7. It is easy to see that in the derivations, However, as we discussed, they are not affected by the transition from (r(21 xr) to some (z(2) x).
only rely on functions 7z (xr) and of(x)) (see eq. (60)).
Therefore, may repeat all the same derivations for some 7(21 (z) using factorized distribution.
19

---

B Algorithms
B.1
General Variational Flow Matching
Algorithm 1 Variational Flow Matching
Training
Sample 21 from data and 2g ~ Po()))
Sample
U(0, t + (1 -]
Compute
Compute loss L - log of (01 xx))
Backpropagate.
Generation
Sample zo Po(oo))
B,plea/zel-a 1-t-e dt
Solve ODE To + 1:
Return
B.2
Categorical Variational Flow Matching (CatFlow)
Algorithm 2 Categorical Variational Flow Matching (CatFlow)
Assume (2 x,) = Il Cat(2f (z,)) where p is learnable
Training
Sample from data and mo Po(zo)
Sample & ~ 24(0 1)
Compute I; tr + (1- t)
Compute cross-entropy loss C == - Ed= Ek 1x k) 1ht (z) Backpropagate.
Generation
Sample 2o po(()))
Solve ODE x = To + pt=1 pe(2e)=Be df 1-845
Return Ii
B.3
Gaussian Variational Flow Matching
Algorithm 3 Gaussian Variational Flow Matching
Assume (x xx) = II Nox v4(x)) 1) where is learnable
Training
Sample ø1 from data and zo Po(u))
Sample f ~ U(00 1)
Compute 2; tw + (1-th t)
Compute mean squared error loss L = Eb (14)))) xf112
Backpropagate.
Generation
Sample xo Po(o))
Solve ODE Do + p(=1 p=(2))-Be di
1-tte
Return I1
20

---

C Additional Results
C.1
Detailed results
Here, we provide the results for CatFlow with standard deviations, as computed in [20] through multiple seeds.
Ego-small
Community-small
4 < [V] v 18
12 < [V] S 20
Degree +
Clustering +
Orbit +
Degree
Clustering +
Orbit
0.013 + 0.007
0.024 + 0.009
0.008 + 0.005
0.018 + 0.012
0.086 = 0.021
0.007 = 0.005
Enzymes
Grid
10 [V] < 125
100 S [V] v 400
Degree 4
Clustering 4
Orbit +
Degree
Clustering
Orbit
0.013 + 0.012
0.062 + 0.011
0.008 + 0.007
0.115 + 0.010
0.004 = 0.002
0.075 + 0.071
QM9
ZINC250K
[ I 9, atom types
[
38, atom types
Valid
Unique t
FCD 1
Valid
Unique 1
FCD
99.81 + 0.03
99.95 + 0.02
0.441 + 0.023
99.21 + 0.04
100.00 + 0.00
13.211 + 0.12
C.2 Extra results SBM and Planar Graphs
Results on Stochastic Block Model and Planar Graphs. We ran extra experiments for (standard) flow matching and Dirichlet flow matching. We observe that CatFFlww obtains SOTA performance on tasks and metrics. Moreover, it is worth noting CatFlow was significantly faster to train than Dirichlet FM due computationally cheaper forward process.
Model
Deg
Clus |
Orb J
V.U.N. 1
Stochastic Block Model
SPECTRE [35]
1.9
1.6
1.6
53%
ConGress [58]
34.1
3.1
4.5
0%
DiGress [58]
1.6
1.5
1.7
74%
Flow Matching [27]
10.2
2.0
3.2
22%
Dirichlet FM [56]
1.7
1.5
1.4
80%
CatFlow
1.5
1.5
1.4
85%
Planar Graphs
SPECTRE [35]
2.5
2.5
2.4
25%
ConGress [58]
23.8
8.8
2590
0%
DiGress [58]
1.4
1.7
75%
Flow Matching [27]
5.1
5.6
5.5
30%
Dirchlet FM [56]
1.5
1.3
1.5
80%
CatFlow
1.4
1.3
1.5
80%
21

---

D Experimental setup
D.1 Model
To ensure comparison on equal terms to baselines, we employ the same graph transformer network as proposed in [12]. which was also used in [58], along with the same summarize the parametrization of our network here.
setup. We
Just done in DiGress, graph transformer takes input graph (H,, H2) and predicts distribution over the clean graphs, using structural and spectral features to improve the network expressivity, which we denote H Each transformer layer does the following operations:
1. Node Features H and Edge Features He:
(a) Linear Transformation: Apply linear transformations to both H,, and H..
(b) Outer Product and Scaling: Compute the outer product of the transformed features and apply scaling.
2. Node Features H
(a) Feature-wise Linear Modulation (FiLM): Apply FiLM the transformed node features using global features Hg.
3. Edge Features H.:
(a) Feature-wise Linear Modulation (FiLM)): Apply FiLM to the transformed edge features using global features Hg.
4. Self-Attention Mechanism:
(a) Linear Transformation: Apply linear transformation to the transformed node features.
(b) Softmax Operation: Compute the attention scores using the softmax function.
(c) Attention Score Calculation: Calculate the weighted sum of the transformed node features based on the attention scores.
5. Global Features y:
(a) Pooling: Apply PNA pooling to the node features H, and edge features H
(b) Summation: Sum the pooled features with the global features H.,
6. Final Outputs:
(a) Node Features H', Obtain updated node features after the attention mechanism.
(b) Edge Features H Obtain updated edge features after the attention mechanism.
(c) Global Features Hy: Obtain updated global features after summation.
Here, the FiLM operation is defined as:
FiL,(MM, M2) M, + (M,)) M2 + M2
for learnable weight matrices W, and W,, and PNA is defined as:
PNA(X) cat(max (X), min(X), mean(X). std( W.
D.2 Hyperparameters and Computational Costs
We report the hyperparameters here:
All models were trained until convergence. Furthermore, all data splits are kept the same as in [20]. and hidden dimensions are kept the same as [58]. All experiments were run on single NVIDIA RTX 6000 and took about day to run.
22

---

Hyperparameter
Abstract
QM9/IINC200
Ablation
Optimizer
AdamW
AdamW Cosine Annealing
AdamW Cosine Annealing
Scheduler
Cosine Annealing
Learning Rate
2 10-4
2 10 10-4
2 2-100- 1 1-10-12
Weight Decay
1 10-12
1 10
EMA
0.999
0.999
0.999
Table 3: Hyperparameter setup.
E Detail CNFs
To compute the resulting distribution pa for CNP, one can use the change of variables formula:
[(t))po(x)) = po ) det
Oz (xr)
(71)
This induces a probability path, i.e. a mapping p: [0, 1] RD E= such that J pe(x)dz 1 for all & E [0, 1]. We say that generates this probability path given a starting distribution po- In theory, one could try and optimize the empirical divergence between the resulting distribution p and target distribution, but obtaining a gradient sample for the loss requires us to solve the ODE at each step during training, making this approach computationally prohibitive.
One way to assess if a vector field generates specific probability path is using the continuity equation, i.e. can assess whether 12 and pr satisfy
0t Pi + V (p((z)er(z)) = 0,
(72)
where V is the divergence operator. Note that by sampling po, a new sample from can be generated by following this ODE, i.c. integrating
I = xD + S (x)))dt
(73)
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our paper follows the structure of the abstract and introduction directly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We added a section addressing limitations section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide all proofs in detail in appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Besides code, we provide the model description and hyperparameters in appendix Moreover, our model and datasets are common the field, e-g. [20], so no novel experimental details are required. Our main contribution, the objective, is clearly stated section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide all code to the experiments. Moreover, all data is publically available, and will be provided in the repository, including data processing.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
Chblic//uides/Codee//e ietpoommsssiooooliiiooeo (elollicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nnpss wwtppcccccc///cccccccccccccc..lt ceallic/guidese http//uudes///eee (CodeSubmissse ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the hyperparameters and experimental detail appendix Moreover, we took all of our choices from previous works of [20]. which is also publicly available. Last, our code will made public.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide error bars and how they are computed in appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurdl: ccp wttpplpcccc//ccccccccccccccttttt Ceteiiic//heeeeeee itt.
Answer:
[Yes]
Justification:
We read and checked the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We reflect on this in our conclusion, i.e. section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not publish such models and as such do not require safeguards for them.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the authors of the relevant assets where used, see section All relevant information about licenses is publicly available and respected in our code.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We provide new objective to train and use previous code/assets for the experiments which are referenced.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not work with subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "y0DAWog0jI", "reviewer": "Reviewer_5EeZ", "summary": "This paper introduces a new variational inference framework of flow matching with a focus on applying the framework on discrete data generation. Instead of using the squared norm in standard flow matching, the paper proposes a variational distribution to the conditional path, which is used in the vector field. The paper shows that when the variational distribution is identical to the conditional path, the approximate vector field equals to the target one. The paper then uses mean-field factorisation of the variational distribution and applies linear conditional vector field. Based on the proposed framework, the paper mainly applies it for discrete data generation including the tasks of abstract graph generation and molecular generation. Experiments show that the proposed method outperforms other diffusion-based or flow-matching-based methods.", "questions": "but not all of them are weaknesses.\n\n1. In standard flow matching, one needs to minimise the squared norm between the parameterised vector field and the true vector field. That's why one needs to compute the (conditional) vector field. But in variational flow matching, it seems that the final objective is the maximisation of the data likelihood in terms of $\\theta$. I wonder how the vector fields come into play in this case. Are they used for computing the variational distribution somehow?\n\n2. Related to the previous question, it would be good to have more details of the parameterisation of $q$ or the implementation of $\\theta$.\n\n3. Mean-field approximation of the variational distribution and linear formulation of the conditional vector field are used, more for efficiency consideration.  Can the authors provide more discussions on when these approximation and assumption will be the bottleneck of performance?\n\n4. Although the code is provided, it might be good to provide an algorithm of pseudo code in the paper.\n\n5. It would be good to have more explanations of Figure 1.", "limitations": "The authors adequately addressed the limitations", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The technical contribution and significance of the paper are good. The paper provides a novel angle of formulating flow matching as a variational inference problem, which is theoretically sound and conceptually intuitive.\n\n2. The paper derives a general framework first and then implement it on the task of discrete data generation. The paper also provides comprehensive theoretical analysis on the connections to related methods.\n\n3. The experimental results of the proposed method are convincing.\n\n4. The paper is well-written in general.", "weaknesses": "In general, the quality of the paper is good. But I have a few questions on clarity. Please see below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 5EeZ,\n\nFirst of all, we would like to thank you very much for your positive words and useful points about clarity. We will first answer each question and then propose concrete changes to the paper to address them in the final version.\n\n> In standard flow matching, one needs to minimise the squared norm between the parameterised vector field and the true vector field. That's why one needs to compute the (conditional) vector field. But in variational flow matching, it seems that the final objective is the maximisation of the data likelihood in terms of $\\theta$. I wonder how the vector fields come into play in this case. Are they used for computing the variational distribution somehow?\n\n\nIf we understand the question correctly, the reviewer is wondering how the role of the conditional vector field in VFM differs from its role in standard FM, particularly during training. \n\nDuring *generation*, we explicitly use $u_t(x | x_1)$ to compute the approximate velocity field  $v_t^{\\theta}(x) = \\mathbb{E}_{q_t}[u_t(x | x_1)]$. \nHowever, during *training*, we indeed do *not* compute it explicitly to evaluate the objective. \n\nWith that said, the conditional velocity field $u_t(x | x_1)$ generates a probability path $p_t(x | x_1)$, which in turn defines a posterior probability path $p_t(x_1 | x)$, which we approximate with a variational distribution $q_\\theta(x_1 | x)$. In other words, the learned variational approximation implicitly depends on the choice of conditional velocity field $u_t(x | x_1)$, even if this field does not show up explicitly in the the objective. Please let us know if this answers your question, we would be happy to clarify further. \n\n**Concrete steps:** We will dedicate a paragraph in the paper to explaining this better. We will also add one algorithm block for training and one for sampling/generation that clearly summarises the steps.\n\n> Related to the previous question, it would be good to have more details of the parameterisation of $q$ or the implementation of $\\theta$.\n> \n\nYes, we fully agree! Since the goal in VFM is to predict $\\mathbb{E}[x_1 \\mid x_t]$, the goal is to train a network to predict the parameter needed to compute this expectation based on input $x_t$ at time $t$, e.g. the mean of a Gaussian distribution or the probability vector of a categorical distribution. \n\n**Concrete steps:** We will add a detailed section in the appendix describing the parameterisation of $q$ and $\\theta$. Specifically, we will do this for 1) the general case, 2) the Gaussian case, and 3) the Categorical case in the final version. \n\n> Mean-field approximation of the variational distribution and linear formulation of the conditional vector field are used, more for efficiency consideration. Can the authors provide more discussions on when these approximation and assumption will be the bottleneck of performance?\n> \n\nAs we understand it, the reviewer has two related questions here,  pertaining how model expressivity/performance are affected by (1) the  mean-field parameterization and (2) the assumption of linearity of the conditional vector field in $x_1$.\n\n*Implications of (1)*: Theorem 1  states that if (2) holds, the mean-field parameterization is not an “approximation’’ as much as it is a “simplification”. \n\nThe approximate flow field $v_\\theta(x)$ matches the flow field $u(x)$ *exactly* whenever for each component of the posterior mean $\\mu_1^d$ under $p_t$ we have that $\\mu_1^d =  \\mathbb{E}_{q_t}[x_1^d | x]$. \nThis is to say that the *only* requirement in VFM under assumption (2) is that each component $d$ of the mean of the variational distribution must match the mean of the posterior probability path. A mean-field parameterization therefore does not compromising expressivity/performance at all in this setting.\n\n*Implications of (2)*: The assumption of linearity in $x_1$ for the conditional flow field holds for most existing flow matching methods and indeed an analogous assumption hold for many diffusion-based models as well. We would therefore not consider this a particularly strong restriction in practice, and certainly not something that might become a bottleneck for model performance relative to existing flow matching and diffusion-based methods. \n\n**Concrete steps:** We will dedicate an extra paragraph in the paper to elaborating on these points. \n\n> Although the code is provided, it might be good to provide an algorithm of pseudo code in the paper.\n> \n\nWe fully agree. This would also help highlight the fact that, from an implementation point of view, VFM is not more complicated than FM. \n\n**Concrete steps:** We will add 1) a general code block, 2) a code block for the Gaussian case, and 3) a code block for the categorical case. We will also add a jupyter notebook with these cases as examples, and we are currently working on a small pytorch library to supply all code which is to be released soon.\n\n> It would be good to have more explanations of Figure 1.\n> \n\nYes, we agree. Moreover, we think that in general some extra figures should added, with the aim of making the difference between VFM and standard FM more clear.\n\n**Concrete steps:** We will add this description in. Moreover, we will add the extra figures mentioned above.\n\nAgain, thank you so much for the time you spent reviewing our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YNIpxRmXFe", "reviewer": "Reviewer_WAZn", "summary": "This work presents variational flow matching framework VFM and introduces CatFlow, an application to generation of categorical data such as graph. The paper reformulates flow matching using variational perspective and with linear assumption on the conditional vector field achieves tractable objective. The authors derive the connection with the score-based models and show that the framework provides variational  bound on the model likelihood.", "questions": "I would appreciate it if the authors could address the weakness above.\nAlso, I recommend more explanation on Figure 1 as it would privde the readers to get better understanding on CatFlow.\n\nIn summary, I believe the paper provides theoretical contributions, but the experimental validation needs more room to improve.", "limitations": "The authors have addressed the limitations at the end of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, the variational framework for flow matching is novel and the provided theoretical works seem solid.\n- Connection with the score-based model provide intuitions on the method.\n- Application to discrete data seems reasonable and the benefits over standard flow matching is well explained.", "weaknesses": "- The reason CatFlow outperforms previous diffusion models is unclear. Is it because VFM provides variational bound on the model likelihood? Ablation studies on why CatFlow outperforms other diffusion models would strengthen the work.\n- CatFlow should be validated on larger datasets used in recent works [1, 2], for example Planar and SBM, since community-small and ego-small datasets consists of very small graphs which is not suitable for evaluating generative models. \n- In particular, validity metric should be used to evaluate the method, instead of only relying on MMD. Evaluating only with MMD is not appropriate as MMD may fail to catch important graph characteristics, for example, small MMD does not guarantee the generated graphs is actually a community graph. Evaluating on Planar or SBM dataset and comparing the results of V.U.N. (valid, unique, and novel) is necessary to strongly argue the advantage of the proposed method.\n- Comparison with discrete diffusion model like DiGress should be done on more datasets other than QM9. This also leads to experiments on Planar or SBM datasets or in molecular generation tasks, like GuacaMol dataset.\n\nMinor correction: citation in line 148 should be fixed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer WAZn,\n\nWe would like to thank you very much for your positive comments and thoughtful questions about our work. We will first answer each question and then propose concrete changes to the paper to address them in the final version. Since some questions are related, we grouped them together in our answer.\n\n> The reason CatFlow outperforms previous diffusion models is unclear. Is it because VFM provides variational bound on the model likelihood? Ablation studies on why CatFlow outperforms other diffusion models would strengthen the work.\n> \n\nThanks a lot for this question. Our hypothesis is that CatFlow outperforms diffusion-based baselines for similar reasons that continuous FM has proven a strong competitor to diffusion in e.g. image generation, which is to say that a linear interpolation in the probability path appears to work well in practice.  It is of course difficult to demonstrate conclusively why any method works better than any other method. Indeed, to our knowledge no such demonstration exists for continuous FM either.\n\n**Concrete steps:** We will discuss the intuition for why CatFlow might outperform diffusion-based baselines  more clearly in the text. \n\n> CatFlow should be validated on larger datasets used in recent works [1, 2], for example Planar and SBM, since community-small and ego-small datasets consists of very small graphs which is not suitable for evaluating generative models.\nIn particular, validity metric should be used to evaluate the method, instead of only relying on MMD. Evaluating only with MMD is not appropriate as MMD may fail to catch important graph characteristics, for example, small MMD does not guarantee the generated graphs is actually a community graph. Evaluating on Planar or SBM dataset and comparing the results of V.U.N. (valid, unique, and novel) is necessary to strongly argue the advantage of the proposed method.\nComparison with discrete diffusion model like DiGress should be done on more datasets other than QM9. This also leads to experiments on Planar or SBM datasets or in molecular generation tasks, like GuacaMol dataset.\n> \n\nThank you for the suggestion – we fully agree! We have run CatFlow on Planar and SBM now and have obtained SOTA performance on these datasets as well, being on par / a tiny bit better than Digress on all metrics, including V.U.N. (all reported in the added PDF). We will report these values in the final version of the paper. Moreover, we would like to highlight that in the submitted manuscript, we also ran large-graph experiments (see appendix B), on which we did obtain strong performance as well. \nSadly, we have not been able to run the GuacaMol dataset in time for the response deadline, but we will aim to include that in the final version of the paper.\n\n**Concrete steps:** Add results Planar and SBM in the final version of the paper, including extra metrics. Where appropriate, we will rerun the existing experiments with more metrics to provide a better comparison to existing methods.\n\nWe will also make sure the citation mentioned will be corrected and the text in the figure will be improved. \n\nLast, thank you so much for your time reviewing our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIbKPJuNjy", "reviewer": "Reviewer_vH9B", "summary": "This paper proposed a variant of flow matching for graph generation from a variational perspective, called CatFlow. The main idea is not to estimate the marginal velocity, but instead to estimate the posterior first and then evaluate the marginal velocity as the expectation of the conditional velocity w.r.t. the posterior. In the case of linear conditional velocity and Gaussian variational approximation, the equivalence to vanilla flow matching has been established and connection to score-based models have been discussed. A simplified mean-field variational approach has also been proposed based on the observation that marginal posterior matching (for individual dimensioins) is enough for posterior mean estimation for linear conditional velocity. Experiments on several graph/molecule generation tasks demonstrate the effectiveness of the CatFlow.", "questions": "1. Although it is reasonable to match the marginal posterior using mean-field approximation, it is still not clear if this simple mean-field approximation is good enough to capture the posterior mean. Can the author provide some intuition when the mean-field approximation is good enough?\n\n2. In flow matching, an error bound for the generated distribution in term of flow matching error is provided. Can a similar result be provided here in terms of the variational approximation error?\n\n3. How would VFM perform when the conditional velocity is not linear?", "limitations": "The limitations has been adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is written clearly. The variational formulation and the connection to standard flow matching is new. The authors have also shown that a weighted average of the training objective of VFM provides a bound on the log-likelihood of the model.", "weaknesses": "The main weakness of the paper is that the core idea of learning the posterior first (i.e., the variational formulation) and evaluating the marginal velocity using the expected conditional velocity w.r.t. the posterior has already been proposed in Dirichlet Flow [1]. Although the author claimed that Dirichlet Flow uses the specific dirichlet distribution as the conditonal probability, this is mainly a design choice which does not affect the generality of Dirichlet Flow for modeling discrete objectives such as graphs and molecules. Given this, the lack of an appropriate comparison to Dirichlet Flow in the experiments represents a significant drawback of the current paper.\n\nThe paper also lacks a brief introduction to the baseline methods, which would help contextualize the results and better highlight the contributions of the proposed approach.\n\n\n[1] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05841, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer vH9B,\n\nWe would like to thank you for the useful questions and positive words about our work. We will first answer each question and then propose concrete changes to the paper to address them in the final version. \n\n> The main weakness of the paper is that the core idea of learning the posterior first (i.e., the variational formulation) and evaluating the marginal velocity using the expected conditional velocity w.r.t. the posterior has already been proposed in Dirichlet Flow [1]. Although the author claimed that Dirichlet Flow uses the specific dirichlet distribution as the conditonal probability, this is mainly a design choice which does not affect the generality of Dirichlet Flow for modeling discrete objectives such as graphs and molecules. Given this, the lack of an appropriate comparison to Dirichlet Flow in the experiments represents a significant drawback of the current paper.\n\nWe agree that this work is directly related to the recently proposed work on Dirichlet FM and that an experimental comparison is warranted in this context. We have included such a comparison in our main response.  \n\nTo briefly comment further on the relationship to Dirichlet FM: the cross-entropy loss and the velocity field in equations (9) and (10) of the Dirichlet FM paper are analogous to those in our paper, which (incidentally) are *also* described in Equations (9) and (10) in our manuscript. From a methodological perspective, the main distinction is that the Dirichlet FM paper is framed in terms of a probability path on the simplex, whereas we arrive at CatFlow by way of a more general variational perspective that admits both categorical and continuous models as special cases. Moreover, CatFlow is not defined through a forward process on the simplex and simply operates in $\\mathbb{R}^n$.\n\n**Concrete Steps:** As shown in the main response, we have now performed additional experiments to compare against Dirichlet FM. As we wrote, we did make an earlier attempt to perform a comparison at the time of submission, but we felt that the (poor) performance we observed in our initial results was not sufficiently representative to merit inclusion. We believe that we have addressed this with these new results and hope this also addresses the main concern raised by the reviewer.\n\n> The paper also lacks a brief introduction to the baseline methods, which would help contextualize the results and better highlight the contributions of the proposed approach.\n\nThat is a fair point! \n\n**Concrete steps:** We will mention the most important baselines in the main text, and add a small description of the other ones in the appendix. \n\n> Although it is reasonable to match the marginal posterior using mean-field approximation, it is still not clear if this simple mean-field approximation is good enough to capture the posterior mean. Can the author provide some intuition when the mean-field approximation is good enough?\n\nWe are not sure why this might be a concern, perhaps the reviewer could elaborate?\n\nMean-field variational inference is known to under-approximate the posterior variance when minimizing an exclusive/reverse KL divergence. This is a consequence of the fact that the variational family cannot capture correlations in the posterior. However, the posterior mean estimates are generally known to be accurate. As such, we have *no* indication that these distributions would *not* be able to provide a good estimate of the mean. This also appears to be born out by empirical results.\n\n**Concrete Steps:** We will amend the paper to provide some discussion on this point. \n\n> In flow matching, an error bound for the generated distribution in term of flow matching error is provided. Can a similar result be provided here in terms of the variational approximation error?\n\n**Answer:**  We are not aware of error bounds in the concurrent works on Flow Matching by Lipman et al.,  Albergo at al., and Liu et al. Is the reviewer referring to the error bounds developed by Benton and colleagues [1]? It might indeed be possible to derive an analogous result starting from an assumed bound on the approximation error in the posterior mean, rather than an approximation error in the velocity field. However, this is not something that we have considered in the context of this submission. \n\n[1] Joe Benton, George Deligiannidis, Arnaud Doucet. Error Bounds for Flow Matching Methods. TMLR 2024. \n\n> How would VFM perform when the conditional velocity is not linear?\n\nThis is a very interesting question. \n\nJust to make sure we are on the same page: The linearity assumption we care about is *the linearity of the conditional field in the endpoint*, i.e. the fact that $u_t(x \\mid x_1)$ is linear in $x_1$. This is subtly different from the fact that in flow matching, $u_t(x \\mid x_1)$ itself is a linear interpolation, which is a design choice in FM typically referred to as the ‘optimal transport’ formulation. \n\nThe linearity in $x_1$ is, however, a standard assumption often made in e.g. diffusion models and flow matching models. We show that under this common linearity assumption, optimising VFM using a 'mean-field approximation' can actually find the exact solution. In this setting the mean-field approximation is not an approximation at all! \n\nThe question of what happens if this linearity assumption does not hold is one which, as far as we know, cannot be tackled by the standard diffusion/flow matching techniques out there. However, using VFM one could simply resolve to learning the loss $\\mathcal{L}(\\theta) = -\\mathbb{E}[\\log q_t^\\theta (x_1 \\mid x)]$ through stochastic gradients for some distribution $q_t^\\theta$ that is not fully factorized. This is actually a promising direction for future research and might be a good example of where a variational point of view can offer new perspectives relative to the standard point of view.\n\nThank you so much for your time and effort reviewing our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thoughtful reading of the manuscript and their detailed comments. We are happy to hear that reviewers overall appear in agreement that this is a clearly written paper that provides a novel variational perspective on flow matching, develops  useful connections with related approaches, and demonstrates good empirical results on graph generation tasks. 

We respond to individual points and questions by each reviewer below. We would like to highlight two points raised by reviewers about additional experimental evaluations. We include results for these evaluations in the attached PDF, and will incorporate these results in the manuscript:

- Reviewer `WAZn` made the helpful suggestion to evaluate on the **Planar** and **SBM** datasets, which we have done. Results show that CatFlow attains SOTA performance on these datasets. We would also like to call attention to additional results on larger graphs in **Appendix B** (which were already in the original submission).
- Reviewer `vH9B` comments that a comparison with **Dirichlet FM** would be helpful. We completely agree. As we wrote in our manuscript, we were unfortunately not able to get this method working sufficiently well in our initial (limited) experiments. We have since invested additional time and effort and are now able to report a comparison that we believe is representative. The results show that even though Dirichlet FM outperforms standard FM in graph generation tasks, CatFlow obtains better performance in the tasks considered in our work. Though not visible in this table, we note that CatFlow also was faster to train in our experiments.
- We added an additional comparison with a baseline trained with the normal **Flow Matching** objective. To adapt this model to categorical data, we simply select the nearest one-hot vector on the simplex at the final step of generation.

Additional changes that reviewers can expect mainly pertain to points of clarification. In each of our responses, we use the text “**Concrete Steps”** to summarise what changes we intend to make in response to reviewer comments.

Meta-review: The work suggests the use of variational framework in flow matching, a natural step in an interesting topic. This work is timely and important to the community and will be an excellent addition to NeurIPS.
Decision: Accept (poster)