Ergodic Generative Flows
Authors: Leo Maxime Brunswic
Mateo Clémente
Rui Heng Yang
Adam Sigal
Amir Rasouli
Yinchuan Li
Venue: ICML 2025 poster (2025)
Keywords: Generative Flows
ergodicity
non-acyclic
diffeomorphisms
GFlowNets
Spectral Gap
Special Linear group
free group action
nasa dataset
generative modeling
imitation learning
reinforcement learning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We introduce a family  Generative Flow in the strong non-acyclic regime of ergodicity; this mixing property is used to provide universality guarantees.

Abstract: Generative Flow Networks (GFNs) were initially introduced on directed non-acyclic graphs to sample from an unnormalized distribution density. Recent works have extended the theoretical framework for generative methods allowing more flexibility and enhancing application range. However, many challenges remain in training GFNs in continuous settings and for imitation learning (IL), including intractability of flow-matching loss, limited tests of non-acyclic training, and the need for a separate reward model in imitation learning. The present work proposes a family of generative flows called Ergodic Generative Flows (EGFs) which are used to address the aforementioned issues. First, we leverage ergodicity to build simple generative flows with finitely many globally defined transformations (diffeomorphisms) with universality guarantees and tractable flow-matching loss (FM loss). Second, we introduce a new loss involving cross-entropy coupled to weak flow-matching control, coined KL-weakFM loss. It is designed for IL training without a separate reward model. We evaluate IL-EGFs on toy 2D tasks and real-world datasets from NASA on the sphere, using the KL-weakFM loss.  Additionally, we conduct toy 2D reinforcement learning experiments with a target reward, using the FM loss.

Ergodic Generative Flows
Leo Maxime Brunswic*
Mateo Clemente
Rui Heng Yang'
Adam Sigal
Amir Rasouli Yinchuan Li?
Abstract
et al., 2018: Bengio et al., 2023) and IL (Goodfellow et al., 2020; Ho al., 2020; Song et al., 2020b; Rozen et al., 2021; Papamakarios et al., 2021) methods are often considered to be separate approaches, it is common practice to first train a reward model from a dataset and then use RI. techniques to solve an IL task (Chung et al., 2024; Zhang et al., 2022b;a; Sendera et al., 2024).
Generative Flow Networks (GFNs) were initially introduced on directed acyelic graphs to sample from an unnormalized distribution density. Re- cent works have extended the theoretical frame- work for generative methods allowing more flexi- bility and enhancing application range. However, many challenges remain in training GFNs in con- tinuous settings and for imitation learning (IL), including intractability of flow-matching loss, lim- ited tests of non-acyclic training, and the need for a separate reward model in imitation learning. The present work proposes a family of generative flows called Ergodic Generative Flows (EGFs) which are used address the aforementioned is- sues. First, we leverage ergodicity to build simple generative flows with finitely many globally de- fined transformations (diffeomorphisms) with uni- versality guarantees and tractable flow-matching loss (FM loss). Second, we introduce a new loss involving cross-entropy coupled to weak flow- matching control, coined KL-weakFM loss. designed for IL training without separate reward model. We evaluate IL-EGFs on toy 2D tasks and real-worle datasets from NASA on the sphere. using the KI-weakFM loss. Additionally, we con- duct toy 2D reinforcement learning experiments with a target reward using the FM loss.
Among II. methods, normalizing flows (NFs), although ca- pable of leveraging advanced neural ODEs (Chen et al., 2018; Grathwohl et al., 2018), have not achieved state-of- the-art performance due to architectural constraints (i.e. the requirement of sequentially combined parameterized dif- feomorphisms) and numerical instabilities (Caterini et al., 2021; Verine al., 2023).
While NFs deterministic models, stochasticity has proven to be crucial component of IL generative mod- els, such as diffusion models as its simplest embodiment (Ho et al., 2020; Song et al., 2020b). However, diffusion models face a challenging trade-off between loss function tractability and the time-consuming denoising process for generation. This trade-off drives continued efforts to accel- erate the diffusion inference process (Lu et al., 2022; Tang et al., 2025; Chen al., 2024). Such issues are particularly important when the computational and time budgets of gen- eration are highly constrained, which is typical on real-time tasks or mobile devices. The present work leverages the flex- ibility of Generative Flow Networks (GFNs) (Bengio et al., 2021; 2023) to address these issues by building simple yet highly expressive models with short sampling trajectories.
1. Introduction
Generative Flow Networks (GFNs), originally formulated for RL tasks, are family of generative methods designed to sample proportionally based on a reward function r = sxx which represents the density of a target distribution k. They were initially restricted to directed acyclic graphs, but have since been extended other settings. Subsequent work has extended GFNs to generalize to continuous state spaces (Li et al., 2023; Lahlou et al., 2023) and non-acyclic structures (Brunswic et al., 2024).
Generative models aim to sample from a target distribution i on a space S. possibly conditioned on additional context or variables. The target distribution x either specified by an unnormalized density with respect to a background measure X, as in reinforcement learning (RL), or defined by dataset of samples DD as in imitation learning (IL). While RL (Brooks et al., 2011; Sutton, 2018; Haarnoja
Equal contribution 'Huawei Technologies Canada, Noah's Ark Laboratories * Huawei. Correspondence to: Lco Maxime Brunswic mleo.maxime.bl brunswic @ h portners.com>
Despite these advancements, GFNs still face four key chal- lenges that hinder their application in RL and IL settings. First, although it has been suggested that GFNs themselves could be used as reward model in the IL setting (see Ben- gio et al. (2023) p37), previous works (Zhang et al., 2022b;
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ergodic Generative Flows
Lahlou et al., 2023) train separate reward model on sam- ples from K, effectively reducing the problem to RL task. The need to train an additional reward model, which results in additional training and computation costs, stems from a lack of control of the so-called flow-matching (FM) property of the flow. Indeed, the FM-loss is zero by construction of the GFN self-defined reward model.
2. Preliminaries: GFN for RL and IL
In this work, we focus on generative flows in the continuous setting, encompassing scenarios where the state space S is either a vector space, 8 = R or potentially a Rieman- nian manifold (Lee, 2018; Gemici et al., 2016). Both are instances of Polish spaces that admit a natural finite back- ground measure À (usually the Lebesgue, Haar (Halmos, 3) or Riemmanian volume measures).
Second, the FM training loss presents significant challenges in continuous settings, its evaluation is intractable in naive implementations (Li et al., 2023), necessitating the training of an imperfect estimator when directly enforeing the FM constraint. Addressing this issue requires either handcrafting or training an additional backward policy mi_, or resorting to higher-variance loss functions, such as the Detailed Balance (DB) (Bengio et al., 2021) and Trajectory Balance (TB) (Malkin et al., 2022) losses.
A measure μa is dominated by v, denoted μ < v if v(A) O = ((A) = for any measurable A C S; then, the Radon-Nikodym derivative of μ with respect to v, denoted du E L' (v),, is the unique v-integrable function ip such that yv = jr. A Markov kernel т X + 3 is a stochastic map denoted ((x). The image measure of by π denoted for is defined by μot (A) = Soex P(r(()) E A)dp(æ) for any measurable c y. Their tensor product is the kernel I @ (A + B) == Saça ((x((x) E B)du(a). Even when i is an unnormalized distribution, we write je for "the law of x is "²)
dy
Third, the acyclicity requirements (Lahlou et al., 2023) mean that additional structures must be manually crafted. Cycles appear naturally in naive implementations and RL environ- ments. Brunswic et al. (2024) offers a theory for non-acyclic flows and stable FM losses, but these have yet to be tested in strongly non-acyclic settings.
We adopt a formulation of generative flows that differs from Lahlou et al. (2023) and Brunswic et al. (2024), providing the necessary flexibility for the subsequent sections. Let (S, A) be a Polish space with a finite background measure. A generative flow on S is the data of a star forward policy, i.c. a Markov kernel T", S + S, and a star outflow, i.c. a finite non-negative measure F", on S.
Lastly, Li et al. (2023) argues that exact cycles are negligi- ble in the continuous setting due to their zero probability, whereas Brunswic et al. (2024) argues that 0-flows, ergodic measures & for nI,, need to be addressed. Otherwise, the divergence-based losses used by Bengio et al. (2023) may become unstable in the presence of g - a prediction that remains untested.
With unnormalized initial and terminal distributions Finit and Fierm it induces Markov chain (2))221 defined by 81 and 8,41 -T,,, (s,). Define the sampling time T E N as the random variable given by² IP 2 1)
We present Ergodic Generative Flows (EGFss, which ad- dresses the aforementioned key limitations. EGFs are built using a policy choosing at a given state from finitely many globally defined transformations, i.e. diffeomorphisms of the state space. This allows for tractable inverse flow policy, hence tractable FM-loss. Since we favor finitely many sim- ple transformations, the universality of EGFs is non trivial. We provide provable guarantee with an ergodicity (Walters, 2000; Kifer, 2012) assumption of the group generated by the EGFs transformations.
and IP(r = f/r 2 t) = 2 a(Feam++1 F1)
dffterm (s.). The sampler
associated with generative flow is obtained by emulating the Markov chain (s,)221 and sampling s, By extending the sampling Markov chain with a source so and sink sy state, we can summarize the flow as follows:
F Bi
Finit
Finrm T 88
80
S
Main contributions:
The following theorem provides theoretical guarantees on the sampler associated with a generative flow.
1. We extend the theoretical framework of generative flows, in particular providing quantitative versions of the sampling theorem for non-acylic generative flows.
Theorem 2.1 ((Bengio et al., 2021: Brunswic et al., 2024)).
Let Finit, and Fremm be unnormalized distributions with Frerm F 0 and let (T²,, be generative flow. If the flow-matching constraint
2. We develop a theory of EGFs, presenting a universality criterion and demonstrating that, in any dimension tori and spheres, this criterion can be fulfilled with four simple transformations.
Finit + F F Frerm + F=,
(2)
'A topological measurable space is Polish its topology completely metrizable and it is endowed with its Borel o-algebra. "Clearly From & Fromm + F so the Madon-Nikodym deriva- tive is well-defined.
3. We propose a coupled KL- We loss to train EGFs directly for IL. This allows to train EGFs for IL without a separate reward model.
2

---

Ergodic Generative Flows
is satisfied then Pini(S) F Fverml (S) and:
zero probability, Brunswic al. (2024) disputes that their generalization called O-flows must still be addressed. 0- flows are measures for which ní, is ergodic (Walters, 2000). Brunswic et al. (2024) predicts that divergence- losses used by Bengio et al. (2023) are unstable if such & exists. This has yet to be tested.
F (S)
E(T) - FS)
+ 1, 87 ~ ~
The pair (F + is trained using gradient descent to min- imize loss function that enforces the flow-matching con- straint in equation (2). Once convergence is achieved, the generative flow sampler yields samples of the random vari- able 8-- Hence, by Theorem 2.1, the generative flow sam- pler samples from Fierm- For example, we can use the most straightforward stable FM-loss:
3. Ergodic Generative Flows for RL and IL
To address the four issues outlined above, we propose Er- godic Generative Flows (EGFs).
Definition 3.1 (Ergodic Generative Flows). Let S be Rie- mannian manifold endowed with its volume measure À and let (#.) be family of diffeomorphisms S + S. An EGF is generative flow, where
Cstable Bxwerrai [(finit + - -ftrrm f4 (2)*(()) (s) FM..
(3)
where finit dA Storm dFtarm and Jz dx
dFiitt
dF=
dx
d(F= +i)
In general, both finit and fi, are determined by the chosen parametrization, with f", being problem-dependent, while f2 requires the computation of an image measure density. However, the previously mentioned methods outlined above face several limitations:
(s) = Σ a (5)0((())) i=1
(5)
for some policy a : S - [0, and such that the group of diffeomorphisms generated by the $; is topolog- ically ergodic. That is for all æ, y E S and any neighbor- hood U of y. there exists a sequence i,, it such that xD, i 1 ... 21 U.
Limitation @:
The first limitation arises in IL. frar is unknown so one has to build density model for Furm- Disregarding solutions based on separate energy models, we can attempt to set Fierm = Finit + F - F,,, assuming FL_ is tractable. We then use a cross-entropy loss to train Fiem to approximate K. Despite the FM loss formally and trivially becoming O, Frerm may not be positive. Therefore, the existing theoretical framework fails and so does the sam- pling. At this stage, we move beyond the existing theoretical framework.
There are two key ideas that motivate this definition:
Tractability of star inflow fi_ is achieved by only using finitely many diffeomorphisms, Let say that S = IR en- dowed with Lebesgue measure À and that each move o; is a parametrizable diffeomorphism. Then, a closed form formula for the backward policy may be deduced from a detailed balance argument, see appendix A for details:
Limitation 2: The second limitation arises in RL. At the inference time, the reward may be unknown, either because of its costly nature or because is simply inaccessible. A straightforward solution consists of using Frerm during training and Fiemm = Fit Fr F*, during inference. However, this reward-less sampling is ill ill-controlled because Fremm may not be positive.
P
Σ
of (s)00, (0)
(6)
d'_(s) E,(a5ft) $ (z
2
(3)
where J, denotes the Jacobian matrix at 8. Therefore, we have a closed-form formula for the density of the star inflow,
Limitation @: Both training and inference are based on the tractability of f². For instance, naive forward policies, such as those used in diffusion models add noise in IRd These policies are typically defined as ( (x) m(x) c, where (O, represents Gaussian noise. This leads to an intractable star inflow as the integral in equation (4) becomes computationally intensive.
f2) > (t) क (s) det J
(8)
m is deterministic model and
All terms equation (8) are tractable, so f" is tractable as long as the number of diffeomorphisms p is small. As a consequence, the flow-matching loss given in Equation (3) is tractable for p small enough.
(V) x [ (x) (- (-(((x)-1) 9)2 dx(z).
JZEZd
21²2
Expressivity can potentially be an issue if the number of diffeomorphisms is small and each move is too simple (Rezende et al., 2020). Ergodicity guarantees that the pa- rameterized family of EGFs able be flow-matching for
(4)
Limitation @: Although Li et al. (2023) argues that exact cycles are negligible in the continuous setting due to their
3

---

Ergodic Generative Flows
any non-zero Fint & X and Fierm <> X even with simple transformations, such as affine maps on tori and rotations on spheres.
3.1.2. ERGODICITY ON TORT
For the sake of simplicity, let T be flat torus of side I endowed with Lebesgue measure À and let P. : R - T be the natural universal covering projection. The simplest family of transformations is the group of affine transforma- tions,
The following sub-sections are organized as follow: Sec- tion 3.1 establishes the universality of EGFs. In Section 3.2, we present theoretical advancements for general gen- erative flows, which forms the basis for the design of the KL-weakFM loss used IL training, described in Section 3.3.
Aff(T)) := fz + Ax + b b E R and A E (L)(Z)),
(10)
3.1. Universality From Ergodicity
with (Z)(Z) denoting the set of square matrices of order d with integral coefficients and determinant 1.
We begin with a formalization of universality in conjunction with the master universality Theorem.
Given that (0+) are all in (RR)), we can build con- tinuous family of generative flows using equation (5), which is then parameterized by the translation part of each Q; to- gether with a model having a softmax head a., and a scalar head f*, We call such a family an affine toroidal family.
Definition 3.2. On a state space (S, A). a family of gener- ative flow family is universal if for any two distributions U.v < À with bounded density there is a flow (f=,. in the family that is flow matching for Finit = V and Fiemm J.
Theorem 3.5. Any affine toroidal family is a universal EGF family provided mild technical assumptions on the (&.)
We begin with the presentation of our master universality theorem in section 1. This theorem is then applied to tori and spheres, providing simple examples of universal families, as detailed in sections 3.1.2 and 3.1.3.
The "technical assumptions" above is in particular satisfied if the group generated by the linear part of the T. is the whole group (Z). It is shown in Conder et al. (2025) that SZ,(Z) has presentation with two generators. Therefore, in any dimension. EGF are universal with p = 4 and well- chosen (0.) the two generators and their inverse.
3.1.1. MASTER UNIVERSALITY THEOREM
The next theorem states that the family contains a suffi- ciently strong ergodic policy n", then the family univer- sal. More precisely,
3.1.3. ERGODICITY ON SPHERES
Consider the round sphere sd = {x E Rd+1 |||||| - 1} endowed with its natural Riemannian volume measure X. Again simple family of diffeomorphisms is provided by the so-called projective transformations,
Definition 3.3. A Markov kernel π on (S))) is summably T,2 mixini if ı admits an invariant measure X equivalent to À and such that the miimixnng coefficients given by,
Na = sup /
PGL (IR = { + Ax/||Aa|| |
A E GLa+1 (R)), (11)
(pTT(ss)² (s)))d((ss)
(9)
PEH JES
where GL. (R) is the group of invertible matrices of order d+1. Similar to tori, we build a continuous family of EGFs
with H = ( (2(A) ||ell2 = 1 and saes ((s) = 0} are summable: Enz0 in < +00.
using equation (5) by parametrizing (#=)_=1 in PGT (R) which is a Lie group, and choose tractable model that have a softmax head a., and a scalar head ft, Such family is called a projective spherical family. We focus on the sub- family of isometry spherical family composed of rotations D E SO (R) for which we have theoretical guarantees. Theorem 3.6. An isometry spherical f family is a universal EGF family, provided technical assumptions on the (Zr)
The technical summably L'-mixing property informally means that iterating the Markov kernel is averaging out functions fast enough ensure that the sum of errors is finite. In contrast, ergodicity alone ensures the convergence of " in the much weaker Cesaro sense.
This property is in particular satisfied if the policy æ induces a Koopman operator á + d(())==
dA on (S), which has so- called spectral gap (Conze & Guivarc' H, 2013). Informally the spectrum of the Koopman operator is bounded away from on the subspace { I Φdd = 0}. In this case, the convergence of iterates by π is exponential.
The technical assumption satisfied if the 4 have algebraic coefficients and are frozen: this is a consequence of the main theorem of Bourgain & Gamburd (2012) (see also Benoist & de Saxcé (2016)) that guarantees spectral gap under this assumption. Furthermore, the main Theorem of Breuillard & Gelander (2003) allows to build a dense subgroup of SOa(R) with two generators and algebraic coefficients. Therefore, EGF are universal on spheres with
Theorem 3.4. A parameterized family of EGF is universal provided that it contains some summably L L²-mixing n", and that the set of f*, dense in 1,2 T,2 X).
4

---

Ergodic Generative Flows
p 4 and well-chosen (0,)=== the two generators and their inverse.
with respect the counting measure. We may also rewrite
TV
R
F(a) Eves P(e)
r(a)
K(S]
Ev'es Evest(o) ((x)
so that if Z = EEES r(s)
then
3.2. Quantitative Sampling Theorem
We present quantitative sampling theorem for generative flows. The key idea of the proof is that any generative flow matches the flow up to transformation of the initial and terminal distributions.
TV
ST
1 +
1 2
|
|
(14)
K(S)
3.3. KL-WeakFNM loss and IL algorithm
Definition 3.7 (Virtual initial and terminal flow), Let [2) be a generative flow and let Pinit, Froem be ini- tial and terminal distributions. Define the initial and ter- minal errors as Fiit == (Fiitt + F Fiemm F F and 8Fterm
Henceforth, we assume that S is a Polish space equipped with a background measure À and k is target probability distribution.
(Fiiit + F= Pierm- - -= respec- tively. Define the positive and negative parts of measure as u²= The virtual initial and terminal distributions are Finit == Finit + & and Feerm == Fterm + &Fterm
Leveraging Theorem 3.8, design the KL-weakFM loss, LKI-WFM. which enables II. training of generative flow without requiring a separate reward model. Let f = min(0, fimit + f-- and Ferm max(0, Jimit + J 12). The KL-weakFMM loss can then be defined as follows:
We note that once the initial and terminal distribu- tions Finit; Feorm are specified, then any generative flow (z) is a FM with respect to the virtual initial and terminal distributions, Fintt and Frerm- This enables the presentation of a quantitative formulation of the sampling theorem for generative flows.
LKL-WFM (0)
bB finit Esne log Feerm (s), (15)
for some training distribution of paths Vtrain and b 0.
Theorem 3.8 (Quantitative Sampling of Generative Flows). Let fi.) be a generative flow and let Finit and Fverm initial and terminal distributions. Assume that Finst # O and consider the sample 8, of the generative flow Markov chain from Finit. to From then:
The name of the loss is derived from two components: on the one hand, the cross-entrope term, which controls the Kullback-eibler divergence between Fverm and k: on the other hand, the term Eg-reetn fmit which resembles the FM-loss but controls only the negative part of the FM defect. We now provide detailed description and motiva- tion for CKL-WFM. The discussion begins with a corollary of Theorem 3.8, based on Pinsker's inequality:
8 (S)
TV 8T
Frerm
(12)
Fierm (S)
Fiemm (S)
Corollary 3.10. Let (T,) be a generative flow. We train the generative flow on target probability distribu- tion with Finit(S) 1. By using Frem R :=
with TV the total variation.
Corollary 3.9. For a generative flow trained on the tar- get K. with (S) 1 and using Fremm = R = (Fiit P -P*)+ during inference, the sampling error
(Fini + Fr. - -Fi)* during inference, the sampling error is bounded as follow:
is controlled by:
8
1 KL.
R
xx x)) |x)
TV
ST
K K(S)
S 1+5
+ TVV
R(S)
<(S)
K
(13)
TV TV (s+ [lm))
1 1
+
++V
R((S)
with d = (Finit + - -2)
with 5= = SSeS &inie (s))((s)
Commonly the training loss is designed to control the total variation term in the upper bound. However, a secondary regularization term controlling Fait may be added to enhance reward-less sampling.
First, the weak-FM term Ey Et= fini (s,)) in equation (15) controls the term 1+6 in equation (16).
Second, since the target k has unknown density, we employ reward-less inference, using Fierm (Finit + F F= It is therefore natural to directly train the density of (Finit + Fr - F to match k using cross-entropy, leading to the second term on the right-hand side of Equation (15). Corollary 3.10 demonstrates that controlling the Kullback- Leibler divergence through cross- also controls the total variation sampling error.
To substantiate this quantitative bound, let's consider the RL case with a reward r defined on finite DAG (V, with V- = S I sy). trained using a flow-matching loss, say Catable Let's take Vraai the uniform distri-
FM,g
bution on the vertices and q 1, then LFM.4 1 (6 E.es (f(s) r(s))) with f(s) the density of R
5

---

Ergodic Generative Flows
Algorithm 1 Ergodic flow: IL. training
(Saxe al., 2013; Hu al., 2020). We use the AdamW optimizer (Kingma & Ba, 2015; Loshchilov & Hutter, 2019) for training.
Input: A list of trainable bi-Lipchitz maps 11
Input: A softmax model : S - (( ... ,,))))))))))))))))
Input: A trainable star outflow model f" S - R+
The KL-WeakFFN loss tends to favor Fierm positive on the whole S, therefore, even with highly localized target dis- tributions, the EGF reward model tends to have small background inducing unwanted outliers. We filter it by replacing frerm with 1 from >y Stem during inference, where 1i chosen to minimize negative likelihood on a validation dataset. See Appendix D.1 for details.
Input: target samplable distribution x
Input: source Finit samplable and of density fiit
repeat
Fill replay buffer B = B_, B+ with B trajectories (n using (Fintt; m for B_, and (1s, m for B with t .
Minimization step of
4.1. RL Experiments
L 2B
fii (s))pe(s) - BBaw log Fuerm (s),
Two dimensional distributions are tested in RL settings, which allow us conduct sanity-check on the tractability of the EGF, its stability and expressivity. An EGF on S T2 is built with 16 transformations (8 translations and 2 ele- ments of (Z) together with their inverse). Their MLPs have 5 hidden layers of width 32 to parameterize f", and T_, This EGF is trained either with stable CPM or unsta- ble Cdiv FM-losses, with or without regularization R (see equations (3), and (20)).
with
pr(s)
II 1 ( (=v))
if B.,
II (1+ (6 f2) M<t
if B
B
until converged
ft + finit 2 (s:)
)
Cdix
E, Σ log
(19)
1i + Srorm
Third, more precisely, the KL term may be decomposed into
R = E, > (L)))
(20)
K
KL.
R
R(S)
The regularization R is motivated by the stability Theory of Brunswic et al. (2024); as long the directional derivative of R is positive on O-flows directions, such regularization helps reducing the unstability of loss. See also Morozov et al. (2025) for a study of the impact of such regularizations for detailed balance losses on graphs.
Egwre log Feerm(( (8) + log {
(s)
(aaa)(s)-H())) H(()),
SES
cross entropy
nommalization
(17)
da (s) is the entropy of k. Since where H(KK) = E.,. log dx the normalization constant is unknown a priori, the cross- entropy insufficient to fully control the KI. divergence. However,
A stability comparison divergence- based FM loss (see fig- ure 1) shows the expressivity of very small EGF well as the unstability Ldvx as predicted by Brunswic et al. (2024)(refer to limitation 4 in section 2).
[ Prerm (s)d)(s)=1)) =1 [ 5mmi SES
(18)
4.2. II. Experiments
We proceed with imitation learning experiments on tori T and spheres S?.
Therefore, the weak-FM term in equation (15) also controls the normalization factor of of Fiemm
On T2, we compare small Moser Flows, DDPM and EGFs, demonstrating that EGF' can generate well-behaved distri- butions even when Moser Flows break down. An EGF is built with the minimal four affine transformations described in section 3.1.2, implemented MLPs with 3 hidden layers of width 32. These transformations parameterize f", and T², The Moser Flow is trained using the implementation provided the authors' GitHub repository (Rozen, 2022) with the only modification being the model size set to 32x3. Our EGP is trained using the KL--weaFFM loss together with
4. Experiments
We proceed with experiments wherein the state space S is either flat torus T or sphere g? An EGF family is built by randomly choosing diffeomorphisms from affine toroidal families on T and from isometry spherical families on $ together with two multi-layer perceptron (MLP) models: one for fü, and one for a The MLPs are tanh hyper- bolic activated and initialized using orthogonal initialization
6

---

Ergodic Generative Flows
a EGF 32x5, density (left) and samples (right).
Moser Flow
unstable FM, regularization
stable FM loss, no regularization
unstable FM, no regularization
100
stable FM loss, no regularization
unstable FM, regularized
stable regularized
unstable FM, regularized
shable FM loss, regularized
b DDPM
RGF 32x5, stability comparison between FM losses. Unregu- larized unstable loss (blue) flow size (dashed lines) and sampling time r (solid lines) blow while stable loss (red) converges. Reg- ularization helps stabilization for both losses.
Figure 1. Checkerboard RL task.
Table 1. Negative log-likelihond scores of the volcano dataset.
Volcano
Earthquake
Flood
Mixture vMF
-0.31
0.59
1.09
EGF
Stereographic
-0.64
0.43
0.99
Figure 2. Comparison of imitation learning on standard toy distri- butions using tiny 32x3 MLP models. Background filter is applied to EGF, similar filter Moser flow would yicld worse results. Without a filter, EGF samples have outliers similar to DDPM. Since DDPM does not give access to density, only samples are provided. For fairness, DDPM is sampled using 100 denoising steps.
Riemannian
-0.97
0.19
0.90
Moser Flow
-2.02
-0.09
0.62
EGFN
-2.31
-0.12
0.56
a regularization as in the RL setting. Figure shows how Moser Flow fails train with such small model while minimal EGF reproduces the target distribution with high fidelity.
On S2, we benchmark EGFs on the earth science volcano dataset (NGDC/WDS, 2025). sample distribution given the dataset in Figure 3 and negative log-likelihoods are given in Table We only use six rotations: rotation of angle क/ /4 around each of the three axes plus their respective inverse. The two core MLPs of EGF are of size 256x5, compared to the 512x6 used by Rozen et al. (2021). The learning rate is le-3 with an exponential decay to le-5 at 3000 epochs of 25 steps. We outperform Moser Flow and all related baselines. Notably, the EGF achieved its reported performance with training time 10 times shorter.
Figure 3. EGF generated points reward field From (left) and whole dataset point with KDE field (right) for the volcano task.

---

Ergodic Generative Flows
5. Related Work
T ( Pk o... 1bn) F-to Sees a (5)dF
(6), where
F-, = P + Freem-
EGFs draw inspiration from NFs for their transformation blocks, as well as from denoising diffusion models (DDMs) (Ho et al., 2020) for incorporating stochasticity. While simi- lar approaches have been explored in previous works (Wu et al., 2020; Zhang& & Chen, 2021), our method stands out by leveraging general generative flows theory to construct a more efficient framework. Compared to previous works, EGFs are more compact and demonstrate faster convergence as illustrated in our experiments. Furthermore, DDMS and their variants (Song et al., 2020a) rely on fixed discretiza- tion of inference denoising trajectories, whereas EGFs en- able trainable sampling times, offering greater flexibility and adaptability.
Building on this formulation, we can draw comparisons between EGFs and three related research directions: Con- tinuously Indexed Normalizing Flows (CINF) (Caterini et al., 2021), Neural Architecture Search (NAS) (Elsken et al., 2019), and Wasserstein Gradient Descent (WGD) (Chizat & Bach, 2018). First, CINF's attempt to overcome the limitations of NFs by using a fixed architecture, with conditioning sampled from a latent distribution. This ap- expectation,
proach generates the target distribution Fierm- = Ex (mimik) where the random NF K is drawn from a continuous distribution of NFs. In contrast, our method samples K from a tree structure. Second, NAS aims to find the optimal neural network architecture for a given task. comparison, our EGF constructs a distribution of suitable NFs, where the sampling time can be interpreted as a learned depth of the resulting architecture. Third, by training over distribution of architectures and considering the expectation of the output (with Finit. as the input and Fierm as the output), our approach parallels the setup of Wasserstein Gradient Descent (WGD), Extending the theo- rems of Chizat & Bach (2018) to EGFs would be a valuable addition to our theoretical framework.
A particularly notable comparison arises when examining the relationship between our EGFs and NFs introduced in Rezende et al. (2020), as both utilize affine toroidal families and projective spherical families. However, while our EGF framework provides universality guarantees, the NF setting lacks such guarantees as it solely relies on affine and projec- tive transforms. Indeed, both projective and affine transfor- mations form groups (compositions of affine transforms are affine), thus limiting the expressivity. This distinction high- lights a fundamental advantage of EGFs, further reinforcing their theoretical robustness and practical effectiveness.
Lastly, from the proof of the universality Theorem 3.4, we observe that f*, can be obtained fixed point of flow operator dependent on n",,, Finit. and Fierm- This operator is a contraction for the affine toroidal family. Although we train f via gradient descent, this approach bears similari- ties to Deep Equilibrium Models (DEQ) (Bai et al., 2019; 2020), where the fixed point is computed by a contraction neural network. The key difference is that we explicitly approximate the fixed point with a neural network, while DEQ models use black-box solver and the implicit func- tion theorem for gradients. Future work could develop a DEQ-EGR, where f", is implicitly obtained as a fixed point instead of via a feedforward network.
Broadly speaking, previous methods, such as Riemannian Continuous Normalizing Flows (Mathieu & Nickel, 2020) and Flow Matching on Manifolds (Chen & Lipman, 2024) rely on either handcrafted non-linear transformations or so- phisticated NeuralODEs (Chen et al., 2018). Our EGF does not incorporate NeuralODEs, as we do not face expressivity limitations that justify their usage. However, these tech- niques remain fully compatible with EGFs and could be integrated into future extensions of our framework to further enhance its ficxibility and adaptability if needed.
Since EGFs' elementary transformation are derived from the transformation blocks of NFs, EGFs can be viewed as stochastic sampler for NF architectures. In par- ticular, any EGFs sampled trajectory can be defined as (s))-1. where the trajectory (st)i= is obtained by com- posing NF transformations as follows: ~ Fini; 82 2x, ($1),
6. Limitations and Future Work
First and foremost, our experiments are limited to low di- mensions. Although ergodicity is easily achievable with a number of generators independent from the dimension (for instance on tori, one randomly initialized non-trainable translation sufficient to ensure ergodicity), the technical assumption of LL-mixing summability needed for univer- sality is more subtle. EGFs suffer from limited proven applicability, further theory is needed to casily enforce L mixing summability in higher dimension together with re- lated experiments. More precisely, a theoretical bound on the minimal number of transformations necessary to achieve universality would be useful for hyperparameter tuning. We give such a bound only for affine toroidal and isometry
87 = - .... 4b, ($1). with the transition probability P(ke+1 kt) = ( Each com- position &x__, o .... 2x, is a diffcomorphism. and the terminal distribution is given by Frrm Ex (Fimt K), where K Φx, o... o 4h represents a random NF. In other words, an EGF with p diffcomorphisms represented as pair mT, where i Tn + DS) is trainable embedding of the p-ary tree into the space of diffeomorphisms on the state space, and x is a ran- dom walk policy on To with a random stopping time. However, the policy ग is intractable as it is defined by
be
8

---

Ergodic Generative Flows
spherical families where two transformations sufficient to achieve universality.
groups of affine transformations of compact nilmanifolds. Annales Scientifiques de l'École Normale Supérieure, 48 (3):607-645, 2015.
Second, the mixing summability property is essential to ensure universality. It is likely that training would be im- proved if some control over this property was to be achieved, say by adding a regularization. There is extensive mathemat- ical literature on the so-called spectral gap (Kontoyiannis &
Bengio, E., Jain, M., Korablyov, M., Precup, D., and Ben- gio, Y. Flow network based generative models for non- iterative diverse candidate generation. In NewrIPS, 2021.
Meyn, 2012; Guivare'I h & Le Page, 2016; Marrakchi, 2018: Bekka & Francini, 2020), which is sufficient condition for mii-mixing summability, and absolutely continuous in- variant measures (Góra & Boyarsky, 2003; Bahsoun, 2004; Galatolo, 2015) including stability results (Froyland et al., 2014). A systematic review of this literature would certainly yield such regularization.
Bengio, Y., Lahlou, S., Deleu, T., Hu, E. J., Tiwari, M., and Bengio, E. Gflownet foundations. The Journal of Machine Learning Research, 24(1):10006-0006) 2023.
Benoist, Y. and de Saxcé, N. A spectral gap theorem in simple lie groups. Inventiones Mathematicae, 205(2): 337-361, 2016.
Lastly, our focus was on the theoretical advancements of generative flow theory through EGF. As result, we did not exploit EGF's high modularity, which enables sophisti- cated transformations, replay buffers, or neural architectures. Additionally, hyperparameter tuning was minimal, leaving room for future work to perform a systematic analysis.
Bourgain, J. and Gamburd, A. A spectral gap theorem in su.(d). Journal of the European Mathematical Society (EMS Publishing), 14(5), 2012.
Breuillard, E. and Gelander, T. On dense free subgroups of lie groups. Journal of Algebra, 261(2):448 467, 2003.
Brooks, S., Gelman, A., Jones, G., and Meng, X.-L. Hand- book of markov chain monte carlo. CRC press, 2011.
7. Conclusion
We propose new family of generative flows that leverage ergodicity to provide universality guarantees while utilizing simple diffeomorphisms and neural networks. Our results demonstrate that EGFs maintain their expressivity even at parameter counts where Moser Flow (Rozen al., 2021) fails. This simplicity enables us outperform our baselines on the NASA volcano dataset with model 30 times smaller.
Brunswic, L., Li, Y., Xu, Y., Feng, Y., Jui, S., and Ma, L. A
theory of non-acyclic generative flow networks. In AAAI, 2024.
Caterini, A., Cornish, R., Sejdinovic, D., and Doucet, A. Variational inference with continuuusly-indexee normal- izing flows. In Uncertainty in Artificial Intelligence, 2021.
Impact Statement
Chen, H., Ren, Y., Ying, L.,, and Rotskoff, G. M. Ac- celerating diffusion models with parallel sampling: In- ference at sub-linear time complexity. arXiv preprint arXiv:2405.1 15986, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, R. T. and Lipman, Y. Flow matching on general geometries. In ICLR, 2024.
References
Chen, R. T., Rubanova, Y.. Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. In NeurlPS, 2018.
Bahsoun, W. Absolutely continuous invariant measures for random maps. PhD thesis, Concordia University, 2004,
Bai, S., Kolter, J. Z., and Koltun, V. Deep equilibrium models. In NeurIPS, 2019.
Chizat, L. and Bach, F. On the global convergence of gradi- ent descent for over-parameterizze models using optimal transport. In NeurIPS, 2018.
Bai, S., Koltun, V., and Kolter, J. Multiscale deep equi- librium models. NeurlPS, 2020.
Chung, H. W., Hou, L.,, Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang. X., Dehghani, M., Brahma, S., et al. Scaling instruction- instraction-finetuned
Bekka, B. and Francini, C. Spectral gap property and strong ergodicity for groups of affine transformations of solenoids. Ergodic Theory and Dynamical Systems, 40 (5):1 1118811199 2020.
finetunedd language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Conder, M., Liversidge, G., and Vsemirnov, M. Generating pairs for (n, 7). Journal of Algebra, 662:123-137, 2025.
Bekka, B. and Guivare'h, Y. On the spectral theory of

---

Ergodic Generative Flows
Conze, J.-P. and Guivarc'H, Y. Ergodicity of group ac- tions and spectral gap; applications random walks and markov shifts. Discrete and Continuous Dynami- cal Seriems-Series A, 33(())42339443)) 4269, 2013.
Kifer, Y. Ergodic theory of random transformations, vol-
ume 10. Springer Science & Business Media, 2012.
Kingma, D. and Ba, J. L. Adam: A method for stochastic gradient descent. ICLR, 2015.
Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Kontoyiannis, I. and Meyn, S. P. Geometric ergodicity and the spectral gap of non-reversible markov chains. Probability Theory and Related Fields, 154(1):327-339, 2012.
Froyland, G., González- González-Tokman, C., and Quas, A. Sta- bility and approximation of random invariant densities for lasota-yorke map cocycles. Nonlinearity, 27(4):647, 2014.
Lahlou, S., Deleu, T., Lemos, P., Zhang, D., Volokhova, A., Hernández-Garcia A., Ezzine, L. N., Bengio, Y., and Malkin, N. theory of continuous generative flow networks. arXiv preprint arXiv:2301.12540 2023.
Galatolo, S. Statistical properties of dynamics. introduc- tion to the functional analytic approach. arXiv preprint arXiv:1510.02615, 2015.
Lee, J. M. Introduction to Riemannian manifolds, volume 2.
Springer, 2018.
Gemici, M. C., Rezende, D., and Mohamed, S. Normal- izing flows riemannian manifolds. arXiv preprint arXiv: 1.02304, 2016.
Li, Y., Luo, S., Wang. H., and Hao, J. Cflownets: Continu- control with generative flow networks, arXiv preprint arXiv:2303.02430 2023.
Goodfellow, I., Pouget- Abadee. J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communicutions of the ACM, (3(11))139-144 2020.
Loshchilov, and Hutter, Decoupled weight decay regu- larization, ICLR, 2019.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurlPS, 2022.
Góra, P. and Boyarsky, A. Absolutely continuous invari- ant measures for random maps with position dependent probabilities. Journal of Mathematical Analysis and Ap- plications, 278(1):225-244 2003.
Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint WxXi:2201.13359, 2022.
Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., and Duvenaud, D. Ffjord: Free-form continuous dy- namics for scalable reversible generative models. arXiv preprint
Marrakchi, A. Strongly ergodic actions have local spectral gap. Proceedings of the American Mathematical Society, 146(9):3887-3893, 2018.
2018.
Mathieu, E. and Nickel, M. Riemannian continuous normal- izing flows. In NeurlPS, 2020.
Guivare'h, Y. and Le Page, E. Spectral gap properties for linear random walks and pareto's asymptoties for affine stochastic recursions. Annales de I l'Institut Henri Poincaré Probabilités et Statistiques, 52(2):503-574, 2016.
Morozov, N., Maksimov, I., Tiapkin, D., and Samsonov, Revisiting non-acyclic gflownets in discrete environments. arXiv preprint arXiv:2502.07735, 2025.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft cctorc algorithms and applications. arXiv preprint 1rXiv:1812.05940 2018
NGDC/WDS.
Global significant volcanic eruptions
database. a access / id=gov cerov.noaaa.o noaa ngde mgg hww.aaaaaaa11aaa.... 2025.
httpp:://ww.nne http://wwww.ceii..a.........
Halmos, P. R. Measure theory, volume 18, chapter XII. Springer, 2013.
Papamakarios, G., Nalisnick, E.,, Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22( 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurIPS, 2020.
Hu, W., Xiao, L., and Pennington, J. Provable benefit of or- thogonal initialization in optimizing deep linear networks. arXiv preprint arXiv:2001. .05992, 2020.
Rezende, D. Papamakarios, G., Racaniere, S., Albergo, M., Kanwar, G., Shanahan, P., and Cranmer, K. Normal- izing flows on tori and spheres. In ICML,, 2020.
10

---

Ergodic Generative Flows
Rozen, N.
Moser
flow.
2022.
Rozen, N., Grover, A., Nickel, M., and Lipman. Y. Moser flow: Divergence- generative modeling on mani- folds. In NeurIPS, 2021.
Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 131.6610, 2013.
Sendera, M., Kim, M., Mittal, S., Lemos, P., Scimeca, L., Rector-Brood Brotor-roook,... J., Adam, A., Bengio, Y., and Malkin, N. Improved off-policy training of diffusion samplers. NeurIPS, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502. 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- baaee-bsed generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 134566, 2020b.
Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018.
Tang, S., Wang, Y., Ding, C., Liang, Y., Li, Y., and Xu, D. Adadiff:: Accelerating diffusion models through step-wise adaptive computation. In ECCV, 2025.
Verine, A., Negrevergne, B., Chevaleyre, Y., and Rossi, F.
On the expressivity of bi-lipschitz normalizing flows. Asian Conference on Machine Learning. 2023.
Walters, An introduction to ergodic theory, volume 79.
Springer Science & Business Media, 2000.
Wu, H., Köhler, J., and Noé, F. Stochastic normalizing flows. In NeurIPS, 2020.
Zhang, D., Chen, R. T., Malkin, N., and Bengio, Uni- fying generative models with gflownets. arXiv preprint arX9v:2209, 02606, 2022a.
Zhang, D., Malkin, N., Liu, Z., Volokhova, A., Courville, A., and Bengio, Y. Generative flow networks for discrete probabilistic modeling. In ICML, 2022b.
Zhang, Q. and Chen, Y. Diffusion normalizing flow. NeurIPS, 2021.

---

Ergodic Generative Flows
A. Fundamentals of EGFs
As stated in section 3, on Differential manifold S endowed with background measure À absolutely continuous with respect to Lebesgue measure, define an EGF as Generative Flow (x F with F f* and n=, (s - Σi=1 a2,(())) (s)0 where 4, are diffeomorphisms of S. With these definitions, the backward policy automatically has closed form formula.
Theorem A.I. Let (m F as above, the induced Generative Flow on S has a backward star policy given by
(s) Σ ol (1
(21)
a (s) = (a..f%) o $, (s) J.
(22)
(s)
ft (a (s) (5)
(23)
Proof. From (Brunswic et al., 2024) section 3.2 and Proposition I in their appendix, we have for any measurable X C S:
FL(Y) O -Y
(24)
7 +X) dF= @ _ (.+
(8)
(25)
d Ox
Fz O +X)
(26)
Therefore, for all measurables s:
(Y (
(27)
J +
(28)
S.e (x(5)
(29)
/ (19
(30)
JSES i=1
Σ JEE
(31)
Σ i=l JEE
(32)
Σ E
1x( "11 IdX(u)
(33)
HEY
1 "o (d)(a))
(34)
WEY i=1
12

---

Ergodic Generative Flows
Since this formula is true for any Y S, we deduce that F
2 - X)
x and that
dF X n. (. + X)
(x)
=
Σ
6 (m)
(1)))
(35)
dλ
1=1
Σ a ( (x)) (0) 150, (x) (X)
(36)
dF= 8 _ + S) (I)
f2(x)
(37)
=
b (m) (1
(38)
=
= (z)
(39)
The result follows.
B. Universality Theorems
B.1. General Universality Theorem
We shall both write the theorems in more formal way and provide rigorous proof of each statements.
The following theorem implies Theorem 3.4.
Theorem B.1. Let (S be a measured Polish space and let n², be Markov kernel acting on S. Assume that x²,, is ergodic on (S, X) with summable 1,2mmxing coefficients, then for any probability distributions Finits Frem & X with density in (A) and any >> O, there exists f E (x) such that the generative flow (= from Fint to Fremm is such that F (S) + oFForm (S) <.
Proof. Let H + DT + Finit - Frem. let UD = Finit - Fuem and let 24+1 = H(v)). By assumption, vn density is in (2), then the density of i4 with respect to À is L" for all E N.
We have
VI HT(w)
(40)
Σ
(Fint (=4))
(41)
1=0
T
Σ
Et
(42)
with et (Fiit Frerm) (. For all (,) 0, hence by assumption Eizo soes ($r (s))² ((s))>dd))))) dX(ss) < +00.
Therefore, VT converges as T -> +00 to some Vso and |uoo22 S Eizo Soes dee (s)) d)(s) < t00, hence vso & À and
du E (A). Furthermore, 3o is fix point of H so = " R + Finit-
Now, Poo does not necessarily provide a suitable fú, because it may happen that the negative part V * 0. Since for any 7 > 0 we have H (1.5 + 11)) = H(vs)) + n = Voo + +1d, we may add f7À to Vso to get another fix point v/2, = v/ +
of H. Define f* d(v2.)*
so that:
dA
Finit + (f²A))=² Frerm 13
Finit + -f2A v²6 + +=² Fromm (SE, 02 v%)
(43)
- v ) - (f-, v22)
(44)
Therefore, 8Fmm : ( - 112 - (fi,) =²))1= and 8 8Feerm == [(fi) -16 - -ffiA- So that
FFmit (S) + Fterr (S) =
- 1)((5) < 2(v2))
(45)
Since Voo0 & A in particular lim (v/() ((S) = 0, the result follows by choosing 1) big enough.
13

---

Ergodic Generative Flows
B.2. Universality on Tori
The universality Theorem on tori is obtained as a consequence of the spectral gap for Affine Toroidal families. We use the following result.
Theorem B.2 (Theorem 5 of (Bekka & Guivare' h, 2015) ). Let H be countable subgroup of Aff(TT). The following properties are equivalent:
(i) The action of H on T does not have a spectral gap.
(ii) There exists non-trivial H-invariant factor torus T such that the projection of H on Aut is amenable.
If condition (ii) of Theorem B.2 is false for the group generated by the transformations (#:) the we deduce any c induced by equation (5) has spectral gap if Vi, aL = Since each 4, keep the Lebesgue measure À invariant, by universality master Theorem B.1, the family is universal. Our technical condition of Theorem 3.5 is then "The set of transforms (9.);-, stable by inverse and the group generated by (2:) violates condition (ii)." In particular, the linear part of H the whole SL(d, Z), projections such the one in condition (ti) are never amenable since SL(d, Z) contains free group with two generators.
B.3. Universality on Spheres
The universality Theorem on spheres is obtained consequence of the spectral gap for isometry spherical families. Before stating the technical mathematical result use, recall that real number is algebraic if there exists non-zero polynomial P E [[X] with integral coefficients such that P(x) O. The set of algebraic real number is denoted Q. We use the following result.
Theorem B.3 (Reformulation of Theorem I of (Bourgain & Gamburd, 2012) ) Assume that (L1, , 2p) E (O(d) n Mataxa(Q).
Q), that the group generated by P1. Dp is dense in SO R) and that Vi E {1...., .p....pp..... .p. e (1.... ,P) $; $. Assume that, Vi, oz = then the associated Markov kernel Th, given by equation (5) has spectral gap.
With the technical condition "The set of transforms ( is stable by inverse, the group generated by (2,) is dense in (O R) and each of the o; has algebraic coefficients", Theorem 3.6 then follows from master Theorem B.1.
C. Quantitative Sampling Theorem
Theorem C.1. Let f2,) be a generative flow and let Fintt and Fvem initial and terminal distributions. Assume that Funt + 0, and consider the sample 8 of the generative flow Markov chain from Fint. to Freru then:
8 Fmin (8)
TV
8r
Prorm
(46)
F Frorm (8)
F Prerm (S)
with TV the total variation.
Proof. For any generative flow F = (x",, FT,), define Tz.7 the kernel + (sr|51 x) for the stopping condition
Frem- The sampling distribution 8, of is then (S) F F"." Furthermore, we have & (S) F (S)
Finit FF)+ - F Freem(65 (S) = Finit (S) - Fiemm (S). . In particular, Feemm() 2 Finit > so Frorm # 0.
We notice that the generative flow IF = (m",,
satisfies the flow-matching constraint from Pinit to Prorm and that 3.' = -.3 Applying the sampling Thcorem IF", we obtain
F Finer Frerm (Fint + F )r"
(47)
Frerm
(48)
Finit n F + & F.
F Frerm
(49)
a 1 (S) Fuu +(1 a) 8 (S)
Frorm
(50)
Peom (S)
Sr.
14

---

Ergodic Generative Flows
with a
(
Frerm (5))-8Fiithh Pearm (S)
F (S)
The result follows.
Frrrm (S)
Furm (S)
D. Implementation considerations
D.I. Density filtering
Since EGFs give access to a tractable density ftem. we can estimate its mean and standard deviation or on the state space filter out any unwanted outliers when sampling. By choosing lower bound saturation value faat ko where k defines how strong the filter should be, then define stricter sampling density frem
Feer if faerm 2 Prorm
fterm
(51)
otherwise
Since the new sampling density fiemm = 0 for all points of low true density, the next state of the Markov Chain is guaranteed not be the sink state. This ensures the EGF only generates samples associated with high density. The factor & can be chosen manually to obtain a more strict or lenient filter, or can be automatically optimized by recalculating the negative tog-likelihood on training dataset to systematically evaluate all values.
-1.2
-1.4
Filter cutoff factor
-1.6
-2.0
-2.
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Figure NLL estimation for range of values of & for given model on the volcano dataset.
Note that this filtering is applied on the density rather than on the samples, meaning that no samples are filtered out. Therefore. this simply has the effect of preventing the trajectories from stopping at low density points. Figure 5 demonstrates that using the optimal filter value concentrates the samples in the zones of high true density, bypassing the uniform background noise that would otherwise be found prior filtering.
15

---

Ergodic Generative Flows
From left right, the depth of f*, kept constant while depth increases.
From left to right, the depth of ax,, is kept constant while f*, depth increases.
Figure 6. Variation of f* and a depths from 2x32 to 6x32. The theoretical prediction is that a.,, may be kept constant while f*, trained. Keeping a., constant indeed yields improvements, however the improvement obtained by increasing the depth of o.,, instead of fi is comparatively better.
Target distribution
Unfiltered samples
Samples with optimal filter
Figure 5. Comparison of the EGF's samples using the model with and without density filter.
E. Ablation study on the size of EGF
16

---

Ergodic Generative Flows
Figure 7. Theoretically minimal EGF with four affine moves on the torus T ie two generators of SL(2, 2) together with their inverse.
The MLPs parameterizing f", a., have size 4x 4x128. see that generated checkerboard has high quality, sharp boundaries but some under densities
17

---

Ergodic Generative Flows
a Generation of spin wheel with variations of MLPs for both f", and a.,, from 2x32 to 5x32 (TOP) and 6x32 to 9x32 (BOT).
h Generation of spin wheel with variations of ML.S for both fi, and o.,, from 2x64 to 5x64 (TOP) and 6x64 to 9x64 (BOT).
Figure 8. Variation of få and a- depths and width from 2x32 to 9x64. Increasing depth and width increases quality until depth 4 then the training seemingly becomes lest stable as shown by the 9x64 experiment. Since we keep the learning rate constant at 1e-3, it likely that the MLPs training becomes unstable. The state space is S = R²2 to ensure F- (S) < +00, constrain f", to be zero outside [-4, 4J2. The BGF has 8 transformations which are translations, while the MLPs for a.,, and fC, vary from 2x32 (2 hidden layer of width 32) to 9x64. Learning rate is kept at 0.001.
18

---

Ergodic Generative Flows
Generation of 4 circles with variations of MLPs for both f*, and Q.,, from 2x32 to 5x32 (TOP) and 6x32 to 9x32 (BOT).
b Generation of 4 circles with variations of MLPs for both fc, and au from 2x64 to 5x64 (TOP) and 6x64 to 9x64 (BOT).
Figure 9, Variation of få, and a- depths and width from 2x32 to 9x64. Increasing depth and width increases quality until depth 4 then the training seemingly becomes lest stable as shown by the 9x64 experiment. Since keep the learning rate constant 1e-3, it likely that the MLPs training becomes unstable. The state space is S = R22 to ensure F-, (S) < +00, constrain f", to be zero outside [-4, 4J2. The EGF has transformations which are translations, while the MLPs for a.,. and fC, vary from 2x32 (2 hidden layer width 32) to 9x64. Learning rate is kept at 0.001.
19

---

Ergodic Generative Flows
Generation of 2spirals with variations of MLPs for both f", and a., from 2x32 to 5x32 (TOP) and 6x32 to 9x32 (BOT).
b Generation of 2 spirals with variations of MLPs for both fc, and a.,, from 2x64 to 5x64 (TOP) and 6x64 to 9x64 (BOT).
Figure 10. Variation of få, and a depths and width from 2x32 to 9x64, Increasing depth and width increases quality until depth 4 then the training scemingly becomes lest stable as shown by the 9x64 experiment. Since we keep the learning rate constant at 1e-3, it likely that the MLPs training becomes unstable. The state space is S = R²2 to ensure F-, (S) < +00, constrain f", to be zero outside [-4, 4J2. The EGF has 8 transformations which are translations, while the MLPs for a.,, and fC, vary from 2x32 (2 hidden layer of width 32) to 9x64. Learning rate is kept at 0.001.
20


Reviews:
[{"review_id": "lyWCnu6yzL", "reviewer": "Reviewer_moVi", "summary": "This paper proposes a new family of generative flows called Ergodic Generative Flows (EGFs) which are capable for both RL and IL tasks in continuous settings. The generative flows are built upon finitely many globally defined transformations, with probable universality over continuous spaces like tori and spheres, enabling explicitly expressing and sampling from the undertraining distribution simultaneously. Further, the authors derived a novel loss, coined KL-weakFM loss for IL training where the target distributions is intractable. The proposed methods are evaluated through toy 2D experiments for both RL and IL tasks, and NASA volcano dataset for IL task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear moVi,\n\nWe thank you for your detailed review and valuable feedback. Please find our responses to your comments below:\n\n1. Methods and Evaluation Criteria\nStable Loss: \nWe appreciate your observation regarding the difference between our stable loss and the one proposed by Brunswic et al. (2024) in equation (19). We would like to clarify that Brunswic et al. introduce a family of stable losses in Section 4.2, denoted by $\\Delta_{f,g,\\nu}(\\alpha, \\beta)$. Our stable loss, as given in Eq. (3), is an instance of this family. Since our focus in this work is on the foundational aspects of Ergodic Generative Flows (EGF), we opted for the simplest stable loss, as it seemed more relevant for our goals. Furthermore, we experimented with Brunswic et al.'s loss on non-acyclic GFlowNets and observed little improvement, with additional difficulties in hyperparameter tuning.\n\nRegularization:\nBrunswic et al. introduced a cycle-killing regularization schedule in Theorem 1 to ensure convergence of non-acyclic flows toward an acyclic flow. Although they do not explicitly suggest using this regularization to stabilize losses in non-acyclic settings, it seemed a natural step for us. We note that a similar approach was also taken by Morosov et al. in their work \"Revisiting Non-Acyclic GFlowNets in Discrete Environments.\"\n\nWhile we believe the first point does not require further discussion in the main paper, we will include a clarification about our architectural and hyperparameter choices in the appendix. We will also ensure the regularization approach is more clearly explained in the main text. Also there was a typo on the Bengio loss (the square was missing).\n\n2. Experimental Designs or Analyses\nPurpose of RL Experiment:\nThe RL experiment serves a dual purpose: A) testing the applicability of EGF in the original context of GFlowNets, and B) addressing limitation 4. While limitation 4 is secondary to the primary discussion on non-acyclic GFlowNets, we think it is an interesting point that warrants clarification.\n\nAdditional Datasets:\nWe agree that the inclusion of flood and earthquake results would improve the completeness of our experiments. We are currently running updated experiments to incorporate these benchmarks. However, we were unable to locate the original fire dataset due to changes in the NASA repository. \n\n3. Supplementary Material\nWe have corrected the issues in the supplementary material as per your suggestions.\n\n4. Other Comments or Suggestions\nWe will make the necessary corrections and reformulations to the notations and definitions as suggested.\n\n5. Sampling Time and Performance\n- Spectral Gap and Sampling Time:\nThe estimation of sampling time can be derived from the spectral gap and the $L_2$ norm of the target distribution. We intend to discuss these relations in a broader follow-up work focused on spectral gap control for EGF. Although this theoretical framework is still in development, the following informal results provide some insight:\n\n    - A spectral gap $\\eta>0$ guarantees exponential decay in $L^2$ deviation from the mean: $||(f-\\int f)\\cdot(\\pi^*_\\rightarrow)^t||  = O((1-\\eta)^t).$ \n\n    - To transition from an initial distribution $f_{init}$ to a terminal distribution $f_{term}$ with error $\\epsilon$, we require: $||(f_{init}-f_{term})(\\pi^*_\\rightarrow)^t||\\leq \\epsilon$.\n    - Thus, the required sampling time $\\tau$ is expected to scale as: $\\tau = O\\left(\\frac{\\log(\\epsilon)-\\log(||f_{init}-f_{term}|| )}{\\log(1-\\eta)}\\right)$.\n    - One could then leverage general lower bounds on spectral gaps such as those of https://arxiv.org/abs/2306.12358 to guarantee a spectral gap scaling as $\\eta \\sim 1/dim$. \n    - Furthermore, the difficulty of the objective is controlled by $||f_{init} - f_{term}||$, the co-dimension $q$ of the manifold supporting the distribution would then leads (with a bit of gaussian dequantization) to $\\log||f_{init} - f_{term}|| \\sim \\log Vol($Ball of radius $r$ in dim $q) $ so that one would end up with $\\tau = O\\left(-\\log(\\epsilon)dim+q\\log(q)dim\\right)$. \n    -  However, this bound is probably very loose: in the small volume limit, the \"effective\" spectral gap is bounded away from 0 (we may discuss this in more details if you wish) so the term $q\\log(q)dim$ becomes $q\\log(q)$.\n    - We stress the need for a proper analysis, hence a later submission.\n- Trainable Parameters and Sampling Time: the simplest answer is via Theorem 1 as the sampling time is directly controlled by $\\int  f^*_{\\rightarrow}$ and the WeakFM loss via equation 17.\nThe master Theorem 3.4 provides a control:  The proof controls the $L^2$ norm of the target $\\||f^*_{\\rightarrow}-\\int f^*_{\\rightarrow} \\|| = O(\\||f_{init}-f_{term}\\||\\sum_t(1-\\eta)^t) = O(\\||f_{init}-f_{term}\\||\\frac{1}{1-\\eta})$. One could link some measure of expressivity  of neural network to such a bound, we did not do it at this stage as we would like to reserve it for a future work (as mentioned above).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The training losses for RL tasks seem problematic. The \"stable FM loss\" referred to in Section 2 of this paper is neither the commonly used one proposed by Bengio et al. (2021) nor the one proposed by Brunswic et al. (2024). Also, the regularization term in Section 4.1 needs further explanation.", "theoretical_claims": "I checked the theoretical claims except for Section 3.1 because I'm not familiar with topology and differential geometry.", "experimental_designs_or_analyses": "Section 4.1 (RL Experiments) is confusing. Despite the issues mentioned above, the purpose of this experiment seems unrelated to the primal claims of this paper, and the presentation of the results (Figure 1(b)) is unclear.\n\nThe experiments and analyses in Section 4.2 are reasonable, but I suggest the authors to present the complete results on the complete earth and climate datasets, including volcano, earthquake, flood and fire.", "relation_to_broader_scientific_literature": "This paper extends the theoretical framework of GFlowNet for continuous settings and IL taks, providing a novel approach for generative models in continuous space.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\nThis paper is well-motivated and easy to follow. The proposed method seems to have a strong theoretical support and is insightful. The novel framework of GFlowNet for continuous settings and IL tasks has great potential.\nWeaknesses\nThe RL part lowers the overall level of this paper.\nThe experiments are limited.", "other_comments_or_suggestions": "Some notations are inconsistent or used without definition. For example $f^*_\\rightarrow$ in (3), $\\delta$ in (5), source and target distribution in Algorithm 1.", "questions_for_authors": "Can the authors explain how the sampling time $\\tau$ is related to the trainable parameters, the dimension of the state space and the performance of the flow network?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89cZvcFTmp", "reviewer": "Reviewer_6zNQ", "summary": "This work introduces Ergodic Generative Flows (EGFs) to tackle several issues not satisfactorily resolved in generative flow networks by using ergodicity to create simple generative flows with globally defined transformations and tractable flow-matching loss. Furthermore, a new KL-weakFM loss is proposed for IL training without a separate reward model. The effectiveness of IL-EGFs is evaluated on toy 2D tasks and real-world NASA datasets, while toy 2D reinforcement learning experiments are conducted using the FM loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear  6zNQ,\n\n   We thank you for your detailed review. \nWe acknowledge the need for higher dimensional experiments, it is part of an ongoing project to scaling up EGFs as well as training conditioned EGFs. \n\nRegarding your questions:\n1. Unfortunately, they do not directly.  A straightforward strategy would be to use Wasserstein bounds together with dequantization. \n    - By Theorem 3.8 and its corollaries, the KL-WeakFM loss controls the TV distance between the EGF sampling distribution and the target dequantized empirical distribution.\n    -  On compact domains W < diameter * TV, the loss then controls the Wasserstein distance between EGF sampling distribution and dequantized empirical target distribution.\n    - Finally, use triangular inequality in W distance for the path \"dequantized empirical target distribution -> empirical target distribution -> true target distribution.\"  Bounds such as https://arxiv.org/pdf/1707.00087 to estimate the Wasserstein distance of the empirical distance to the target distribution are leveraged at this step.\n\n    Such a computation is straightforward and may be added in appendix. The unbounded case requires extra care and assumption for the second step.\n\n2. Regarding the curse of dimensionality,\n    - to begin with, we refer to the answer given to reviewer moVi about the control of the sampling time (short answer yes in a tame way). \n    - more to the point, the curse of dimensionality is likely to arise in the estimation of the weakFM loss since we need to enforce a flow-matching condition. We expect the spectral gap to reduce the asymptotic theoretical complexity of the Monte Carlo estimator compared to naive translation moves (non-ergodic), but more theoretical work is needed. A more sophisticated replay buffer is also probably required.  We note this question for future theoretical investigations.\n\n3. On neural networks approximations\n- Neural networks approximate: \n    - the star outflow $f_\\rightarrow^*$,\n    - the policy softmax part $\\alpha_\\rightarrow^i$.\n    - the translation part of the affine transforms on tori (but with no inputs since they do not depend on the state)\n\n- The answer to your question then depends on whether we are in the RL or IL setting. In the IL setting, Corollary 3.9, together with equation 16, ensures that the KL-WeakFM loss controls the TV error. In the RL setting, we should add a paragraph explaining the following: \n    - an L1 flow matching error controls the TV bound, itself bounded the L2 flow matching error controlled by the loss, which in turn depends (in a tractable way) on the trainable models $f_\\rightarrow^*$, $\\alpha_\\rightarrow^i$ and the fixed models $f_{init},f_{term}$. More abstractly, the FM loss we use is an estimator of a twisted L2 distance of the density of the approximated flow bimeasure $F_\\rightarrow \\otimes \\pi_{\\rightarrow}$  to the affine subspace of target flow-matching bimeasures, where $F_\\rightarrow = (f_\\rightarrow^* + f_{term})\\lambda + f_{init}\\delta_{s_0}$ and $\\pi_{\\rightarrow}(x) =  \\frac{  f_\\rightarrow^*(x)}{ f_{term}(x) + f_\\rightarrow^*(x)} \\pi_{\\rightarrow}^*(x) + \\frac{ f_{term}(x)}{ f_{term}(x) + f_\\rightarrow^*(x)}\\delta_{s_f}$. This L2 distance controls the L1 distance if we assume the state space to have a finite volume. The L1 distance is $\\delta F_{init} (\\mathcal S)$. \nAs a result, the stable FM-loss controls the right-hand side of Theorem  3.8. \n    \nThus, we will add our answer to question 1 in the appendix and a paragraph in the main text to answer question 3.", "claims_and_evidence": "This paper is well-written and presents rigorous theoretical results for a class of generative models. The authors provide clear and convincing evidence to support their findings.", "methods_and_evaluation_criteria": "This paper primarily focuses on theory and methodology, offering substantial theoretical insights and proposing new methods. However, the experimental evaluation of the proposed method is somewhat limited, with only a few experiments conducted to assess its effectiveness. Expanding the experimental section with more extensive tests could further validate the practical applicability and performance of the method.", "theoretical_claims": "I have gone over the theoretical results and proofs presented in the paper, and they appear to be correct. The logical flow and mathematical rigor of the arguments are sound, and the conclusions drawn from the proofs are well-supported.", "experimental_designs_or_analyses": "The experiments conducted to assess the effectiveness of the proposed method are limited to two-dimensional datasets. While these experiments provide initial insights into the method's performance, evaluating it on higher-dimensional datasets would offer a better understanding of the method.", "relation_to_broader_scientific_literature": "Generative Flow Networks (GFNs) were initially developed for sampling from unnormalized distribution densities on directed non-acyclic graphs. While recent advancements have expanded their theoretical framework, challenges persist in training GFNs in continuous settings and for imitation learning (IL), such as intractable flow-matching loss and the need for a separate reward model. The key contributions of this paper is that it addresses these issues and proposed a new approach.", "essential_references_not_discussed": "It appears that the essential related works are adequately cited in this paper.", "other_strengths_and_weaknesses": "No additional comments here.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "I have the following questions for the authors:\n\n1. In the standard generative learning scenario, where only a random sample of size n is available, do Theorem 3.8 and Corollary 3.9 provide any insights into how the error bounds are influenced by the sample size?\n\n2. How do the error bounds in Theorem 3.8 and Corollary 3.9 relate to the dimensionality of the data? Do these results suffer from the curse of dimensionality?\n\n3. Which functions are approximated by neural networks in your framework, and how are the approximation errors addressed in Theorem 3.8?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtDMA324xU", "reviewer": "Reviewer_mQ7G", "summary": "This paper introduces Ergodic Generative Flows (EGFs), a novel framework that extends Generative Flow Networks (GFNs) to address key challenges in training generative models for both reinforcement learning (RL) and imitation learning (IL). The authors identify four main challenges with existing GFNs: intractability of flow-matching loss, limited tests of non-acyclic training, the need for a separate reward model in imitation learning, and challenges in continuous settings.\nThe key innovations of this work are: (1) leveraging ergodicity to build generative flows with finitely many globally defined transformations (diffeomorphisms), providing both universality guarantees and tractable flow-matching loss; (2) introducing a new KL-weakFM loss that couples cross-entropy with weak flow-matching control, enabling IL training without a separate reward model; and (3) developing a mathematical framework for EGFs that ensures expressivity even with simple transformations like translations on tori and rotations on spheres.\nThe authors empirically validate their approach on toy 2D tasks and real-world datasets from NASA on the sphere using the KL-weakFM loss. Additionally, they conduct toy 2D reinforcement learning experiments with a target reward using the flow-matching (FM) loss. Their results demonstrate that EGFs can effectively address the identified challenges and outperform baseline methods, particularly in settings where computational and time budgets for generation are highly constrained.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear mQ7G,\n \nWe thank you for your particularly detailed review. \nTo begin with, we will add clear references to proofs in the appendix.\n\n3. Theoretical Claims:\n- Your interpretation of Definition 3.3 is correct. We may move the paragraph after Theorem 3.4 to after Definition 3.3 and expand it to better explain this definition by referring to the mixing properties of the Markov kernel.\n\nWe agree that understanding Theorem 3.8 would benefit from a specialization on graphs. Your suggestion is in line with our answer to reviewer 6zNQ regarding neural network approximations. We shall insert a paragraph answering both points in the RL setting.\n\n\n5. Scientific Literature: \n\n - We apologize for not citing Zhang et al.  We will cite it properly.\n\n6. Strengths And Weaknesses: \n\n- Equation 7, you are right; this equation should be introduced more gently. It may be derived from \n$\\pi^*_\\leftarrow (x\\rightarrow A) = \\frac{d (F_{\\rightarrow}^*\\otimes \\pi^*_\\rightarrow)(A\\rightarrow \\cdot)}{d F_{\\leftarrow}^*}(x)$ \ntaken from section 3.2 of Brunswic et al. https://arxiv.org/pdf/2312.15246. This equation indeed formalizes a detailed balance condition in the measurable setting. Also, Proposition 1 in the appendix clarifies the link between bi-measures and Markov kernels. We will add a sentence to account for this interpretation and add proof in the appendix.  More generally, we will enhance the introduction of Theorems of definitions with comments on their intuition.\n\n7. Questions:\n\n- The main point of Theorem 3.4 is that EGFs reduce the generative problem of finding a scalar field on the state space. Theorems 3.5 and 3.6 provide sufficient conditions for L2-exponential mixing, then an L2 error on $f_{\\rightarrow}^*$ translates into an increase in  $\\delta$ in Corollary 3.9 and an increase of the TV, which is easy to relate to the L1 error of $f_{\\rightarrow}^*$.  \nWe agree that the \"expressive enough\" formulation is vague and requires a comment.\n\n- Regarding Algorithm 1 and importance densities. The densities were used as is for the results presented in the submission.  In later experiments, we added a small constant (1e-2) to stabilize training; however, no ablation studies were conducted to assess its relevance or effectiveness. Flow collapse was the most common problem encountered but seemed more related to too strong a regularization than to collapsing densities. A lower bound on $\\int f_{init}$ proved sufficient to force exploration.\n\n- Your comment on the regularization echoes that of moVi. The regularization is inspired by Brunswic et al. regularization; their Theorem states that a decaying regularization may be used to enforce convergence toward an acyclic flow on graphs when using a stable FM loss. From an optimization viewpoint, the regularization pushes the derivative of the loss to be positive in the direction of any 0-flow, hence leading to a stable loss in the sense of definition 3 of Brunswic et al.  In our setting, any 0-flow component is expected to vanish (but the theorem mentioned above does not apply to our setting due to their enumerable state space assumption). A side effect is reducing the sampling length.  We will enhance the RL section with a discussion of this regularization. \n\n\n- We conducted comparisons to diffusion models (DDM) and normalizing flows (FFJORD), but we felt that the comparison to Moser flow already carried our point. We will add qualitative comparisons in the appendix on 2D toys on tiny 32x2 models. On the NASA datasets, we already include NLL comparisons to baselines for volcanoes; we will add the results on earthquakes and floods, also the results for Riemannian diffusion, as well as a comment on the relative size of each baseline.\n    - The experiments presented in the paper were solved with manual hyperparameter tuning; the only caveats were:\n        - too high regularization scale leads to flow collapse\n        - too high WeakFM scale tends to make learning the optimum harder especially for small models.\n\nSo far, we have only trained EGFs up to dimension 20 on distributions (toy and sparse reward Markov decision tasks). However, we prefer to limit the scope of the present work to proof of concept as hyperparameter tuning proved less straightforward in higher dimensions. \n    - Regarding training time,  we did not benchmark the training speed. We shall include a comparison table in the appendix.  Our wall clock evaluation led us to conclude that our training for NASA datasets was faster than that of Moser Flow by several orders of magnitude. Diffusion models tend to train slightly faster than EGFs on 2D toys, while FFJORD is significantly slower. However, hyperparameters have been shown to impact convergence speed significantly.", "claims_and_evidence": "Most claims in the paper appear to be well-supported by theoretical analysis and empirical evidence. The authors provide a comprehensive mathematical framework that logically connects EGFs to existing generative modeling approaches.\n\nHowever, some claims could benefit from additional clarification or support:\n\n- The claim in Theorem 3.4 regarding the expressivity of EGFs lacks specificity about what \"expressive enough\" means in terms of parameterization of $f^*_{\\rightarrow}$. While the theorem states that EGFs can approximate any smooth density, the conditions under which this holds true could be more precisely defined.\n- The experimental results in Section 4.1 demonstrate the stability of EGFs with regularization, but the paper doesn't thoroughly explore why regularization has this stabilizing effect—whether it's simply controlling flow size or inducing more meaningful behavioral changes.", "methods_and_evaluation_criteria": "The methods proposed in the paper are appropriate for addressing the challenges in training GFNs. The authors' formulation of EGFs using a finite set of diffeomorphisms with topological ergodicity provides a mathematically sound foundation that enables tractable flow matching while maintaining expressivity.\nThe introduction of the KL-weakFM loss is particularly noteworthy as it eliminates the need for a separate reward model in imitation learning settings, addressing one of the key challenges identified. By coupling cross-entropy with weak flow-matching control, this approach offers a more direct path to training generative models from demonstration data.\nThe evaluation criteria used in the experiments are sensible. For the toy 2D tasks, the authors assess the expressivity and stability of EGFs, which are important properties for generative models. For the NASA datasets on the sphere, they use the KL-weakFM loss and evaluate sample quality, demonstrating the real-world applicability of their approach. The additional reinforcement learning experiments with a target reward further validate the flexibility of the framework.\n\nHowever, the evaluation could be strengthened by:\n\n- Including direct comparisons to standard generative models like diffusion models in computationally constrained settings, which would better highlight the claimed advantages of EGFs.\n- Providing more comprehensive metrics such as training time, sampling efficiency, and robustness to hyperparameter choices.\nMore extensive testing on higher-dimensional problems to demonstrate scalability beyond 2D and spherical domains.", "theoretical_claims": "The paper presents several theoretical results, including:\n\n- Theorem 3.4 on the expressivity of EGFs\n- Definition 3.3 of $L^2$-exponential ergodicity\n- Theorem 3.8 and Corollary 3.9 on sampling error bounds\n\nThe proofs are in the appendix, which should be explicitly mentioned in the main text. While the core ideas behind these results seem sound, Definition 3.3 could be explained more clearly. As I understand it, the concept of $L^2$-exponential ergodicity ensures that the Markov kernel decorrelates functions over time, meaning the influence of initial conditions diminishes rapidly in an $L^2$-sense.\n\nThe formulation of Theorem 3.8, which provides bounds on sampling error in terms of total variation distance, is valuable but would be strengthened if expressed for acyclic discrete GFlowNets, for example, using common notation in the field, which would make the connection to existing work more explicit.", "experimental_designs_or_analyses": "The experimental designs in the paper are generally sound. The authors test their approach on both synthetic and real-world data, which helps validate the method's applicability.\n\nIn Section 4.1, the authors test EGFs on a checkerboard distribution, which is a good choice for evaluating expressivity. However, the analysis of why regularization stabilizes training could be more thorough.\n\nFor the experiment in Section 4.2 on the volcano dataset, the authors appropriately use the KL-weakFM loss from Algorithm 1 and provide comparisons to baselines. The positive results here are compelling evidence for the effectiveness of EGFs.\n\nOne potential issue is that Algorithm 1 uses importance weighting terms involving ratios of densities $f^*_{\\rightarrow}$ and $f^*_{\\leftarrow}$. If these densities are poorly estimated or close to zero in some regions, the weights could become unstable. It would be valuable to know if the authors encountered this issue in their experiments and, if so, how they addressed it.", "relation_to_broader_scientific_literature": "The paper builds upon and extends several strands of research in generative modeling:\n\n- It draws connections to normalizing flows, which model distributions through invertible transformations.\n- It relates to diffusion models, which generate samples by reversing a diffusion process.\n- It extends the framework of GFlowNets, which were initially designed for generative reinforcement learning.\n\nThe paper contributes to unifying these disparate approaches under a common theoretical framework, which is valuable for advancing the field's understanding of generative modeling.\n\nHowever, the authors should acknowledge that Zhang et al. (2022) in \"Unifying generative models with GFlowNets and beyond\" already attempted to extend GFlowNets to imitation learning using the Trajectory Balance Consistency objective, achieving positive results. This prior work is relevant to understanding the context of the current paper's contributions.", "essential_references_not_discussed": "Besides the Zhang et al. (2022) paper mentioned above, which should be cited given its relevance to extending GFlowNets to imitation learning, I don't have specific knowledge of other missing essential references.", "other_strengths_and_weaknesses": "Strengths:\n- The paper presents a compelling unification of different generative modeling approaches, which is valuable for advancing theoretical understanding in the field.\n- The introduction tells a coherent and gradual story, building from normalizing flows to diffusion models to GFlowNets, which helps contextualize the work.\n- The proposed Algorithm 1 elegantly trains an ergodic generative flow using imitation learning with a balanced loss function.\n\nWeaknesses:\n- The presentation of some mathematical concepts is dense and could be more accessible. For example, Equation 7 appears to stem from the detailed balance condition and conservation of probability, but these connections aren't explicitly stated. Most theorems and definitions would benefit from a more gentle introduction for the typical ICML paper reader.\n- The background section on GFlowNets could be clearer, particularly in explaining how the stopping probability $\\frac{dF_{\\text{term}}}{d(F_{\\text{term}} + F^*_\\rightarrow)}(s_t)$ ensures that the Markov chain naturally stops in regions that match the terminal distribution (in the case of discrete acyclic GFlowNets).\n- The paper would benefit from a more thorough comparison to existing generative models in terms of sample quality, training time, and robustness.", "other_comments_or_suggestions": "- The background section should more clearly explain the stopping probability in GFlowNets and how it relates to the terminal distribution. Specifically, it should clarify that when $F_{\\text{term}}$ has a lot of mass, the process is likely to stop, and when $F^*_{\\rightarrow}$ dominates, it is likely to continue.\n- The derivation of Equation 7 should explicitly mention its connection to the detailed balance condition ($\\text{Backward flow density} \\times \\text{Backward transition probability} = \\text{Forward flow density} \\times \\text{Forward transition probability}$) and conservation of probability.\n- Definition 3.3 could benefit from a more intuitive explanation, perhaps using the interpretation that it ensures the Markov kernel decorrelates functions over time.\n- The authors should explicitly mention that proofs are in the appendix.", "questions_for_authors": "- In Theorem 3.4, could you clarify what \"expressive enough\" means in terms of the parameterization of $f^*_{\\rightarrow}$? What specific conditions must be met for EGFs to approximate arbitrary smooth densities?\n- In Algorithm 1, have you encountered issues with importance weighting terms becoming unstable when densities $f^*_{\\rightarrow}$ and $f^*_{\\leftarrow}$ are poorly estimated or close to zero? If so, how did you address this challenge?\n- Could you elaborate on why regularization stabilizes training in the experiments in Section 4.1? Is it merely controlling the flow size, or does it induce more meaningful changes in model behavior?\n- Have you compared EGFs and standard generative models like normalizing flows and diffusion models? How do EGFs compare in terms of sample quality, training time, or robustness?\n\nI'd be happy to raise my score if my questions are addressed and if the authors commit to improving the clarity and readability of their (great) paper, by providing a reasonable plan.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "weV4yOTTVZ", "reviewer": "Reviewer_HCTd", "summary": "The paper studies the problem of generative modelling and sampling (a bit confusingly referred to in the paper as IL and RL). The authors focus on the framework on GFlowNets in continuous spaces and learning directly from samples. The paper presents an alternate theoretical framework of Ergodic Generative Flows thats redefines the transition mechanism using a finite set of globally defined diffeomorphisms—smooth, bijective transformations with smooth inverses—that exhibit ergodic properties. Ergodicity ensures that iterative application of these transformations enables comprehensive exploration of the state space, allowing EGFs to approximate any point within it over time. This framing results in the flow matching objective requiring only a discrete sum over the finite set of diffeomorphisms instead of an integral. The paper establishes theoretically that EGFs can approximate any continuous distribution on compact manifolds such as tori and spheres under some assumptions. The paper also introduces KL-weakFM, which integrates a Kullback-Leibler (KL) divergence-based cross-entropy term with a relaxed version of the flow-matching condition, allowing learning directly from samples. The paper also includes some experimental results on low-dimensional problems, with EGFs showing strong performance.\n\n### Update after rebuttal\n\nThe authors provided some clarifications to my some of my questions. However, I do not agree with some of the claims made in the rebuttal (about calling what the method is doing imitation learning over generative modeling and TB not being a \"pure GFN\" and thus not a valid baseline). The authors also did include a baseline with a learned density which is discussed in the paper. Regardless of these, I think the paper makes interesting contributions so I keep my positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear HCTd,\n     \nWe thank you for your detailed review. Before going into detail, allow us to emphasize that the present submission is an \"Exercise in Style\": how far can we go with \"pure\" GFN without a separate energy model? We understand it may be restrictive, but this is the game we decided to play.\n\n\n1. You are right to point to Sendera et al., 2024, which deals with GFN generalization of diffusion models. To the best of my knowledge, Sendera et al. are restricted to reward and energy model objectives. As such, we should cite them together with Zhang et al., 2022. \n\n2. Experimental Designs Or Analyses (1): We agree that DB or TB losses comparison is important, we made the initial choice to restrict ourselves to FM-losses for consistency. Also, the main purpose of the RL section is to check the basics of EGF in a standard GFN setting and confirm the 0-flow problem noted by Brunswic et al. DB or TB losses require changes in our implementation, but we will try to incorporate stability results by the end of the rebuttal discussion.\n\n3. Experimental Designs Or Analyses (2): We fail to understand the comment as Figure 2 provides such a comparison.\n\n4. Tractability of TB.   Since TB objective does not enforce flow-matching property (there is no flow, only policies) it's not a \"pure\" GFN. Furthermore, to the best of our knowledge, TB requires a separate energy model since it has no flow. On the other hand, DB may be in the scope, but the expectancy of the DB loss is the same as the expectancy of the FM loss if the backward policy is chosen according to formulas (6-7), except that DB has a higher variance. \n \n5. The other reviewers also made several suggestions and remarks regarding rewording. Please see our answers to the other reviews for the intended changes.\n\n6. We wanted to emphasize the distinction between \"target distribution with density\" and \"target distribution with samples.\" We felt that generative modeling in AI covers many practices that intersect both worlds (the work of Sendera et al., for instance), and we struggled to find the right terminology. Although imperfect choices, \"Imitation Learning\" carries the idea of \"examples\" while \"Reinforcement Learning\" is strongly associated with \"reward\" in the community. We cannot change this choice for the present submission, but we are open to suggestions for terminology that you may provide.", "claims_and_evidence": "C1: Extending the theory of GFNs with quantitative results about sampling in the non-acyclic case. \n\nTo the best of my understanding, the results in TB (Malkin et al. 2022) are applicable with a weaker assumption of termination w.p. 1 instead of acyclicity. Aside from Brunswic et al. 2024 which is discussed briefly in the paper, the result in Theorem 3.8 provides a novel quantitative result for GFNs in non-acyclic domains, providing a bound on the total variation. Although I note that the bound is not tested numerically. \n\nC2: Develop a theory of EGFs which induces a tractable flow matching loss\n\nIn Section 3, the authors introduce EGFs which provide an alternate parameterization for policies in terms of a finite set of diffeomorphisms, and establish universality in Theorem 3.4, Theorem 3.5 and Theorem 3.6. The authors also discuss the dependency of the result on the $L^2$ mixing summability property in the Limitations. \n\nC3: Propose a coupled KL-weakFM loss to train EGFs for generative modeling\n\nIn Section 3.3, the authors propose and discuss the KL-weakFM loss and validate it numerically in experiments in Section 4 in the generative modeling setting. This is the claim with the weakest evidence as I will elaborate further below. In particular, the experimental choices could be improved significantly.", "methods_and_evaluation_criteria": "* A shortcoming already discussed in the paper is that the empirical evaluations are limited to low dimensional problems.", "theoretical_claims": "The key contributions of the paper are theoretical in nature. I have read through all the results and looked at the proofs for Theorem 3.4, 3.5, 3.6 and 3.8. I have tried my best to check the correctness but it Is possible that I might have misunderstood some aspects.", "experimental_designs_or_analyses": "* Additionally I find the baselines in the experiments lacking. For instance, in the RL case there is no comparison with other tractable GFN losses (for continuous spaces) like trajectory balance. \n* While the authors talk extensively about the shortcomings of learned diversity, the IL experiments have there is no comparison with a learned density baseline.", "relation_to_broader_scientific_literature": "The contributions of the paper add to the literature on GFlowNets, In particular the quantitative bound in Theorem 3.8 is useful addition to the broader literature on GFlowNets.", "essential_references_not_discussed": "In the context of tractable continuous GFlowNets, Sendera et al. 2024 leverages the view of GFNs are diffusion models to enable scaling to high dimensional problems and would be useful to discuss here.", "other_strengths_and_weaknesses": "Strengths:\n* I find the idea of replacing policies with a finite set of diffeomorphisms quite novel and interesting, as it opens up new avenues to explore. \n\nWeaknesses\n* (Minor) The paper focuses quite heavily on the intractability of flow matching in continuous spaces and equates it as a weakness of GFlowNets in general, even though there are tractable objectives like TB. I think the framing could be improved in that regards.", "other_comments_or_suggestions": "* (Minor) As the paper is mainly theoretical, the writing can be a bit dense at times. Some rewording, especially in Section 3 to make the results a bit easier to follow would be helpful. \n* (Minor) What the authors call imitation learning is really just a problem of generative modelling, and I find calling it IL a bit confusing.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel view on GFlowNets / hierarchical generative models in continuous state spaces. Motivated by some limitations of continuous parametric kernels (as, for example, in the GFN training algorithms for diffusion samplers), it is proposed to model the transitions by a finite collection of trainable diffeomorphisms of the underlying space and to train policies that select which of them to use. Under an ergodicity condition on the diffeomorphisms, this allows to sample any well-behaved distribution in the limit. Losses are described and tested in both the distribution-matching and data likelihood maximisation settings.

Overall, reviewers found the idea elegant, and I concur. In the responses, the authors committed to improving the clarity of the writing so as to make the main text less dense. The experiments were found borderline-sufficient, and the authors are encouraged to add discussion and comparison with more standard settings where GFNs have been used (e.g., diffusion samplers). 

Note: Is there an error in the first sentence of the abstract: GFNs were initially introduced on **acyclic** settings?
Decision: Accept (poster)