MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets
Authors: Dominic Phillips
Flaviu Cipcigan
Venue: Accepted by TMLR ()

Abstract: Generative Flow Networks (GFlowNets) are a class of generative models that sample objects in proportion to a specified reward function through a learned policy. They can be trained either on-policy or off-policy, needing a balance between exploration and exploitation for fast convergence to a target distribution. While exploration strategies for discrete GFlowNets have been studied, exploration in the continuous case remains to be investigated, despite the potential for novel exploration algorithms due to the local connectedness of continuous domains. Here, we introduce Adapted Metadynamics, a variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains. We use Adapted Metadynamics as an exploration strategy for continuous GFlowNets. We show several continuous domains where the resulting algorithm, MetaGFN, accelerates convergence to the target distribution and discovers more distant reward modes than previous off-policy exploration strategies used for training GFlowNets.

Published in Transactions on Machine Learning Research (09/2025)
MetaGFN: Exploring Distant Modes with Adapted Metady- namics for Continuous GFlowNets
Dominic Phillips
dominic..hssllppstedd.ccc
University of Edinburgh, Edinburgh
Flaviu Cipcigan
flaviu flavssskfffavinccppessa.ccm
IBM Research Europe, UK
Reviewed on OpenReview:
Abstract
Generative Flow Networks (GFlowNets) are a class generative models that sample objects in proportion to a specified reward function through a learned policy. They can be trained cither on-policy or off-policy, needing balance between exploration and exploitation for fast convergence to target distribution. While exploration strategies for discrete GFlowNets have been studied, exploration in the continuous the potential for novel exploration algorithms due to the local connectedness of continuous domains. Here, we introduce Adapted Metadynamics, variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains. We use Adapted Metadynamics as an exploration strategy for continuous GFlowNets. We show several continuous domains where the resulting algorithm, MctaGFN, accolerates convergence to the target distribution and discovers more distant reward modes than previous off- off-policy exploration strategies used for training GFlowNets.
remains to be investigated, despite
1 Introduction
Generative Flow Networks (GFlowNets) type of generative model that samples from discrete space X by sequentially constructing objects via a sequence of actions taken from learned policy Pr {Bengio et al., 2021a). The policy PF(s,s' s') specifies the probability of transitioning from some intermediate state to some other state s'. The policy is paramcterised and trained so that, at convergence, the probability of eventually sampling an object X proportional a specified reward function R(x).
GelowNets offer advantages over more traditional sampling methods, such as Markov chain Monte Carlo (MCMC), by learning amortised sampler, capable of singlo-shot generation of samples from the desired distribution. Since GFlowNets learn parametric policy, they can generalise across states, resulting in higher
performance across various tasks (Bengio et al. 2021a
Malkin et al.
2022
Zhang et al. 2022 Jain et al.
2022 Deleu et al. 2022 Jain et al. 2023 Hu et al.
2023 Zhang et al.]
2023
Shen et al. 2023b) Applications
explored so far include conditioned molecular generation (Shen et al. 2023b maximum likelihood estimation in discrete latent. variable models (Hu al.
2023 3, structure learning of Bayesian networks (Deleu al.,
2022), schoduling computational operations (Zhang et al. 2023), and discovering reticular matcrials for carbon capture CCipcigan et al. 2023).
Although originally conceived for discrete state spaces, GFlowNets have been extended to more general state spaces, such as entirely continuous spaces, spaces that are hybrid discroto-comtimuons (Lahlou et al.) 2023). In the inntinuous setting, given the current stale, the policy specifies a continuous probability distribution over subsequent states, and the probability density over states x E X' sampled with the policy is proportional to reward density function r(x). The continuous domain opens up more applications for GFlowNets, such as molecular conformation sampling (Volokhova et al.) 2023 and continuous control problems (Luo al.
2024

---

Published in Transactions on Machine Learning Research (09/2025)
GFlowNets are trained similar to reinforcement learning agents. Trajectories of states are generated either on-policy or off-policy, with the terminating state x E X' providing a reward signal to inform a gradient step on the policy parameters. GFlowNets therefore suffer from the same training pitfalls as reinforcement learning. One such issuc is slow temporal credit assignment, which has thus far been addressed by designing more effective loss functions, such as detailed balance (Bengio et al., 2021bp, trajectory balance (Malkin et al. 2022 and sub-trajectory balance (Madan et al.) (022). This latter approach has recently been extended by providing an energy function induetive bias the early stages of training (Pan et al., 2023).
In addition to the choice of loss function, GFlowNet training also depends on the exploration strategy used to collect samples. Sampling purely on-policy learning is usually inefficient, as it can quickly become trapped in a local optimum and fail to discover novel, high-reward objects. More successful strategies, therefore, rely on a degree of off-policy exploration. For the discrete setting, numerous exploration strategies have been proposed, including c-noisy with a uniform random policy, tempering, Generative Augmented Flow Networks (GAFN) (Pan et al. 2022), Thompson sampling (Rector-Brode Brooks al. 2023), and Local Search GFlowNets (Kim et al. 2024). Although these approaches can be generalised to the continuous domain, there is limited literature benchmarking their effectiveness in this setting.
Sampling in the continuous setting is a common occurrence in various domains such as molecular modelling (Hawkins 2017 Yang et al. 2019) and Bayesian inference (Shahriari et al. 2016). The local connectedness of a continuous domain allows for novel exploration strategies that are not directly applicable in the discrete setting. Sendera et al. (2024) compared several exploration strategies in the context of diffusion samplers and proposed alternating between on-policy sampling and backward sampling from a fixed set of MCMC samples when training continuous GFlowNets. These samples were selected prior to training, thereby inducing fixed pre-training cost but keeping the cost of each training trajectory constant. The authors showed effective exploration, even in high-dimensional problems. However, their MCMC approach requires access to cheap gradients of the reward landscape, which may not always be readily available (Rengarajan et al. 2022). In addition, there are some scttings, such as in molecular conformation sampling, where MCMC approachos are known to require significantly longer limescales to overcome energy barriers than methods based on molecular dynamics (MD) simulations (Abrams and Bussi 2014), thereby making the pre-training cost prohibitively expensive.
We introduce MetaGFN, new exploration algorithm for continuous Gelow inspired by meladynamics, an enhanced sampling method widely used in molecular modeling (Laio and Parrinello 2002). Unlike MCMC-based approaches, MetaGFN operates in a general black-box setting: it queries only reward values from an oracle and does not require reward gradients. As in standard metadynamics, MetaGFN converges rapidly when a low-dimensional representation of the reward function, specified through a so-called collective variable (CV) basis, is known. This makes it particularly effective for small molecular systems, where such CVs are well established and typically low dimensional (Fiorin et al. 2013). Moreover, MetaGFN requires pretraining and adds only small, constant overhead per training trajectory, which becomes negligible when the cost of evaluating the policy or reward is non-trivial, which is the case if either of these is implemented via a deep learning model.
The main contributions of this work are:
Introducing MetaGEN, an algorithm that adapts metadynamics to black box rewards and continuous GFlowNets;
Proving that the Adapted Metadynamics formulation underlying MetaGFN is consistent and reduces to standard metadynamics in the appropriate limit;
Showing empirically that MetaGFN outperforms existing GFlow exploration strategies in various continuous environments, including alanine dipeptide conformation sampling.
The rest of the paper is as follows. In Section 2 we review the theory of discrete and continuous GFlowNets, as well as metadynamics and collective variables. In Section g we presont the Adapted Metadynamics and MetaGFN algorithms. In Section g| we evaluate MetaGFN against other approaches, showing that MetaGFN generally outperforms existing exploration strategies in various continuous environments. We
2

---

Published in Transactions on Machine Learning Research (09/2025)
finish with limitations and conclusions in Sections and 6 Code for MetaGFN is available at https
7/gi //github.com htt.a//ccoomdmmiiiiiicaaaaaaaaaa contsff.....ttt.ggggg...
2
Preliminaries
2.1 Discrete GFlowNets
In G.lowNet, the network refers to directed acyclic graph (DAG), denoted G AS. Nodes represent states E S, and edges represent actions s + s' E A denoting one-way transitions between states. The DAG has two distinguishable states: a unique source state So, which has no incoming edges, and unique sink state L, that has no outgoing edges.
The set of states X' C S that are directly connected to the sink state are known as terminaling slates. GFlowNets learn forward transition probabilities, known as forward policy P=(s'|s), along the edges of the DAG that the resulting marginal distribution over the terminal states complete trajectories (the terminal distribulion). denoted p-(x), is proportional to given reward funclion R X + R. GFlowNets also introduce additional learnable objects, such as backward policy Ps(s|s'), which is distribution over the parents of any state of the DAG, to create losses that train the forward policy. Objective functions for GFlowNets include flow matching (FM), detailed balance (DB), trajectory balance (TB) and subtrajectory balance (STB) [Bengio et al.] 2021a b Malkin et al. 2022: Madan et al., 2022). During training, the parameters of the flow objects are updated with stochastic gradients of the objective function applied to batches of trajectories. These trajectory batches can be obtained either directly from the current forward
policy from alternative algorithm that encourages exploration. These approaches are known on-policy
and off policy training respectively.
2.2 Continuous GFlowNets
Continuous GelowNets extend the generative problem continuous spaces (Lahlou et al. 2023 where the quantity analogous to the DAG is measurable pointed graph (MPG) (Nummelln 1984). MPGs model continuous spaces (e.g., Fuclidean space, spheres, tori), as well hybrid spaces, with a mix of discrete and continuous components, as often encountered in robotics, finance, and biology (Bortolussi and Policriti 2008 Swiler et al. 2012 Neunert et al. 2020).
Definition 2.1 (Measurable pointed graph (MPG)). Let (S,T) be a topological space, where S is the state space, T is the set of open subsets of S, and Σ is the Borel o-algebra. associated with the topology of S. Within this space, we identify the source state So E S and sink state LE S, both distinct and isolated from the rest of the space. In this space, we define a reference transition kernel k S + [0, +20) and a backward reference transition kernel K² S Σ + [0, +20). The support of k(s.-) are all open sets accessible from The support of (s) are all open sets from which is accessible. Additionally, these objects must. be well-behaved in the following sense:
(i) Continuity: For all B E E, the mapping s + k(s, B) is continuous.
(ii) No way back from the source: The backward reference kernel has zero support at the source state, that is, for all B E E, ns,Sn, B) B)=0) O.
(iii) No way forward from the sink: When at the sink, applying the forward kernel keeps you there, that. is, x(1.-) = (-),), where d1 is the Dirac measure of the sink state.
(iv) A fully-erplorable space: The number of steps required to possibly reach any measurable B Σ from the source state, and to guarantee to reach the sink state, with the forward reference kernel is bounded.
The set of objects (S, E, So. 1, n²) then defines an MPG.
Note that the support of kss..) and n² (s, are analogous to the child and parent sets of state in DAG.
3

---

Published in Transactions on Machine Learning Research (09/2025)
The set of terminating states X' are the states that transition to the sink, given by X {s E S s(s, (1)) > 0}, where S = {8o}. Trajectories T are sequences of states that run from source to sink, = (So...., Sn, Ln. The forward Markov kernel Pp : S x Σ + o0) and backward Markov kernel PB: : S x Σ - [0, oo) have the same support as k(s,-) and (s(,,) .) respectively. As Markov kernels, (s.s,)) and (B(s,.) define probability distributions over next states and must therefore satisfy the normalisation condition: for all s S, we have Ss (s ds') = Ss Ps(s, ds') 1. A flow F a tuple F (f, Pr), where is flow measure, satisfying [([1))= - ((s) Z, where Z is the total flow.
- oo)
The reward measure is positive and finite measure R over the terminating states X. The density of R is denoted as r(x), for x E X. A flow F is said to satisfy the reward matching conditions if
R(dx) f(((f() {1}).
If a flow satisfies the reward matching conditions and trajectories are generated by recursively sampling the Markov kernel Pp starting at so, the resulting measure over the terminating stotes, P-(B), is proportional to the reward: P+ = π(x]
R(B) for any B in the o-algebra of terminating states (Lahlou et al. 2023
Objective functions for discrete GFlowNets gencralise to contimuous GFlowNets. However, in the contimuous case, the forward policy Pr S S + [0, oo); backward policy PB S S + [0, oc) and parameterised flow f: + [0, oo) parameterise the P',, PB transition kernels and flow measure f on an MPG. Discrete GFlowNets paramcterise log transition probabilities and flows on a DAG. In this work, we consider DB, TB and STB losses. For a complete trajectory T, the TB loss can be written as
LTB(T) = log Ze II o (5), S441;0) SL+1))
r(s,) 1== (n
where Zo E IR is the paramctorised total flow (sec Appendix A for the DB and STB loss functions).
exploration algorithm designed for training continuous GFNs in the black-bor setting; where we do not have access to the internals of the function parametorisation of r(x), hence a closed form expression for Vr(æ) is unavailable. Thus, if gradicnts of r(x) are needed, they must typically be computed via finile differences, requiring O(d) evaluations of r(x), where d dim(X). Computing Vr(x) can then easily become computational bottleneck. For instance, in drug discovery, r(z) may quantify the binding affinity of a molecular conformer x given target protein, and each evaluation of r(z) might require costly molecular dynamics simulations of the protein system (e.g., via alchemical free-energy perturbation Mey et al. 20201).
Black-box continuous GFNs. Later in this work, we introduce
2.3 Exploration strategies for GFlowNets
GFlowNets can reliably learn using off-policy trajectories, key advantage over hierarchical variational models (Malkin et al.] 2023). For optimal training, it is common use replay buffer and alternate between on-policy and off- off-policy (exploration) batches (Shen et al.) 2023a). In the discrete case, proposed techniques to encourage exploration include e-noisy exploration, tempering, and the incorporation of intermediate rewards (Bengio et al. 2021aa Pan et al.] 2022). Exploration strategies for continuous GFlowNets are less well studied in the literature, but several modh designed for discrete GFlowNets can be adapted (Sendera et al. 2024 In this work, we consider the following.
Local Search GFlowNett (Kim al. 2024)): Explores by backtracking and resampling on-policy trajectories. Reconstructed trajectories with higher reward than the original are used for training, thus encouraging the learning of high-reward modes.
Thompson sampling. (Rector-Brooks et al. 2023):
Explores higl-uncertaintt regions using an ensemble of policy heads with a shared torso. A random head generates the on-policy trajectory, and the loss computed by averaging contributions over heads, where each head is independently included with probability p.
Noisy exploration: Explores by increasing policy uncertainty. A small constant is added to the policy variance which is gradually reduced to zero over the course of training.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Nested sampling (Lemos al. 2023 A Markov-Chain Monte Carlo (MCMC) algorithm is used to sample from the reward distribution. A collection of these samples are stored in a replay buffer prior to training. Then, during training, off-policy trajectories are generated by backward sampling from these states using the current backward policy.
2.4 Metadynamics and collective Variables
Molecular dynamics (MD) simulates that dynamics of molecules using Langevin dynamics (LD) (Pavliotis 2014), stochastic differential equation that models the motion of particles under friction and random noise. LD trajectories ergodically sample the Gibbs measure of the molecule, p3(x) x ( where E X represents atomic positions, V(x) is the molecular potential, and 3 is the thermodynamic beta However, when (x) contains multiple, deep local minima, as is common in biomolecules, then LD becomes inefficient, as it tends to get trapped in these minima, slowing the exploration of the full state space.
Metadynamiics overcomes this by progressively modifying the potential landscape to discourage visits to already explored regions (Laio and Parrinello 2002] 2 It achieves this by regularly depositing repulsive Gaussian biases, centered at the current state of the evolving LD trajectory. This modifies the potential to Viotal () V(x)= + Viiia(x, 1); where Vhins (x, 1) is the cumulative bias at time 4. Intuitively, meladynamics thus progressively transforms the landscape into a more level surface, in which the system can diffuse more freely, thereby accelerating exploration (Figure 1p. In the limit + oo, the dynamics approximates free diffusion and uniformly samples the domain of V(x).
Since biases are typically specified on a numerical grid, this gives rise to a memory cost that grows exponentially with dimension. Therefore, biases are typically applied along low-dimensional coordinates known as collective variables (CVs). A collective variable z(x) : X + Z is any mapping from the original (tigh-dimension) state space X' to lower-dimensional space 2. For potential V(x) and collective variables 2, metadynamics is guar- anteed to eventually uniformly sample the domain of the maryinal potential V(zz) := Sx 5(z' - (x)xxx In molecular contexts, ideal CV choices correspond to the physical reaction coordinates that underly rare event, transitions (e.g., backbone dihedrals, interatomic distances) (Laio and Gervasio 2008 De Vivo et al. 2016). In more general settings, idcal CVs should resolve rare events dynamics on V(z), simplfy the landscape (rc- taining its essential features, e.g. barriers and basins), and minimise the dimensionality of this representation. For simple systems, CVs can be derived from known symmetries describe state changes. However, if the choice of CVs is not obvious, then they can learnt through data-driven methods such as Time-L Independent Component Analysis (TICA) (Molgedey and Schuster, 1994). manifold learning algorithms, neural net autoencoders or variational methods (Mardt et al.) 2018 Bonati et al. 2021 Ramaswamy et al. 2021). For an in-depth review on these CV-learning approaches, see Sidky ee al. 2020). Metadynamics has seen numerous extensions (Bussi and Laio 2020). In the next section, we adapt the original metadynamics algorithm to the continuous black-box setting. We will assumee suitable CVs are given, but the theory we present is agnostic to their form-be it analytical expressions, neural networks, non-parametric tabular mappings.
macroscopic order parameters that
3 MetaGFN: Adapted Metadynamics for GFlowNets
Metadynamics has two properties that make it well-snited as an exploration strategy. Firstly, it eventually uniformly samples the domain, ensuring exploration of all local minima. In contrast, techniques such as Local Search GFlowNets, Thompson sampling, and noisy exploration rely on localised strategies that do not have global exploration guarantees and, as such, are prone to mode locking (Section 4p. Secondly, since metadynamics gradually diffuses from the starting configuration, it allows for incremental exploration, which may result in more stable training than global approaches like nested sampling.
To apply metadynamics to GFlowNets, we must first translate the reward density into a potential energy function. We assume that X' is a manifold and that the reward density r(x) is bounded and L1 intograble
We review Langevin dynamics in Appendix Note that the 'meta' in metadynamics stems Trom chemistry concepts, and unrealed meta-learning in computer science. The motivation for the name is that metadynamics accelerates transitions between minima (metastable states) of V(x).
5

---

Published in Transactions on Machine Learning Research (09/2025)
Total Potential
Potential Energy, V(z)
Original Potential Latest Sample
Step 5
Step 20
Step 40
Collective Variable, z
Collective Variable, z
Collective Variable, z
Figure 1: Illustration of metadynamics in multi-well potential three different snapshots in time. Regular deposition of a bias leads to total potential that gradually flattens, encouraging exploration. Red arrows illustrate the general effect of the bias potential on the particle dynamics at that snapshot.
over X with at most finitely many discontimuities. Thus, the target density over the terminal states, p(x) == r(x)/ Jx rx'x)''', dx', can be expressed as Gibbs distribution: p(x) exp(=3VV(x)) Jx r(x') dx', where we identify V(x) = g. In r(x), for some constant 3' > This formulation ensures that high-reward areas correspond to low-potential-nerryy minima. For simplicity, we set f equal to thermodynamic beta.
However, similar to nested sampling, metadynamics has the drawback of requiring access to not only the potential, but also its gradients, which are not directly available in a black-box GFlowNet setting (Section 2.2) We now address this challenge by introducing two algorithms: Adapted Metadynamics (AM) for exploring the potential energy landscape find high-reward states (Section 3.1 and MelaGFN for guiding the GFN to learn from these states and eventually sample all modes of the reward density (Section 3.2
3.1 Adapted Metadynamics
Metadynamics requires the gradient of the total potential, where (X, 8) -V((xx)) + Vbias (x, t)). Using the above assumptions, we have VV(x)= (F(x)/(F'r(x)),, However, r(x) is often a computationally expensive black-box function, and its gradient, Vr(x), is unknown. While finite differences can estimate Vr(x) for smooth, low-dimensional reward distributions, this approach is impractical in high-dimensional spaces. Below, we explain how to adapt metadynamics to this black-box setting and avoid finite difference gradient estimates by storing a dynamically-updated kernel density estimate (KDE) of the potential. We call the modified meladynamics algorithm Adapted Meladynamiics (AM) (Algorithin
Let X, denote the metadynamics sample at time t and z(x) variables, where each 2, is a one-dimensional coordinate, and E Z is d-dimensional. Let 24,1 (xx) denote the corresponding 7th CV coordinate at time and assume that the Jacobian Vxz is well defined for all x. We store a discretisation of the KDE potential and bias potentials in CV space Z. Since these are stored in memory, gradient computations are cheap and require no further ovaluations of r(x).
be a given set of collective
Let (2, 1) represent the KDE of the marginal potential at time L. To compute V 1), we maintain two separate KDEs: (zz t) for visited states (line { and R(z, t) for cumulative rewards (line gb. We update these KDEs on-the-fly at the same time the bias potential is updated, which occurs every integer n steps of Langevin dynamics (line 7 If Z is isomorphic to Euclidean space, we use Gaussian kernels with kernel width o = (01,..., ok) RR matching the width of the Gaussian bias (line 11 This is a reasonable as both are set by the variability length scale of (x).. Finally, the KDE potential V(z, 1) is then calculated as ((2 t) = - log (z.()+e +c), for a fixed constant c <> 0 (linc 100. We found empirically that e ensured numerical stability by preventing division by zero and bounding V from above. Defining V' through the ratio
R(z.t)
MMaingg this choice means that the single parameter & uniquely controls the (unbiased) transition rates between minima of the potential - from the Kramer formula transition rate x 1/ exp(3AV), where AV x $- TT 2 isomorphic d-torus, use von Mises distributions.

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: Adapted Metadynamics
Input
:anniflld environment of terminating states A' with reward density r: X - R. Initial state
(x), E & x Tx, (X). Collective variables z (Z1....,2.) Parameters: Gaussian width (01,...,,0u) E R Gaussian height w 0. Stride Z+ LD parameters: 7, 3. Timeslep Al.
Ñ <-0 0
R 0
((z) 0
Vzina(z) 0
every timestep At:
Z: z(x:)
every n timesteps nAt:
N + N exp
- -i=1 (x1-2)1))
of
â - R + r(x)) cxp (-2 -1 (#1-21.1)))
o?
V - - log R + ()) +()
10
Vbias (z) &- Viias (z) + n At . w exp Σ (11-3,11
11
12
compute forces:
F (ZV(z))z=z + V Vhins
hins (z)lx=n.)
V
propagate x, Pt by AL using Langevin dynamics with computed force F (Alg. 3 Appendix B
R/ ensures that it rapidly and smoothly adjusts whenever new modes are discovered. Furthermore, we prove that V.r eventually discovers all reward modes in the CV space. More precisely, Theorem 3.1. If the collective variable z(x) is analytic with bounded domain, then
lim lim lim (z,t)=14 e+0 00
(1)
where V = V(z) Sx 0(z (Z)d) is the ground truth free-energy surface in CV space.
The proof is in Appendix
Note that the algorithm can be extended to a batch of trajectories, where each metadynamics trajectory evolves independently, but with shared J and Voias which receive updates from every trajectory in the batch. We found empirically that this accelerates exploration and reduces stochastic gradient noise during training and this the version we use in our experiments (Section 4
3.2 MetaGFN
Each Adapted Metadynamics sample x; E N' is an off-policy terminal state sample. To train a GFlowNet, complete trajectories are required. We generate these by backward sampling from the terminal state, giving a trajectory = (So: s1...., 8, 8n = x)), where each state 8,-1 sampled from the current backward policy distribution (s; 0), for from n reasonable credit according the loss function, thus providing useful learning signal. However, since this requires a backward policy, this is compatible with DB, TB, and STB losses, but not FM loss. Given the superior credit assignment of the former losses, this is not limitation (Madan et al. 2022).
This approach means that the generated trajectory T has
In addition, we use a replay buffer. Due to the theoretical guarantee that AM will eventually sample all collective variable space (Theorem 3.1). AM samples are ideal candidates for storing in a replay buffer. When storing these trajectorics in the replay buffer, there are two obvious choices: (1) Store the entire trajectory the first time it is generated, (2) Store only the AM sample and regenerate trajectories using the current backward policy when retrieving from the replay buffer. We investigated both options in preliminary experiments
7

---

Published in Transactions on Machine Learning Research (09/202))
144
12
1.0
F0oe Mini 10 05 mro
0.8
3.6
2
02
2
10
15
20
-10
-5
20
Figure 2: Line environment re- ward density.
Figure 3: Free energy surface of alanine dipeptide.
Figure 4: Grid environment re- ward density in dimensson
(Appendix D.2 Option 2 was found to be uniformly superior and it is the version we use in our main experiments (Section We call the overall training algorithm MetaGFN, with pseudocode presented in Algorithm below.
Algorithm 2: MetaGFN
Input
:Fowwrdd policy Pr. Backwards policy Pn- Loss function L. Parameters: How often to run Adapted Metadynamics batches, freqMD. How often to replay buffer batches, freqRB. Batch size, b. Stride, E Z+. Time step, At > 0
for each episode do:
if episode number is divisible by freqMD:
Run Adapted Metadynamics (batch size b) for time n&t, obtain samples {xI.... X²} Push (xI....,X., ] to the replay buller
Backward sample from {xI...., x using current PB to obtain trajectories [TI..... To}
clif episode number is divisible by freqRB:
Random sample (xi....,X,, Xa from the replay buffer
Backward sample from {xi,..., xx} using current Pn to obtain trajectories [TI....,To else:
10
Generate trajectorics (rI....,To) no} on-policy
Compute loss Ei L(T,, PF, PB) Take gradient step on loss
4 Experiments
We compare MetaGFN with Thompson sampling, noisy, Local Search GFlowNets and nested sampling (Section 2.3 We run experiments in five continuous environments, summarised below. For each exploration strategy, we use a replay buffer and alternate between exploration and replay buffer batches. For MctaGFN, use freqRB 2, freqMD 10. The forward and backward kernels are Gaussian/ /on Mises mixture distributions, with distribution parameters specified at each state by MLP. In all environments, the additional computational expense of running Adapted Metadynamics was negligible (<5%) compared to the training time of the models. Full experimental details are in Appendix D
one-dimensional environment with state space = R (tEN,1St 3), where indexes the position of a state a trajectory. The source state is 89 (0, 0). The terminal states are therefore R {3) ~ R. The collective variable the identity, i.c. x. The reward density, plotted in Figure 2 consists of asymmetric bimodal peak near the origin and additional distant lone peak. It is
Line environment:
8

---

Published in Transactions on Machine Learning Research (09/2025)
given by the Gaussian mixture distribution:
2 + N((20,0000+
r(x)
((2.0,0.6) + +.(2000,0))
-5 - s> 2 otherwise,
0;
where N(n, is a Gaussian density with mean lr and variance o².
Alanine dipeptide environment: One application of continuous GFlow is molecular conformation sampling (Volokhova et al., (003).. Here, we train GFlowNet to sample conformational states of alanine a small biomolecule of 23 atoms that plays key role in modelling the dynamics of proteins [Hermans] 2011 The metastable states of AD are distinguished in 2D CV space defined by the backbone
dipeptide AAD):
dihedral angles & and v.. The resulting free energy surface Vd,, b) in explicit water, obtained after extensive sampling with long molecular dynamics simulations, is shown in Figure 3 The metastable states, in increasing
energy, are P1I OR: CG a', ag, and ap- The state space is S = =T x (t S 3), with source state so PI = ( (11.2,22 2.68). Terminal states are X' = T x [3] = T². The reward function is the Boltzmann weight, r(o, + = 1 exp(-5V (Ø, v)), where Z is the normalisation constant.
Grid environments: We consider (hyper)grids in = 2, 3 and dimensions. The state space is S = [-15, 15 x (1 N, S < 33. The k-dimensional (hyper)grid consists of 2* modes, located at the corners of the (hyper)cube [-10, 10 The (hyper)grid is centered on the origin with an edge width of 20. The reward modes are Gaussians with variance o² (Figure 4 We use trajectory lengths of 3, 5, and 6 for dimensions 2, 3, and 4 respectively. The collective variable is the identity, i.e. x. The source state is the origin and terminal states are in X= R
4.1 Results
In each environment, we run experiments with three different loss functions: Detailed Balance (DB), Trajectory Balance (TB) and Subtrajectory Balance (STB). We evaluated performance by computing the L1 error between the known reward distribution and the empirical on-policy distribution during training resulting from 10t independent samples. The results, averaged over 10 repeats, are shown in Figure 5 In 12 of 15 experiments, MctaGFN outperformed all other exploration strategics. We give further analysis below.
Line Environment: Among losses, TB shows the lowest variance and, for all losses, MetaGEN always converges to the lowest e Indeed, MetaGFN is the only method that consistently samples the distant reward peak at 20 (Appendix D.2p. Although other strategies occasionally sampled the distant peak, MetaGFN alone converges because it continuously samples this peak, even if the forward policy starts to lock onto the central modes, thus ensuring that the replay buffer is always populated with diverse samples. The slight increase in the loss of MetaGFNN around batch number x 10x occurs as the on-policy distribution widens when Adapted Metadynamics first discovers the distant peak. Appendix D.2 provides further analysis of Adapted Metadynamics in this environment and compares different MetaGFN variants, with and without noise, and with and without trajectory regeneration. We confirm that the variant of MetaGFN presented in Figure 5 [no added noise, always regenerate trajectories) is the most robust.
Alanine Dipeptide Environment: For each loss, MetaGFN generally converges to a lower error than all other exploration strategics. However, for TB loss, the average L1 error is marginally higher than on-policy training, but this conceals the fact that the best-case error smaller. To better understand this result, we examined the best and worst training runs (as measured by L1 error) for TB on-policy and TB MetaGFN, shown Figure 6 Note that, unlike on-policy, the best MetaGFN run learns to sample the rare az mode, In the worst case, MetaGFN fails to converge (although this is rare; only one of 10 runs failed). In Table [ quantify how often the different AD modes were sampled over the different repeats (a mode is considered sampled if the on-policy distribution has a mode within the correct basin of attraction). TB loss with MetaGFN is the only combination that consistently samples the majority of modes. The only mode not sampled by any method aD, which has a natural abundance approximately 10 times less frequent than az.
We do show results for nested sampling on the line environment as the implementation used did not support one- dimensional environments, see Appendix D>
9

---

Published in Transactions on Machine Learning Research (09/2025)
Line (1D)
Alanine Dipeptide (2D)
(2D)
Grid (3D)
Grid (4D)
0.6
0.0
0.5
0.5
0.8
0.8
0.7
0.4
0.6
0.6
0.7
0.6
DB
0.3
0.6
0.5
0.2
0.4
0.4
0.4
0.5
0.1
0.2
0.2
0.4
0.0
0.2
0.3
0.0
Error
0.6
TB
0.3
0.2
0.4
0.4
0.4
03
0.3
0.2
0.2
0.3
0.4
0.0
0.2
0.3
0.6
0.9
0.9
0.5
0.8
0.0
0.8
0.8
0.7
0.4
0.6
0.6
0.7
0.6
Onipoliyy
STB
0.3
0.6
Thompson sampting
0.2
0.4
0.4-
0.4
0.5
Neley
MetaGrN
0.1
02
6.2
0.4
LocalSearch
NestedSampling
10F
10
10a
10a 102
162
10a
10a
10
16
10t
10=
10*
10
Batches Sampled
Batches Sampled
Batches Sampled
Batches Sampled
Batches Sampled
Figure The L1 difference between on-policy and reward distribution during training for different loss functions and exploration strategies. The mean plotted with standard error over 10 repeats. DB Detailed Balance loss, TB Trajectory Balance loss, STB Subtrajectory Balance loss.
Table 1: Number of correct samples of AD modes in trained GFlowNets over 10 independent repeats for DB, STB, and TB loss functions. OP On-policy and MD MetaGFNN The ap mode wasn't. sampled in any model due to its low natural frequency.
DB
STB
TB
OP
MD
OP
MD
OP
MD
P
1
7
6
5
10
8
6
9
7
10
10
9
C.
2
7
5
6
10
8
a
6
9
5
10
aL
0
1
10

---

Published in Transactions on Machine Learning Research (09/2025)
Worst On-policy
Best On-policy
Worst MetaGFN
Best MetaGFN
I
a
n/2
0
n/2
0.0
1.4
0.0
1.0
0.0
0.10
0.0
1.0
-77 - n/2
0
1/2
π
-1
- n/2
m//
n -I
7t/2
0
m/2
n n
m12
0
n/2
π
Φ
Figure 6: Learned on-policy distribution for TB on-policy and TB MctaGFN training runs. The colour bar shows the probability density. Red histograms show the marginal distribution along the angular coordinates. Black curves show the marginal distributions of the ground truth. In the best case, MetaGFN is able to learn the ag mode. In the worst case, MetaGFN fails to converge. On-policy training, although more consistent, fails to learn to sample from the ay mode.
Grid Enwironnentt:: MetaGFN achieves the lowest error in 7 of 9 experiments, although note that the performance of all methods generally decreases with increasing dimension. This unsurprising, since the exploration task becomes more difficult. For MetaGFN, this is also an expected outcome of the curse of dimensionality of the replay buffer grid; in high-dimensional spaces metadynamies samples encounter high-reward modes less frequently, leading to replay buffer with less sample diversity. This is known limitation of metadynamics (Laio and Parrinello 22022). Nested Sampling outperforms MetaGFN in one case (Grid 3D, DB loss) but populating the replay buffer with nested samples from the terminal distribution relies on costly MCMC-based pretraining. The cost of this pretraining scales exponentially with dimension and requires reward gradients, unlike MetaGFN. The generally favourable performance of MctaGFN could be due the method discovering an overall larger diversity of oll-policy samples, that are refreshed and updated during training, in contrast the static samples when using Nested Sampling approach.
5 Limitations
For metadynamics to be an effective sampler, the CVs must be low-dimensional and have bounded support. If they are not low-dimensional, then the memory cost of storing the CV's on the bias grid is too prohibitive, growing exponentially with dimension. If they have unbounded support, then exploration is not guaranteed to converge. Additionally, either these CVs should be known analytically (as in our experiments) or learnt from data (Sidky et. al. 2020). Alternatively, CVs could be adaptively learnt by parameterising x(x; 0) by neural network and updating its parameters by back-propagating through the GFlowNet loss during training. The metadynamics algorithm could also be replaced with variant with smoother convergence properties or slightly better dimension scaling, such as well-tempered metadynamics (Barducci et al. 2008). reconnaissance metadynamics (Tribello et al. 2010), or on-the-fly probabilit-enhannce sampling (OPES) [Ivernizzii 2021). The bias potential could also be defined parametrically, lowering memory overhead. We leave these as extensions for future work.
6 Conclusions
While exploration strategies for discrete Generative Flow Networks (GFlowNets) have been widely studied, methodologies for continuous GFlowNets remain largely unexplored. To bridge this gap, we demonstrated how metadynamics, a well-established enhanced sampling technique in molecular dynamics, can be adapted as an effective exploration strategy for continuous GFlowNets.
In molecular dynamics, atomic forces are derived from the gradient of the potential, whereas continuous GelowNets address problems where the reward function is black box, making gradients generally inaccessible.
11

---

Published in Transactions on Machine Learning Research (09/2025)
We showed that this challenge can be overcome by dynamically updating kernel density estimate of the reward function, proving that this approach ensures proper exploration in the appropriate limit. Our empirical results highlight that MetaGFN provides a computationally efficient way to discover new modes in environments where prior knowledge of collective variables available.
More broadly, this work advocates for leveraging techniques from molecular modeling advance machine learning. Looking ahead, anticipate that this cross-disciplinary perspective will open new directions, enabling further applications of enhanced sampling methods in generative modeling and reinforcement learning.
12

---

Published in Transactions on Machine Learning Research (09/2025)
References
Cameron Abrams and Giovanni Bussi. Enhanced Sampling in Molecular Dynamics Using Metadynamics, Replica-Exchange. and Cenpeerature-Accloratte Entropy, 16(1):163-199 January 2014. ISSN 1099-
4300. doi: 10. 10.3390/e16010163 URL hattps://wws http:::/www.mmp..co///// httppp..coo//0099-8880////1/11/1 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute.
Alessandro Barducci, Giovanni Bussi, and Michele Parrinello. Tell-Tempered Metadynamics: A Smoothly Converging and Tunable Preo-Energy Method. Physical Review Letters, 100(():020603, Jamuary 2008.
ISSN 0031-9007, 1079-7114. doi: 10.
URL http:://iiinnaa..rr./..//
1103/PhysRevLett.. 020603
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Y. Bengio. Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation. ArXiv, June 2021a.
Yoshua Bengio, T. Deleu, J. E. Hu, Salem Lahlon, Mo Tiwari, and Emmanuel Bengio. GFlowNet Foundations. ArXiv, November 2021b.
Luigi Bonati, GiovanniMaria Piccini, and Michele Parrinello. Doep learning the slow modes for rare events sampling. Proceedings of the National Academy of Sciences, 118(44)):21113533311 18, November 2021. doi: 10. 1073/pun http//aaasss101 21153311,,33111. URL httpp:///ww/ http::://wwwppaa....r
www.ppaa..ppg org/ /ttp://o//fu1111.10////ppaa http://///.//ppaaoorr//////////0
2113533118
Publisher:
Proceedings of the National Academy of Sciences.
Massimiliano Bonomi, Giovanni Bussi, Carlo Camilloni, Gareth A. Tribello, Pavel Banáš, Alessandro Barducci, Mattia Bernetti, Peter G. Bolhuis, Sandro Bottaro, Davide Branduardi, Riccardo Capelli, Paolo Carloni, Michele Ceriotti, Andrea Cesari, Haochuan Chen, Wei Chen, Francesco Colizzi, Sandip De, Marco De La Pierre, Davide Donadio, Viktor Drobot, Bernd Ensing, Andrew L. Ferguson, Marta Filizola, James S. Fraser, Haohao Fu, Piero Gasparotto, Francesco Luigi Gervasio, Federico Giberti, Alejandro Gil-Ley, Toni Giorgino, Gabriella T. Heller, Glen M. Hocky, Marcella Tannuzzi, Michele Invernizzi, Kim E. Jelfs, Alexander Jussupow, Evgeny Kirilin, Alessandro Laio, Vittorio Limongelli, Kresten Lindorff-Larsen, Thomas Löhr, Fabrizio Marinelli, Layla Martin-Samos, Matteo Masctti, Ralf Meyer, Angelos Michaelides, Carla Molteni, Tetsuya Morishita, Marco Nava, and The PLUMED consortium. Promoting transparency and reproducibility in enhanced molecular simulations. Nature Methods, 16(8))670 673, August 2019. ISSN 1548-7091. doi: 10. (0.1008///41522-19---55
Luca Bortolussi and Alberto Policriti. Hybrid Systems and Biology. In Marco Bernardo, Pierpaolo Degano, and Gianluigi Zavatlaro, editors, Formal Melhods for Computational Systems Biology, pages 424-40 424-448, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-68594
Giovanni Bussi and Alessandro Laio. Using metadynamics to explore complex free-energy landscapes. Nature Reviews Physics, 2:200 212, March 2020. doi: 10. 40.1033//41225-00000033 URL hattps://i.aaat harvard edu/ /b2 2. 200B ADS Bibcode: 2020NatRP.....2200
Flaviu Cipcigan, Jonathan Booth, Rodrigo Neumann Barros Ferreira, Carine Ribeiro dos Santos, and Mathias Steiner. Discovery of Novel Reticular Materials for Carbon Dioxide Capture using GFlowNets, October 2023. URL http:://axii...tt org/ abs/2310. 03111.77777711111111111117711111 arXiv:2310.07671 [cond-mat].
Marco De Vivo, Matteo Masetti, Giovanni Bottegoni, and Andrea Cavalli. Role of Molecular Dynamics and Related Methods in Drug Discovery. Journal of Medicinal Chemistry, 59 5909):4035-4461 May 2016. ISSN 0022- 2623. doi: 10.1021 jessjmeddhem...b0 Publisher: American Chemical Society.
URL bttpps://do... http:///t.ooog.....// 1021//10.0021/// jwwdchemeecmmmmmm 5b01684
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian Structure Learning with Generative Flow Networks, June 2022. URL http:
23900
arXiv:2202.13990 Jes, stat].
Peter Eastman, Jason Swails, John D. Chodera, Robert T. McGibbon, Yutong Zhao, Kyle A. Beauchamp, Lee-Ping Wang, Andrew C. Simmonett, Matthew P. Harrigan, Chaya D. Stern, Rafal P. Wiewiora,
13

---

Published in Transactions on Machine Learning Research (09/2025)
Bernard R. Brooks, and Vijay S. Pande. OpenMM 7: Rapid development of high performance algorithms for molecular dynamics. PLoS compulational biology, 13(7)=e1005659, July 2017. ISSN 1553-7358. doi: /oa.ra////laall.pb.....
Giacomo Fiorin, Michacl L. Klein, and Jérôme Hénin. Using collective variables to drive molecular dynamics simulations. Molecular Physics, 111(22-23):33455 3362, December 2013. ISSN 0026-8976. doi: 10.1080/ 00268976.2013.8 URL https: http::///oooo.o//////oo 10800 www./////0000000.000 813594 Publisher: Taylor &
Francis eprint: https://doo.oog//00000000/022220
Paul C. D. Hawkins. Conformation Generation: The State of the Art. Journal of Chemical Information /ttp//cc...mmm.....
and Modeling, 57(8):1747 1756, August 2017. ISSN 1549-9596. doi:
URL
https://doi.
7b00221
Publisher: American Chemical Society.
Jan Hermans. The amino acid dipeptide: Small but still influential after 50 years. Proceedings of the National Academy of Sciences, 108(8):3005--3096 February 2011. doi: 1a.107/ppnas.. 1019.1019470108. URL 1019470108 Publisher: Proceedings of the National
htaps://wwa.. http://wwwwpppa....../.aa pnas omag..rg/aa.. Academy of Sciences.
Edward J. Hu, Nikolay Malkin, Moksh Jain, Katie Everett, Alexandros Graikos, and Yoshua Bengio. GFlowNet-EM for learning compositional latent variable models, June 2023. URL http://arxiv.org 0tt//2332.06677 arXiv:2302.06570 [cs, stat].
Michele Invernizzi. OPES: On-the-fly Probability Enhanced Sampling Method, 1l Nuovo Cimento C, 44 (405):1-4, September 2021. ISSN 03905551, 03905551. doi:
URL http:
/ //axxiv..rrgaab//200..0008 06991 arXiv:2101.06991 [physics].
Moksh Jain, Emmanuel Bengio, Alex Hernandez- Garcia, Jarrid Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological Sequence Design with GFlowNets. In Procecdings of the 39th International Conference on Machine Learning, pages 9786 9801. PMLR, June 2022. URL htttps://ppoceeeeeee gress/v162/jainde jiin2aa. html ISSN: 2640-3498.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernandez-Garria,, Jarrid Rector- Brotor-Brooos, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-Objective GFlowNets, July 2023. URL httt:///rxxv..ort abs abs/ 12765 arXiv:2210.12765 [cs, stat].
William L. Jorgensen, Jayaraman Chandrasekhar, Jeffry D. Madura, Roger W. Impey, and Michael L. Klein. Comparison of simple potential functions for simulating liquid water. The Journal of Chemical Physics, 79(2): 926 935, July 1983. ISSN 0021-9606. doi: 10.1063/1.4440 1445666 URL https:ttt///
94589
Minsu Kim, Tacyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local Search GFlowNets, March 2024. URL http://arxiv httt:://axxiv.orr/abb////2
arXiv:2310.02710
(cs, stat].
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-Garcde
Léna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. Proceedings of the 40th International Conference on Machine Learning, 2023. doi: 10.48550/ AA/// 12594. URL https: //tp:i//aiivvvvee orgo
12594 Publisher: arXiv Version Number: 2.
Alessandro Laio and Francesco L. Gervasio. Metadynamics: method to simulate rare events and reconstruct the free energy in biophysics, chemistry and material science. Reports on Progress in Physics, 71(12): 126601, November 2008. ISSN 0034-4885. doi: 0008/00034-185/7/1122/11199 URL http::ttt::////c doi 0rf/200008/00344-8///11222221212
Alessandro Laio and Michele Parrinello. Escaping freo-energy minima. Proceedings of the National Academy of Sciences, 9252)) -25666 October 2002. doi: 10.1073 10.1073/pnas.202222 20242720,,,,399 URL httpp:://ww/tt pnnss Publisher: Proceedings of the National Academy of Sciences.
org/doi/ful1/00 1073//1...11/0/111 http://m.....22229..... 202427399
14

---

Published in Transactions on Machine Learning Research (09/2025)
Pablo Lemos, Nikolay Malkin, Will Handley, Yoshua Bengio, Yashar Hezaveh, and Laurence Perreault- Levasseur. Improving Gradient-guided Nested Sampling for Posterior Inference, December 2023. URL
http://arxiv.on/ hwrg/abs/23115 03911 arXiv:2312.03) [cs, stat].
Shuang Luo, Yinchuan Li, Shumyu Liu, Xu Zhang, Yunfeng Shao, and Chao Wu. Multi-agent Contin- Hous Control with Generative Flow Networks. Neural Networks, 174:106243, June 2024. ISSN 0893- 6080. doi:
URL httpp://ww..cienccdireett htt.:///ww.ccc/ccccccncrr/taiiit
p11/50883368024001667
Kanika Madan, Jarrid Rector- Brooos, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. Proceedings of the 40th International Conference on Machine Learning, 2022.
doi: 10.48550/AR35 ARWIV 1279. URL /ttts:///axiv. orgoorgg//. http/bbb/2200.... 12782 Version Number: 3.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Y. Bengio. Trajectory Balance: Improved Credit Assigument in GelowNets. ArXiv, January 2022.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. GelowNets and variational inference, March 2023. URL http://aaxiv..ot org/ httpg//b//21.....e// 2210 00580 arXiv:2210.00500 [cs, stat].
Andreas Mardt, Luca Pasquali, Hao Wu, and Frank Noé. VAMPnets for deep learning of molecular kinetics. Nature Communications, 9(1):5, Jamuary 2018 ISSN 2041-1723. doi: (0088//44477007-02220 Publisher: Nature Publishing Group.
URL
https: //www. natura. com/ ar aaticcls//411677-7--008888
Antonia S. J. S. Mey, Bryce K. Allen, Hannah F. Bruce Macdonald, John D. Chodera, David F. Hahn, Maximilian Kuhn, Julien Michel, David L. Mobley, Levi N. Naden, Samarjeet Prasad, Andrea Rizzi, Jenke Scheen, Michael R. Shirts, Gary Tresadern, and Huafeng Xu. Best Practices for Alchemical Free Energy Calculations. Living journal of computational molecular science, 2(1):18378, 2020. ISSN 2575-6324, doi: 10.33011/livcconss 1.1188788. URL battps://vee ncbi nlm nih httpph/www....ecaaaa..ee////aaaa http:///www...l.lkcco.hcc//////r
L. Molgedey and H. G. Schuster. Separation of a mixture of independent signals using time delayed correlations. Plysical Review Letters, 32(23)33634 3637, June 1994. ISSN 0031-9007. doi: 10.1103/PhyssReeLLttt......
URL https: : aps org /o1/ http:////...../////eeeee.ettt...
3634
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuous-Discreee Reinforcement Learning for Hybrid Control in Robotics. In Proceedings of the Conference Robot Learning, pages 735-751. PMLR, May 2020. CRLAttps://prroeediii mlr wtt...pres/vvvoo// 1atps//vvvo//nuee22000 neunert htmll ISSN:
2640-3498.
Esa Nummelin. bridge Tracts in Mathematics. Cambridge University Press, Cambridge, 1984.
General Irreducible Markov Chains and Non-Negative Operators.
Cam-
ISBN
978-0-521-6049949.
doi: 10.1017/CB0998051 1526237.
URL https: /www. fambridge
Ling Pan, Dinghuai Zhang, Aaron Courville, Longbo Huang, and Yoshua Bengio. Generative Augmented
Flow Networks, October 2022. URL http://arxiv wttpr/iwww.rg//b///2/1111 03308 arXiv:2210.03308 [cs].
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better Training of GFlowNets with Local Credit and Incomplete Trajectorics. In Proceedings of the 40th International Conference on Machine Learning, pages -28900 PMIRR, July 2023. URL htttps:/pproceedings mlr. ww..m.rrreeet.e p20.s/////2/2//a22 html ISSN: 2640-3498.
Grigorios A. Pavliotis. Stochastic Processes and Applications: Diffusion Processes, the Fokker-Plande and Langevin Equations, volume 60 of Terts Applied Mathematics. Springer, New York, NY, 2014. ISBN 978-1-4939-1322-0 doi: 10.1007/978 1-4939- 1323-7. URL https://link: www.ingr.ccn/mo.onoo...
978-1-4939-1323-7
15

---

Published in Transactions on Machine Learning Research (09/2025)
Venkata K. Ramaswamy, Samuel C. Musson, Chris G. Willcocks, and Matteo T. Degiacomi. Deep Learning Protein Conformational Space with Convolutions and Latent Interpolations. Physical Review X, 11(1): 011052, March 2021. doi:
URL https://link. http://.iiaaaa.........mm. htps.orgg/oi/1..11100 11. Publisher: American Physical Society.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in GelowNets, June
2023. URL hattp://azae www.aaiwwwww.aaaaaaaaaaaaoooeeee htt///bb//3000/ 1769283666.736... 2XXiv:2306.17690 [cs].
Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, and Srinivas Shakkottai. Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration, February 2022. URL http:
//arxiv. hwtp://rrga/bb/////ee/ee 04622
arXiv:2202.04625
Romelia Galomon-Ferrees David A. Case, and Ross C. Walker. An overview of the Amber biomolecular simulation package. WIREs Computational Molecular Science, 3(2):198-214) 2013. ISSN 1759-0884. doi: 10.1002/wcms.1120 1121. URL ptttps://ooninnelliee
http...oo////////////////// 100////w00.000//mmm/ 1121 eprint: https://onllnelibaary.. www.ey.coo//o/////////ooo httpp/df/....///////1 10.1000 1121.
Marcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos, Luca Scimeca, Jarrid Rector- Rector-Brooks, Alexandre Adam, Yoshua Bengio, and Nikolay Malkin. Improved off-policy training of diffusion samplers, May 2024.
URL http://arxiv org/ wwgg/bs///g
ab////bs/22222/ 2402 2402
arXiv:2402.05098 [es, stat].
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando De Freitas. Taking the Human Out of the Loop: Review of Bayesian Optimization. Proceedings of the IEEE, 104(1):148-175, Jamuary 2016. ISSN 0018-9219, 1558-2256. doi: 10.1109/1PRO0::015.2.223 2414444. URL https://iooexplore. ieee org/ org//dccueent/3552066/
Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tom- maso Biancalani. Towards Understanding and Improving GFlowNet Training. Proceedings of the 40th International Conference on Machine Learning, 2023a. doi:
URL
https://arxis
07170 Publisher: arXiv Version Number: 1.
Tony Shen, Mohit Pandey, Jason Smith, Artem Cherkasov, and Martin Ester. TacoGFN; Target Conditioned GFlowNet for Based Drug Design, December 2023b. URL http://arxiv..rrr org /tt./////11/ 03223 arXiv:2310.03223 [cs).
Hythem Sidky, Wei Chen, and Andrew L. Ferguson. Machine learning for collective variable discovery and enhanced sampling in biomolecular simulation. Molecular Physics, 118(5), March 2020. ISSN 0026-8976. doi: 1.1100000026997 200.7...22201..7...7. 1737742. URL https: http://wwwwsstooo osti hwwp:/aags/bbbbll//6111111 Institution: Argonne National Laboratory (ANL), Argonne, IL (United States) Publisher: Taylor & Francis.
Laura Painton Swiler, Patricia Diane Hough, Peter Qian, Xu Xu, Curtis B. Storlie, and Herbert K. H. Lee. Surrogate models for mixed discrete-contimuong variables. Technical Report SAND2012-0491, Sandia National Laboratories (SNL.,, Albuquerque, NM, and Livermore, CA (United States), August 2012. URL
gov/ http://wwiiiigobbbbb///////63//
Gareth A. Tribello, Michele Ceriotti, and Michele Parrinello. A self-learning algorithm for biased molec- ular dynamics. Proceedings of the National Academy of Sciences, 107(41) October 2010. URL Proceedings of the National Academy of Sciences.
doi:
http://///paaa.1111111100111000. Publisher:
Alexandra Volokhova, Michal Koziarski, Alex Hernández-Gariia, Cheng-Hao Liu, Santiago Miret, Pablo Temos, Luca Thiede, Zichao Yan, Alán Aspura-Guzik, and Yoshua Bengio. Towards equilibrium molecular conformation generation with GFlowNets, October 2023. URL http://aaxx..orr///s///11..111 14782 arXiv:2310.14722 [cs].
Yi Isaac Yang, Qiang Shao, Jun Zhang, Lijiang Yang, and Yi Qin Gao. Enhancod sampling in molccular dynamics. The Journal of Chemical Physics, 151(():070902 August 2019. ISSN 0021-9606. doi: 10.1063/1. 5109531. URL http://...rr///...00/11111..1...
16

---

Published in Transactions on Machine Learning Research (09/2025)
David W. Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust Scheduling with GFlowNets, February 2023. URT. http:://axxii..rrttttt/
05446 arXiv:2302.05446 [es].
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio.
Generative Flow Networks for Discrete Probabilistic Modeling. In Proceedings of the 39th International Conference Machine Learning, pages 26412 26428. PMLR, June 2022. URL https://prooeeeding
mlr. ppwss/v162 http///vv2222ha22vvv html ISSN: 2640-3498.
A
Loss functions
For complete trajectory 7, the detailed balanced loss (DB)
n-1
f(s: (fpr S441:0)
LDB(T)= = Σ log
t=0
where ((8+1100) is replaced with r(s.) if Sn is terminal.
The subtrajectory balance loss (STB) is
LSTB(T)= Eosiesen CT-(TT)T
Eosicsen N-i
CSTB(TES) = log f(s;; 0) 11 dr (s2+ (S
[s 0)
f(s;; 0) IT PB
where f(s;; 0) is replaced with r(sy) sy is terminal. In the above, >< 0 hyperparameter. The limit
X + 0 leads to average detailed balance. The X limit gives the trajectory balance objective. We use
A 0.9 our experiments.
B Langevin dynamics
Langevin dynamics (LD), is defined through the Stochastic Differential Equation (SDE):
dx pdd
(3)
dp f(x)dt - <pdt + 2
(4)
In the above, x, E RD are vectors of instantaneous position and momenta respectively, f: F. - force function, w(t) is vector of D independent Wiener processes, M E RD x R. is constant diagonal mass matrix, and 7, B > 0 are constant scalars which can be interpreted temperature respectively. In conventional Langevin dynamics, the force function is given by the gradient. of the potential energy function, f(x) = -x)(xx), where V RDD + R and the dynamics are ergodic with density
+ R is a
friction coefficient and inverse
respect to the
p= - (8H(x.))
where ((x,))=p'MM "pppp--p/++// p/2+
is the Hamiltonian. Since the Hamiltonian is separable in position and momenta terms, the marginal Gibbs-Boltzmann density is position space is simply Pa Px(x) xx -8V(()) often referred to as the Gibbs density.
We present the MarcraMa numerican scheme for integrating equations below. This is called line 14 of Adapted Metadynamics (Algorithm 1
17

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 3: Fuler-Maruyama Langevin Dynamics Step
Input Current state (xe: P,). Force
Parameters: Friction coefficient "y. Thermodynamic beta 3. Timestop At.
Output State (X Pi+ the next timestep.
Sample random vector R. with the same dimension Xt, where each element is an independent sample from standard normal.
Xi-A: = + P,
Pą+Δ: =, fåt 7P,++ 2 R
return (X++Ar, PL+AA)
C Proofs
Lemma C.1. Let (fn(z)) and (gn(x)) be sequences of real functions where lim fn(x) 00, fn(a) ga(x)
nim (n(x) = ao and lim
= h(z). Then, for all c <> O, we have lim cx)++ fa(x) = h(x).
Proof.
[x(x)
lim fn(x) + n-to0 (x) (
lim (n(x)+
and the hand side is the product of two functions whose limit exists so, by the product rule of limits
lim fn(z) lim 1 = h(z) 1= h(x),
2=000
(x) n
1 c/9n(x)
so done.
Lemma C.2. Let (X,) be sequence of continuous random variables that take values on bounded domain
D c Rd that asymplotically approaches the uniform random variable U on D, iee X; - U uniformly. Further, suppose
h(x)= = Di f(x))ga,
Ex I(,,I;)
exists, where E D is sample from X; and f(x) and g(x,x²) are analytic functions on D and D x D respectively. Then,
M(x) E f(us)g(z,u))
E 1zz u.)
where the samples from U. We make assumption of independence of samples.
Proof. Fix probability space (2,F, P) which (X,) and U are defined. Recall that continuous random variable X that takes values on D C IRd is measurable function X : Ω + D where (D, is measure space and B is the Borel o-algebra on D. Let w C Ω denote an arbitrary element of the sample space. The requirement that X, + U uniformly can be written formally as:
Ve > (c) s.t. Vi ((e), Yw c (i(w)) U(w) <e
We prove the Lemma by showing that equality holds for all possible sequences of outcomes W1, ....... That is, we prove:
Di f(X,(x)))))) X,(,,) [[,
(w:))))))) ((()))) Beng(a,U (w:)))
(5)
Since these are ratios of infinite series, to prove their equality it sufficient show that the numerator of the LHS is asymptotically equivalent the numerator of the RHS and that the denominator of the LHS is asymptotically equivalent. to the denominator of the RHS. Recall that two sequences of real functions (a.))
18

---

Published in Transactions on Machine Learning Research (09/2025)
and (b,) are asymptotically equivalent if oo (x) holds with
an (n(a)
where is constant. First, we prove that this
On = E [(x(ax)))g(xxx((x)))
(6)
and
bn Σ (xx
(7)
We write
ENO (Xx( Ei ((X X;
bn
EN f(U(w)))((,) ((w))) + EieN(+ ((U(w))))) U(w)))
Dividing by Σ Ei=N(-)-1 f(U(w)))g(a, (ww))) and taking the limit n so we have
EL F(X( U(a)))
an
lim
lim
2i 21-000
bn 2000
Since f and g are analytic and i (e) for all terms the sums we have, by Taylor expansion, ((Xx(wi)) f(U(wi)) O(e) and g(v, Xi(w))) = g(z, U(w)) + Oce), hence
nO(c) Ei J(U U(w))) )
) =1 lim nO(c) O(n)
lim an lim
1+ 1
1 0
ba -0000
Finally, since can be made arbitrarily small by partitioning the sum N(e)) that sufficiently large, we conclude that lim by = hence (an) and (bn) defined in equation and equation 7 are asymptotically equivalent. By a similar argument, it can be shown that
Cn Σ g(a, X,(w)))
and
dn =
g(z, U (w:)))
are also asymptotically oquivalent. This proves equation
Below, we present. the proof of Theorem 3.1 that appears in the main text.
Proof. For concreteness, throughout this proof we assume that the kernel function is Gaussian. We explain at the appropriate stage in the proof, indicated by (*), how this assumption can be relazed.
First we take the n + oo limit. Recall the notation from Section i.c. assume collective variables z(x) = (x), (x)) where z X' + Z, and Z is d-dimensional. Since the log function is continuous, the limit and log can interchanged and we have
R(zz,tn) tn.) ((z + EE)
lim V(z, tn) gr n+0
log
lim n no
where, from Algorithm I
d
(2g 2j (x,,))²
f(z,t,) = Σ r(xx.) exp
(8)
20
1=1
1=1
19

---

Published in Transactions on Machine Learning Research (09/2025)
n
d (2j (x,, ))²
N N(z (n) = Σ exp
20
Since the domain is bounded, know that for fixed z, both equation and equation have limit infinity, Le. oo R(z,ta tn) oo and lim (z, tn) oo. Hence, by Lemma C.1
have
lim M(z,tn) tn) N(2, tn) + n-000 t,)
lim R(z,ta) kn)
provided the limit on the RHS exists. Next, we show that this limit exists by computing it explicitly. The limit can be written
E r(xe, exp
(- Ei (3 1)²
lim R(z,tn) 20000 tn)
203
lim 21+30
E exp
Ej=1 (2) (x1=))(x1;))=
20
Recall that metadynamics eventually leads to uniform sampling over the domain, independent of the potential. Hence, since R and zxx) are analytic, by Lemma C.2 may replace the metadynamics samples Xt, with samples u, from uniform distribution over X:
E r(u))exp (-E (امب
A(z,tn) lim 100000
2
lim n 100 (n)
Ei= exp
(2) (u,))) -ド==
20
In the limit, the ratio of sums with uniform sampling becomes ratio of integrals:
r(x') exp
(- (-E (a) (x))) dx'
lim A22,,, (n) Ñ(z tn) exp Σd (2)-2,(x')) dx
20
203
The limit is therefore a (scaled) convolution of the reward function with a Gaussian in the collective variable space with width vector a. Taking the limit σj + 0 for all j E f d), the Gaussian convergences to a delta distribution the collective variable space and we have
lim lim R(z,ta) 2 (Z,tt))
J r(x))(zz (i(e))xxx'''
(*) This step also holds for any kernel that becomes distributionally equivalent to Dirac delta function in the limit that its voriance parameter goes to zero. In particular, it also holds for the von Mises distribution that we use in our alanine dipeptide experiment in T2.
Finally, we take the limit e + to obtain
1 lim log ( r(x))((z z(x') (dxx +c)
lim lim lim ((z, tn)
(10)
e0 00000-0000 n-00
3r c
3 I log (r(x')) 8(z z(x'))dxx
1
(11)
/ x
V(x')o(z 2((0)=((x')''' = V(2),
(12)
where have used the definition V(x') - log(r(x')) and in the last step we used the definition of the marginal potential energy the collective variable space. If 2xx) invertible, then the delta function simplifies to a delta function in the original space and we obtain the original potential instead of the marginal potential.
20

---

Published in Transactions on Machine Learning Research (09/2025)
D Experiment details
We detail the experimental setup. In Table we summarise the common hyperparameters used in all experiments. In Table we summarise the experiment-specfie values of the MetaGFN hyperparameters.
A detailed discussion of the setup follows in sections D.1 D.4 below, after some general remarks.
Table 2: Common Hyperparamcters for All Experiments
Parameter
Value
General Training
Training Batches (B)
105
Optimizer
Adam with gradient clipping
Learning Rate Schedule
Linear decay from 10 to 0
log Ze Learning Rate (TB)
Linear decay from 10-1 to
STB Loss À
0.9
Min Nog-Reward Clip
-10
Thompson Sampling Heads
10
Thompson Sampling Bootstrap (p)
0.3
Replay Buffer
Capacity
10 trajectorics
Storage Reward Threshold
10-3
Sampling Strategy
Biased (50% from top 30%, 50% from bottom 70%)
MetaGFN
freqRB
2
freqMD
10
Table 3: Experiment Specific MetaGFN Hyperparameters
Seperimenn-SSpecffi
Hyperparameter
Line
Alanine Dipeptide
Grid (2D/3D/41
Al (MD time step)
0.05
0.01
0.35
n (MD steps)
3
B (Inverse temp.)
0.4009
7 (Langevin friction)
0.1
2
(Hill width)
0.15
10-5
0.10
(KDE bandwidth)
o= 0.1
k= 10
2
e (Reward threshold)
10-3 3
10
10 10-4
Grid Spacing
0.01
0.1
0.075/0.3/04
General remarks on setting hyperparameters
In practice, despite having a large number of parameters, the performance of MetaGFN is not highly sensitive to the specific values. The most important hyperparameters are the KDE bandwidth a and the bias height w.
Setting a: Determined by the length-scale of variation of the problem, which usually known or can be estimated in practice; for the line environment we set o 0= (smallest variance peak was 0.1) and for the grid environments a (variance of peaks is 2). These ball-park values immediately gave the expected metadynamics behaviour and did not require any further tuning.
Setting w: This controls the deposition rate of the Gaussian bias and is linked to freqMD, which determines how often biases are applied. Simulations with and freqMD such that w/freqMD = const show similar behaviour. If w too low, exploration slows, and if too high, metadynamics becomes unstable. Intermediate w values yield sensible results. We set w by examining an on-policy training run and sctting w such that metadynamics explores new modes in a similar timescale the time it takes the policy learn correctly sample the previous mode. As such, it didn't require extensive hyperparameter tuning. An alternative way
21

---

Published in Transactions on Machine Learning Research (09/2025)
to set w would be to schedule it to decrease overtime, as is done in well-tempered metadynamics Barducci et al. 2008 We opted for standard metadynamics to avoid unnecessary complexity in our presentation.
Remarks the computational overhead of MetaGFN
We compare the runtime for training 1000 batches in each environment using an on-policy GFN and MetaGFN in Table 4 Overall, the computational overhead of MetaGFN is practically negligible (typically < 5%) and often statistically insignificant.
We now reason why we expect the cost overhead to be negligible. Let d be the spatial dimension, B be the buffer size and b be the batch size. The KDE + bias update is ((d) computational cost. The cost of adding to the buffer is O(B) and sampling from the buffer is O(b). These costs also scale lincarly with the update frequency, which we kept constant throughout all environments (one metad update every 10 backpropagations). In terms of policy error, this frequency was, within statistical significance, just as good as performing a metad update every steps (Figure S). Importantly, however, an ADAM update step is also O(d) computational cost, since the input size of the first layer scales linearly with d. We have found empirically that ADAM update layer) was more expensive than metadynamics update, for the same dimension. Further, since ADAM and metad both have cost O(d), this means that we expect MetaGFN to continue to have negligible computational overhead, even in higher dimensions. This is supported by our empirical results in Table !
typical multi-layer perceptron (3 hidden layers, 512 hidden units per
Compute resources: Experiments are performed in PyTorch on a desktop with 32Gb of RAM and a 12-core 2.60GHz 17-10750H CPU. For fairness of comparison in runtimes, we use CPU since our meladynamics update code has not yet been implemented efficiently for GPU architectures.
Table 4: Comparison of runtimes over 5 independent repeats of 1000 batches. The runtime is reported as mean + standard deviation in seconds. The final column shows the percentage overhead of MetaGFN or notes if the difference is not statistically significant. All experiments were run using the hyperparameters in Table
Environment
On-policy GFN (s)
MctaGFN (s)
Overhcad / Difference
Line
46.5 0.7
45.5 0.8
Not statistically different
Alanine
130.5 + 1.4
129.3 + 2.7
Not statistically different
Grid 2D
73.4 + 1.2
76.3 0.8
+4%
Grid 3D
150.9 + 1.3
156.1 0.6
+3%
Grid 4D
283.8 1.5
282.6 6 1.6
Not statistically different
D.1 Common experimental parameters
Training parameters: For all environments we train for B = 105 batches use a learning rate with a lincar schedule, starting at 10-3 and finishing at For the TB loss, we train the log Zo with a higher initial learning rate of 10-1 (also linearly scheduled). For the STB loss, we use À = 0.9, a value that has worked well the discrete setting (Madan et al. 2022). We use the Adam optimiser with gradient clipping. For all loss functions, clip the minimum log-reward signal at -10. This enables the model to learn despite regions of near-zero reward between the modes of r(z).
We use a replay buffer with capacity for 10s trajectories. Trajectories are stored in the replay buffer only if the terminal state's reward exceods 10-3, When drawing a replay buffer batch, trajectorics are bias-sampled: 50% randomly drawn from the upper 30% of trajectories with the highest rewards, and the remaining 50% randomly drawn from the lower 70%.
Replay buffer:
Noisy exploration: An additional constant, o, is added to the standard deviations of the Gaussian distributions of the forward and backward policies. Specifically, the forward policy becomes 3t-1; $t
Ei w,N(H;, (a, + 3)2), and similarly for the backward policy, where le is the number of Gaussians in the mixture. We schedule the value of ō so that it decreases during training according to exponential-flat.
22

---

Published in Transactions on Machine Learning Research (09/2025)
schedule:
σ (e 2jc/(B/2) -e 22) j B/2
(13)
j B/2.
where j (I,..., B) is the batch number and oo = 2 is the imitial noise, plotted in Figure Note that, for the alanine dipeptide environment, the policy is a mixture of bivariate von Mises distributions and the noise a is added to the concentration parameter K, where concentration is related to standard deviation through
Thompson sampling: We use 10 heads with the bootstrapping probability parameter set to 0.3.
2.0
Local Search GFN: Local search samples trajectories using a forward-backward roconstruction mothod. First, forward policy samples actions to generate batch size on-policy trajectories. Then, the backward policy is used to re-sample the last. K steps of each trajectory. Finally, the forward policy is used to reconstruct the last K steps, and trajectories in the batch with higher rewards from reconstruction replace the originals. We use == for the line environment, alanine dipeptide environment and two-dimensional grid. We use K = 2 and K = 3 for the three and four dimensional (hyper)grids, respectively.
1.5
1.0-
0.5
2000
4000
6000
8000
10000
Nested sampling: We use the Gradient Based Nestod Sam- pling (GBNS) (Lemos al. 2023) with publicly available im- For each environment, GBNS generates samples from the reward distribution. During training, we alternate between two types of backpropagation: one fully on-policy, and the other using backward-sampled trajectories from the generated samples, guided by the current. backward policy.
Figure 7: Exponential noise schedule.
plementation: https:// github. commPaaloo-eeoos/Gon
D.2 Line environment details
The forward and backward policies are a mixture of three Gaussians. We parameterise dF, DB and the flow f through an MLP with 3 hidden layers, 256 hidden units per layer. We use the GELU activation function and dropout probability 0.2 after each layer. This defines the torso of the MLP. Connecting from this common torso, the MLP has three single-layer, fully-connected heads. The first two heads have output dimension 9 and parameterise the 3 means (u), standard deviations (a) and weights (w) of the mixture of Gaussians for the forward and backward policies respectively. The third head has output dimension 1 and parameterises the flow function f. The mean and standard deviation outputs are passed through a sigmoid function and transformed so that they map to the ranges μa E (-14, 14) and a E (0.1, 1). The mixture weights are normalised with the softmax function. The exception to this parameterisation is the backward transition to the source state which is fixed to be the Dirac delta distribution centred on the source, i.e. ( (1: 0) = 6sp- For the TB loss, we treat log Zo a separate learnable parameter. We use batch size 6 = 64. It takes approximately 1 hour to train continuous Net in this environment with
Parameterisation:
B = 10 batches.
0.1, MetaGFN: We use A/ = 0.05, n n= 2, 2,,, = 0.15, a= 0.1, 6= 10-3 The domain of Adapted Metadynamics is restricted to [-5, 23] and reflection conditions are imposed at the boundary. The bias and KDE potentials are stored on a uniform grid with grid spacing 0.01. Initial metadynamics samples are drawn from Gaussian distribution, mean 0 and variance 1, and initial momenta from Gaussian distribution, mean 0 and variance 0.5.
The LI error between the known reward distribution, p(x) r(x)/Z, and the empirical on-policy distribution, denoted ((r), estimated by sampling 10 on-policy trajectories and computing the empirical distribution over terminal states. Specifically, we compute error 1/2 1 p(x) - r(x) dz, where the integral is estimated by discrete sum with grid spacing 0.01. Note that this error is normalised such that for all valid probability distributions p(x), have 0 & error & 1.
Evaluation:
23

---

Published in Transactions on Machine Learning Research (09/2025)
Sensitivity Analysis of Policy Errors
FreaMD
2
16
FreqMD
+
FreqRB
0.45
0.40
0.35
Average Policy Error
0.30
0.25
0.20
0.15
0.10
FreqRB
Figure 8: Mean and standard deviation L.1 loss after 10 training batches over 10 repeats in the line environment. Blue line: changing freqRB while freqMD 10. Red line: changing freqMD while freqRB 2.
On-policy distributions: Figure 9 shows the forward policy and replay buffer distributions (with bias sampling) after training for 10 iterations with TB loss. MetaGFN is the only method that can uniformly populate the replay buffer and consistently learn all three peaks.
Adapted Metadynamics: Figure 10a shows the L1 error between the density implied by the KDE potential and the true reward distribution during typical training run. Figure 10b shows the resulting V. and Vaias the end of the training. By (1), Adapted Metadynamics has fully explored the central peaks. At (2), the third peak is discovered, prompting rapid adjustment of the KDE potential. By 2.5 x 10a iterations, steady state is reached and the algorithm is sampling the domain uniformly.
Sensitivity analysis of hyperparameters: We compare MetaGFN performance for various freqRB and freqMD values, showing the mean and standard deviation of the final L1 error after 10 batches (Figure Intuitively, we expect that if freqRB set too large, then MetaGFN focuses predominantly on the on-policy dynamics and may fail learn sample distant modes. By contrast, setting freqRB means exclusively training the replay buffer, without any on-policy training trajectories, which we would expect to lead to slow convergence. Therefore, an intermediate value of freqRB should be optimal. We found that setting freqRB- = 2 (giving an equal weighting to both on-policy and replay buffer trajectories) showed the lowest loss. By contrast, performance is not too sensitive to freqMD. freqMD should be set so that Adapted Metadynamics explores the reward landscape in a comparable time it takes to train the GFN to convergence, hence only the order of magnitude of freqMD important. For concreteness, we choose freqMD 10 but the model performance is not too sensitive to this parameter.
Comparing MetaGFN variants:
We consider three MetaGFN variants. The first variant, always backwards sample, regenerates the entire trajectory using the current backward policy when pulling from the replay buffer. The second variant, reuse initial backwards sample, generates the trajectory when first added to the replay buller and reuses the entire trajectory if subsequently sampled. The third variant, with noise, is to always backwards sample with noisy exploration as per equation equation 13 We plot the Ll policy errors in Figure I1 We observe that always backward sample better than reuse initial backwards sample for all loss functions. For DB and TB losses, there is no evidence for any benefit of adding noise, whereas noise improves training for STB loss performing very similarly to TB loss without noise.
24

---

Published in Transactions on Machine Learning Research (09/2025)
On-policy
Noisy
Thompson Sampling
MetaGFN
Replay Buffer Forward Policy
Step Domsity
0.3
0.2
0.0
Step Density
0.3
0.2
0.0.0
5 5 10
15 20-25-5
5
5 10 15
20 25 5 15 20-25-5 0 10 15 20-25
X
X
Figure 9: Forward policy and replay buffer distributions after training for 10th iterations with TB loss. MetaGFN is the only method that can consistently learn all three peaks.
0.7
Vous
0.6
Samples
7.5
(1)
(2)
5.0
error
0.4
Potential
0.3
mm
0.0
0.2
-2.
0.1
-7.5
10a
10a
10a
10
-5
10
15
20
Iteration
X
(a)
(b)
Figure 10: (a) L1 error between (x))= and the reward distribution, r(x)/Z (b) kde potential, bias potential, and positions of final samples after 2.5 10 training iterations.
DB
STB
TB
6x10
40
L1 Error
1
Intral
Whsh
107
10a
10"
10a
10?
10a
10'
10" 10?
10a
10"
10
Iteration
Iteration
Iteration
Figure 11: Comparing MetaGFN variants with and without backward sampling in the line environment
25

---

Published in Transactions on Machine Learning Research (09/2025)
D.3 Alanine dipeptide environment details
Computing the free energy surface: To obtain ground-truth free energy surface (FES) in Φ- space, we ran a 250ns NPT well-tempered metadynamics MD simulation of alaninc dipeptide at temperature 300K (3 = 0.4009), pressure 1bar with the TIP3P explicit water model (Jorgensen et al. 1983 We used the PLUMMED plugin (Bonomi al. 2019) for
00
OpenMM Eastman et al. 2017) with the AMBERI4 force field (Salomon-
Ferrer al. 2013
The forward and backward policies are a mixture of three bivariate von Mises distributions. We parameterise Ωr, ΩB and f throuuh three heads of an MLP with hidden layers with 512 hidden units per layer, with GeLU activations and dropout probability 0.2. The first two heads have output dimensions of 15, paramcterising the mcans, 6 concentrations, and 3 weights of the mixture of von Mises policy. The third head has output dimension 1 and parameterises the flow function f. The means are mapped to the range (- π) through 2arctan(-). Concentrations are parameterised in log space and passed through a sigmoid to map to the range In(x) E (0. Mixture weights are normalised with the softmax function. We use batch size b= = 64. It. takes approximately 10 hours to train a continuous GFlowNet in this environment with = 10 batches.
Parameterisation:
Figure 12: The potential KDE V learnt using MetaGFN on the ala- nine dipeptide environment, con- centration k = 10.
MetaGFN: We use freqRB 2, freqMD 10, At = 0.01, n = 2, 3 0.4009, 7 0.1, w 10-5, 10, 6= 10-6. The bias and KDF potentials stored on uniform grid with grid spacing 0.1. Initial samples are drawn from Gaussian distribution, mean centred P1), variance o² 0..1, and initial momenta from Gaussian, mean = (0,0), variance o² (0.05, 0.05). In Figure 12 show the resulting learnt KDE potential with these parameters.
Evaluation: The L1 error of a histogram of on-policy samples, p v) is computed via a two-dimensional gencralisation of equation D.2 error 1 f_, 1_, ((0, v) r(o.v) dodu, estimated by a discrete sum with grid spacing 0.1.
D.4 Grid environments details
Parameterisation: For the two-dimensional grid, we mixture of Gaussians. For the three and four dimensional (hyper)grids, we
mixture of 2 Gaussians (but use longer trajectories, Section 4 We parameterise fr, ÅB and the flow f through an MLP with 3 hidden layers, 512 hidden units per layer, GELU activation function and dropout probability 0.2. The mcans and standard deviations mapped to the range и (-15, 15) and E (0.1,7) through sigmoid functions. Mixture weights are normalised with the softmax function. We use batch size b = 128. It takes approximately 10 hours to train a continuous GFlowNet in this environment with B = 10 batches.
We use freqRB 2, freqMD 10, A 0.35, л 3, w- 0.10, o a= 2, = 10-4. The bias and KDE, potentials are stored on uniform grid with grid spacing 0.075/0.3/0.0 (2, 3, and 4 dimensions respectively). Initial samples are drawn from a Gaussian distribution, mean centred at the origin, isotropic variance 6² 1. Momenta are initialised to zero.
MetaGFN:
The I.1 error between the histogram of the terminal states of 10a on-policy samples and the exact distribution computed, e.g. for the 2D grid, error 1 5 - p(z, y) r(a-))) discrete sum with grid spacing 0.075.
Evaluation:
drdy, estimated by
2
26


Reviews:
[{"review_id": "qV8E9rF7bS", "reviewer": "Reviewer_6BLR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- It introduces a principled way to incorporate metadynamics–style bias into continuous GFlowNets, bridging sampling methods from molecular dynamics with generative modeling.\n- It provides a rigorous consistency proof (Theorem 3.1) for the KDE‐based estimator of the repulsive potential.\n\n**Weaknesses**\n- All experiments are in low dimensions; it remains unclear how the grid‐based KDE and replay buffer will scale to moderate or high dimensions.\n- Critical parameters (KDE bandwidth σ, bias height w, update stride) lack systematic sensitivity analysis or adaptive tuning strategies.\n- Overhead is only briefly mentioned (< 5 % in low‐D); a detailed breakdown of computational and memory costs versus dimension would aid reproducibility and deployment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes MetaGFN, a novel method that adapts metadynamics-inspired exploration strategies to continuous Generative Flow Networks (GFlowNets). The key idea is to bias sampling trajectories using a KDE-based repulsive potential in a learned or user-supplied collective variable (CV) space, enabling better discovery of distant high-reward modes. The method is theoretically grounded with a consistency result (Theorem 3.1) and is empirically validated on synthetic multimodal benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be good to consider a table to summarize all hyperparameters used in the experiments (e.g., bias height, width, stride, and replay buffer size).\n2. While the paper states that the computational overhead of Adapted Metadynamics is \"<5%\" in all experiments, this claim is not quantitatively backed. (1) Which parts of the algorithm (KDE computation, bias update, buffer maintenance) contribute most to runtime and memory cost? (2) Whether the overhead remains negligible in higher dimensions or with more frequent bias updates? (3) How does the overhead compare between MetaGFN and baseline GFlowNet across tasks?\n3. Check out some typos like on Page 6 \"access to not only the potential potential\", Page 8 \"a a small biomolecule of 23 atoms\".", "broader_impact_concerns": "I do not foresee major ethical or societal concerns arising from this work. On the contrary, the method may have positive implications in scientific discovery tasks, like molecular design, rare-event simulation, or multimodal inference.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uM1kApMRL1", "reviewer": "Reviewer_rMtr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis work introduces a theoretically grounded technique to enhance the exploration capabilities of continuous GFlowNets, supported by experimental validation.\n\nWeaknesses: \nThe experiments are limited to low-dimensional toy problems, providing insufficient evidence for the method’s effectiveness in more challenging, real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper integrates metadynamics into the framework of GFlowNets to enhance the exploration of distant modes in continuous settings. It is shown that the proposed method outperforms existing baseline exploration techniques in toy continuous experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is recommended to include additional experiments on high-dimensional, real-world scenarios.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "InchDkwOEy", "reviewer": "Reviewer_f3nt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n(+) The paper is generally well presented and takes on a novel perspective for exploration when training continuous state space GFlowNets. I believe this is a valuable contribution to the broader area of GFlowNet literature that can be extended and further investigated in future work.\n(+) To add, the authors support the claims of their work (MetaGFN) with both theoretical contributions and empirical evidence through a set of several experiments.\n(+) From my understanding, I believe this is one of the first works to explore exploration strategies for GFlowNets in continuous state spaces.\n\n**Weaknesses:**\n\n- Some of the language could use clarification in certain parts of the text. For example, the authors state, \"However, similar to nested sampling, metadynamics has the drawback of requiring a potential and its gradients, which are not directly available in a black-box GFlowNet setting\". It is not clear to me what is meant by this. Do you mean that training via metadynamics exploration requires the gradients of the energy potential? And this is non-trivial to acquire? If this is the case, could the authors expand on why this is the case? \n- In a similar vein, after reading the paper, what remains slightly unclear to me is the \"meta\" aspect of MetaGFN. I understand that at a high-level, the \"landscape\" of the reward (energy potential) is being adjusted throughout optimization (correct me if my understanding is wrong). However, in the text, it is not entirely clear why this is \"meta\"? It would be useful to have a sentence or two clearly stating and/or explaining the definition of this term.\n- Metrics for performance evaluation are limited -- i.e. only L1 error between the true reward distribution and the on-policy estimated distribution is reported. This makes it difficult to evaluate the robustness of the claims. I would consider either (or both) the (i) KL-divergence between true reward distribution and the on-policy estimated distribution and/or (2) the Jensen-Shannon divergence between true reward distribution and the on-policy estimated distribution. These will give you a non-linear metric for the distance between the estimated and true distributions. \n    - In addition, some clarification on the current metric would be beneficial. Is the L1 error computed in the log-probability space or directly in probability space? \n- Is my understanding correct that the highest dimensional environment considered in the empirical experiments is 4 dimensions (in the hyper-grid environment)? Have the authors explored the behavior of MetaGFN in even higher dimensions? 10-d, or even 50-d? Or possibly provide some intuition on the scaling behavior of MetaGFN.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present MetaGFN, a novel exploration strategy for continuous Generative Flow Networks (GFlowNets, GFNs) based on adaptive Metadynamics. They provide theoretical backing for MetaGFN and empirically demonstrate their proposed approach's effectiveness for exploration in continuous state space environments. The authors showcase that MetaGFN can outperform and accelerate convergence to target reward distributions relative to counterpart exploration strategies. They empirically demonstrate this by considering two synthetic environments, a 1-D \"line\" environment and continuous hyper-grid environments of varying dimensions, as well as the Alanine dipeptide environment for sampling molecular conformations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think this is a very interesting work! I recommend the authors try and make some of the changes/additions pertaining to the items I outlined above in weaknesses. Specifically, adding an additional evaluation metric(s), either one of the metrics I suggested above, or another one from the literature. As well as elaborating on language in some parts of the text.", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)