Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization
Authors: Timofei Gritsaev
Nikita Morozov
Sergey Samsonov
Daniil Tiapkin
Venue: ICLR 2025 Poster (2025)
Keywords: generative flow networks
gflownets
reinforcement learning
sampling
generative models
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: backward policy optimization algorithm for GFlowNets that directly maximizes value in an entropy-regularized MDP

Abstract: Generative Flow Networks (GFlowNets) are a family of generative models that learn to sample objects with probabilities proportional to a given reward function. The key concept behind GFlowNets is the use of two stochastic policies: a forward policy, which incrementally constructs compositional objects, and a backward policy, which sequentially deconstructs them. Recent results show a close relationship between GFlowNet training and entropy-regularized reinforcement learning (RL) problems with a particular reward design. However, this connection applies only in the setting of a fixed backward policy, which might be a significant limitation. As a remedy to this problem, we introduce a simple backward policy optimization algorithm that involves direct maximization of the value function in an entropy-regularized Markov Decision Process (MDP) over intermediate rewards. We provide an extensive experimental evaluation of the proposed approach across various benchmarks in combination with both RL and GFlowNet algorithms and demonstrate its faster convergence and mode discovery in complex environments.

Published as a conference paper at ICLR 2025
OPTIMIZING BACKWARD POLICIES IN GFLOWNETS VIA TRAJECTORY LIKELIHOOD MAXIMIZATION
Timofei Gritsaev
Nikita Morozov HSE University
Sergey Samsonov HSE University
HSE University
Constructor University, Bremen tgritssevvgmal com
nvmorozovéhse.ru
swsamsonov@hse.
Daniil Tiapkin
CMAP -MAP-CNRS - École polytechnique Institut Polytechnique de Paris Université Paris-Saclay, CNRS, Laboratoire de mathématiques d' O'Orayy daniil. wtw..ittttiieeeloeeeeee.ee
ABSTRACT
Generative Flow Networks (GFlowNetts) are a family of generative models that learn to sample objects with probabilities proportional to given reward func- tion. The key concept behind GFlowNets is the use of two stochastic policies: a forward policy, which incrementally constructs compositional objects, and back- ward policy, which sequentially deconstructs them. Recent results show close relationship between GFlowNet training and entropy-regularrized reinforcement learning (RL) problems with particular reward design. However, this connection applies only the setting of a fixed backward policy, which might be a significant limitation. As remedy to this problem, we introduce a simple backward policy optimization algorithm that involves direct maximization of the value function entropy regullareed Markoy Decision Process (MDP) over intermediate rewards. We provide an extensive experimental evaluation of the proposed approach across various benchmarks in combination with both RL and GFlowNet algorithms and demonstrate its faster convergence and mode discovery in complex environments.
INTRODUCTION
Generative Flow Networks (GFlowNets, Bengio et al., 2021) are models designed to sample compo- sitional discrete objects, e.g., graphs, from distributions defined by unnormalized probability mass functions. They operate by constructing an object through a sequence of stochastic transitions defined by a forward policy. This policy is trained match the marginal distribution over constructed objects with the target distribution of interest. Since this marginal distribution is generally intractable, an auxiliary backward policy is introduced, and the problem is reduced to one of matching distributions over complete trajectories, bearing similarities with variational inference (Malkin er al., 2023).
GFlowNets have found success in various areas, such biological sequence design (Jain al., 2022), material discovery Srrmandez-Garciin et al., 2023), molecular optimization (Zhu et al., 2024), recommender systems (Liu al., 2024), large language model (LLM) and diffusion model fine-tuning (Hu et al., 2023; Venkatraman et al., 2024; Uchara et al., 2024; Zhang et al., 2025), neural architecture search (Chen& & Mauch, 2023), combinatorial optimization (Zhang al., 2023), and causal discovery (Atanackovic et al., 2024).
Theoretical foundations of GFlowNets have been established in the seminal works of Bengio et al. (2021; 2023). Most of the literature has since focused on practical applications of these models, leaving their theoretical properties largely unexplored, with few exceptions (Krichel al., 2024; Silva et al., 2024). However, a recent line of works has brought attention to connections between GFlowNets and reinforcement learning (Tiapkin et al., 2024; Mohammadpour et al., 2024; Deleu al., 2024; He et al., 2024a), showing that the GFlowNet learning problem is equivalent to specific RL problem with entropy regularization (also called soft RL, Neu et al., 2017; Geist et al., 2019). 1).. This opened new perspective for understanding GFlowNets. The importance of these findings is supported by empirical evidence, as various RL algorithms have proven useful for improving GFlowNets (Tiapkin et al., 2024; Morozov al., 2024; Lau al., 2025).

---

Published as a conference paper at ICLR 2025
However, these connections still have limitation related to GFlowNet backward policies. While GFlowNets can be trained with a fixed backward policy, standard GFlowNet algorithms allow the training of the backward policy together with the forward policy (Bengio et al., 2023; Malkin et al., 2022; Madan et al., 2023), resulting in faster convergence of the optimization process. Other algo- rithms for optimizing backward policies have been proposed in the literature as well (Mohammadpour et al., 2024; Jang et al., 2024), showing benefits for GelowNet performance. The theory connecting GFlowNets and erguopy-regularized RL based on using the backward policy add "correction" to GFlowNet rewards and shows the equivalence between two problems only when the backward policy is fixed. Thus, understanding the backward policy optimization remains a missing piece of this puzzle. Moreover, Tiapkin et al. (2024) demonstrated that this theoretical gap has practical relevance, as optimizing the backward policy using the same RL objective as the forward policy can either fail to improve even slow down convergence, highlighting the need for more refined approach.
In this study, we introduce the trajectory likelihood maximization (TLM) approach for backward policy optimization, which can be integrated with any existing GelowNet method, including entropy- regularized RL approaches.
We first formulate the GFlowNet training problem as a unified objective involving both forward and backward policies. We then propose an alternating minimization procedure consisting of two steps: (I) maximizing the backward policy likelihood of trajectories sampled from the forward policy and (2) optimizing the forward policy within an entropy-regulariized Markov decision process that corresponds the updated backward policy. The latter step be achieved by any existing GFlowNet or soft RL algorithm, as it was outlined by Deleu et al. (2024). By approximating these two steps through single stochastic gradient update, we derive an adaptive approach for combining backward policy optimization with any GFlowNet method, including soft RL methods.
Our main contributions are as follows:
We derive the trajectory likelihood maximization (TLM) method for backward policy optimization.
The proposed method represents the first unified approach for adaptive backward policy opti- mization in soft RL-based GFlowNet methods. The method is easy to implement and can be integrated with any existing GFlowNet training algorithm.
We provide an extensive experimental evaluation of TLM four tasks, confirming the findings of Mohammadpour et al. (2024), which emphasize the benefits of training the backward policy in a complex environment with less structure.
Source code: ginnb..com/griiseev/ffoootttttt
BACKGROUND
2.1 GELOWNETS
We aim at sampling from probability distribution over a finite discrete space X' that is given an unnormalized probability mass function R: X-> Rzo, which we call the GFlowNel reward. We denote 7. = -Exex R(x) to be an (unknown) normalizing constant.
To formally define generation process in GFlowNets, we introduce a directed acyclic graph (DAG) = (S, where S is a state space and C x S is a set of edges (or transitions). There is exactly one state, 8o, with no incoming edges, which we refer to as the initial state. All other states can be reached from Su. and the of terminal states with no outgoing edges coincides with the space of interest X' Non-termin
states & X correspond to "incomplete" objects and edges + s' represent adding "new components" to such objects, transforming $ into s'. Let T denote the set of all complete trajectories = (sn, 81, .8.,, in the graph, where is sequence of states such that (s; >> 1)+1) E & and that starts at 8g and finishes at some terminal state 8n, E X. As result, any complete trajectory can be viewed as sequence of actions that constructs the object corresponding to sn, starting from the "empty object" so-
We say that state s' is child of state if there is edge ss) E E. In this case, we also say that s is a parent of s'. Next, for any state s, we introduce the forward policy, denoted by (s('(ss) for (s + s') E, an arbitrary probability distribution over the set of children of the state In similar fashion, we define the backward policy as an arbitrary probability distribution over the parents of state and denote it as Pss('1ls),, where (s' + *8) E E.
2

---

Published as a conference paper at ICLR 2025
Given these two definitions, the main goal of GFlowNet training is search for a pair of policies such that the induced distributions over complete trajectories in the forward and backward directions coincide:
R(sn.) ) Z
PF st-1)
PI st), Vr STE.
==1
The relation (1) is known as the trajectory balance constraint (Malkin et al., 2022). We refer to the left and right-hand sides of (1) to the forward and backward trajectory distributions and denote them as
nr PP (T) = II (P(8))s.-1).
R(sn=)
nr Pn(L..-11s)
P (T)
where 7 = (so: $1,.... Sn,) Sn.) ET If the condition (1) is satisfied for all complete trajectories. sampling trajectory in the forward direction using Pp will result in a terminal state being sampled with probability RZz)/Z. We will call such PF a proper GelowNet forward policy.
In practice, we train a model (usually neural network) that parameterizes the forward policy (and possibly other auxiliary functions) to minimize an objective function that enforces the constraint (1) or its equivalent. The main existing objectives are Trajectory Balance (T3, Malkin et al., 2022), Detailed Balance (DE, Bengio et al., 2023) and Subtrajectory Balance (SubTB, Madan et al., 2023). The Sub TE objective is defined
Wikh log Fole, ( Pr(selet-1.0) Felsy) Ili Pa(s:111a..0)
(SubTB(O;T)= =
Σ OSicksn,
where (s(s) is a neural network that approximates the flow function of the state see (Bengio et al., 2023; Madan et al., 2023) for more details on the flow-based formalization of the GFlowNet problem. Here Fo (s) is substituted with R(s) for terminal states s, and wyk is usually taken to be * and then normalized to to 1. TB and DE objectives can be viewed as special cases of (3), which are obtained by only taking the term corresponding to the full trajectory or to individual transitions, respectively. All objectives allow either training the model in on-policy regime using the trajectories sampled from Pp or in an off off-policy mode using the replay buffer or some exploration techniques. In addition, is possible either optimize Pp along with Pp to use fixed PB, e.g.. the uniform distribution over parents of each state. One can show that given any fixed PB. there exists unique PF that satisfies (I); see, e.g., (Malkin al., 2022).
2.2 GETS AS SOFT RI.
In reinforcement learning (Sutton & Barto, 2018), a typical performance measure of agent is a value function, which is defined as an expected discounted sum of rewards when acting via a given policy. Entropy-rreuuaarize
reinforcement learning (RL), also known as soft RL (Neu al. 2017; Geist al. 2019; Haarnoja et al. 2017), incorporates Shannon entropy H(Pr(- s)) A E log PF(s' (s' []] into the value function. This addition encourages the optimal policy to be more exploratory:
(P"(s:) A Epe
(r(s())))1)) + XH(Pp( sc))) so =
(4)
where A 2 0 is the regularization coefficient. Note that we use the next state instead of the more conventional action representation, as there a one-to-one correspondence between the action taken and the resulting next state in DAG environments. Similarly, the regularized Q-values QP" (s, s') are defined as the expected discounted sum of rewards, augmented by Shannon entropy, given an initial state Sg = and the next state $1 s'. The regularized optimal policy Pt is the policy that maximizes vPPo( (s) for any state Note: in standard RL notation, policy is typically denoted as Here, we use Pr. consistently with GFlowNet notation, to avoid notational clutter.
It was proven by Tiapkin et al. (2024) that training GFlowNet policy Pp with a fixed Pp can be formulated as a soft RL problem in a deterministic MDP represented by a directed acyclic graph G, where actions correspond to transitions over edges. For transitions (s - x) that lead terminal states, the RL rewards are defined as ,Ps rPs((s x) = log (n x)) + log R(x), while for intermediate transitions (s s'), the rewards are PP
(s, 8') log Pa(( 8'). By setting À = 1 and 7 it can
3

---

Published as a conference paper at ICLR 2025
be shown that the optimal policy PF Pf in this regularized MDP coincides with the proper GFlowNet forward policy Pr( s), which is uniquely determined by Pn and R (Theorem 1, Tiapkin et al., 2024), thereby inducing the correct marginal distribution over terminal states Rxz)/Z.
In addition, Proposition of Tiapkin al. (2024) provides connection between the corresponding regularized value function at the initial state so for any forward policy Pp (not necessarily proper) and KL-divergence between the induced trajectory distributions:
V (#0; r Pb) = log Z - (T, (IP
The main practical corollary of these results is that any RI. algorithm that works with entropy regularization can be utilized train GFlowNets when PB fixed. For example, Tiapkin al. (2024) demonstrated the efficiency of the classical SožtDQN algorithm (Haarnoja et al., 2017) and its modified variant called Munchausente enDN (Vieillard et al., 2020). Moreover, it turns out that, under this framework, the existing GFlowNet training algorithms can be derived from existing RL algorithms. Tiapkin al. (2024) showed that on-policy TE corresponds to policy gradient algorithms. and DB corresponds to dueling variant of SoftDQN. At the same time, Deleu et al. (2024) showed that TB, DE and SubTB algorithms can be derived from path consistency learning (PCL, Nachum et al., 2017) under the assumption of a fixed backward policy.
2.3 BACKWARD POLICIES IN GFLOWNETS
The idea of backward policy optimization is essential to understanding the GFlowNets training procedure. In particular, the most straightforward approach used in GFlowNet literature (Malkin et al., 2022; Bengio et al., 2023) proposes to optimize the forward and backward policies directly through the same GFlowNet objective (e.g., (3)). This approach can accelerate the speed of conver- gence Malkin et al. (2022), at the same time potentially leading to less stable training (Zhang et al., 2022). This phenomenon motivates studying the backward policy optimization in the recent works of Mohammadpour et al. (2024) and Jang al. (2024).
Mohammadpour et al. (2024) suggested using the backward policy with maximum possible trajectory entropy, thus focusing the exploration challenges of GFlowNets. Such policy is proven to be PB(s s') = n( n2ss/n(s')) where ns)) is the number of trajectories which starts at su and end at s. It corresponds to the uniform one if the number of paths to all parent nodes is equal. When n(s) cannot be computed analytically, Mohammadpour et al. (2024) propose to learn log (((),, 8 E S alongside the forward policy using its relation to the value function of the soft Bellman equation in the inverted MDP (see Definition 2 in Mohammadpour et al. (2024)). Mohammadpour et al. (2024) utilize RL as a tool find the maximum entropy backward policy and make the connection to RL solely for such policies. In contrast, our work theoretically considers the simultaneous optimization of the forward and the backward policy from the RL perspective, and develops an optimization algorithm grounded in it. The approach of Mohammadpour et al. (2024) showed consistently better results in less structured tasks, like QM9 molecule generation (see Section 4.3 for a detailed description). At the same time, more structured environments with a less challenging exploration counterpart show less benefits of the proposed backward training approach.
In another line of work, Jang et al. (2024) claim that the existing GFlowNets training procedures tend to under-exploit the high-reward objects and propose a Pessimistic Backward Policy approach. Thus, the primary aim of Jang et al. (2024) is to focus on the exploitation of the current information about high-reward trajectories. Towards this aim, they focus on maximizing the observed backward flow pf (7) (see (2)) for trajectories leading to high-reward objects, which are stored in a replay buffer. Our approach shares similarities with this method as both compute the loss over whole trajectories but differs in theoretical motivation and the choice of trajectories for PB optimization.
As limitation of both Mohammadpour et al. (2024) and Jang et al. (2024), mention the fact that only a single GelowNet training objective is used in both papers (SubTB and TB respectively) to evaluate approaches for backward policy optimization, while we carry out experimental evaluation with various GFlowNet training objectives (see Section 4).
3 TRAJECTORY LIKELIHOOD MAXIMIZATION
The objective of our method is to formalize the optimization process for the backward policy for reinforcement learning-based approaches. It is worth mentioning that soft RL methods cannot address
4

---

Published as a conference paper at ICLR 2025
the changing of the reward function, except for reward shaping schemes (Ng et al., 1999) that preserve the total reward any trajectory. Therefore, we need to return to the underlying GFlowNet problem. Let us look at the following optimization problem:
minppeulo,Phelll (PPP
(5)
where Te and IT represent the spaces of forward and backward policies, respectively, and P and 7 are defined in (2). is easy to see that any solution to the problem (5) satisfies the trajectory balance constraint (1) and thus induces a proper GFlowNet forward policy. Additionally, it is worth mentioning that for any fixed Pn the problem (5) is equivalent to maximizing value
V (s); Ps) = log KL (IP
over Pp E Tp in entropy-regularizzed MDP, established in Tiapkin et al. (2024, Proposition 1). Thus, the joint optimization resembles the RL formulation with non-stationary rewards. To leverage the problem's 's block structure, we propose a meta-algorithm consisting of two iterative steps, repeated until convergence:
PB = arg minp, KI. |
|P?)))
Pft arg minp, KL (PP |PP DC+1
(6)
worth noting that if these optimization problems are solved exactly, the algorithm converges after the first iteration. This occurs because, for every fixed backward policy PB, there is unique forward policy Pr, such that Prr Pf" see, e.g., (Malkin et al., 2022), ensuring that the loss function reaches its global minimum. In the following sections, we provide implementation details on approximating these two steps.
First Step: Trajectory Likelihood Maximization. Using the connection between forward KL divergence minimization and maximum likelihood estimation (MLE), formulate the following trajectory likelihood maximization objective:
0 arg ming Ep (0:
Crun(0:):==->>> log PB(s:-1|ss, 0).
(7)
In this formulation, P.. This step seeks to update the backward policy by minimizing the negative log-likelihood of trajectories generated from the forward policy. Additionally, instead of solving (7) for exact arg min for every t, perform one stochastic gradient update
(s0, 81....; ,,, denotes a trajectory generated by the forward policy
0
f/ - -VeoTu((0
To approximate the second step of (6), exploit the equivalence between the GFlowNet framework and the entropy-regularized RL problem. This leads the following expression:
Second Step: Non-Stationary Soft RL Problem.
Pfl z arg minp,ell, KL KL(P? (p? |PP
Pf+ = arg maxp VPr (oo ) (8)
where PPs is the RL reward function corresponding to the backward policy PB. This step can be solved using any soft RL method, such as SoftDON (Haarnoja et al., 2018) 8). Additionally, it is noteworthy that all existing GFlowNet algorithms with fixed backward policy can be viewed as variations of existing RI. methods, see, e.g., (Deleu et al., 2024). Thus, they can be used to solve the optimization problem in (8).
To mitigate the computational overhead of searching for exact arg min in (8), we also propose to perform single stochastic gradient update in the corresponding GFlowNet training algorithm
of 0f v) V VoLnng (0); ( P(+1),
where Lxig represents the loss function associated with a GFlowNet or soft RL method, such as SubTB SoftDON. Here, denotes (possibly off off-policy) trajectory.
The complete procedure can be interpreted as soft RL method with changing rewards. Our suggested method is summarized in Algorithm and can be paired with any GFlowNet training method Alg (e.g., DB, TB, SubTB, or So SoftDON). While for PB training our approach uses on on-policy trajectories, Pr can still be trained off-policy, e.g., by utilizing a replay buffer that stores trajectories or transitions.
5

---

Published as a conference paper at ICLR 2025
Algorithm Trajectory Likelihood Maximization
1: Input: Forward and backward parameters 0p. f.. any GFlowNet loss function Laig. (optional) experience replay buffer B;
2: for t- to Niers do
3: Sample batch of trajectories (T k== from the forward policy Pr(-1:;
4: (optional) Update B with (Tx k=1
5: Update 0z!1 0g 7t π Ex=1 VLrux(0 TB (1),, see (7):
=k=1 from B; 6: (optional) Resample batch of trajectories (Th (t) 1
7: Update 0ft1 0f Th R > VLsig(0p (0 Tk PB(+1, ²));
8: end for
Convergence of the method Next, show why this method indeed solves the GFlowNet learning problem. First, we introduce a non-stationary soft reinforcement learning problem of minimizing the worst-case dynamic average regret
+I V " V. (soir"),
(9)
where {r') herry is sequence of reward functions, and y² is revealed learner before selecting policy Pt. Following Zahavy et al. (2021), we conjecture that existing RL algorithms are adaptive to the setting of known but non-stationary reward sequences. The implementation of Sampler player of Ent algorithm by Tiapkin et al. (2023) is an example of such a regret minimization algorithm. Additionally, we notice that the optimization of dynamic regret is well-studied in the online learning literature, even in a more challenging setting of revealing the corresponding reward function after playing policy (Zinkevich, 2003; Besbes et al., 2015).
Next, we provide the convergence result for our two-step procedure, using a stability argument for the first step. The proof is given in Appendix A.I.
Theorem 3.1. Assume that (1) the backward updates are stable, i.e., sup PP - -1
as T + oC, and (2) the forward updates follow non-stationary regret minimization algorithm, i.e. H + as T - 00. Then, there exists proper GFlowNet forward policy Pp that induces the marginal distribution R(x) over terminal states, such that + E.= Pf + PPC as T 30.
During numerical experiments, observed that enforcing stability in backward updates, particularly by using a decaying learning rate and a target network, significantly improves convergence in practice. Furthermore, as the theorem shows, this stability is essential for theoretical convergence. We discuss stability techniques we utilize alongside our algorithm in Appendix A.2.
EXPERIMENTS
We conduct our experimental evaluation on hypergrid (Bengio et al., 2021) and bit sequence (Malkin al., 2022) environments, as well on two molecule design environments: sEH (Bengio al., 2021) and QM9 (Jain et al., 2023). Additional experimental details and hyperparameter choices are provided in Appendix A.3.
We evaluate four GFlowNet training methods: MunchausenDgN (based on the framework of Tiap- kin et al. (2024)), DB (Bengio et al., 2023), TB (Malkin et al., 2022), and SubTB (Madan et al., 2023), which we refer as GFlowNet algorithms (denoted as Cxig in the previous section). On hypergrids, we also include results for So (Tiapkin et al., 2024). Alongside these algorithms, we consider five strategies for learning selecting the backward policy:
our approach (TLM);
fixed uniform backward policy (uniform)
simultaneously learning the backward policy with Pp using the same objective (naive);
maximum entropy backward policy (maxent) (Mohammadpour al., 2024);
pessimistic backward policy (pess mi st (Jang al., 2024).

---

Published as a conference paper at ICLR 2025
hypergrid 20x20x20x20
30-3
ScnDON
Munchassen DQN
DB
T3
SubTB
1.00
0.25
hypergrid 20x20x20r200 (hard)
26.-5
SuNDON
Munclaesen DQN
DB
TB
SubTB
0totbal 62200
1.00
0.25
13
10a
10
12a
101
10
#faxjactocse
Figure I: L distance between target and empirical sample distributions over the course of training on the standard (top row) and hard (bottom row) hypergrid environments for each method. Lower values indicate better performance.
We refer to these strategies as backward approaches. In this section, denote the distribution induced by a forward policy Pp over terminal states as (x(x) for x E X, representing the probability of sampling from our GelowNet.
4.1 HYPERGRID
We start experiments with synthetic hypergrid environments introduced by Bengio et al. (2021). These environments are sufficiently small to compute target distribution in the closed form, allowing us directly examine the convergence of (xxx) R(x)/Z.
The environment is d-dimensional hypercube with side length equal to H. The state space is represented as dddiimensionl vectors ( 8a) E (O..... . H 1] with the initial state being (O..... 0)". For each state (s1..... $8-1), there are at most d + actions. The first action always corresponds to an exit action that transfers the state to its terminal copy, and the rest of actions correspond to incrementing
coordinate by without leaving the grid. The number of terminal states here is | = H There are 2d regions with high rewards near the corners of the grid, while states outside have much lower rewards. The exact expression for the rewards is given in Appendix A.3.2.
We explore environments with the reward parameters taken from Malkin al. (2022), referred as "standard case", and with the reward parameters from Madan et al. (2023), referred to as a "hard case". In the second case, background rewards are lower, which makes mode exploration more challenging. We conduct experiments on a 4-dimensional hypercube with a side length of 20, As an evaluation metric, we use _ distance between the true reward distribution and the empirical distribution of the last 2- 10t terminal states sampled during training.
Figure presents the results. For SoftDON, MunchausenDON, and DB, TLM shows the fastest convergence for both "standard" and "hard" reward designs. For the SubTB algorithm, TLM shows similar performance to naive and outperforms uni rm, ma xent and pess mi ic. known to have difficulties in this environment (Madan et al., 2023), all approaches fail to converge under the "hard" reward design. At the same time, with the "standard" one, maxent backward shows the best convergence. An important note is that results reproduce the findings of Tiapkin et al. (2024): for SoftDON and MurchausseDDN training with un form backward converges faster than with naive ve algorithm, while TLM shows stable improvement over uni form. The results and the ranking of algorithms are almost the same for SoftDON and MunchausenDON, so we leave only MunchausenDgy out of two for further experiments.
7

---

Published as a conference paper at ICLR 2025
Bit Seguences
Munchoasen DQN
DR
TH
SubTR
Sound Modee
TIM
20
10000
20003 30000 40300 50000
10000 20363 35000
43030 50000
15050
20030
10200
40203 S0000
18000 20005
16350
(0303
53000
#lanttinn
Pitwestione
Munchausen
5300
TIM
2000
salve
manent
1200
pessiziatic
6/00
6:00
10000 etforation:
20000
1500 attoration:
ettorations
etterstoo:
SEN
Munchausen DQN
DB
TB
SubTB
00000
uniform
40000
T.M
20000
prociiattic
10000 15000
5000
15000
5000
L5009
5000
20000
Aiterations
Pitcrations
#iteration
#bbrrtions
Figure 2: Top row: Bit Sequences, the number of discovered modes out of total of 60 modes for different methods. Center row: QM9, the number of -aaaartto-epparated modes with reward higher or equal to 1.125 for different methods. Bottom row: sEH, the number of Taaiimto-ssppaattd modes with reward higher or equal to 0.875 for different methods. Higher values indicate better performance. For each pair of GFlowNet algorithm and a backward approach, the results are presented for the best learning rate chosen in terms of the total number of discovered modes.
4.2 BIT SEQUENCES
In this section, we consider the bit sequence generation task introduced by Malkin et al. (2022). Following the experimental setup of Tiapkin et al. (2024), we modify the state and action spaces to create a non-tree DAG structure, similar to the approach introduced in Zhang et al. (2022).
This task is to generate binary sequences of fixed length n. using vocabulary of k-bit blocks. The state space of this environment corresponds to sequences of n/k words, and each word in these sequences is either empty word @ one of 2 possible k-bit words. The initial state so corresponds to a sequence of empty words. The possible actions in each state are replacing an empty word @ with one of 2 non-empty words in the vocabulary. The set of terminal states X' consists of sequences without empty words and corresponds to binary strings of length n. The reward function is defined R(x)) exp( 2 min, d(xr,a x')), where M
of modes and d is Hamming distance. We fix = 120 and A = 8 for our experiments. The terminal state space size is |X| 2120 Importantly, for this environment, the uni mifform backward coincides with maxent, see Proposition I of Zhang et al. (2022) and Remark 3 of Theorem 3 of Mohammadpour et al. (2024).
To evaluate the performance, we use the same metrics in Malkin al. (2022) and Tiapkin et al. (2024): the number of modes found during training (number of sequences from M for which a terminal state within a distance of 30 has been sampled) and Spearman correlation on the test set between R(x:) and an estimate of Pg. Since computing the exact probability of sampling a terminal state intractable due to a large number of paths leading to it, we use Monte Carlo estimate following the approach of Zhang et al. (2022); also Appendix A.3. We train all models with various choices of the learning rate, treating as a hyperparameter, and provide the results depending on its value, similarly to Madan et al. (2023).
Figure shows the number of modes for different GelowNet algorithms and backward approaches found over the course of training. We observe that TLM shows significant improvement for DB and a minor one for MunchausenDor case all backward approaches find all 60 modes, TB and SubTB also find almost all modes, and
in comparison to other backward approaches, where in the latter
8

---

Published as a conference paper at ICLR 2025
Bit Sequences
Munchousen DON
DR
SubTA
zaltermmmassen
TLMM
rabre
peedrriet:
5-10
18
2-15-3 5-10-1
10 lourning rale
2-10-1
5.10-
13-3
2-10- 10
10-3 leaming sale
2-10-1
Jeamung rale
learsing
Poar Oun tlgin vaja)
Munchauses DQN
Subra
0.4
pendiiaati
5.10
10
5:10
5:10
10
5 10
5-10
16
5:19
5:10
10
5-30
leassing cale
loatting r01o
Joamin Fale
leeaning s0to
Den (bpk kugas)
Mancchausen DQN
DB
sEIl
TB
SubTB
0.6
uniffrm
6.4
naliee
0.2
maxent
pisoimistie
5-10
10-1
5 -0
10
5.10
10
5-10 10-3
5-10
10th
5.10 10-3
10
10t1
5-10-1
10
Jaaming rate
learning rabe
leaming rate
Ineming rate
Figure 3: Top row: Bit Sequences, Spearman correlation between R. and Po a test set for different methods and varying learning rate E 15-10-4, 100 2.
10- Center row: QM9, Pearson
correlation between log R and log Po the fixed subset of the QM9 dataset (Ramakrishnan et al., 2014) for different methods and varying lcarning rate E (5 10 10
4.5 5- 10 1,10-,, Bottom row: sEH, Pearson correlation between log R and log Po on the test set from Bengio et al. (2021) for different methods and varying learning rate E (5 10-6 1,10-1 5. 10-4, 10-3) 1 HHigher values indicate better performance. We note here that pessimii st backward policy can be very sensitive the choice of learning rate.
TLM does not affect the results much in comparison to uni form and naive, while outperforming pess immmsiiii c in case of SubTB. Full plots for modes across varying learning rates presented in Figure 5 in Appendix. Figure 3 (top) presents Spearman correlation between R and Po on the test set for the same GFlowNet algorithms and varying learning rates. TLM shows better similar performance to the baselines across all GFlowNet algorithms if the optimal learning rate is chosen. Moreover, for and SubTB, TLM shows steady improvement over the baselines for all learning rates.
4.3 MOLECULE DESIGN, SEH AND QM9
Our final experiments carried out on molecule design tasks of sEH (Bengio al., 2021) and QM9 (Jain et al., 2023). In both tasks, the goal is to generate molccular graphs, with reward emphasizing some desirable property. For both problems, we use pre-trained reward proxy neural networks. For the sEH task, the model is trained to predict the binding energy of molecule to a particular protein target (soluble epoxide hydrolase) (Bengio et al., 2021). For the QM9 task, the proxy is trained on the QM9 dataset (Ramakrishnan et al., 2014) to predict the HOMO-LUMO gap (Zhang et al., 2020).
For the sEH task, we follow the framework proposed by Jin al. (2020) and generate molecules using a predefined vocabulary of fragments. We use the same 72 fragments as in the seminal work of Bengio et al. (2021). essential to mention that these fragments are explicitly selected for the sEH task to simplify high-quality object generation. The states represented as trees of fragments. The actions correspond to choosing a new fragment and then choosing an atom to which the fragment will be attached. There is also special stop action that moves the state its terminal copy and stops the generation process.
For the QM9 task, molecules generated atom-by-ate by-tomm and bond-by-bond, Every state is connected graph, and actions either add a new node and edge or set an attribute on an edge. Thus, the graph-building environment is much more expressive than the fragment-based tree-building

---

Published as a conference paper at ICLR 2025
environment, but it results in more complex generation task and can lead to construction of invalid molecules. It is worth mentioning that there exists a simpler fragment-basee version of this environment in GFlowNet literature (Shen et al., 2023; Chen & Mauch, 2023), while consider the more complex atom-by-atom setup from Mohammadpour et al. (2024).
We use the same evaluation metrics for both tasks as proposed in previous works (Bengio et al., 2021; Madan et al., 2023; Tiapkin et al., 2024). We track the number of Tanimoto-separatee modes above a certain reward threshold captured over the course of training, and Pearson correlation on the test set between log Rxx) and log (2r). For sEH task we use the same test set as in Bengio et al. (2021), and for QM9 we use a subset of the dataset introduced in Ramakrishnan et al. (2014). We train all models with various choices of the learning rate, treating it as a hyperparameter, and provide the results depending on its value, similarly to Madan et al. (2023).
Figure 2 (center and bottom) shows the number of modes for different GelowNet algorithms and backward approaches found over the course of training. TLM speeds up mode discovery on QM9 when utilized alongside DB and DB and performs for Mun cuunchausenDQN and SubTE. but shows similar or worse performance when compared to other backward approaches on sEH. However, we note that on sEH no backward approach shows significant improvement over uni f Form in terms of mode discovery. Full plots for modes across varying learning rates are presented in Figure 6 in Appendix. It is worth noting that on the QM9 task, TLM shows robust improvement over the baselines in the majority of configurations. Figure 3 (center and bottom) shows Pearson correlation between log R and log Po estimate measured on the test set for various learning rates. TLM results in better correlations when paired with SubTB, shows comaprable results with the best baselines when paired with DB and T3, and falls behind pe wimimmisti when paired with Mur. indicates, pess imi stic backward policy can be very sensitive to the choice of the learning rate and the GFlowNet algorithm that is used for forward policy training.
par with the best baseline backward approaches
while outperforming other backward wark approaches. However, as Figure 3
4.4 DISCUSSION
From the plots above, one can see that across all GFlowNet algorithms (forward policy training objectives), TLM generally shows performance that is better comparable to other backward approaches. The sole exception is the number of discovered modes in the sEH environment, where TLM can fall behind other backward approaches. To explain this shorteoming, we hypothesize that TIM is more beneficial in environments with less structure. This is supported by the major improvements to mode exploration that it obtains on QM9, while sometimes even degrading the same metric on sEH. Indeed, molecules in the sEH task are constructed from the predefined set of blocks, while the QM9 task, they are created from atoms. This manually predefined set adds a lot of structure into the environment, in which one creates a junction tree from these blocks. In comparison, the process of creating an arbitrary graph of atoms imposes less structure and can even lead to construction of invalid molecules in certain cases. We suppose that the strong methodological bias is a possible reason why it is of little utility to consider non-trivial backward approaches in the sEH task and why the uni form backward approach often has the best or at least comparable performance according to Figure 6. This is exactly the hypothesis that was initially put forward by Mohammadpour et al. (2024), and our results align with it well.
5 CONCLUSION
In this work, we propose new method for backward policy optimization that enhances mode ex- ploration and accelerates convergence in complex GFlowNet environments. TLM represents the first principled method for learning a backward policy in soft reinforcement learning-based Gel algorithms, such as SoftDQN and MunchausenDgN We provide an extensive experimental evalu- ation, demonstrating benefits of TIM when it is paired with various forward policy training methods, and analyze its shortcomings, arguing that our results support the hypothesis of Mohammadpour et al. (2024) about benefits of backward policy optimization environments with less structure.
A promising further work direction is using backward policy for exploration as proposed in Kim et al. (2023) and He et al. (2024b). Indeed, the ability to sample trajectories starting from high- reward terminal states via PB provides an opportumity to improve mode exploration. We expect that combining such methods with TLM-like approaches will additionally improve their performance.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors are grateful Dmitry Vetrov for valuable discussions and feedback. The paper was prepared within the framework of the HSE University Basic Research Program. This research was supported in part through computational resources of HPC facilities at HSE University (Kostenetskiy et al., 2021). The work of Daniil Tiapkin was supported by the Paris fle-de-France Région in the framework of DIM AI4IDF.
REFERENCES
Lazar Atanackovic, Alexander Tong, Bo Wang, Leo Lee, Yoshua Bengio, and Jason S Hartford. Dyngfn: Towards bayesian inference gene regulatory networks with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27323 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio.
Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations research, 1275)1127-, 2015.
Yihang Chen and Lukas Mauch. Order-preserving gflownets. In The Twelfth International Conference on Learning Representations, 2023.
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete proba- bilistic inference as control in multi-path environments. In The 40th Conference on Uncertainty in Artificial Intelligence, 2024.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference Machine Learning, pp. 2160-2169, PMLR, 2019.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018
Haoran He, Emmanuel Bengio, Qingpeng Cai, and Ling Pan. Rectifying reinforcement learning for reward matching. arXiv preprint arXiv:2406. 022 13, 2024a.
Haoran He, Can Chang, Huazhe Xu, and Ling Pan. Looking backward: Retrospective backward
synthesis for goal-conditioned gflownets. arXiv preprint arXiv:2406.01 150, 2024b.
Alex Meenandez-Garrci. Alexandre Duval, Alexandra Volokhova, Yoshua Bengio, Divya Sharma, Pierre Luc Carrier, Michat Koziarski, and Victor Schmidt. Crystal-gfnn sampling crystals with desirable properties and constraints. In 37th Conference on Neural Information Processing Systems (NeurIPS (0023)-A144AA workshop. 2023.
Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In The Twelfth International Conference on Learning Representations, 2023.
Moksh Jain, Emmanuel Bengio, Alex Hermandez-Garde Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Moksh Jain, Sharath Chandra Raparthy, Alex Herándee- Garcia, Jarrid Rector- Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International Conference Machine Learning, pp. 14631-14653. PMLR, 2023.
Hyosoon Jang, Yunhui Jang, Minsu Kim, Jinkyoo Park, and Sungsoo Ahn. Pessimistic backward policy for gflownets. In Advances in Neural Information Processing Systems, volume 37, pp. 107087-107111 Curran Associates, Inc., 2024.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Artificial Intelligence in Drug Discovery, pp. 228-249. The Royal Society of Chemistry, 2020.
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. In The Twelfth International Conference on Learning Representations, 2023.
PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpe resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, pp. 012050. IOP Publishing, 2021.
Anas Krichel, Nikolay Malkin, Salem Lahlou, and Yoshua Bengio. On generalization for generative flow networks. arXiv preprint arXiv:2407.03105, 2024.
Elaine Lau, Stephen Lu, Ling Pan, Doina Precup, and Emmanuel Bengio. Qgfn: Controllable greediness with action values. Advances in Neural Information Processing Systems, 07:81645- 81676, 2025.
Ziru Liu, Shuchang Liu, Bin Yang, Zhenghai Xue, Qingpeng Cai, Xiangyu Zhao, Zijian Zhang. Lantao Hu, Han Li, and Peng Jiang. Modeling user retention through generative flow networks. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5497-5508, 2024.
Kanika Madan, Jarrid Rector-Brooke Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483, PMLR, 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35: 5955 5955-5967, 2022.
Nikolay Malkin, Salem Lahlou, Tristan Delcu, Xu Ji. Edward Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. Gflownets and variational inference. In The Eleventh International Conference Learning Representations, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 51807540):599-333 2015.
Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, and Pierre-Lue Bacon. Maximum entropy gflownets with soft g-learning In International Conference on Artificial Intelligence and Statistics, pp. 2593-2601. PMIR, 2024.
Nikita Morozov, Daniil Tiapkin, Sergey Samsonov, Alexey Naumov, and Dmitry Vetrov. Improving gflownets with monte carlo tree search. arXiv preprint arXiv:2406. 1114406..6655 13655, 2024.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of retroay-re markov decision processes. arXiv preprint arXiv: 2017.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In leml, volume 99, pp. 278-287, 1999.
12

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp. and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, (I)):140022, 2014.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
Tiago Silva, Eliezer de Souza da Silva, Rodrigo Barreto Alves, Luiz Max Carvalho, Amauri H Souza, Samuel Kaski, Vikas Garg, and Diego Mesquita. Analyzing gflownets: Stability, expressiveness, and assessment. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp- 387-395. Pmir, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Éric Moulines, Rémi Munos, Alexey Naumov, Pierre Perrault, Yunhao Tang, Micha Valko, and Pierre Ménard. Fast rates for maximum entropy exploration. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, 23, JMLR.org. 2023.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry P Vetrov. Generative flow networks as entropy-r rl. In International Conference on Artificial Intelligence and Statistics, pp. 4213-4221. PMLR, 2024.
Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding rein- forcement learning- based fine-tuning of diffusion models: A tutorial and review. arXiv preprint arXiv:2407.13734 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector-Browd Brooks, Yoshua Bengio, Glen Berseth, and Nikolay Malkin. Amortizing intractable inference in diffusion models for vision, language, and control. In Advances in Neural Information Processing Systems, volume 37, pp. 76080-76114 14. Curran Associates, Inc., 2024.
Nino Vieillard, Olivier Pietquín, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235-4246. 2020.
Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems, 34:25746-2551 25759, 2021.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. International Conference on Machine Learning, pp. 26412-26422 PMLR, 2022.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. In Advances in Neural Information Processing Systems, volume 36, pp. 11952-11969 2023.
13

---

Published as a conference paper at ICLR 2025
Dinghuai Zhang, Yizhe Zhang, Jiatao Gu, Ruixiang ZHANG, Joshua M. Susskind, Navdeep Jaitly, and Shuangfei Zhai. Improving GFlownets for text-to-image diffusion alignment. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-driven graph neural network with multiplex graph for molecular structures. arXiv preprini arXiv:2011.07457, 2020.
Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample-efficient multi-objective molecular optimization with gflownets. Advances Neural Information Processing Systems, 36, 2024.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pp. 928-936, 2003.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 OMITTED PROOFS
Proof of Theorem 3.1. From the stability of backward updates, the Cauchy criterion implies that there is Pg E Ilg such that pg + PB. At the same time, by the choice or rewards rf = r Proposition 1 by Tiapkin al. (2024) and joint convexity of KL-divergence
T T
T
KL P
Pi
N KL
1
1
n
T
T
1=1
Notice that a mapping PB J P PB is continuous, thus P. also converge to P Finally, applying Pinkser's inequality, we have
+ P PB
and, as a result, averages of P.
T
HELLP--
P, -
29
11 Σ
T
P
11.
The right-hand side of the inequality tends to T +00, thus $ ΣT Pf + P Finally, since for any PB there is Pp: such that P P we conclude the statement.
A.2 STABILITY TECHNIQUES
In this section, highlight important practical techniques and design choices motivated by Theo- rem 3.1 that we use alongside our TLM algorithm to enforce stability into the training of PB-
First, we found it beneficial to either over the course of training (see the next sections for detailed descriptions).
lower learning rate for the backward policy or decay
Second, akin to how the Deep Q-Network algorithm (Mnih et al., 2015) utilizes target network to estimate the value of the next state, we utilize target networks for the backward policy when calculating the loss for the forward policy. For example, (3) transforms into
Σ wikk log Fo(s,) Th Paisil 1.0) (SubTB(O;T) osjeken.
2
(10)
Po(sk) I1 Ps(se-i(st.!)
where the parameters 0 of PP 18, 0) are updated via exponential moving average (EMA) the online parameters 0 of PPs( 0). So the loss for the backward policy LruM is computed using an online backward policy PB( (). and the loss for the forward policy Lxig is computed using a target backward policy PB(st- 0). which is frozen during the gradient update of Pp.
Finally, we find it helpful to initialize Pn to the uniform distribution at the beginning of training, which is done by pero-initialization of the last linear layer weight and bias.
We ablate the impact of the proposed techniques on QM9, where try to turn off each of the three separately. Results are presented in Figure 4. We observe that using a target model and a lower learning rate is crucial, whereas disabling uniform initialization increases variance and shows slightly worse results. For this experiment, we choose DB as the base algorithm because TLM overall shows the most significant impact when applied with compared to TB, SubTB, and MunchausenDON.
A.3 EXPERIMENTAL DETAILS
We utilize PyTorch (Paszke al., 2019) in our experiments. For hypergrid and bit sequence environments, we base our implementation upon the published code of Tiapkin et al. (2024). For molecule design experiments, our implementations are based on the open source library by Recursion Pharmaceuticals. In all our experiments, PP and PB share the same neural network backbone. predicting logits via two separate linear heads. For all experiments, mean and std values are computed over three random seeds.
'https:// www..com/..c..uu..cccooeaamm wtt..///wmerrrrriiooaaaaaaaaaaaa onpharma /f
15

---

Published as a conference paper at ICLR 2025
DB, learning rate 5 10
2500
1.125
TLM
TLM, w/o lower backward Ir
2000
TLM, w/o backward target model
Found Modes
1500
TLM, w/o uniform backward init
1000
500
5000
10000
15000
20000
#iterations
Figure 4: Ablation study of stability techniques on QM9. The number of Tanimoto-separated modes with reward at least 1. 1.125 is shown. As base algorithm, we use DB with learning rate of 5 10
An important note is that we do one stochastic gradient update over the replay buffer for pess mi itti backward policy approach (Jang et al., 2024) each training iteration to allow for a fair comparison with other methods. Jang et al. (2024) perform 8 stochastic gradient updates for pess imi stic PB at each training iteration in the experiments in their paper, which results in discrepancy in the number of gradient steps when comparing the approach to other methods.
A.3.1 HYPERGRID
The reward at a terminal state s with coordinates (s' "s.....,''"
s)) is defined as
s²
D
R(s) Ro R1
II
0.25
0.5
+ R2
II
0.3
0.5
0.4
H
H
Standard reward uses parameters (Ro 10 10-3 Rı 0.5, R2 2.0) and hard reward uses (Ro 10 R = 1.0, Rg = 3.0), taken from Bengio et al. (2021) and Madan et al. (2023) respectively.
All models are parameterized using MLP with 2 hidden layers and 256 hidden units. We the Adam optimizer with a learning rate of 10-3 and a batch size of 16 trajectories. For SubTE, we set use prioritized replay buffer (Schaul et al., 2016) and adopt the same hyperparameters as in Tiapkin et al. (2024). We sample 256 transitions from the buffer compute the loss for these methods.
= 0.9, following Madan et al. (2023). For So DQN and MunchausenDDgn
For the TLM backward policy, we use the same initial learning rate of 10-3 with an exponential scheduler, tuning y from {0.999, 0.9999]. The backward policy target network for TLM uses soft up- dates with a parameter = 0.25 (Silver et al., 2014). Since the environment is small, we precompute log n(s) for states, allowing us to obtain the maxent backward exactly. For pessint st backward policy we store trajectories from the last 20 training iterations in the replay buffer and sample 16 trajectories to compute the loss.
Hypergrid experiments were performed on CPUs. Table summarizes the chosen hyperparameters.
A.3.2 BIT SEQUENCES
The set of modes M is defined as in Malkin et al. (2022), and we choose the same size, |M| = 60. We set H = ('00000000', 11111111', 1111110110000 0000101111 (00111000', Each sequence in M is generated by randomly selecting n/88 elements from H with replacement and then concatenating them. The test set for evaluating reward correlations is generated by taking mode and flipping random bits in it, where this repeated for every mode and for each 0 < < n.
We utilize the same Monte Carlo estimate for Po presented in Tiapkin et al. (2024) with N 10:
PF(T 0) PB(T
N PF(T" 0) x)
Po(x) = Epa(r m)
x)
~
N
PB(T
+' Pe(T 1x).
16

---

Published as a conference paper at ICLR 2025
Table 1: Hyperparameter choices for hypergrid experiments.
Hyperparatemeter
Value
Training trajectories
10
Learning rate
10-3
Z learning rate for TB
10-1
Adam optimizer 3, e, X Number of hidden layers
(0.9, 0.999), 10 8.0
Hidden embedding size
256
SubTB À
0.9
MunchaussenDQN a
0.15
MunchausenDON lo
-100
Prioritized RB size
10
Prioritized RB /
0.5, 0.0
Prioritized RB batch size
256
pess i www.miit...... itti buffer size
20 x 16
TLM backward learning rate
10-3
TLM backward Ir decay 7
(0.999, 0.9999)
TLM backward target model T
0.25
Notice that any valid Pn can be used here, but for each model, we take the PB that was fixed/trained alongside the corresponding Pp since such a choice will lead lower estimate variance. However, note that the metric is still very noisy, for each training run, we compute the metric for all model checkpoints (done every 2000 iterations) and use the maximum value.
All models are parameterized as Transformers Vaswani al. (2017) with 3 hidden layers, 8 attention heads, and hidden dimension of 64. Each model is trained for 50,000 iterations and a batch size of 16 with Adam optimizer. We provide results for learning rates from (5 x 10-4 10-3 For SubTB we use À = 0.9. For MunchausenDQN we use a prioritized replay buffer (Schaul et al., 2016) and take the same hyperparameters as in Tiapkin al. (2024). To compute the loss, we sample 256 transitions from the buffer for MunchausenDON.
10-3).
For the TLM backward policy, we use the same initial learning rate as for the forward policy and utilize an exponential scheduler, where y is tuned from {0.9997, 0.9999}. The backward policy target 0.1 (Silver al., 2014). For pes iimi
network for TLM uses soft updates with a parameter backward policy we store trajectories from the last 20 training iterations in the replay buffer and sample 16 trajectories to compute the loss.
To closely follow the setting of previous works (Malkin et al., 2022; Madan et al., 2023; Tiapkin al., 2024), we use e-uniform exploration with & 10-3 We note that this can introduce a small bias into the gradient estimate of V VoLuLn(Of, since will not be sampled exactly from Pr.
Each bit sequence experiment was performed on single NVIDIA V100 GPU. Table summarizes the chosen hyperparameters.
A.3.3 MOLECULES
For sEH, we use the test set from Bengio et al. (2021). For QM9, we select subset of 773 molecules from the QM9 dataset (Ramakrishnan et al., 2014) containing between and atoms. The subset is constructed to ensure approximately equal representation of different molecule sizes. To compute correlation, we use the same Monte Carlo estimate as the bit sequence task. important to note that Mohammadpour et al. (2024) used different evaluation approach, computing correlation sampled molecules rather than on a fixed dataset.
In the sEH task, rewards are divided by a constant of 8, and the reward exponent is set to 10. For QM9, we subtract the 95th percentile from all rewards, resulting in the majority of rewards being distributed between 0 and 1, with 5% of molecules having reward greater than The reward exponent is also set to 10. We set R(x) exp(-75.0) for invalid molecules x in QM9. We track the number of Tanaaato-separated modes described in Bengio al. (2023), using Tanimoto
17

---

Published as a conference paper at ICLR 2025
Table 2: Hyperparameter choices for bit sequence experiments.
Hyperparatemeter
Value
Training iterations
x 10*
Learning rate
{5 x
10
1,003 10
10-3}
learning rate for TB
10-1
Adam optimizer B. E-uniform exploration
(0.9, 0.999), 10 1
10
Number of transformer layers
3
Hidden embedding size
64
Dropout
0.1
SubTE À
0.9
MunchausenDQN a
0.15
MunchausenDQN
-100
MunchausenDQN leaf coefficient
5
Prioritized RB size
10
Prioritized RB a, 8
0.9, 0.1
Prioritized RB batch size
256
pess stis buffer size
20 x
TLM backward learning rate
learning rate
TLM backward Ir decay y
0.9999
TLM backward target model T
0.1
similarity threshold of 0.7. After normalization, the reward thresholds are 0.875 and 1.125 for sEH and QM9, respectively.
We use the graph transformer architecture from Jain et al. (2023) with layers and 256 embedding dimensions for both tasks. Each model is trained for 20,000 iterations using the Adam optimizer. We present results for learning rates in the set {5 x 10-4 10-4, 5 x 10-4, 10-$). The learning rate for is fixed 10-3 for all experiments. We apply exponential schedulers to all learning rates each training step and refer readers to the hyperparameter table for the exact decay values. The batch sizes are 256 for sEH and 128 for QM9, For SubTB, set 1.0, following Madan al. (2023), For MaurchausenDOo we do not use a replay buffer, training the model on-policy while using the same other hyperparameters in Tiapkin et al. (2024).
For TLM, the learning rate for PB is initially set to be 10 times smaller than that for Pr, while other approaches use the same learning rate for PB as for Pp. The backward policy target network for TLM uses soft updates with parameter T 0.05 (Silver et al., 2014). To learn log n(s) )) for the maxent backward policy, we follow the approach from Mohammadpour et al. (2024), implemented in the Recursion Pharmaceuticals GFlowNet library. For pes simi stic backward policy store trajectories from the last 20 training iterations the replay buffer and sample 256 trajectories to compute the loss for sEH and 128 for QM9 (the same batch size for these environments).
In line with previous works (Malkin et al., 2022; Madan et al., 2023; Tiapkin et al., 2024), we use E-uniform exploration with 0.05. To account for the bias introduced into the gradient estimate of VaLTun (6:77 T). we linearly anneal to zero over the course of training. This annealing approach is applied to all GelowNet algorithms and backward methods ensure fair comparison terms of the number of discovered modes.
Each molecule generation experiment conducted single NVIDIA A100 GPU. Table summarizes the chosen hyperparameters.
18

---

Published as a conference paper at ICLR 2025
Table 3: Hyperparameter choices for molecule experiments.
Hyperparatemeter
sEH
QM9
Reward exponent
10
Batch size
256
128
Training iterations
10a
Learning rate
{5 10
1004,,4 5, X 10
10
10-33
Learning rate decay
2 2-1/20000
learning rate for TB
10-3
learning rate for TB decay Adam optimizer 8, À s-uniform exploration
1/500000
(0.9, 0.999), 10 10
0.05 linearly annealed to 0.0
Number of transformer layers
8 256
Hidden embedding size
SubTB
1.0
MunchausenDDg
0.15
MunchausenDQN 4
500
MunchausenDQN leaf coefficient
10
pessimistic buffer size
20 x 256
20 x 128
TLM backward learning rate
0.1x X learning rate
TLM backward Ir decay ?
2 1/5000
1/20000
TLM backward target model
0.05
19

---

Published as a conference paper at ICLR 2025
A.4
FULL PLOTS
Bit Sequences
Jearning rate 10-4
learning rate 10-3
learning rate 2-10-3
60
Munchassen DON Modes Found
40
unffrm/mamnt
20
T1M
naive possimistie
10000 20000 30000 40000 50000 #iterations
10000 20000 30000 40000 50000 10000 20000 30000 40000 50000
diterations
#iterations
Modes Found
DB
20
10000
20000 30000 40000 50000 10000 20000 30000 40000 50000 10000 20000 30000 40000 50000
#iterations
#iterations
#:terations
60
Modes Found
40
T2
20
10000
20000 30000 40000 50000 10000 20000 30000 40000 50000 10000 20000 30000 40000 50000
#iterntions
#iberations
#tterations
Modes Found
40
Sab
10000 20000 30000 40000 #iterations
50000
10000 20000 30000 40000 50000
10000 20000 30000 40000 50000 #'terations
#lberations
Figure 5: Bit Sequences, the number of modes discovered over the course of training for different methods and a learning rate E (5- 10 1,103,3 3,2 10 10-3). Some results for the learning rates of 10-3 and 10-3 are not full because of exploding gradients at certain points training.
20

---

Published as a conference paper at ICLR 2025
leaming
10
Haming 10"4
leamiin
leamiing
Pouad Modaee 12
2003
Mordhanm DON
TLM
10020
20000
5003
10300
15003
2000
10000
15000
20000
5502
13300
4.125
2203
Pount Mesio
1,002
6910
10030
10030
20000
5003
10000
15000
20003
1000
1,000
15000
20000
5002
10200
10000
20000
1.125
Foun Modio
1500
503
5000
1.0000
15020
20000
5003
10000
15003
20003
5000
15000
13000
20000
5200
10200
:3200
20200
1.125
2003
Salt
Foond Medda
1.000
5050
10030
15030
2030
5003
10003
1500
2003
5000
15000
15000
20000
5303
10300
15300
28200
sEH
learning rate --003
learning rate 10-4
learning rate 5-10-1
learning 10-
0.815
60000
Fous No0ss
40000
Salse
monent
20000
possimistic
5000
10000
15000
20000
5000
10000
15000 20000
500n 100000
15000
20000
5000
10000 #iteratonn
15000
20000
#xtcrotions
etterotions
etteritionss
0.870
10000
Found M00ES
30000
5000
10000
15000
20000
5000
10000
15000 20000
5000 10000
15000
20000
5000 10000
15000
20000
#Rerotions
Ptterations
etteritions
#iterations
8,97
00000
Found Soute
5000
10000
15000 20000
5000
10000
15000 20000
5000 100000
15000 20000
5000
10000 15000
20000
#Xterotions
#tterations
otterationn
#iteratSons
9,855
Found boote
5000
10000
15000
20000
5000
10000
15000 20000
5000
10000
15000 20000
5000 10000
15000 20000
#
#tterations
etterations
#iteratsons
Figure 6: Top row: QM9, the number of Tanimoto-scparatee modes discovered over the course of training with reward higher equal 1.125 for different methods and learning rate in (5- 10-5, 10 10-4,,--004 50 1,103)) Bottom row: sEH, the number of Banimoto-separater modes discovered over the course of training with reward higher or equal 0.875 for different methods and learning rate E (5- 10 15-10-b 10 10-4 10-4 10-3).
21


Reviews:
[{"review_id": "L0AcTwnz3M", "reviewer": "Reviewer_q9aZ", "summary": "This paper proposes a novel method to optimize the backward policy of the GFlowNet during training. The proposed method can be combined with any GFlowNet training method, and the experimental results demonstrate that it can bring performance improvement in some tasks.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written. The presentation and structure of this paper are clear.\n- The proposed method, which employs an EM-style mechanism to optimize the training objective, is interesting. Theoretical analysis guarantees its convergence, and it can be used in any GFlowNet algorithm.", "weaknesses": "- The performance improvement is marginal. On the sEH task, the proposed method even harms the performance, as explained in line 485. Moreover, the experiments (e.g., Qm9 and sEH) included in this paper are somewhat toy. I encourage the authors to consider more complex and large-scale benchmarks, such as AMP and RMA generation [1].\n\n\n[1] Biological Sequence Design with GFlowNets, ICML 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60RSOQPval", "reviewer": "Reviewer_6Sem", "summary": "This paper introduces a method called Trajectory Likelihood Maximization (TLM) for optimizing backward policies in Generative Flow Networks (GFlowNets). The authors formulate the GFlowNet training problem as a unified objective involving both forward and backward policies and propose an alternating minimization procedure. The method is validated through extensive experimental evaluation across various benchmarks.", "questions": "1. Could the authors provide a direct experimental comparison with the pessimistic backward policy approach proposed by Jang et al. (2024)? This would help establish the relative merits of TLM in similar settings.\n2. Can the authors provide an explanation for the performance of TLM with different choices of the GFlowNet training objective (TB, DB, SubTB)? Are there certain objectives where TLM shows particularly strong or weak performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed TLM method is designed to be easily implementable and compatible with existing GFlowNet training algorithms.\n- The authors validate their approach across four different tasks, demonstrating its effectiveness in various scenarios.\n- The paper is well-structured and clearly written, with careful explanations of technical concepts and methodological choices.", "weaknesses": "- Limited comparative analysis: While the paper acknowledges recent work by Jang et al. (2024) on pessimistic backward policies, it doesn't provide a direct experimental comparison with this highly relevant approach. Such a comparison would help readers understand the relative advantages of TLM. Though the experiments are extensive, it would be valuable to see more detailed analysis of how TLM performs compared to other backward policy optimization methods across different types of environments.\n- Ablation studies: The paper could benefit from more detailed ablation studies to understand the contribution of different components of the proposed method.\n- Results presentation: It will be better to combine the last three columns in Figure 1 to a single figure as they are just different variants of GFlowNets training objectives. Including the best-performing loss will better present the results (e.g., row 2).\n\nHyosoon Jang, Yunhui Jang, Minsu Kim, Jinkyoo Park, and Sungsoo Ahn. Pessimistic backward policy for gflownets. arXiv preprint arXiv:2405.16012, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekXoGWIX3H", "reviewer": "Reviewer_A6Yj", "summary": "This paper studies the connection between GFlowNet training and entropy-regularized reinforcement learning problems–also known as soft RL. It specifically tackles this relationship in the setting of the fixed backward policy which is considered a significant limitation in the literature. For this indeed, they propose a simple backward policy optimization algorithm; Trajectory Likelihood Maximization (TLM)  that primarily involves a direct maximization of the value function in an entropy-regularized Markov Decision Process (MDP) over intermediate rewards. They formulate the GFlowNet training problem as a unified objective involving both forward and backward policies and propose an alternating minimization procedure. Their main contribution is deriving the first unified approach for adaptive backward policy optimization in soft RL-based GFlowNet methods, i.e. the TLM. They carry out an experimental evaluation on Hypergrid and Bit Sequence as well as two molecule design environments: sEH and QM9 to conclude that their algorithm converges faster and that can be integrated with any existing GFlowNet training algorithm. Lastly, there are empirical experiments done to show the performance of TLM under different hyperparameter choices.\nIn general, this paper could be a good contribution, with the caveat of some clarifications on the theory and experiments. Given these clarifications in an author's response, I would be willing to increase the score.", "questions": "For the related work, I still cannot understand what connections you are trying to make and which research you are trying to use and build upon. More specifically, what limitations are you trying to overcome? What are the similarities between your algorithm and the celebrated EM-algorithm (Dempster et al., 1977) and Hinton’s Wake-Sleep algorithm (Hinton et al., 1995) and how does TLM differ from their algorithm?\nThe experimental work does not seem well justified, but there is no discussion of the varying performances across the environments; what conclusions are you deriving? Do you have any explanations for the deficiencies your algorithm has which were shown in some parts of the empirical results? Do you have any suggestions in order to overcome those shortages? \nThe experiments do not provide convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches. There are so many missing details it is difficult to draw many conclusions:\n- Why did you use an online backward policy to compute the loss for the backward policy L_TLM?\n- Why did you use a target backward policy to compute the loss for the forward policy L_Alg?\n- In Hypergrid, all models are parameterized by MLP with 2 hidden layers, why? There seems no study of the decaying learning rate conducted on the Hypergrid environment, any comments?\n- Why did you choose transformers to parameterize your models in the Bit Sequences environment?\n- There is no clear rule for choosing your hyperparameters; it appears that you are simply taking the values established in previous related work. Why?\nThe differences in the “full plots” compared to the original ones are very small when choosing decaying learning rates. Can you provide some context to understand the significance of this difference?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors study a very important problem in the GFlowNets and entropy-regularized RL literature which is backward policy optimization. They propose the TLM method that represents the first unified approach for adaptive backward policy optimization in soft RL-based GFlowNet methods which enhances mode exploration and accelerates convergence in complex GFlowNet environments. They formulate the GFlowNet training problem as a unified objective involving both forward and backward policies, proposing an alternating minimization procedure. For novelty, they approximate these two steps through a single stochastic gradient update and derive an adaptive approach for combining backward policy optimization with any GFlowNet method, including soft RL methods. The significance of their work arises from working on the limitations of two main previous research; Mohammadpour et al. (2024) and Jang et al. (2024). Furthermore, their approach can be connected to cooperative game theory by interpreting the forward policy PF as a forward player and the backward policy PB as a backward player, with both players attempting to minimize KL-divergence between corresponding trajectory distributions. For theory, they introduce a new theorem that shows that stability is essential for theoretical convergence. The quality of the proposed method is shown by their numerical experiments. TLM greatly speeds up mode discovery on the QM9 environment for all GFlowNet algorithms. Their results support the findings of Mohammadpour et al. (2024), where it was also observed that the backward policy optimization enhances GFlowNets on more complex and less structured environments like QM9. As a last note, the math behind their algorithm is neat and well-explained.\n\nThe main strengths are as follows:\n- Deriving the trajectory likelihood maximization (TLM) method for backward policy optimization.\n- The proposed method represents the first unified approach for adaptive backward policy optimization in soft RL-based GFlowNet methods. The method can be integrated with any existing GFlowNet training algorithm.\n- Providing an extensive experimental evaluation of TLM in four tasks, confirming the findings of\nMohammadpour et al. (2024), emphasized the benefits of training the backward policy in complex environments with less structure.\n- Providing the basis for implementing and extending their approach in different domains such as cooperative game theory.", "weaknesses": "The paper does not do a great job of demonstrating the deficiencies of the proposed algorithm. No section in this paper clearly explains and shows the difference between the proposed algorithm and other related work. In other words, the reader may not understand what specifically the work you are trying to build your work on. The authors point to references to establish closely related work such as Mohammadpour et al. (2024) and Jang et al. (2024), but do not clearly and plainly show what exactly they are improving, and what previous limitations they are trying to overcome. Furthermore, the authors mention that there are some similarities between their algorithm and a celebrated EM-algorithm (Dempster et al., 1977) and Hinton’s Wake-Sleep algorithm (Hinton et al., 1995), but did not mention what aspects those similarities are.\nThe experiments do not seem well justified, in the Hypergrid environment, Figure 1, the authors do not clearly explain why their method TLM fails to outperform the naive method in the TB training. In the Bit Sequences environment, the authors again do not show a sufficient explanation for the different results shown in Figure 2-Top row such as the improvement in DB and why TLM does not affect the results much in TB and SubTB. In addition, there seems to be no clear reason for producing results using two different metrics: Found Modes, and the Spearman Correlation between R and P_theta in three different environments, Bit sequences, sEH, and QM9. No explanation for Figure 2 (center and bottom) where TLM greatly speeds up mode discovery on QM9 for all GFlowNet algorithms, but shows similar or worse performance when compared to other backward approaches on sEH. There is no justified explanation of the poor performance of TLM on sEH in terms of mode discovery in all four Gflownets training algorithms.\nLastly, in Figure 3, their conclusion “TLM results in better correlations when paired with MunchausenDQN and SubTB, and shows similar results to the baselines when paired with DB and TB.” is not well justified. In what aspects does that affect your method? It would be much better to explain their method’s performance alongside its deficiencies in a more active discussion section. \n\n\nSuggestions:\n- Specify related work, similarities, and differences in a separate section.\n- Explain your empirical results, i.e. your algorithm’s deficiencies and mention suggestions to enhance its performance.\n\nMinor comments:\n1. Appendix. A.1 Omitted Proofs P. 14 Line 2, “by the choice of rewards”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6swwfEZLez", "reviewer": "Reviewer_EZ4M", "summary": "This paper introduces a novel method called Trajectory Likelihood Maximization (TLM) for optimizing the backward policy in Generative Flow Networks (GFlowNets). GFlowNets are a class of generative models, and the TLM method accelerates convergence in complex tasks by directly maximizing the likelihood of trajectories. Additionally, it significantly enhances exploration capabilities in high-dimensional tasks.", "questions": "1. The paper highlights the advantages of the TLM method in complex environments, but its performance is suboptimal in simpler environments, such as the sEH task. Can the reasons behind this underperformance in simpler tasks be explored in more depth? Is it due to limitations of the model or characteristics inherent to the environment?\n\n2. The paper notes that TLM struggles to handle non-stationary rewards effectively. Could a variant of TLM be designed specifically for dynamic reward structures to address this issue? How would such an adaptation impact TLM's performance in non-stationary reward environments?\n\n3. Maximum entropy policies are beneficial for enhancing exploration. Could the TLM method be combined with maximum entropy policies to improve exploration capabilities in certain tasks? If so, how might these two approaches be effectively integrated for optimal collaboration?\n\n4. Could the TLM method be integrated with other types of reinforcement learning algorithms, such as hard-constrained RL, to address different task requirements? Would such a combination further improve backward policy optimization and overall performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The TLM method shares a strong theoretical connection with existing reinforcement learning approaches, such as entropy-regularized RL. It introduces a novel joint optimization framework that enhances the efficiency of learning the backward policy. These theoretical connections are validated through experimental results presented in the paper.\n\n2. The method is grounded in the maximum likelihood estimation problem from reinforcement learning and proposes a unified approach for optimizing the backward policy. This provides a robust theoretical basis for addressing the longstanding challenge of backward policy optimization in GFlowNets.\n\n3. The paper is written in a clear and accessible manner, particularly in its detailed explanations of complex theoretical concepts, making it easier for readers to comprehend.", "weaknesses": "1. The paper primarily focuses on the introduction and validation of the TLM method, but provides limited discussion on comparisons with other backward policy methods, such as those proposed by Jang et al. and Mohammadpour et al. Notably, the improvement in backward policy optimization is not significant in more structured tasks, such as the sEH task.\n\n2. While the paper demonstrates the advantages of the TLM method in complex environments, it does not sufficiently explore scenarios where TLM might underperform, particularly in simpler or highly structured tasks.\n\n3. Some of the experimental environments, such as the sEH task, are relatively simple. Although the TLM method performs well in these tasks, validation in more complex or realistic environments would better illustrate the practical value of the approach.\n\n4. Although the paper emphasizes the TLM method, exploring combinations with other backward policy approaches, such as maximum entropy policies, could offer more innovative directions for future research.\n\n5. The paper could further discuss the impact of dynamic reward structures on the TLM method, particularly in handling complex, non-stationary reward distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)