Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Authors: Haoran He
Emmanuel Bengio
Qingpeng Cai
Ling Pan
Venue: ICML 2025 poster (2025)
Keywords: Generative Flow Network (GFlowNets)
Policy Evaluation
Primary area: probabilistic_methods
TL;DR: We uncover a new connection between GFlowNets and reinforcement learning which is previously overlooked by investigating their equivalence through uniform policy evaluation.

Abstract: The Generative Flow Network (GFlowNet) is a probabilistic framework in which an agent learns a stochastic policy and flow functions to sample objects with probability proportional to an unnormalized reward function. GFlowNets share a strong connection with reinforcement learning (RL) that typically aims to maximize reward. A number of recent works explored connections between GFlowNets and maximum entropy (MaxEnt) RL, which incorporates entropy regularization into the standard RL objective. However, the relationship between GFlowNets and standard RL remains largely unexplored, despite the inherent similarities in their sequential decision-making nature.
While GFlowNets can discover diverse solutions through specialized flow-matching objectives, connecting them to standard RL can simplify their implementation through well-established RL principles and also improve RL's capabilities in diverse solution discovery (a critical requirement in many real-world applications), and bridging this gap can further unlock the potential of both fields. In this paper, we bridge this gap by revealing a fundamental connection between GFlowNets and one of the most basic components of RL -- policy evaluation. Surprisingly, we find that the value function obtained from evaluating a uniform policy is closely associated with the flow functions in GFlowNets. Building upon these insights, we introduce a rectified random policy evaluation (RPE) algorithm, which achieves the same reward-matching effect as GFlowNets based on simply evaluating a fixed random policy, offering a new perspective. 
Empirical results across extensive benchmarks demonstrate that RPE achieves competitive results compared to previous approaches, shedding light on the previously overlooked connection between (non-MaxEnt) RL and GFlowNets.

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Haoran He' Emmanuel Bengio? Qingpeng Cai
Ling Pan
Abstract
1. Introduction
The Generative Flow Network (GFlowNet) is a probabilistic framework in which an agent learns stochastic policy and flow functions to sam- ple objects proportionally to an unnormalized reward function. A number of recent works explored connections between GFlowNets and maximum entropy (MaxEnt) RL, which modi- fies the standard objective of RL agents by learn- objective. However, the relationship between GFlowNets and standard RL remains largely unexplored, despite the in- herent similarities in their sequential decision- making nature. While GFlowNets can discover di- verse solutions through specialized now-matching objectives, connecting them can simplify their implementation through established RL princi- ples and improve RL's diverse solution discov- cry capabilities. In this paper, we bridge this gap by revealing a fundamental connection be- tween GFlowNets and one RL's most basic com- ponents - policy evaluation. Surprisingly, we find that the value function obtained from eval- uating uniform policy is closely associated with the flow functions in GFlowNets through the lens of flow iteration under certain struc- tural conditions. Building upon these insights, we introduce a rectified random policy evalua- tion (RPE) algorithm, which achieves the same reward-matching effect as GelowNets based on simply evaluating fixed random policy in these cases, offering a new perspective. Empirical re- sults across extensive benchmarks demonstrate that RPE achieves competitive results compared to previous approaches, shedding light on the previously overlooked connection between (non- MaxEnt) RL and GFlowNets.
Generative Flow Networks (GFlowNets) (Bengio et al., 2023) have emerged as a powerful probabilistic frame- work for object generation and sampling from complex distributions, which can be seen variant of amortized variational inference methods (Ganguly et al., 2022). In GFlowNets, an agent learns stochastic policy π(2) and flow functions to sample objects x E X' proportionally to an unnormalized reward function R(x). GFlowNets are related to Markov chain Monte Carlo (MCMC) methods (Metropo- lis et al., 1953; Hastings, 1970; Andrieu et al., 2003), but do not rely Markov chains that make small and local steps, which often leads to inefficient sampling in high- dimensional discrete spaces due to their local exploration nature. Therefore, they can generalize and amortize the cost of sampling without suffering from the mixing prob- lem (Salakhutdinov, 2009; Bengio et al., 2013; 2021).
ing an
sequen- tial eeciisionmmakin process: objects x E X are constructed incrementally through a sequence of steps, where at each step the GFlowNets agent adds an element to the current construction. The sequential nature of GFlowNets is closely related to the decision makiing processes in reinforcement learning (RL) (Sutton & Barto, 1998), whose training ob- jectives (Bengio al., 2021) were also motivated by the temporal difference methods. However, GFlowNets pur- sue a different goal: instead of maximizing rewards in standard RL, they aim to match the underlying reward dis- tribution (x(z) xx R(x)) (Bengio et al., 2021). This enables GFlowNets to discover diverse, high-reward candidates by sampling them proportionally to their rewards (reward- matching), proving particularly valuable in scenarios with uncertain imperfect rewards, such as drug discovery (Jain et al., 2023a). This capability has led to significant ad- vances in various challenging domains, including molecule generation (Bengio et al., 2021), biological sequence de- sign (Jain et al., 2022; Chen & Mauch, 2023), Bayesian structure learning (Deleu et al., 2022), and combinatorial optimization (Zhang et al., 2024: 2023b).
GFlowNets reformulate the sampling problem
'Hong Kong University of Science and Technology Valence Labs "Kaaishou Technology. Correspondence to: Ling Pan <ling- pan@ust..c uan@ust.hk>.
Recent works have explored the connections between GFlowNets and maximum entropy (MaxEnt) RL. (Tiap- kin et al., 2024; Deleu et al., 2024; Mohammadpour et al., 2024), a variant of RL that modifies the standard objective
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
by incorporating an entropy regularization term (Haarnoja al., 2017b). These works reveal that GFlowNets' reward- matching behavior can emerge from entropy-regularized objectives, providing valuable insights into the relationship between the two frameworks. However, the connection between GFlowNets and standard (non-MaxEnt) RL re- mains largely unexplored and poorly understood, despite the fact that both are rooted in sequential decision-makinn Understanding this relationship can unlock new possibil- ities and further advance both fields by combining their strengths: enabling GetowNets to leverage well-established RL techniques for improved sampling efficiency and sta- bility (Lau et al., 2024), while providing new perspectives on exploration and diversity in RL through GFlowNets reward- matchhig behavior (Hu et al., 2024).
objective of GetowNets is to learn a stochastic policy a that constructs discrete objects x E X' with probability propor- tional to the reward function: R. i.e. T(x) x R(x). The agent generates objects through sequential process, and adds a new element to the current state at each timestep
t. The sequence of states transitions from the initial state to a terminal state is referred to as a trajectory, denoted by T (sg + + s,,), where T E T belongs to the set of all possible trajectories TT. Bengio al. (2021) intro- duce the definition of the trajectory flow, represented by the function :: T R2o, which assigns non-negative real value to each trajectory. The state flow, denoted by F(s), is defined as the sum of flows of all trajectories passing through state 8, i.e., F(s) = Erg, F(r). The edge flow F(s + s') the sum of flows of all trajectories contain- ing the transition from state s to state s', which is defined as F(s + 5') Er F(T). We can then define the forward policy Pr(8'1s)= = F(s - termines the transition probabilities from a state s to its possible children states s'. In addition, we define the back- ward policy PB[(sss) = F(s + s))/F(s), which specifies the likelihood of reaching the parent state from the current state s'. A flow is considered consistent if the total incom- ing flow for a state matches the total outgoing flow for all internal states 8, i.e.,
In this paper, we uncover new connection that bridges this gap through one of the most basic components of RL: policy evaluation. To establish this connection, we first refor- mulate GelowNets training from a dynamic programming perspective through flow iteration, which paves the way for understanding their relationship. While previous works have primarily focused on modifications to RL objectives and introducing additional entropy regularization terms, we show that policy evaluation, which is often viewed as fun- damental building block for estimating the expected value of a given policy, can be naturally connected to GelowNets. Specifically, we discover that the resulting value function obtained from evaluating a uniform random policy under reward transformation is closely associated with the flow functions in GelowNets under specific structural conditions. Our findings reveal an unexpected connection and bridge the gap between these two frameworks, offering more comprehensive understanding of their underlying connec- tions than previously recognized. Building upon this insight, we introduce a rectified random policy evaluation (RPE) algorithm based on simply evaluating fixed random pol- icy, providing a straightforward implementation path for GelowNets in these applicable settings while maintaining the same reward-macching capability.
which de-
Σ Fss = Σ F(s
(I)
It is proven in (Bengio et al., 2021; 2023) that for consis- tent flows, the policy can sample objects with probability proportional to R(x) and therefore match the underlying reward distribution.
Flow Matching, Flow matching (FM) (Bengio et al., 2021) parameterizes the edge flow function by Fe(s, with A denoting the learnable parameters, and aims to optimize Fo(s, 8') for satisfying the flow consistency constraint. The FM loss is defined as Lim((s for non-
(log Err Fo(s", s) - log terminal states, which is the squared difference between the sum of incoming flows and the sum of outgoing flows (optimized in the log-scale due to stability issues). The term En Fa(s, 8') is replaced by R(s) if s is a terminal state.
F6 (s,, Fo,,
To validate our findings, we conduct extensive experiments and compare RPE with GelowNets (Bengio et al., 2021; Malkin et al., 2022; Bengio et al., 2023; Madan et al., 2023) and MaxEnt RL (Haarnoja et al., 2017a; Vieillard et al., 2020). Our results demonstrate that RPE achieves competi- tive performance compared to previous approaches in such domains, highlighting the effectiveness of our proposed method, and also shed light on the previously overlooked yet fundamental connection between RL and GetowNets.
Detailed Balance. Detailed balance (DB) parameter- izes a state flow model Fo. a forward policy model Pr.- and backward policy model PB, (Bengio et al., 2023), which aims to minimize the loss defined as Lps(s, s')
(log (F) (s'|s)) log(Fe (s,) (s) consider- ing the flow consistency constraint at the edge level, and also guarantees correct sampling from the target distribution.
2. Background
(Sub) Trajectory Balance. Malkin et al. (2022) pro- pose a trajectory-level optimization which is analogous to the Monte Carlo approach (Hastings, 1970) in RL, defined as LTT(T) = (log Zo ITi= PF, (8441
2.1. Generative Flow Networks (GFlowNets)
Consider directed acyelic graph (DAG) g = (8, where S and A represent the state and action spaces. The
2

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
log R(x)) IT220 PB. (s1))) that involves training the total flow Z, the forward and backward policies. To miti- gate the large variance, SubTB (Madan et al., 2023) opti- mizes the flow consistency constraint in sub-trajectory levels. Specifically, it considers all possible (n²²) sub-trajectories Tay = {8%,- (, ,3, and obtain the objective defined as (T) = Er,err Wij (log F(e.) I Pe(es+1ss)) 2 where Wij represents the weight for Ti:j.
2017; Haarnoja et al., 2017a; Geist et al., 2019) considers entropy-regularize objective augmented by the Shannon entropy, i.e.,
Son (s) Ex
Σ rr(s), + XH(T((s)))ss =
(2)
F(ay) IT Proteleet1))
where is the coefficient for entropy regularization. Schul-
man et al. (2017a) show that corresponds to Qoon (s. a)
r(s,a) + >.E''P(e.aai (Algea)) exp(Qson- (5', a') [J)] with the Boltzmann softmax policy Tott ( (als) exp( x Son - (s)). Littman (1996) introduces the generalized Q-function, which considers using a generalized operator @ for updating the Q-values, i.e. Q(s, a) + r(s,a) a) + +w P(o'|s, a) Sa Q(s',a'). Song et al. (2019); Pan et al. (2020b;a; 2021) study the Boltzmann softmax operator, defined as (sQ(ss
2.2. Reinforcement Learning (RL)
A Markov decision process (MDP) defined as a 5-tuple A, r)))), where S represents the set of states, A rep- resents the set of actions, P S A + S denotes the transition dynamics, r is the reward function, and is the discount factor. In an MDP, the RL agent interacts with the environment by following a policy т. which maps state 8 for a pol-
E, exp( where 3 denotes the temperature L. exp exp (Q((a) (usually set with non-zero value). When / approaches 00, it corresponds to the max operator as typically used in standard Q-learning (Mnih et al., 2013). On the other hand, when 3 approaches 0, it corresponds to the mean or average operator.
states to actions. The value function icy x is defined as the expected discounted cumulative reward the agent receives starting from the state 8 i.e., (s))) = E., 1 ('s, ds)]s0 s]. The goal of RL is to find an optimal policy that maximizes the value function at all states. We consider the RL setting consistent with GFlowNets (as in Tiapkin et al. (2024)), with deterministic transitions and the discount factor to be 1, and without in- termediate rewards as in Bengio et al. (2013). It is worth noting that GelowNets can also be extended to stochastic tasks (Pan et al., 2023b; Zhang et al., 2023a), albeit we fo- cus on the more standard deterministic setting in this work. In GelowNets, the reward is obtained at the terminal state, while the reward typically occurs at transitions in RI. To bridge this gap, we define the value of terminal states (xx) as R(x).
3. Related Work
Generative Flow Networks (GFlowNets). Bengio et al. (2021) introduce GFlowNets as a framework for learning stochastic policies that generate objects x through a se- quence of decision-making steps, aiming to sample x with probability proportional to the reward function. GFlowNets have demonstrated remarkable success in various domains, including molecule generation (Bengio et al., 2021), bio- logical sequence design (Jain et al., 2022), Bayesian struc- ture learning (Deleu et al., 2022), combinatorial optimiza- tion (Zhang et al., 2023b; 2024), and fine- -uuing language models (Li et al., 2023; Hu et al., 2024; Lee et al., 2024; Yun et al., 2025), showcasing their potential for discovering high- quality and diverse solutions. Recent research has focused on providing theoretical understandings of GFlowNets by exploring their connections to variational inference (Zim- mermann et al., 2022; Malkin et al., 2023; Niu et al., 2024), generative models (Zhang et al., 2022), and Markov chain Monte Carlo methods (Deleu & Bengio, 2023). Addition- ally, since the introduction of the flow matching learning objective (Bengio et al., 2021), efforts have been made to enhance the learning efficiency of GFlowNets. tackle large variance (Malkin al., 2022: Bengio et al., 2023; Madan et al., 2023), improve exploration (Pan et al., 2022; Lau et al., 2024), enable more efficient credit assignment (Pan et al., 2023a; Jang et al., 2024), and extend to stochastic practical environments (Pan et al., 2023b; Zhang et al., 2023a), largely motivated by the development in the RL literature. Temporal differecce methods in reinforcement
Policy evaluation (Sutton & Barto, 1998) in the dynamic programming literature considers how to compute the value function for an arbitrary policy म, which is also referred to as the prediction problem. The iterative policy evaluation algorithm is summarized in Algorithm 1.
Algorithm 1 Policy Evaluation
input The policy π to be evaluated; small threshold 0 for the accuracy of estimation
I: Initialize value functions (s) arbitrarily for s E S, and V(x)= = R(x) for x X
2: repeat
3: A-0
4:
for S \
X do vt v+V(s) V(s)
5:
6:
V(s) - E. (rals)(r
7:
A + max A |u
8:
end for
9:
until <
Maximum entropy RL. Maximum entropy RL (Neu et al.,
3

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
learning (RL) (Sutton, 1988) serve as significant inspira- tion for GFlowNets (Bengio et al., 2023). There have been number of recent works drawing connections between GFlowNets and maximum entropy (MaxEnt) RL (Tiapkin et al., 2024; Mohammadpour et al., 2024; Deleu et al., 2024), but they are limited to considering an entropy-rege regulaierggl objective that differs from the goal of standard RL. This work establishes a link between GFlowNets and standard (non-MaxEnt) RL through one of its most basic building blocks of policy evaluation for random policy.
frameworks being rooted in temporal difference learning and sequential decision-making.
In this section, we establish a surprisingly simple yet fun- damental connection between GFlowNets and standard RL by returning to one of its most basic building blocks: pol- icy evaluation. We present a novel connection between GFlowNets and policy evaluation under random policies, by analyzing the theoretical equivalence under certain struc- tural conditions between flow functions through flow itera- tion and (scaled) value functions obtained from evaluating a fixed random policy in Section 4.1, which holds uncondition- ally for tree structures and extends to non-tree DAGs con- sidering uniform backward policies under path-invariance conditions. Building upon these insights, we introduce a rectified random policy evaluation method that provides a simple equivalent alternative to existing GFlowNets train- ing methods in cases where the equivalence holds while achieving the same reward-matching effect as GFlowNets in Section 4.2.
Reinforcement Learning (RL). In RL, the problem is typi- cally formulated as a Markov decision process (MDP) with states and actions defined similarly to the directed acyclic graph representation in GFlowNets. The agent learns a deterministic optimal policy to maximize the cumulative return (Sutton& & Barto, 1998). Maximum-entrop (Max- Ent) RI. (Haarnoja et al., 2017a), also known as soft RL or entropy-regularrized RL, optimizes an entropy-regularized objective (Fox et al., 2015; Haarnoja et al., 2017b), where the agent seeks to maximize both the reward and action entropy, which falls under the broader domain of regular- ized MDPs (Neu et al., 2017; Geist et al., 2019). Soft Q- learning (Haarnoja al., 2017b) is a popular instance of MaxEnt RL, which employs log-sum-exp operator instead of the max operator commonly used in Q-learning (Mnih et al., 2013), along with a Boltzmann softmax policy (Schul- man et al., 2017a; Pan et al., 2020a). Related studies have investigated alternative operators for learning the value function, demonstrating that the Boltzmann softmax op- erator (Song et al., 2019; Pan et al., 2020a) can mitigate the estimation bias (Pan et al., 2020a; 2021) in popular RL algorithms, when using a non-zero temperature parame- ter. Recently, Laidlaw et al. (2023) have shown that acting greedily with respect to the value function for a uniform pol- icy can be as competitive as proximal policy optimization (PPO) (Schulman et al., 2017b) in several standard game environments, which highlights the potential of simple, un- informed learning strategies to achieve strong performance.
4.1. Connections
To bridge the gap between GFlowNets and RL, we first establish GFlowNets training from a dynamic programming (DP) (Barto, 1995) perspective, which paves the way for understanding their relationship as DP principles form the foundation of most RL algorithms (Sutton & Barto, 1998).
Algorithm 2 Flow Iteration
input The backward policy PB; a small threshold 0 for estimation accuracy
1: Initialize flow functions F(s) arbitrarily for E S, and
F(z) R(x) for X
2: repeat
3:
A 0+
4:
for s E E X' do
5:
I F(s)
F(s) + E, PB(ss))F(s))
7:
A <- max F(s))
8: end for
4. Rectified Random Policy Evaluation
9:
until A <0
There have been number of recent works (Tiapkin al., 2024; Deleu et al., 2024) exploring the connections of GFlowNets (Bengio et al., 2023) and maximum entropy (MaxEnt) or soft RL (Haarnoja et 2017b; 2018; Geist et al., 2019)), variant of RL that modifies the standard objective with entropy regularization. Specifically, Tiap- kin et al. (2024) show that GFlowNets can be viewed as MaxEnt RL with a particular intermediate reward correc- tion r(s + 8')= = log Pn (s|s"), where the soft value func- tion (s) corresponds to the logarithm of the state flows F(s) in GFlowNets. i.e., Van = log Fss). Despite these findings, the connection berween GFlowNets and standard Lnon-MaXEm) RL remains largely unexplored, despite both
To formalize the DP perspective of GFlowNets, we in- troduce the Flow Iteration algorithm as outlined in Algo- rithm 2. Specifically, Flow Iteration estimates the state flow F(s) for non-terminal states s based on its possible children states s' and the backward policy, which is de- fined as F(s) = Σ.' PP((s(s)F(s)) (following the flow consistency principle in the state-edge level). This formula- tion shares certain computational characteristies with policy evaluation in RL: Flow iteration propagates flow values from child to parent states using the backward policy PB. while policy evaluation propagates values from successor to current states using the (forward) policy т. Note that we consider uniform backward policies for flow iteration in

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
non-tree DAGS in this work.
specific form of value estimation in RL with a specific trans- formation. It also broadens the understanding of what can be achieved with policy evaluation only, unlike typical RL that requires iterative policy evaluation and policy improvement (policy iteration) for reward maximization (Sutton, 1988).
Based on this, we now investigate the relationship between flow functions and value functions considering the structural similarities between flow iteration and policy evaluation. We systematically investigate this relationship, beginning with the simpler case of tree-structured graphs (Bengio et al., 2021) and then exploring the more challenging non-tree- structured directed acyclic graph (DAG) cases.
VM
9154) -1
(VA)) -2
4.1.1. TREE DAG
A
We begin our analysis with tree- structurdd DAGs (Hu et al., 2024) that serve as a foundation for analyzing more general DAGS. Our key insight is that GFlowNets' flow function F(s) and value functions Vss) in RL share an interesting connection when considering the simplest possible policy- uniform random policy, and principled reward transfor- mation considering a scaling factor of the number of avail- able actions in different states. This connection emerges naturally from the structural similarities between our flow iteration and policy evaluation. Specifically, if we scale the original reward function R(x)) for terminal states x in flow iteration by the product of available actions along the path to x, which accounts for the branching structure of the deci- sion process, we observe equivalence between F(s) and (s).. This relationship not only provides new interpreta- tion of GFlowNets but also reveals how standard RL can be repurposed to achieve the same reward-matching behavior of GFlowNets. In Theorem 4.1, we restate and generalize the relation between V (s) (obtained by policy evaluation for a uniform policy) and Fs)) (obtained by our proposed flow iteration), extending initial observations made by Bengio et al. (2021), where the proof can be found in Appendix A.
RGx) -4
(a) Tree MDP.
(b) Results of F(s) and V(s).
Figure I: Comparison of random policy evaluation and flow
iteration in tree-structured DAG example.
Empirical Validation. Consider a tree-structured DAG as shown in Figure (a)), where terminal states x (blue squares) are associated with rewards R(x). To validate our theoreti- cal findings, we compare the flow function F(s) obtained through flow iteration using the original reward R(x) with the value function ((s) computed through policy evalua- tion of a uniform policy using transformed rewards R' (z). The resulting flows F(s) and values V(s) for each state illustrated in Figure 2(a) and Figure 2(c), respectively. As summarized in Figure 1(b), both F(so) and V(su) yield identical values, correctly estimating the total flow. For all other states, F(s) exactly matches (s) after accounting for the scaling factor IT:=00 A(:)). This empirically confirms that standard policy evaluation for a simple random pol- icy, when configured with our transformed reward structure, learns the same flow functions for achieving the reward- matching capability as GFlowNets. Our approach differs from (Tiapkin et al., 2024) in that we do not consider log PB as intermediate rewards and operate in the log-scale, where we instead directly use a transformed terminal reward and a scaling factor computed when we collect the trajectory. This can be illustrated in Figure 2(b), which shows the re- sult of the MaxEnt RL formulation (Tiapkin et al., 2024) that requires intermediate reward corrections (r = 0 here due to the tree structure, which can be non-zero in non-tree structured DAG cases) at each transition and operates in log-scale.
Theorem 4.1 (Generalization of Bengio et al. (2021)). Let A(s) denote the number of available actions at state s, and R(z) be the reward function for terminal states. Let F(s)) be the state flow function obtained from GFlowNer's flow iteration that samples proportionally from R(x)), and V (st) be the value function under a uniform policy with transformed rewards R' (x:) R(x)) 1-1 A(s;). Then, for all st, V(x)) = F( 1 A(s:).
Remark. This reveals an interesting connection: GFlowNets' sophisticated reward- mattardmttccinngaaccc capability can be achieved through a basic operation in RL. RL- policy evalu- ation of a fixed uniform policy with an appropriately trans- formed reward structure. Flow iteration, when viewed through the lens of policy evaluation, establishes a bridge between GFlowNets and standard RL. This provides differ- ent perspective from number of previous works connecting GFlowNets to MaxEnt RL (Tiapkin et al., 2024; Delcu et al., 2024), which incorporates entropy regularization into the standard RL objective. Our finding suggests that the flow- matching objective in GFlowNets can be reinterpreted as a
VAU
Pay
Par-
VTU -2
700-32
(a)
(
Figure 2: Ilussrattoon of flow and value for tree cases. (a) Flow iteration. (b) MaxEnt RL with intermediate reward correction in log-scale. (c) Random policy evaluation with transformed rewards.
5

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
4.1.2. NON-TREE DAG
cases, it prevents the establishment of direct relationship of GFlowNets with policy evaluation and limits universal applicability, and our analysis characterizes the conditions under which these two frameworks align. We view this limitation as a natural consequence of the simplicity of the method, which is based on simple policy evaluation for a fixed uniform policy. By identifying the structural con- ditions necessary for equivalence, our work deepens the understanding of both frameworks and reveals meaning- ful class of problems where simplified training strategies become feasible and offers new insights.
Building upon the insights gained from the simpler tree- structured case, now extend our analysis to the more gen- eral and challenging setting of non-tree tree-structured DAGs, which is natural setting for GFlowNets applications where states can have multiple parent states.
In Theorem 4.2, we present a connection between GFlowNets' flow iteration procedure with uniform back- ward policy and random policy evaluation in the context of general non-tree DAGs, and establish conditions under which the equivalence holds. The proof can be found in Appendix A.
Empirical Validation. To validate this equivalence, we consider the set generation task studied by Pan et al. (2023a). In this task, the agent sequentially generates a set with size |S] from [U] elements. At each timestep, the agent selects element from U and adds it the current set (without replacement). The agent receives the reward for constructing the set with exactly |S| elements.
Theorem 4.2. Let A(s) and B(s) denote the number of outgoing and incoming actions at state s respectively. For any trajectory visiting state St, its branching ratio up
to St is defined as g(T, st) = T A(x,) Let F(s)) be the state flow function obtained from GFlowNet's 's flow
B(S
iteration with uniform backward policy that samples pro- portionally from the reward function R(x), and V (s:) be the value function under a uniform policy with transformed rewards R'' (x) = (T, ana trajectories Tı and t2 that visits 8t satisfy St) = 9(22, st), then for all st, V (8x) = F(st)g(T,s))
Figure presents the results in the tabular case, where values computed by random policy evaluation or flows computed by flow iteration with uniform backward policies are repre- sented in tables as the state and action spaces are enumer- able. This tabular representation eliminates the influence of neural network approximation and sampling errors, pro- viding a clear comparison between the state flow function and the scaled state value function. In Figure 3, the x-axis corresponds to different states, topologically sorted, and the y-axis corresponds to flows or values V. We compare the resulting state flow function F(s). the value function (s) under the original rewards R(x) from random policy eval- uation, and the value function under transformed rewards R' (x)) also from random policy evaluation. We observe that the scaled value function, denoted as scaled Vs)), aligns perfectly with the state flow function F(s), validating the equivalence.
Remark. Theorem 4.2 establishes the relationship between GFlowNets' flow iteration with uniform backward policy and random policy evaluation to non-tree DAG cases under a iath-invariance condition, which reveals consistent pattern: the state flow function F(s) can be expressed as a scaled version the state value function (s)) under a uniform policy with a transformed reward function. The scaling factor g(T, 8t) accounts for both outgoing and incoming actions each state along the trajectory. Different from the tree case which imposes no constraints (due to its unique path property), the non-tree case requires the path-invariance condition, i.e., g(T1, +t) = 9(72, st), which means that any trajectories Tı and T2 passing through state s, should yield identical g(T,s) s) values. This ensures that the scaling factor remains consistent across different paths to the same state, which is essential for establishing the equivalence.
F(s) and scaled V(s)
This assumption naturally holds in several practical GelowNets benchmark environments, e.g., the set genera- tion task studied by Pan et al. (2023a) (where the number of parent or children states remains independent of the state and trajectory itself). While this equivalence applies to this meaningful class of DAG structures, it does not extend uni- versally to all possible DAGs, such as the HyperGrid (Ben- gio et al., 2021) environment, where its boundary effects create path-dependent g- values for the same state. In such
FSd from ii W Visi an re oi wat
Scates Sootioe Reulun
Figure 3: Comparison of random policy evaluation and flow iteration with uniform backward policies in the set generation task.
4.2. Rectified Random Policy Evaluation
Based on the above theoretical analysis, we leverage this interesting insight from the connections between flows and values to develop Rectified Random Policy Evalua-
Under the path-invariance condition, g(7, ) is identical for all trajectories 7 reaching 5c, and the right-hand side is therefore state-dependent only.
6

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
tion (RPE). By rectifying policy evaluation for a uniform policy, RPE achieves the same reward matchiin capabili- ties as GFlowNets that sample proportionally to the rewards (T(x) x R(x)) and discover diverse candidates, while main- taining the simplicity of standard policy evaluation.
which can reduce the potential approximation error from function approximators (Shen al., 2023). As discussed in Section 4.1.2, the equivalence holds unconditionally in tree- structured problems, making RPE universally applica- ble to the tree cases. For non-tree DAGs, this equivalence is guaranteed only when the path-invariance property is satisfied, which holds in tasks where state transitions have similar structural properties (e.g., set generation). While this assumption encompasses broad range of practical applica- tions, investigating scenarios where it may not hold presents promising direction for future research building upon our analysis. This reformulation reveals previously overlooked connections between GFlowNets and policy evaluation, of- fering novel perspectives that bridge this gap and advance our understanding of both frameworks.
The key insight of RPE stems from our theoretical equiv- alence: by reparameterizing the value function as (ss) = F(s)g(s) and scaling terminal rewards by the g-function, we can transform GFlowNet's sophisticated flow- wow-matching objective into a simpler policy evaluation task with trans- formed rewards R² (z) for a fixed random policy. This trans- formation is practical as the g-function is readily available in standard benchmarks. It is simply the number of avail- able actions at each state along the path in tree-structuure problems, while in non-tree DAGs, it is the ratio of out- going to incoming actions along the path, both of which are predefined by the problem's 's action space specification. To enable sampling from the learned distribution, we can obtain the forward policy (s(s'|s) [s) F('') (Bengio
5. Experiments
5.1. Experimental Setup
F(*)
et al., 2023) that inherits GFlowNet's 's reward-matching prop-
Baselines. We extensively compare RPE with GFlowNets with different learning objectives including Flow Matching (FM) (Bengio et al., 2021), Detailed Balance (DB) (Bengio et al., 2023), Trajectory Balance (TB) (Malkin et al., 2022), and Sub- Trajerrayy Balance (SubTB) (Madan et al., 2023) as introduced in Section 2.1. We use the learned backward poli- cies for GelowNets baselines (DB, TB, and SubTB) the default setting, with results using fixed uniform backward policies provided in Appendix C. Additionally, we compare RPE with the maximum entropy (MaxEnt) RL algorithms, i.e., soft DQN (Haarnoja et al., 2017b) and Munchausen DQN MMMDQN;(Vieilll))) et al., 2020; Tiapkin et al., 2024)), as described in Section 2.2.
erties with a uniform backward policy PB. The complete procedure is detailed in Algorithm 3.
Algorithm 3 Rectified Policy Evaluation
input Flows Fø(s) parameterized by 0, uniform policy π output Sampling policy (s(ss(s)= Fo(d') (with
Fo(a)
uniform PB))
1: for to (1, (T) do
2: Sample trajectory T = {so..... J using क
3: Calculate g(s) for states
4: for & r do
5: it's is not a terminal state then
6:
(s)) - E. (()))))
Metries. We follow standard evaluation metrics and eval- uate each method in terms of the accuracy metric (Shen et al., 2023; Kim et al., 2023), which quantifies how well the learned policy distribution aligns with target reward distribution. Accuracy is calculated by computing the rel- ative error between the sample mean of the reward func- tion R(x) under the learned policy distribution Pr Pr(x)) and the expected value of R(x) under the target distribution. The calculation of accuracy is given as Acc(Pp(1)) = 100 x min [Eppto)[8(e) 1). where p'x) = R(x)/Z rep- [,( resents the target distribution. We also analyze the number of modes discovered during the course of training (Bengio et al., 2021; Jain et al., 2022), which measures the ability to identify multiple high-reward regions in the solution space.
7:
else
8:
((s) - R(s)
9: end if
10: end for
11: 0 f- arg min (MsE(g(s) (ss) by Adam
optimizer
12: end for
Discussion. RPE reformulates the GFlowNets training into a random policy evaluation process with rectification, while maintaining equivalent through our established flow-value connections. In RPE, the policy to be evaluated is a fixed uniform policy т,, con- trast to standard GelowNets and MaxEnt RL that requires estimating flows/values for continuously evolving policies during training, leading to more significant non-stationarity challenge in the learning process (Van Hasselt et al., 2018; Laidlaw et al., 2023). In addition, RPE adopts a simpli- fied parameterizatton that learns only the flow function Fe. from which the sampling policy can be directly derived,
capabilities
Tasks. We compare RPE against GFlowNets and MaxEnt RL baselines across several GFlowNets tasks, including TF- Bind generation (Shen et al., 2023), RNA design (Kim et al., 2023), and molecule generation (Shen et al., 2023). We con- sider minimally modified transition dynamics that ensure the path-invariance assumption required in non-tree DAGs
7

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
APE (ours)
SubTO-GFN
TD-GFN
DB-GFN
FM-GFN
MaxEnt RL (Soft DQN)
MaxEnt RL (M-DQN)
323
Numbber of Mottes
Accuacy 8
Towriig themeions Sone
104
Traning Iterations
204
(a) TFbind 1 (Tree)
(b) TFBind (Tree)
Figure Number of modes discovered and accuracy over training across 3 random seeds.
Numbeer of Mades
Accuncy
Accuracy
of
K3 2.8
0.4
8.3
100
Taaning
3mc
Traning Moy Moratarrs
200
(a) TFBind (DAG)
(b) TFBind 2 (DAG)
Figure 5: Number of modes discovered and accuracy of each method.
can be satisfied for comprehensive evaluation, while main- taining the essential characteristics of these tasks. Specifi- cally, when the sequences contain the same amino acids for adding, we restrict the construction process to only append amino acids. Note that this modification only affects a few border cases, and it maintains the essential characteristics of the original tasks.
parent state. We then investigate a variant of the prepend- append MDP (PA-MDP) (Shen et al., 2023) for the TF Bind generation task, where consider structured version where minimally modified transition dynamics ensure the condition as introduced in Section 5.1 (more detailed description can be found in Appendix B). This for- mulation results in a more complex directed acyclic graph, as opposed to a simple tree, due to the existence of multi- ple trajectories for each object x, which poses significant challenges in the learning process.
Implementations. We implement baselines based on open-source codes from Kim et al. (2023)² and Tiapkin et al. (2024)3. We run each algorithm with three random seeds and report both the mean and standard deviation of their performance metrics. To ensure a fair comparison, our method and all baselines use the same network architecture, batch size, and other relevant hyperparameters, with a more detailed description in Appendix due to space limitation.
Figures 4-5 summarize the results in terms of accuracy and the number of modes discovered during training for each method, considering tree-structured generation and DAG-structured TF Bind generation, respectively. These figures present the results for two reward functions, while the complete results for all four reward functions can be found in Appendix C due space constraints. As shown, GFlowNets (including FM, DB, TB, SubTB learning objec- tives), MaxEnt RI. (including Soft DQN and Munchausen DQN (Tiapkin et al., 2024)), and our RPE algorithm achieve comparable performance in terms of the number of modes discovered, indicating their ability to effectively capture the multi-modal nature of the reward function, due their equivalence. In terms of accuracy, we observe that RPE, simplified learning process of evaluating fixed random policies under appropriate transformation, generally outper- forms other baselines by a small margin.
5.2. TF Bind Generation
We first explore the task of generating DNA sequences that exhibit high binding activity with human transcription fac- tors (Jain al., 2022). The objective of the agent is to discover a diverse set of promising candidates that demon- strate strong binding affinity to the target transcription factor. At each timestep, the agent selects an amino acid and in- corporates it into the currently generated partial sequence. We consider four reward functions from Lorenz et al. (201 1). We first study the task a left-to-right generation of the TF Bind sequence as studied in Malkin et al. (2022), where the agent chooses append amino acid a to the end of the current state. This choice of constructive actions leads to a tree-structured problem, as each state only has one
5.3. RNA Sequence Generation
In this section, we study a larger practical task of generating RNA sequences. We consider four distinct target transcrip- tions employing the ViennaRNA package (Lorenz et al., 2011) as studied in Pan et al. (2024), where each task eval-
http://aw...ccmmccoouuuddd////// gfn "https://gittan http:///.//odiaipa/////o/ooooooa
8

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
RNA1
RNAZ
RNA3
RNM
Number of Modes
47
40
30
20
32
10
022
05
60
0.0
0.6
03
04
13
04
Training Iterations 1e4
164
164
164
100
22
80
90
Accuracy
70
60
50
9.0
0.2
60
0.0
0.6
0.7
0.4
3.0
0.4
1.0
Training Iterations le4
1e4
104
164
4PC (earg)
18: 0PV
D8 6P%
Moater Dggu
Maxite HON)
Figure 6: Top row: number modes discovered over training across 3 random seeds. Bottom row: Accuracy over training across 3 random seeds.
uates the binding energy with unique target serving the reward signal for the agent (Lorenz et al., 201 1). We follow the experimental setup as in Section 5.1, and study the variant of grepend-append MDP introduced by Shen al. (2023). More detailed descriptions of the setup can be found in Appendix B.
QM4
QM9
100
Number of Modes
0.
1.2
The performance of each method in terms of accuracy and the number of modes discovered for each task is shown in Figure 6. The results show that RPE captures the multi- modal reward landscape well, maintaining the key capability discovering diverse, high-reward solutions. In addition, in this more challenging task with larger state spaces, we observe that RPE demonstrates stronger perfor- mance across both metrics, achieving nearly 100% accuracy while discovering more modes than the baselines, as RPE provides a simpler parameterization for evaluating a fixed random policy different from baseline methods.
Training Iteraiins
Training Iterations 1+3
Figure 7: Results in the QM9 molecule generation tasks.
6. Conclusion
of GFlowNett
In this paper, we establish a new connection between GFlowNets and core reinforcement learning concepts through the lens of flow iteration and policy evaluation. Our results reveal that the value function under uniform policy is intrinsically linked to the flow functions in GFlowNets under certain structural conditions. Based on this insight, we develop Rectified Policy Evaluation (RPE), which refor- mulates GFlowNets' objectives through simplified policy evaluation framework. Extensive empirical evaluations on GFlowNets benchmarks demonstrate that RPE is able to capture multi-modal rewards and achieve GFlowNets' so- phisticated reward- reward-matching capability through evaluating fixed uniform policy, providing new insights into under- standing both fields.
5.4. Molecule Generation
In this section, we study the task of generating molecule graphs. We study the variant of the QM9 molecule task as studied in prior GFlowNets work (Jain et al., 2023b; Shen et al., 2023; Kim et al., 2023) following the experimental setup as Section 5.1, where the reward function is defined as the energy gap between the highest occupied molecular orbital and lowest unoccupied orbital (HOMO-LUMO)) We employ pre-trained molecular property prediction model, MXMNet (Zhang et al., 2020), as the reward proxy.
While there are no structural constraints required for tree- structured environments, the extension to non-tree DAGs requires the path-invariance condition, which may not hold universally across all problem domains. Investigat- ing whether the relationship can be extended to broader classes of environments without restrictive assumptions, exploring connections with non-uniform policies, and de- veloping adaptive methods that can automatically satisfy structural requirements in general settings are interesting future directions.
The results are summarized in Figure 7, where RPE consis- tently maintains the ability to capture multi-moda. rewards, demonstrating comparable accuracy GelowNet and Max- Ent RL baselines. RPE also discovers modes faster from its fixed policy evaluation framework.
9

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Acknowledgment
Geist, M., Scherrer, B., and Pietquin, O. A theory of regu- larized markov decision processes. In International Con- ference Machine Learning, 2 2160-2169. PMLR, 2019.
This work supported by the National Natural Science Foundation of China 62406266.
Impact Statement
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-based policies, In International conference on machine learning, pp. 1352- 1361. PMLR, 2017a.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy- policies. In International conference on muchine learning, pp. 1352- 1361. PMLR, 2017b.
References
Andrieu, C., De Freitas, N., Doucet, A., and Jordan, M. An introduction to meme for machine learning. Machine learning, 50:5-43, 2003.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor-critic: Off-pooliy maximum entropy deep reinforce- ment learning with stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018
Barto, A. G. Reinforcement learning and dynamic pro- gramming. In Analysis, Design and Evaluation of Man- Machine Systems 1995, pp. 407-412. Elsevier, 1995.
Bengio, E., Jain, M., Korablyov, M., Precup, D., and Ben- gio, Y. Flow network based generative models for non- iterative diverse candidate generation. Advances in Neu- ral Information Processing Systems, 34:27381-27394, 2021.
Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. 1970.
Hu, E. J., Jain, M., Elmoznino, E.,, Kaddar, Y., Lajoie, G., Bengio, Y., and Malkin, N. Amortizing intractable infer- ence in large language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. Better mixing via deep representations. In International confer- ence on machine learning, pp. 552-560. PMIR,, 2013.
Jain, M., Bengio, E., Hernandez- Garcia, A., Rector- Brooos, J., Dossou, B. F., Ekbote, C. A., Fu, J., Zhang, T., Kil- gour, M., Zhang, D., et al. Biological sequence design with gflownets. In International Conference on Machine Learning. pp. 9786-9801. PMLR, 2022.
Bengio, Y., Lahlou, S., Deleu, T., Hu, E. J., Tiwari, M., and Bengio, E. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
Chen, Y. and Mauch, Order-press dedee-preservig gflownets. arXiv preprint arXiv:2310.0033 10.0688, 2023.
Jain, M., Deleu, T., Hartford, J., Liu, C.-H., Hernandez- Garcia, A., and Bengio. Gflownets for ai-driven scien- tific discovery. Digital Discovery, 2(3):557-577, 2023a.
Deleu, T. and Bengio, Y.
Generative flow net- works: markov chain perspective. arXiv preprint arXiv:2307.01422, 2023.
Jain, M., Raparthy, S. C., Hernández-Garrc A., Rector- Brooks, J., Bengio, Y., Miret, S., and Bengio, E. Multi- objective gflownets. In International conference PMLR, 2023b.
Deleu, Т. Góis, A., Emezue, C., Rankawat, M., Lacoste- Julien, S., Bauer, S., and Bengio, Y. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
chine learning, pp. 1
Jang, H., Kim, M., and Ahn, S. Learning energy decompo- sitions for partial inference of gflownets. In The Twelfth International Conference on Learning Representations, 2024.
Deleu, T., Nouri, P., Malkin, N., Precup, D., and Bengio, Y. Discrete probabilistic inference as control in multi-path environments. arXiv preprint arXiv:2402. 10309, 2024.
Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08862, 2015.
Kim, M., Yun, T., Bengio, E., Zhang, D., Bengio, Y., Ahn, S., and Park, J. Local search gflownets. In The Twelfth International Conference on Learning Representations, 2023.
Ganguly, A., Jain, S., and Watchareeruetai, U. Amortized variational inference: Towards the mathematical founda- tion and review. arXiv preprint arXiv:2209. 10888, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14X2, 12806 2014.
10

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Laidlaw, C., Russell, S. J., and Dragan, A. Bridging rl theory and practice with the effective horizon. Advances in Neural Information Processing Systems, 36:58953- 59007, 2023.
Neu, G., Jonsson, A., and Gómez, V. A unified view of entropy-regularized markov decision processes. arXiv preprint 1rxXiv:705.079990 2017.
Niu, P., Wu, S., Fan, M., and Qian, X. Gflownet training by policy gradients. In Forty-first International Conference on Machine Learning. 2024.
Lau, E., Lu, S. Z., Pan, L., Precup, D., and Bengio, E. Qgfn: Controllable greediness with action values. arXiv preprint arXiv:24022005254 2024.
Pan, L.. Cai, Q. and Huang, L. Softmax deep double deter- ministic policy gradients. Advances in neural information processing systems, 13:11767-11777. 2020a.
Lee, S., Kim, M., Cherif, L., Dobre, D., Lee, J., Hwang, S. J., Kawaguchi, K., Gidel, G., Bengio, Y., Malkin, N., et al. Learning diverse attacks on large language models for robust red-teaming and safety tuning. arXiv preprint arXiv:2405. 18540, 2024.
Pan, L., Cai, Q., Meng, Q., Chen, W., and Huang, L. Re- inforcement learning with dynamic boltzmann softmax updates. In Bessiere, C. (ed.), Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intel- ligence, IJCAI-20, pp. 1992-1998. International Joint Conferences Artificial Intelligence Organization, 7 2020b. doi: 10.24963//jcci. 2020/276. URL https: 2020/226.. Main jcaiiiii/iiai..o http:/aaaaa..22//1112
Li, Y., Luo, S., Shao, Y., and Hao, J. Gflownets with human feedback. arXiv preprint arXiv:2305.07036 2023.
Littman, M. L. Algorithms for sequential decision-making, Brown University, 1996.
/ oooi.orr//e 9323 track.
Lorenz, R., Bernhart, S. H., Höner zu Siederdissen, C., Tafer, H., Flamm, C., Stadler, P. F., and Hofacker, I. L. Viennarna package 2.0. Algorithms for molecular biology, 6:1-14, 2011.
Pan, L., Rashid, T., Peng, B., Huang, and Whiteson, Regularized softmax deep multi-agent q-learning. Ad- vances Neural Information Processing Systems, 34: 1365-1377, 2021.
Madan, K., Rector-Brooks, J., Korablyov, M., Bengio, E., Jain, M., Nica, A. C.,, Bosc, T., Bengio, Y., and Malkin, N. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pp. 23467-23483. PMLR, 2023.
Pan, L., Zhang, D., Courville, A., Huang, L.,, and Bengio, Y. Generative augmented flow networks. In The Eleventh International Conference on Learning Representations, 2022.
Pan, L., Malkin, N., Zhang, D., and Bengio, Y. Better training of gflownets with local credit and incomplete trajectories. In International Conference Machine PMLR, 2023a.
Malkin, N., Jain, M., Bengio, E., Sun, C., and Bengio, Y. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955-5966 2022.
Learning, pp.
Pan, L., Zhang, D., Jain, M., Huang, L., and Bengio, Y. Stochastic generative flow networks. In Uncertainty in Artificial Intelligence, pp. 1628-1638. PMIR, 2023b.
Malkin, N., Lahlou, S., Deleu, T., Ji, Hu, E. J., Ev- erett, K. E., Zhang, D., and Bengio, Y. Gflownets and variational inference. In The Eleventh International Con- ference Learning Representations, 2023.
Pan, I.,, Jain, M., Madan, K., and Bengio, Y. Pre-training and fine-tuning generative flow networks. In The Twelfth International Conference on Learning Representations, 2024.
Metropolis, N., Rosenbluth, A. W. Rosenbluth, M. N., Teller, A. H., and Teller, F. Equation of state calculations by fast computing machines. The journal of chemical physics, 21 1186)10087 1953.
Salakhutdinov, R. R. Learning in markov random fields us- ing tempered transitions. Advances in neural information processing systems, 22, 2009.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, arXiv preprint arXiv: 1312.600, 2013.
Schulman, J., Chen, X., and Abbeel, P. Equivalence be- tween policy gradients and soft g-learning, arXiv preprint arXiv: 7704 704.06440, 2017a.
Mohammadpour, S., Bengio, E., Frejinger, E., and Bacon, P.-L. Maximum entropy gflownets with soft q geaarnnnn. In International Conference on Artificial Intelligence and Statistics, pp. 2593-2601. PMLR, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1777:0177. 2017b.

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Shen, M. W., Bengio, E., Hajiramezanali, E., Loukas, A., Cho, K., and Biancalani, T. Towards understanding and improving gilownet training. In International Conference on Machine Learning, pp. 30956-30975. PMLR, 2023.
Song. Z., Parr, R., and Carin, L. Revisiting the softmax bellman operator: New benefits and new perspective. In International conference on machine learning, pp. 5916- 5925. PMLR, 2019.
Sutton, S. Learning predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. 1998.
Tiapkin, D., Morozov, N., Naumov, A., and Vetrov, D. P. Generative flow networks as entropy- regropy-regularized rl. International Conference on Artificial Intelligence and Statistics, pp. 4223-4221. PMLR, 2024.
Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad. arXiv preprint 1rXXiv:812.02665 2018.
Vieillard, N., Pietquin, O., and Geist, M. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235 4246, 2020.
Xu, B., Wang, N., Chen, T., and Li, M. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1ww.Xxv::0500000
2015.
Yun, T., Zhang, D., Park, J., and Pan, L. Learning to sample effective and diverse prompts for text-to-image genera- tion. arXiv preprint 1rX1v:2502.11477, 2025.
Zhang, D., Chen, R. T., Malkin, N., and Bengio, Y. Unifying generative models with gflownets and beyond. arXiv preprint arXiv:2209.0206
2022.
Zhang, D., Pan, L., Chen, R. T., Courville, A., and Bengio, Y. Distributional gflownets with quantile flows. arXiv preprint ar arXiv:2302.0577) 2023a.
Zhang, D., Dai, H., Malkin, N., Courville, A. C., Bengio, Y., and Pan, L. Let the flows tell: Solving graph combi- natorial problems with gflownets. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, W., Rainone, C., Peschl, M., and Bondesan, R. Robust scheduling with gflownets. arXiv preprint arXiv:2302.05444 05446. 2023b.
Zhang, S., Liu, Y. and Xie, L. Molecular mechanics-driven graph neural network with multiplex graph for molecular structures. arXiv preprint arXiv:2011.07457, 2020.
Zimmermann, H., Lindsten, F., van de Meent, J.WW., and Naesseth, A. variational perspective on generative flow networks. arXiv preprint arXiv:22 2022.
12

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
A. Proofs of Theorems 4.1-4.2
can be simply verified that the tree-structured DAG (Theorem 4.1) is a special case of the non-tree-structured DAG (Theorem 4.2), with (()++1) 1 and there is only one path from Sq any states, and Pa is trivial in this case. Therefore, we prove the general non-tree conttree case here with mathematical induction (and the proof for Theorem 4.1 can also be obtained via the following proof since it is special case).
Theorem 4.2 Let A(s) and ((s) denote the number of outgoing and incoming actions at state 8 respectively. For any trajectory 7 visiting state St, its branching ratio up to St is defined g(T, - [A(&:) [[s++1]]]' Let F())) be the state flow function from GFlowNet's s flow iteration with uniform backward policy that samples proportionally from the reward function R()),, and (st) be the value function under a uniform policy with transformed rewards R'(x:) R(x)g(T, If any trajectories Tı and To that visits St satisfy g(T,, 8t)) g((2, 8c), then for all 8t, V(+e) F(st)g(T,8) 8T)
Proof. For all terminal states Sn. by definition, have that
((nn) (s.) R(sn)g((T,))) sn).
(3)
F(s,) R(s.))
(4)
Thus, V(s.) ((sn)g(T, sn) holds for all terminal states.
Then, for any other state assume all of its children satisfy
V(*+) = (K)))
(5)
By the definition of the policy evaluation procedure, we have that
V(sk)
V(s)) = Σ
(6)
A(x)
81 +5
Combining (5) and (6), we get
F(SH)g(T,8)) (A(st)
V(si) = Σ
31 38k
By definition, we have that
k-1
A(ss;)
g(T,8t) II
(8)
B(s++1)l
1=0
Thus, we obtain that
g(T, 8k) [A(s)) [A(sk-1)]
g(T,
I1 k-2 IA(s:)
|A(st)
(9)
1==0 [B(s+1))
[B(s)]
B(s++1)
As
AA(s:)
g(T, = ([(T88=1)
II
(10)
B B(s+1)]
==0
and combing Eq.(7).(9), and (10), we have that
F(Sk)
V(s)) Σ
g(T, st). B(sx)]
(11)
By the definition of the flow iteration procedure considering uniform backward policies (PB(s)(s)) [B(ec)] ) we have that
F(s2)
F(st) = Σ
(12)
B(s))l
81 Bu
13

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
Combing Eq.(11) and (12), we obtain that
V ((s)) = F(s:))(T,
(13)
Therefore, the condition holds for st. Finally, by induction, V(sr)= = F(st)g(T, s:) holds for all states. Note that under the path-invariance condition, g(T, st) is identical for all trajectories reaching 8t, and the right-hand side therefore state-dependent only.
B. Experimental Setup
B.1. Implementation Details
We use MLP network that consists of 2 hidden layers with 2048 hidden units and ReLU activation (Xu et al., 2015) to estimate flow function Fo.
We encode each state into one-hot encoding vector and feed them into the MLP network.
We clip gradient norms to a maximum of 10.0 prevent unstable gradient updates.
We run all the experiments in this paper with RTX 3090 GPU.
Below we introduce separate details for different benchmarks used in this paper. For thorough comparison across methods considering the nath-invariance property, we consider minimally modified transition dynamics that ensure the assumption required for GFlowNets and RL can be satisfied for comprehensive evaluation. Specifically, when the sequences contain the same amino acids for adding, we restrict the construction process to only append amino acids. This modification only affects few border cases, and maintains the essential characteristics of the original tasks.
TF Bind generation. For the tree-structured TF Bind task, we follow the experimental setup described in Jain et al. (2022); For the graph- graph-structured TF Bind task, we follow the experimental setup described in Shen et al. (2023). We select four tasks defined by different reward functions in Lorenz al. (2011), i.e., S/X6 TI65A RI 8mees, ARX P353L R2 8mers, PAX3_Y90H_RI _8mess and WTI_REF LIEF_RI_AA _8mess. In this task. train our model for 1e4 steps, using the Adam optimizer (Kingma & Ba, 2014) with a 3e-3 learning rate. We set the reward threshold as 0.8 and the distance threshold as 3 to compute the number of modes discovered during training.
RNA Sequence generation. We consider the PA-MDP to generate strings of 14 nucleobases. Following Pan et al. (2024), we present four different tasks characterized by different reward functions, i.c., RNAI, RNA2, RNA3 and RNA4. In this task, we train our model for 1e4 steps, using the Adam optimizer (Kingma & Ba, 2014) with a 3e-3 3 learning rate. We set the reward exponent as 3. We set the reward threshold as 0.8 and the distance threshold as 3 to compute the number of modes discovered during training. We normalize the reward into [0.001, 10) during training.
Molecule generation. The goal of QM9 is to generate a molecule with 5 blocks from 12 building blocks with 2 stems. Following the experimental setup described in Kim et al. (2023), we set the reward exponent as 5. We train our model for 2c3 steps, using the Adam optimizer (Kingma Ba, 2014) with 1e-3 learning rate. To compute the number of modes discovered during training, set the reward threshold 1.4 and the distance threshold 3. We normalize the reward into [0.001, 1000). This normalization is beneficial for flow regression.
Baselines. We implement the MaxEnt RL baselines, including soft DQN and Munchausen DQN, by borrowing the code (Tiapkin al., 2024). We implement the baselines of
from httpt::t///........ / htt///gtttu.... thub. com htmm//tttaaiin/gflloo/...o GFlowNets based on the codes from hattts://githhiie com/ dbsxodud- 11/1s_gffn (Kim et al., 2023).
C. Additional Experimental Results
TF Bind We provide the other two tasks of TF Bind generation benchmark in Fig 8. We find that RPE performs competitively in both tree and graph TFBind tasks.
14

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
RNA generation. Table I and Table summarize the results of the final model in the RNA generation tasks, which clearly show the superior performance of our method RPE in terms of both accuracy and number of modes discovered.
Baselines with uniform Pn. Furthermore, we conduct comparison between RPE and GFlowNets methods using uniform PB. As depicted in Fig. 9, the PB values in TB-GFN, SubTB-GFN, and DB-GFN are consistently fixed to be uniform. Our observations indicate that while these GelowNets methods with uniform P3 exhibit rapid convergence, they generally yield inferior performance compared to instances where PB is learned.
(ours)
SUBTB-GFN
DE-GFN
(Soft DQN)
(M-DQN)
100
100
Number of Modes
Number Modes
98
10
Accuracy
90
20
29-
92
80
20
90
13
10
70
20
04
0.0
0.2
5.4
0.5
10.0
0.2
0.4
0.0
0.8
1.0
0.0
0.2
5.4
0.5
0.0
0.0
0.2
0.4
0.66
0.0
Training iterations
104
104
Training Iterations
1p4
104
(a) Tree TFbind 3
(b) Tree TFBind
100
8
Number of Modes
Number of Modes
38
80
Accuracy
20
96
8
94
13
70
15
00
60
8.0
Training Iterattion ons
1e4
Training Iterationn
1e4
(c) Graph TFBind 3
(d) Graph TFBind
Figure 8: Number modes discovered and accuracy over training across 3 random seeds.
Table 1: Accuracy in different RNA generation tasks.
L14 RNAI
L14 RNA2
L14 RNA3
L14 RNA4
FM-GFN
83.54 + 1.46
81.16 + 1.03
71.82 + 0.57
66.69 + 1.13
DB-GFN
88.48 + 0.41
0.33 88.49 0.33
76.35 + 0.44
70.53 + 0.12
TB-GEN
86.81 = 0.24
87.52 0.18
= $1.80
79.14 + 0.78
SubTB-GFI
85.25 0.46
86.67 0.57
78.34 + 0.67
74.10 + 0.10
MaxEnt RL (Soft DQN)
80.27 + 0.52
79.01 = 0.03
73.52 + 0.72
70.11 + 1.13
MaxEnt RL (M-DQN)
89.55 + 0.51
89.99 = 0.43
78.28 + 0.98
72.40 + 0.73
RPE (Ours)
99.81 + 0.19
100.0 + 3.97
100.0 2.45
97.68 +-2 3.03
D. Discussion
In this section, we discuss the limitations of RPE and the broader impact of our work. Although RPE obtains stronger performance compared with thorough set of baselines in various benchmarks, there are several directions for future improvements. The current implementation computes (=(s(|s) through separate forward passes for each child state s', which presents opportunities for computational optimization. Additionally, like many local credit assignment methods, RPE's effectiveness in environments with very long trajectories could further improved, building upon recent advances
15

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
RNA1
RNA2
RNA3
RNA4
Number of Modes
40
30
20
10
10
0.2
1.0
0.5
0.8
0.2
1.0
0.0
0.6
0.8
Training iterations 1e4
1c4
1e4
1e4
100
100
1.00
100
90
90
Accuracy
80
70
70
660
60
50
50
40
20
20
20
0.0
0.2
0.4
9.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Training Iterations 1e4
1e4
1e4
RPE (ours)
SubTB GFN
TB GPN
DB GFN
PM GFN
MoxEnt RL (Soft DQNI
MaxEnt RL IN DON)
Figure 9: PB is fixed to be uniform for GFlowNets methods. Top row: number of modes discovered over training across random seeds. Bottom row: Accuracy over training across 3 random seeds.
Table 2: The number of modes discovered in different RNA generation tasks.
L14 RNAI
L14 RNA2
L14
RNA3
L14 RNA4
FM-GFN
31 2
28 2
21 +
53 + 5
DB-GFN
3
30 2
25 4
61 + 2
TB-GFN
34
31 31
31
79 + 7
SubTB-GFN
32
29 +1
27
68 3
MaxEnt RL (Soft DQN)
30
26 2
22
61 5
MaxEnt RL (M-DQN)
34
32 +
22
60 10
RPE (ours)
42 + 2
46
52 $2:9 9
90 6
in temporal credit assignment (Malkin al., 2022; Madan et al., 2023; Pan et al., 2023a). While specific scenarios g-values, our work uncovers
(e.g., the HyperGrid task) may require additional consideration due to previously overlooked connections between GelowNets and policy evaluation. This theoretical bridge not only deepens our understanding of both frameworks, but also opens up new research directions for future work to build upon, as demonstrated by our simplified yet effective training strategies.
Differences between RPE and Soft Policy Iteration. Our proposed RPE shares conceptual connections with Soft Policy Iteration (SPI), while establishing key distinctions that highlight the novelty of our work: (1) SPI necessitates iterative cycles of policy evaluation and policy improvement to derive the desired policy, while RPE evaluates only the uniform policy, transforming its value function to flow functions to achieve reward matching. (2) SPI and "control as inference' framework both explicitly incorporate entropy regularization terms to ensure policy stochasticity, while RPE achieves superior reward- reward-matching performance through standard policy evaluation utilizing a summarization operator, eliminating the need for explicit regularization. Our experimental results demonstrate RPE's superior performance compared both SoftDQN and MunchausenDQQ (which are advanced soft RL algorithms for discrete environments) in terms of the standard GFlowNets evaluation protocol.
Assumptions in Theorem 4.2. Theorems &.12 are formulated within the context of GFlowNets. which address constructive tasks by sampling compositional objects through a sequence constructive actions. These tasks are modeled as Directed Acyelic Graphs (DAGs), where agents do not revisit the same state within a single episode, ensuring the graph remains acyclic.
16

---

Random Policy Evaluation Uncovers Policies of Generative Flow Networks
The path-invariance condition in Theorem 4.2 requires that the branching ratio g(T, 8t) is identical for all trajectories reaching the same state 8t- This assumption is satisfied in broad range of domains within the GFNs literature, beyond just tree-structured problems (e.g., sequence/language generation (Hu et al., 2024)), where the local branching structure exhibits symmetry properties inherent to several compositional generation tasks.
However, as we discuss in Section 4.1.2 there are certain cases, e.g. the HyperGrrid task with boundary conditions (where task symmetry fails for states at edges), where this assumption does not hold. We view this limitation as a natural consequence of the simplicity of our method. Yet, it is precisely this simplicity that makes our findings both surprising and impactful, they provide the first rigorous characterization of when GFNs and policy evaluation align.
17


Reviews:
[{"review_id": "NPDh0Qj0L6", "reviewer": "Reviewer_UFa1", "summary": "This paper explores a connection between GFlowNets and policy evaluation in the specific setting of undiscounted Reinforcement Learning with only terminal rewards. The central idea revolves around the flow constraint in GFlowNets, which shows that the flow out of a state must equal the total in-flow from its successor states. The authors draw a parallel between this constraint and the objective of policy evaluation, where the value of a state is the average of the values of its descendant states. The key difference identified is a normalization factor (the number of descendants) in policy evaluation. The paper proposes scaling the flow function in GFlowNets to account for this normalization, suggesting a potential way to perform policy evaluation using GFlowNets, which is shown to work for tree-structured state spaces.\n\n## update after rebuttal\n\nAfter discussing with the author, I will maintain my original assessment given the limitations of the methods and their similarity to existing works", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions and feedback! We carefully address your concerns as follows:\n>Q1: $\\tau$-dependence regarding transformed rewards $R(x)g(\\tau,x)$\n\nThanks for your question! This concern is addressed by the assumption in Theorem 4.2: \"For any trajectories $\\tau_1$ and $\\tau_2$ that visit $s_t$, they must satisfy the condition $g(\\tau_1, s_t) = g(\\tau_2, s_t)$.\" Under this assumption, the transformation from $R(x)$ to $R'(x)=R(x)g(\\tau,x)$ is well-defined, since the transformed reward function $R'(x)$ remains consistent regardless of the specific trajectory taken to reach the terminal state $x$ (and this property naturally holds in a number of standard GFlowNet benchmarks -- Q2). We will also update the notation to eliminate any ambiguity. \n>Q2: questions about the condition g(τ_1,s_t)=g(τ_2,s_t)\n\nThanks for your question. Indeed, this condition is a necessary assumption for the validity of Theorem 4.2. \n\nWe remark that this assumption can be satisfied in a number of standard GFlowNets problems. Please refer to our response to Reviewer ghFJ's Q1 for a detailed discussion.\n>Q3: Definition of P_B\n\nThanks for the question. This is not a typo, and $P_B$ is defined as $P_B(s|s')=\\frac{F(s\\to s')}{F(s')}$ following the GFlowNets literature (Eq. (17) in [1]), representing the probability of having come from state $s$ given that we are currently at state $s'$ (defined in terms of the flow).\n>Q4: Concerns regarding motivations and contributions.\n\nThe motivation for connecting GFlowNets and policy evaluation stems from a critical gap in the current understanding of GFlowNets' relationship with RL frameworks. While previous research has extensively explored connections between GFlowNets and various ML methods, their link to RL has largely been limited to MaxEnt RL [2,3] via modified objectives with entropy regularization. However, the connection between GFlowNets and standard (non-MaxEnt) RL remains largely unexplored, despite the shared foundation of sequential decision-making. This gap limits cross-disciplinary insights (e.g., leveraging RL’s efficiency for GFlowNets or enriching RL with GFlowNets’ diversity).\n\nOur work fills this crucial gap and makes the following key contributions: (1) Surprisingly, we find that the flow functions in GFlowNets naturally correspond to value functions in uniform policy evaluation. This insight leads to our key finding that reward-matching effects, previously achieved through complex GFlowNets algorithms [1,4], **can be accomplished through simple policy evaluation.** (2) Based on this theoretical foundation, our proposed RPE method achieves competitive performance with reduced complexity. (3) Our work challenges the prevailing belief that GFlowNets are intrinsically tied to MaxEnt RL and provides a fundamentally new perspective on GFlowNets learning dynamics.\n\nIn addition, traditional policy evaluation is merely a component for policy improvement; RPE reframes it to generate diverse, high-quality candidates via uniform policy evaluation with transformed rewards, expanding its utility to domains requiring both quality and diversity.\n>Q5: relations to Soft Policy Iteration (SPI) and the \"control as inference\" framework\n\nThanks for your insightful question. Our work shares conceptual connections with SPI, while establishing key distinctions that highlight the novelty of our work: (1) SPI necessitates iterative cycles of policy evaluation and policy improvement to derive the desired policy, while RPE evaluates only the uniform policy, transforming its value function to flow functions to achieve reward matching. (2) SPI and 'control as inference' framework both explicitly incorporate entropy regularization terms to ensure policy stochasticity, while RPE achieves superior reward-matching performance through standard policy evaluation utilizing a summarization operator, eliminating the need for explicit regularization. Our experimental results demonstrate RPE's superior performance compared to both SoftDQN and MunchausenDQN (which are advanced soft RL algorithms for discrete environments) in terms of the standard GFlowNets evaluation protocol. We will add detailed discussions between them in the revision.\n\n>Q6: Comparison on a standard RF setting\n\nSince the goal of our work is to validate that a simple RPE algorithm can achieve competitive reward-matching performance as existing complex GFlowNets, we evaluate RPE and extensive baselines using well-established benchmarks in the GFlowNets literature [8,9], e.g., TFBind/RNA/molecule generation, which requires a critical balance between diversity and quality.\n\n[1] Yoshua Bengio et. al., GFlowNet Foundations, JMLR 2023.\n\n[2] Tiapkin et. al., Generative Flow Networks as Entropy-Regularized RL, AISTATS 2024\n\n[3] Tristan et al., Discrete probabilistic inference as control in multi-path environments, UAI 2024\n\n[4] Bengio et. al., Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation, NeurIPS 2021", "claims_and_evidence": "The paper claims a connection between GFlowNets and policy evaluation and proposes a method to bridge this gap, particularly for undiscounted RL with only terminal rewards. The theoretical result for the tree case (Theorem 4.1) seems plausible based on the explanation. However, I'm not sure about if it's correct for the general DAG case (Theorem 4.2).", "methods_and_evaluation_criteria": "The proposed method involves scaling the flow function F(s) in GFlowNets based on the number of \"branches\" or cumulative inverse of the number of descendants to align with the policy evaluation objective. The evaluation seems to primarily focus on the theoretical derivations, with limited mention of empirical evaluation or specific evaluation criteria. I'm not sure about the motivation behind this specific approach and its advantages, as well as its relation to existing policy evaluation methods like Soft Policy Iteration.", "theoretical_claims": "I'm unsure about the validity of Theorem 4.2, which extends the connection to the general DAG case. The definition of g(τ,s_t) is clear. However, the subsequent definition of R′(x)=R(x)g(τ,x) is ambiguous. In a DAG, multiple trajectories τ can lead to the same terminal state x. This raises a concern as the left side of the equation (R′(x)) should ideally be independent of the specific trajectory taken, while the right side (R(x)g(τ,x)) explicitly depends on τ through g(τ,x). This suggests a potential flaw in the formulation for the general DAG case. The correctness of the proof for Theorem 4.2 is therefore in question.", "experimental_designs_or_analyses": "The experiments are conducted on predicting DNA or RNA sequences andn molecule generation. I wonder why the comparison on a standard RF setting is not presented.", "relation_to_broader_scientific_literature": "This work is related to Soft Policy Iteration (control as inference), where the policy is indeed often treated as uniform. The paper should discuss how this approach compares to and differs from existing methods in the context of policy evaluation and control.", "essential_references_not_discussed": "However, a more comprehensive understanding of the GFlowNet and policy evaluation literature might reveal missing relevant works.", "other_strengths_and_weaknesses": "Strengths: The paper attempts to establish a novel connection between GFlowNets and a fundamental problem in Reinforcement Learning, policy evaluation. The result for the tree case (Theorem 4.1) seems promising.\n\nWeaknesses: The main weakness lies in the ambiguity and potential flaw in the extension to the general DAG case (Theorem 4.2), particularly concerning the τ-dependence in the definition of R′(x). The lack of clarity in definitions and assumptions, such as the condition in line 629-630, also detracts from the paper's strength. Furthermore, the motivation for this specific approach and its relation to existing RL methods are not clearly articulated. The paper also lacks details on experimental validation.", "other_comments_or_suggestions": "- The authors should rigorously define all variables in the equations to improve clarity.\n- The ambiguity surrounding the definition of R′(x)=R(x)g(τ,x) in the context of DAGs with multiple trajectories reaching the same terminal state needs to be addressed. How is the τ-dependence resolved?\n- The condition mentioned in lines 629-630, \"If any trajectories τ_1 and τ_2 that visits s_t satisfy g(τ_1,s_t)=g(τ_2,s_t),\" needs to be clearly stated whether it is an assumption and, if so, how it can be generally satisfied in a DAG. In line 068 (2nd column), the backward probability P_B(s∣s′) seems to have a typo and should likely be P_B(s′∣s).\n- The paper would benefit from a clearer explanation of the motivation behind using this specific approach for policy evaluation. What are the potential advantages or insights it offers compared to existing methods?\n- A discussion on how this work relates to Soft Policy Iteration and the \"control as inference\" framework would help contextualize the contribution.", "questions_for_authors": "1. In the definition of R′(x)=R(x)g(τ,x) for the general DAG case (Theorem 4.2), how is the τ-dependence handled given that multiple trajectories can reach the same terminal state x? Please clarify why R′(x) on the left-hand side would be τ-independent.\n2. Regarding the condition stated in lines 629-630: \"If any trajectories τ_1 and τ_2 that visits s_t satisfy g(τ_1,s_t)=g(τ_2,s_t),\" is this an assumption that needs to hold for your theoretical results to be valid? If so, under what conditions on the DAG structure and the transition probabilities would this assumption be satisfied in general?\n3. In line 068 (2nd column), for the backward probability, is P_B(s∣s′)=F(s→s′)/F(s′) a typo? Should it be P_B(s′∣s)=F(s→s′)/F(s) or P_B(s′∣s)=F(s→s′)/F(s′)?\n4. What is the specific motivation for using this approach to connect GFlowNets and policy evaluation? What advantages does it offer over existing policy evaluation techniques, particularly in the context of undiscounted RL with only terminal rewards?\n5. How does this work relate to the literature on Soft Policy Iteration and the \"control as inference\" paradigm, where policies are often treated as uniform distributions? What are the key differences and similarities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ZH4y9qhKP", "reviewer": "Reviewer_ghFJ", "summary": "The paper presents a connection between GFlowNets and non max-ent RL. It leverages insights in the special case with a uniform policy to establish the connection, which leads to the development of the RPE algorithm. Empirical results suggest that RPE achieves competitive performance with existing GFlowNet training and entropy-regularized RL baselines.\n\n### Update after rebuttal\n\nI thank the authors for their extensive efforts in the rebuttal. The additional clarifications on the applicability of scaling factors and additional evidence on the empirical advantage of the proposed method well addressed my concerns. \n\nWhile I'm not familiar with the GFlowNet literature and only have an understanding of MaxEnt RL, revealing the connection between value functions and flow functions in non-DAG scenarios seems novel and serves as an important step to bridge the research efforts along these two topics. The assumptions in the framework seem to be lenient enough to cover a wide range of environments in the literature.\n\nFor the reasons above, I kept my score to be positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and for a positive assessment of our work! We carefully address your concerns as follows:\n\n>Q1: Concerns about path invariance scaling factor $g$\n\nThanks for your question. We would like to emphasize that this condition is satisfied in a broad range of domains of GFlowNets literature, including most standard benchmarks and important application areas: 1) all tree-structured problems naturally satisfy this condition, e.g., sequence/language generation [1,3], red-teaming [2], and prompt optimization [4]; 2) non-tree structured DAGs where the number of parent and children states remains independent of the specific trajectory to the state, like the widely-studied set generation tasks[4], and real-world applications including feature selection[5], recommender systems[6], experimental design optimization[1], portfolio construction, etc. The key requirement is that the task can be modeled as a DAG where the ratio of children states or parent states remains consistent regardless of the specific path taken to reach the state. These examples demonstrate that the condition applies to a wide range of practical generative tasks within the GFlowNet framework.\n\nHowever, as noted in Section 4.1.2, we acknowledge that RPE's simplicity imposes limitations in certain environments, such as the toy HyperGrid task with boundary conditions. Yet, this simplicity is precisely what makes our findings both surprising and significant. Our work provides the first rigorous characterization of when GFlowNets and policy evaluation align, bridging a critical gap in understanding the relationship between (non-MaxEnt) RL and GFlowNets. This fundamental insight unlocks new possibilities and advances both fields by leveraging their complementary strengths.\n\nDespite being based on basic policy evaluation principles for a uniform policy, our method achieves performance comparable to many well-developed GFlowNet algorithms while maintaining remarkable simplicity across diverse environments, making it a valuable foundation for future exploration in GFlowNets and RL research.\n>Q2: Questions about training variance and stability\n\nThanks for your question. As demonstrated in Figures 3,4,5,6 in the main text, RPE exhibits both consistently improving performance and remarkably stable learning throughout the training process. It is noteworthy that RPE has a fundamental algorithmic design difference (advantage): it evaluates a fixed uniform policy $\\pi$, while both standard GFlowNets and MaxEnt RL algorithms need to estimate flows/values for continuously evolving policies. This distinctive approach helps RPE eliminate key sources of non-stationarity that can contribute to training instability.\n\nTo ensure statistical reliability, all experimental results are averaged across multiple random seeds following evaluation standards in previous works [7]. For the RNA1 generation task, `RPE demonstrates the lowest standard deviation among all baseline methods` (please see Table 1 in https://anonymous.4open.science/r/RPE-added-experiments-57A9). The reduced variance of RPE can be attributed to reduced complexity in the learning objective through fixed policy evaluation and simplified parameterization that learns only the flow function $F_{\\theta}$, different from privious state-of-the-art GFlowNets method like trajectory balance (TB) [7], which can incur high variance due to the reliance on Monte-Carlo estimation techniques.\n\n>Q3: Performance of RPE beyond training iterations 1e4.\n\nTo further evaluate RPE's sustainability beyond the standard training regime, we extend our experimental analysis by training for 1e5 iterations (10$\\times$) on the RNA1 task, and compare it against the most competitive algorithms from both Soft RL and GFlowNets paradigms, including Munchausen-DQN and TB in this task. The results presented in Fig.1 in https://anonymous.4open.science/r/RPE-added-experiments-57A9 demonstrate that RPE maintains its performance advantage throughout the extended training period, which consistently outperforms these top-performing baselines, validating its effectiveness as a stable and high-performing approach.\n\n**References**\n\n[1] Jain et. al., GFlowNets for AI-Driven Scientific Discovery, Digital Discovery, 2023\n\n[2] Lee et. al., Learning diverse attacks on large language models for robust red-teaming and safety tuning, ICLR 2025\n\n[3] Hu et. al., Amortizing intractable inference in large language models, ICLR 2024\n\n[4] Yun et. al.,Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation, CVPR 2025\n\n[4] Pan et. al., Better Training of GFlowNets with Local Credit and Incomplete Trajectories, ICML 2023\n\n[5] Ren et. al., ID-RDRL: a deep reinforcement learning-based feature selection intrusion detection model, Scientific Reports 2022\n\n[6] Liu et. al., Generative Flow Network for Listwise Recommendation, KDD 2023\n\n[7] Malkin et. al.,Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022", "claims_and_evidence": "Claims are well-supported.", "methods_and_evaluation_criteria": "* The approach is well-motivated, but it is unclear if RPE retains good performance in environments with non-uniform backward policies.", "theoretical_claims": "* Theoretical claims are sound.", "experimental_designs_or_analyses": "Empirical validations involve multiple standard benchmark environments. However, it's not shown how the asymptotic performance of the proposed algorithm compares to baselines, for RNA1-RNA4 (Fig. 5) beyond training iterations 1e4.", "relation_to_broader_scientific_literature": "Relation to the literature is well explained in the paper, by introducing a few works linking GFlowNets to MaxEnt RL and highlights the contribution of connecting GFlowNets with standard RL in this work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n* In the non-tree DAG case, requiring the scaling factor to be constant across paths to any state is a strong assumption that holds in the benchmark being considered but not necessarily beyond. This is discussed in Section 4.2 and does not decrease the value of the theoretical insights, but limits the applicability of the proposed algorithm in practice.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* How does the algorithm compare with baselines in terms of training variance and stability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXoqNxJqvO", "reviewer": "Reviewer_wvJJ", "summary": "The authors present a relationship between Generative Flow networks and RL via policy evaluation. Specifically, the authors claim that the value function obtained from evaluating a uniform policy is closely related to the flow function in GFlowNets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful feedback and positive assessment of our work, and for noting that our work builds a simple, practical, and novel algorithm that bridges RL and GFlowNets. We carefully address your concerns as follows:\n>Q1: Questions about Max Entropy RL baselines\n\nThanks for your insightful question! Our choice of Soft DQN and Munchausen DQN as the max-entropy RL baselines was guided by prior research [1], considering the discrete action spaces in our experimental settings. While SAC was originally designed for continuous control problems and would require discretization in our context, Soft DQN is naturally suited for discrete spaces, with Munchausen DQN representing an enhanced variant with documented performance advantages.\n    \nTo directly address the reviewer's concern about SAC, we conduct additional experiments incorporating SAC as a baseline on the RNA1 generation task, following the cleanRL's implementation. The results illustrated in Figure 2 in https://anonymous.4open.science/r/RPE-added-experiments-57A9 show that SAC achieves comparable performance to Soft DQN in terms of mode discovery but is less efficient in terms of accuracy, which is consistent to prior research [1] demonstrating that discrete SAC typically exhibits inferior performance compared to Soft DQN and Munchausen DQN in discrete action spaces for similar tasks. **Notably, our proposed RPE method achieves superior performance across all metrics against these established baselines.**\n\n[1] Tiapkin et. al., Generative Flow Networks as Entropy-Regularized RL, AISTATS 2024", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper broadly relates to solving sequential decision making problems with reinforcement learning and/or generative flow networks. Here the authors specifically investigate benchmarks where the solution space is multimodal i.e. there are many diverse solutions that achieve high reward since generative flow networks are well equipped to capture the multiple modalities.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "**Strengths**\n- The authors’ insight that evaluating a random policy approximately matches the flow function leads to a simple, practical, and novel algorithm that bridges RL and GFlowNets. \n- The proposed RPE algorithm empirically validates the claim that random policy evaluation matches the flow function with high accuracy results on various benchmarks. \n- The algorithm achieves SOTA results on discovering diverse solutions while maintaining high accuracy when compared to both GFlowNets and max-entropy RL. \n\n**Weaknesses**\n- The Max Entropy RL baselines, specifically DQN, are not state of the art. Why was the proposed method not compared against a more recent method like Soft Actor-Critic (SAC)?", "other_comments_or_suggestions": "n/a", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m4VrYfocfx", "reviewer": "Reviewer_trFY", "summary": "This paper studies generative flow networks (GFlowNet) where one aims to learn a policy which generates states with probability proportional to the rewards of these states, contrast with the goal of reward maximization in typical reinforcement learning. This work tries to build the connection between value function and flow function in this framework. By transforming the reward, they can show an equality between a scaled value function and the flow function. With this finding, they interpret GFlowNet from the perspective of value iteration. What's more, they implemented experiments to show this connection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the time and effort in the feedback. We would like to clarify the novelty and significance of our contributions as follows:\n>Q1: Concerns regarding insights: Theorem 4.1 didn't bring anything new about the forward policy.\n\nWhile the reward transformation is a key component in our algorithm, our work’s novelty and impact extend far beyond this step. The focus of our work is not to connect the 'value functions and flow functions in the GFlowNet', but to address a critical gap in the literature by bridging **GFlowNets** and **standard (non-MaxEnt) RL**, an underexplored and overlooked connection that holds broad implications for both theory and practice. Unlike previous works that necessitate flow-matching or entropy regularization [1,2,3], we demonstrate that a GFlowNet forward policy with reward-matching properties can emerge naturally from standard RL components like policy evaluation.\n\nThe core of GFlowNets' research is not about defining forward policies in different formulations ($\\pi(x) \\propto R(x)$), but rather developing effective methods to learn these policies that sample proportionally to rewards. Proposition 1 in [1] explains why MaxEnt RL (Buesing et al., 2019 Haarnoja et al., 2017) are biased by the number of trajectories leading to each terminal state in non-injective environments (i.e., $\\pi(x)=\\frac{n(x)R(x)}{\\sum_{s' \\in X}n(x')R(x')}$). In contrast, our Theorems 4.1-4.2 provide distinctive contributions, serving as the foundation of RPE to demonstrate how a standard RL component (policy evaluation) can surprisingly achieve reward-matching properties without requiring specialized and sophisticated flow-matching objectives or entropy regularization. This fundamentally redefines how GFlowNets can be understood and implemented.\n\n- **Novel theoretical contributions**: We uncover an unexpected bridge between standard (non-MaxEnt) RL and GFlowNets, showing that a basic component in RL, i.e., policy evaluation, can naturally achieve the same reward-matching property as in GFlowNets with minimal modifications. **Our work is not a trivial reformulation or mere reward transformation, but offers a fundamentally new insight into the capabilities of standard RL components**. Our analysis demonstrates that GFlowNets’ core mechanisms can be reinterpreted through the lens of random policy evaluation in standard RL. This reshapes our understanding of GFlowNets’ learning dynamics, resolving an open question in the field.\n- **Algorithmic contributions**: Based on our main theoretical results in Theorems 4.1-4.2, we develop a simple yet effective RPE algorithm using only policy evaluation, which eliminates the need for complex flow-matching conditions [1,2,3]. Despite its simplicity, our extensive experiments show that RPE matches or exceeds the performance of well-established GFlowNet methods [1,2,3], validating both our theoretical insights and practical utility.\n- **Expanding understanding**: Our work provides novel insights into how reward-matching properties can naturally emerge from policy evaluation of a random policy under appropriate reward transformation. This expands the important understanding that was overlooked and opens up promising directions for future research at the intersection of RL and GFlowNets.\n\nIn summary, our work makes substantial contributions by revealing a novel, simpler way to achieve reward-matching policies in GFlowNets through standard RL components under appropriate conditions.\n\n[1] Bengio et. al., Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation, NeurIPS 2021\n\n[2] Bengio et. al., GFlowNet Foundations, JMLR 2023.\n\n[3] Malkin et. al., Trajectory balance: Improved credit assignment in GFlowNets, NeurIPS 2022\n>Q2: Concerns regarding the assumption in Theorem 4.2\n  \nThanks for your question. Please refer to our response to Reviewer ghFJ's Q1 for a detailed discussion due to space limitations.\n>Q3: Questions about Appendix A.\n\nThanks for your question. We will elaborate on more details in Appendix A in the revision to explain the connections between the flow/value functions in GFlowNets, MaxEnt RL, and our RPE approach.\n>Q4: About proof for Theorem 4.1\n\nThanks for your question. We clarified in Appendix B (lines 674-675) that Theorem 4.1 represents a special case of the more general Theorem 4.2. For the sake of conciseness and to avoid redundancy in the presentation, we chose to omit the explicit proof of Theorem 4.1 in the appendix (the proof can be readily derived by following the same logical framework presented in the proof of Theorem 4.2, with $B(s_{i+1}) = 1$ and there is exists exactly one path from $s_0$ to any state). To ensure completeness and facilitate better understanding, we will include the proof for this special case for Theorem 4.2 in the revision.\n\nWe hope these clarifications effectively address your concerns and highlight our key contributions. We welcome any further questions or discussions.", "claims_and_evidence": "Yes, the claims made in this paper are supported by the main theorems (Theorem 4.1 and Theorem 4.2).", "methods_and_evaluation_criteria": "It is not applicable to this paper. The focus of this paper is to establish the connection between two existing concept (or method), value function in value iteration and flow function in GFlowNet.", "theoretical_claims": "Yes, I checked the correctness of the main theorems in this work, i.e. Theorem 4.1 and 4.2. They seem correct to me.", "experimental_designs_or_analyses": "No, I didn't check the soundness of the experiments.", "relation_to_broader_scientific_literature": "The key contribution in this work is to build the connection between the value function and flow function in GFlowNet. The most related result in previous works is Proposition 1 in [1]. This work provides more straightforward interpretation between value function and flow function beyond that result.\n\n[1]. Bengio E, Jain M, Korablyov M, et al. Flow network based generative models for non-iterative diverse candidate generation[J]. Advances in Neural Information Processing Systems, 2021, 34: 27381-27394.", "essential_references_not_discussed": "No, I didn't see any essential related works that are not discussed.", "other_strengths_and_weaknesses": "Strength:\n1. To some extent, this work shows potential results that policy evaluation can achieve beyond the evaluation itself or serving for policy improvement.\n\nWeaknesses:\n1. The theory developed in this paper provides very limited new insights compared with the existing results. For example, in Theorem 4.1 (the main theorem), they build the connection $V(s_t)=F(s_t)\\Pi_{i=0}^{t-1}A(s_i)$ by transforming rewards. However, with this flow function, one can derive the forward policy $\\pi(s'|s)=\\frac{V(s')}{\\sum_{s''}V(s'')}$ where $s''$ denotes all possible subsequent state from $s$. It is a known result already established in Proposition 1 in [1]. What's new here is just transforming the reward such that one can directly write  the flow function to the value function. It is not significant enough to serve as main contributions for a paper, let alone in GFlowNet, people care more about the forward policy, Theorem 4.1 didn't bring anything new about the forward policy.\n2. In Theorem 4.2, the constraint that any trajectories $\\tau_1$ and $\\tau_2$ that visit $s_t$ should satisfy $g(\\tau_1, s_t)=g(\\tau_2,s_t)$ is too strong, which makes the interpretation only applicable to very limited graphs. The authors should dig deeper into it.\n\n[1]. Bengio E, Jain M, Korablyov M, et al. Flow network based generative models for non-iterative diverse candidate generation[J]. Advances in Neural Information Processing Systems, 2021, 34: 27381-27394.", "other_comments_or_suggestions": "1. In Appendix A, the authors include an illustration to show the connection of value function and flow functions, but they only give a figure title, it would be better to at least elaborate it.\n2. Missing proof for Theorem 4.1 in Appendix.", "questions_for_authors": "My main concerns are listed in the weaknesses part, at the moment I don't have questions towards the understanding of this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper brings theoretical and algorithmic contributions. Notably, it establishes a connection between the value function of a random policy and the flow function of a GFlowNet which is novel. From this, they propose a new algorithm that they test on non-standard RL benchmarks (DNA and RNA sequence generation) 

The strength of the paper is to create bridges between standard RL and GFlowNets which also makes it uneasy to review as knowledge in both fields are needed. Yet, the reviewers appreciated that the paper convincingly creates connections between the fields. The experiments seem to support the claim that the algorithmic contribution is real and outperforms other methods on the benchmarks.

Yet, some reviewers have raised concerns about the validity of hypotheses beyond special cases. The discussion didn't completely resolve these issues and the authors should take this into account in a revised version of the paper to adapt the claims.
Decision: Accept (poster)