Evaluating GFlowNet from partial episodes for stable and flexible policy-based training
Authors: Puhua Niu
Shili Wu
Xiaoning Qian
Venue: ICLR 2026 Poster (2026)
Keywords: Probabilistic Inference
Variational Inference
Reinforcement Learning
Combinatorial Optimization
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Generative Flow Networks (GFlowNets) were developed to learn policies for efficiently sampling combinatorial candidates by interpreting their generative processes as trajectories in directed acyclic graphs. In the value-based training workflow, the objective is to enforce the balance over partial episodes between the flows of the learned policy and the estimated flows of the desired policy, implicitly encouraging policy divergence minimization. The policy-based strategy alternates between estimating the policy divergence and updating the policy, but reliable estimation of the divergence under directed acyclic graphs remains a major challenge. This work bridges the two perspectives by showing that flow balance also yields a principled policy evaluator that measures the divergence, and an evaluation balance objective over partial episodes is proposed for learning the evaluator.  As demonstrated on both synthetic and real-world tasks, evaluation balance not only strengthens the reliability of policy-based training but also broadens its flexibility by seamlessly supporting parameterized backward policies and enabling the integration of offline data-collection techniques. Our code is available at [github.com/niupuhua1234/Sub-EB](https://github.com/niupuhua1234/Sub-EB).

Published as a conference paper at ICLR 2026
EVALUATING GFLOWNET FROM PARTIAL EPISODES FOR STABLE AND FLEXIBLE POLICY-BASED TRAINING
Puhua Niu
Shili Wu
Texas A&M University College Station, TX 77843, USA niupuhuu. 1238 @ .emuuddu
Texas A&M University College Station, TX 77843, USA shi Whiliwwetamu.co
Xiaoning Qian
Texas A&M University College Station, TX 77843, USA Brookhaven National Laboratory Upton, NY 11973, USA xqian@t amu. edu
ABSTRACT
Generative Flow Networks (GFlowNets) were developed to learn policies for ef- ficiently sampling combinatorial candidates by interpreting their generative pro- cesses as trajectories in directed acyclic graphs. In the value-based training work- flow, the objective is enforce the balance over partial episodes between the flows of the learned policy and the estimated flows of the desired policy, im- plicitly encouraging policy divergence minimization. The policy-based strategy alternates between estimating the policy divergence and updating the policy, but reliable estimation of the divergence under directed acyclic graphs remains a ma- jor challenge. This work bridges the two perspectives by showing that flow bal- ance also yields a principled policy evaluator that measures the divergence, and evaluation balance objective evaluator. As demonstrated on both synthetic and real-world tasks, evaluation bal- ance not only strengthens the reliability of policy-based training but also broadens its flexibility by seamlessly supporting parameterized backward policies and en- abling the integration of offline data-collection techniques. Our code is available at github. gitggb...o//niigee com/niupuhual htt.aauuuuuuuuuuuuuubaaaa 233//ubb-BB
partial episodes is proposed for learning the
INTRODUCTION
Generative Flow Networks (GFlowNets) are generative models on combinatorial spaces X, such as graphs formed organizing nodes and edges in particular way, strings composed of alphabets in a specific ordering. GFlowNets aim at sampling x E X' with probability x R(x), where R(x)
non-negative score function. The task challenging as |X can be too large to compute the normalization constant Z := Σzex R(x) and the distribution modes can be highly isolated due the combinatorial nature for efficient exploration. In GFlowNets (Bengio et al., 202T 2023). generating or sampling null state, pass through intermediate states, and end at x as the desired terminating state. These trajectories E T can be viewed the paths along Directed Acyclic Graph (DAG) with state S and edge (s - 8') E E. Positive measures (unnormalized probability) of trajectories are viewed as the amount of flows along the DAG, and R(z) is the total flow of trajectories ending at x, so that sampled trajectories will end at r with the probability x R(r).
X' is decomposed into incremental trajectories (episodes) that start from
The core of the GFlowNet training problem can be understood minimizing the discrepancy of the forward trajectory distribution (T( induced by a forward policy (s'sss)) towards the backward trajectory distribution Po(r) Pp(T x: R x - induced by given backward policy T (s|a ss) and R(x) (Bengio et al. 2023 Malkin et al. 2022b). This is motivated by the fact that in real-world applications, sequential generation must be done in forward manner. Besides, the marginaliza-

---

Published as a conference paper at ICLR 2026
Si
X
X
Figure 1: graphical illustration of DAG (left) and its graded version (right). Dotted circles represent dummy states, added during the conversion to graded DAG.
tion (z)x) = E+ PT(T) trivially holds for all x E X, so any backward policy can be used to define a target backward trajectory distribution. Since evaluating Z' and thereby the normal- ized distribution (T( is considered intractable, directly optimizing some distributional divergence of (F(T) and (T(T) is not feasible. To circumvent this, value-based methods reformulate the problem of distributional matching as a flow- www-matching problem and leverage the balance condi- tions of flow values to derive training objectives. The basic condition Br) [log FF (s'sss F()) =
(s) [log Fs(s) means that the edge flow value F(s - s') T ( F()) matches the target edge flow value in the reverse direction, where Pp(s) is the marginal state distribution induced by the data collection policy (s(s'ss). This basic condition leads various training objectives that implicitly encourage the alignment between forward and backward distributions. These objectives variants, providing diverse estimations of
range from edge-wise formulations to subtrajectory-wiit
target flows and quantifications of flow imbalances (Bengio et al. 2023] Madan et al. 2023 Malkin cct al.. 2022a). Alternatively, policy- methods (Malkin et al.) 2022b Niu et al. | 2024; Zim-
mermann al. 2022) introduce evaluation function V(s) of the forward policy TF (r(ss) which
approximates the Kullback-Leibler (KL) divergence between the distribution forward subtrajec- tories (partial episodes) starting from state and that of backward subtrajectories ending Then (s)) is used to update the forward policy =-(ss). While policy updating based on (ss) has been well-studied, how reliably learn (ss) remains open problem.
In light of the success of value-based methods that leverage flow balance conditions and the anal- ogy between the state flow function Fs)) as the total amount of trajectory flows passing state s, and the evaluation function (ss as a measurement of distributional divergence at state it is worthwhile to investigate the relationship between the two quantities. It is known that the optimal forward policy and state flow function can be uniquely determined by the flow balance condition Exp [log F = Exp [log ( We find that, for any fixed RT, the solution of the state flow to this condition coincides with the exact KL divergence starting from state that is, the true evaluation function of RF. This paves the way to derive balance-base objectives to learn (s) reliably. Our contributions as follows:
establish the connection between balance conditions with respect to (w.rt.) the state flow function F and the evaluation function V. Fixing TF, the expected balancing conditions of log F directly lead to a sufficient condition for V, which we call the Subtrajcctory Evaluation Balance (Sub-EB) condition.
We introduce the Sub-EB objective for reliably learning the evaluation function V, where sub- trajectories (partial episodes) serve the basic unit of flow balance.
Experimental results on both simulated and real-world datasets, including hypergrid model- ing, biological and molecular sequence design, and Bayesian network structure learning, have demonstrated the effectiveness and reliability of Sub-EB for policy evaluation.
PRELIMINARIES
We restrict DAGs of GFlowNets to be graded as any DAG can be equivalently converted to be graded by adding dummy non-terminating states (Malkin et al.) 2022b). In DAG G (S, E), element e denotes a state, and element (s + 8') E ETS S x S) denotes a directed edge. We define the index set of time horizons [ (O...., .,. H). Being acyelic means the state space S
2

---

Published as a conference paper at ICLR 2026
can be partitioned into disjoint subspaces: Su... ........ Su....,SH+1. Si++. where each element of S,, is denoted as Sh for à E [ + 1]. Being graded further implies that actions are only allowed from S₁, to Sh+1- The initial and final states are unique, that is, So { and SH+1 {sy}. The terminating state set, Sh := X' is associated with a score function R X - IR+t Furthermore, the (complete) trajectory set is defined T == (T (so
8f) |V (+++8') E (8->') E E). For < j E [H + 1], Tad. Ts: and T. denote subtrajectories (partial episodes) from s; to 8., from s; to so, and from to 8j, respectively. Denoting the natural o-algebra on T as (T), P(T)=
Ter P(T) for any event
I' T (T). An exemplary DAG and its graded version are shown in Fig.
GFlowNet training aims to align the forward trajectory distribution Pr(r): := II (1++11))55 with the backward trajectory distribution PB(T) := II=-0 (B( where TH and Tg are for- ward and backward Markovian policies (transition distributions), respectively, and are assumed to satisfy PF(T) > and (r( > 0 for any т T. Accordingly, Pr (r(Tr_((s) = II
TF (Bn++1(sA)) and (T(Ti_ls;) = II= - BB (sa Except that (x)s)))) == R(z)/Z" for any x E X, (s(sss") can be arbitrary for intermediate edges, as (b(x)) = - PB(T)= R(z)/Z* always holds. There- fore, the optimal forward policy Tpp. satisfying (T) PB(T) for all r E T, induces its marginal Pp- R(x:)/",, thereby achieving the ultimate goal of GFlowNet. Moreover, under the Marko- vian assumption (Bengio et al. 2023), the optimal forward policy प corresponding to TT is unique.
2.1 VALUE-BASED TRAINING
Traditional Variational Inference (VI) approaches that minimize the KL divergence LKL (0) = PkI((P(Ts P)] (T())) is infeasible as 2" is considered intractable (Malkin et al., 2022b). To circumvent the intractable Z", value-based methods introduce an unnormalized and Markovian dis- tribution F(r) := (r(r) Z. called (trajectory) flow (Bengio et al. 2023), together with total flow Z. Accordingly, for any event I E (T), its flow F(T) == E.er P(T). In particular, the edge flow and state flow are fs + 8') = Er F(T) and F(3) == Σ. F(s + 8'). Further- more, F(00) = F(s)) = Z, and Ty (s'ls) F(s (()/F(s))(()) Analogously, the optimal flow is (T) == PB(T) = R(()))) Therefore, the optimality condition of GFlowNets is con- verted to VT E T F(T) = (T). Thanks to the Markovian assumption, the condition can be relaxed and reformulated
the Sub- Tua-Trajcctory Balance (Sub-TB) condition (Madan al. 2023).
requires vi <j E [H + 1]:
Epo(re)) [log (F(s:) (F(s)))((Te -PPo(r))) [log (F(s))P((Te)lss))
(1)
where ((s)))(( R(x) for E X, and PD(TE.) Ersr., Pb(T) is induced by the assumed that PD(T) > 0 for any E T Malkin et al,, 20226). Without the assumption, stricter alternative is to enforce flow balance for each individual subtrajectory, as in Madan et al. (2023). The condition above can be interpreted as the flow value of a subtrajectory should match the target flow value in the reverse direction, which represents an (approximated) desired flow value.
offline data-collection policy TD.
Leveraging the balance condition, value-based methods typically use the Sub-TB objective to op- timize the parameterized policy (s' s; 0) and state flow F(s; 0) toward their optimal solutions (s'ss) and (s). The objective is defined as follows:
(G) = Epp(r)
Σ (f 8F(Te5;0) 0) := log
Pr (Teg 86 F(; 0) PB (Trs sy; ()
(2)
Tiss
where Wi-i denotes the non-zero weight coefficient for subtrajectories that consist of j
i edges. For practical suddient-based optimization, (G) are approximated by (f) =
En Emser W (] where D (T" = P k== is the set of samples.
K LED
2.2 POLICY-BASED TRAINING
Policy- based training methods for GFlowNets (Niu et al. 2024) resemble the policy gradient al- gorithm in Reinforcement Learning (RL) and aim to minimize the KL divergence (k)(0) in traditional VI approaches. The methods follow the actor-critic framework (Sutton & Barto 2018
Haarnoja et al. 2018).
3

---

Published as a conference paper at ICLR 2026
Critic In each training iteration, the critic (true evaluation function) Vt of actor Te is first com- puted to capture the policy gaps in terms of KL divergences over subtrajectories, thereby facilitating the computation of the overall trajectory divergence. For any h [H], (s,; 8) is defined as
(pr(ni ()) log (B(TT:)
F* (8h) DKL (T sa: BB(::
PP(TA, (s;0))
The equality be easily verified (Niu al. 2024 shown in 19 in the Appendix. It should be noted that the explicit computation of 2" is not required the above definition as T (x)())) :=
R(x). Analytical computation of Vt is usually infeasible as [7] can be enormous, so Vt modeled by an evaluation function V with learnable parameters. We will discuss the learning objective of V in the next section.
Actor To minimize L (0), noted that Vo VoVi(so: 0) - -VLKs.(0) since (=0) Z*) is constant. Consequently, -V' (sn: 0) can be surrogate to the KL divergence. Further applying the policy gradient theorems in RL (Agarwal et al. 2021). Vso: 0) can be simplified and computed
in terms of V as follows:
H Σ h=0
(sn > 78 Vo log (s6++ 110::=+11000000000 0].
Epp())
(4)
where dy (s) == log xa Vxpp
ex( with (B(()))) exp V(sy) exp
= R(x), -hh+
H! Di 7' 7² -v 81+1), and the hyperparameter y [0, 1] enables the variance-bias trade-off
during stochastic gradient descent (Schulman et al. 2016 2017). analogous to wo in the Sub-TB method.
SUBTRAJECTORY EVALUATION BALANCE
As discussed in the previous sections, the evaluation function V plays central role in policy-based methods. In this section, we first introduce the Sub-EB condition that characterizes the relationship between TF and V, which closely resembles the Sub-TB conditions between TF and F. We then present the Sub-EB objective for learning V. Further, we extend these results with corresponding conditions and objectives for Th and the backward evaluation function W.
3.1 THE BALANCE CONDITION AND OBJECTIVE FOR FORWARD POLICIES
Given forward policy such that (>(r) > 0 for all E T, the Subtrajectory Evaluation Balance (Sub-EB) condition for the associated evaluation function V can be written as Vi < E [H + 1]: (5)
Epo(Teg) log (PF(Tusl:) exp V (s:)) = (pr((og) log (Pa(Te)ls) exp V (ss))),
where P exp V(8,) = R(x)) for x E X. The condition can be reformulated as: (p)())))) (s:)) - V(s))] = (Pp() [log Pa(tiyle)) PF(Tx:)
H(P)(s:)) H(Pr(s;))
(P)(ee))DLL (Pp(T. (T(r,l,))) where H(.) denotes the entropy of a distribution. There- fore, this condition intuitively requires that the difference of the learned divergences at s; and sj matches the true divergence over subtrajectories between the two states.
Theorem 3.1. Suppose V is an arbitrary evaluation function over S, and F* is the optimal flow induced by a backward policy тв. Given forward policy TF,
vh E [H] (ss) log F* (sn) -
(6)
if and only V satisfies the Sub-EB condition 5
The corresponding proof can be found in Appendix A.1
Theorem 3.2. Suppose is an arbitrary state flow function over S, ue is an arbitrary forward policy, and (F*, .T)) ) is the optimal flow and forward policy induced by a backward policy πg. Then, Vh E [M] log F(sx) -log F* (sn) ) DrL. (P7
(P (Th. [s))) (Th)lsn)
and KF T if and only if (F, TF) satisfies the Sub-TB condition 7
Niu et al. (2024) consider additional total flow estimator 7(0) to scale down the magnitude of Vi, For notational compactness, we defer the discussion of integrating Z into method Appendix A.4
4

---

Published as a conference paper at ICLR 2026
The corresponding proofs can be found in Appendix A.2 While the sufficiency and necessity of the Sub- TB condition have been studied in prior work (Bengio et al. 2023 Malkin et al. 2022a), Theorem 3.2 offers an alternative perspective that more clearly elucidates the connection between the flow function and the evaluation function. It should be noted that the minimum of the KI. term above is zero, as both Pr(r) and (T( are Markovian. Moreover, by Proposition 23 in Bengio et al. (2023). the trajectory flow F(7) = (T) that achieves the zero KL term is unique.
Leveraging the balancing condition, we define the Sub-EB objective for optimizing a parameterized () as:
evaluation function
(δv (T(()):))) (T()):: =log Pr(Tegls;) exp V(s;; Φ) PP(TEplssjo b) exp (sy;
(0) = (r)(r)
E "
(8)
Tup
where wy i the weight for sub-trajectories of i edges, as in the Sub-TB objective. While the traditional Caamporal-DDifereenee (TD) objective (52) detailed in Appendix A.5 (Niu et al. 2024 Schulman et al. 2016) focuses on learning V( Φ) only from events starting at step h and edge- wise mismatches (s + s'), the Sub-EB objective incorporates events both before and after h and leverages subtrajectory-wiss mismatches, yielding more balanced learning of V(sni o). Moreover, Sub-EB allows freely weighting schemes, whereas the scheme of X-TD is restricted to the X-decay form. A detailed comparison between our Sub-EB and X-TD objectives is provided in Appendix A.5
It should be noted that the Sub-EB objective is specifically designed for estimating V of xr. In each optimization iteration, its gradient w.r.t. 6 is computed to update V(::Ø), while is frozen. In contrast, the Sub-TB objective 2 is used to jointly update (-(-1.;0) and log F): marize the workflow of our policy-based method for GFlowNet training in Algorithm
P We sum- where
(o) == K ErED Enger Wi wj-rdv (Tays +)] and VoVi(so: 0) = K E 8h+ 1) V log (s 1115:+++11s;0;;
k=
A7 (8h
3.2 PARAMETERIZED BACKWARD POLICY
In this and the following sections, we further discuss two key advantages introduced by the Sub-EB condition, which provide the improved flexibility of policy-based training. The A-TD objective (52) requires Tg to remain fixed throughout optimization, as V is treated as constant w.r.t. Φ. To ad- dress this limitation, Niu et al. (2024) proposed two-phase algorithm, where each training iteration includes a forward phase and a backward phase. In the forward phase, we sample D PF(T), update V based on D. and update FF based on V and D. In the backward phase, we sample D' ~ (x()))))x<())) update the backward evaluation function W that approximates the true back- ward evaluation function Wi of TB. and update TB based on W. The definitions of Wt and W will be detailed in Section 3.3 In parallel, Grritsav al., Jang et al. (2024) introduced an addi- tional objective for TB within the forward phase. When applied to the policy-based framework, their approach first samples batch D ~ PF(r) and updates Bh by maximizing its log- log-likelhhood, Erep log (T). Then, with TB held fixed, V and FF are updated using the same batch D, follow- ing the procedure described above.
In contrast, both the Sub-TB (2) and Sub-EB objectives allow for updating parameterized n without introdueing a separate backward phase or an additional objective. To be more specific, TB is jointly updated with V under the Sub-EB objective. and with (TF, log F) under the Sub- TB objective. This leads to a more streamlined and efficient training process while enabling the backward policy to adapt dynamically during optimization. Moreover, beyond basic volicy-gradient methods, the Sub-EB objective also enables a parameterized TB within more advanced policy-based methods, such the Trust-Region Policy Optimization (TRPO) method (Niu et al. 2024), where TB was previously required to remain fixed under the X-TD objective.
3.3 OFFLINE POLICY-BASED TRAINING
Both the single-phase and two-phase algorithms by Niu et al. (2024) operate in online manner, where we can not use a data-collection policy TD * TT during training. To overcome this limita- tion, we introduce an offline policy-based method made possible by the flexibility of the Sub-EB
Herr, 0 and & are introduced as separate parameter sets. 0 corresponds to functions updated jointly with TF. while @ corresponds to functions that not.
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Offline Folicy-based Workflow
Algorithm 1 Online Policy-based Workflow
Require: -B(-1-:0 Ø), Te(-1.:0) W(::6), of- fline policy TD. K. N for 11 l,.... do
Require: [=).):;)) batch size K, number of total iterations N
V(+:Ø),
for ,..., N. do
D - (xt x Ett Ih ~ PD(T)]
D < [T 7 Pr(T))
D +
ED ED,,
Based on D, update & by VoLv(ø)
Based on D. update @ by VeLw(0)
Based on D and V, update 0 by
Based on D and W, update Φ by
-V;V'
-VpEp (o)()) (Wi
end for return TF(-1:;0), Nb( (=(1)=)) V(:
end for return TB FB(-1:0), TF (2:11::0....0 ((::0)
objective. We define w (so) log Fso). For any h e [4]) w (sy: o) is defined as:
E (r((alsa)))) log PF(TA Fo)
PB shi Φ) =log F(s) DKL (PB(((T))))) PF
The equality can be easily verified, shown in (22) in the Appendix. It is known that minimiz- ing E.p(w) (
:= -pp((e) DKL. (T (T x;; Φ) ( [r])] also reduces the discrepancy be- tween PB(T) and (r( (Niu et al. 2024 Since V log F F(z) = 0 and -V -VpBpp(a) [w Φ) = V VolEo..) [oxz(Ølx)] [))x)]. it follows that (rr) [w o)] can be surrogate to the expected KL divergence. In analogy to the forward case, we use a parameterized evaluation function W to ap-
proximate Wi, and V (x)E
(W:x;; (
is computed as follows:
H-1
Epp Bpg(r)
Σ h=0
As Vogg
(10)
where (s + 8²) = log (xpp' Pp(T))= exp
(sh sh+1) E Bw 81+1) and
Given backward policy ma, the Sub-EB condition for W can be written as Vi < [H 1]: (11)
Epg(rag) log (Pr(Tey exp ((s))) = =Epg (res)
log (Pa(Te))s)) exp W (s,))
where Pxx W(s)) == R(x) for x X.
The condition can be reformulated as requiring learned divergence difference
(r( (pg(s,)) (s.,s) (s;),) - Ws,)) = (ros)))) log Pa(roals)) at 8; and sj match the true divergence between them.
Theorem 3.3. Suppose W is arbitrary evaluation function over S SM{ss,, and F is the flow associated with a forward policy ny. Given backward policy aB,
vh H 11] ((s+11) log F(sh+1)
(12)
and (xx) = log R(x) if and only if W satisfies the backward Sub-EB condition
The corresponding proof can be found in Appendix A.3 Based on the backward Sub-EB condition,
we present the backward Sub-EB objective for W ((::6 I) as follows:
(::8)
PF (T( ((;:) (13) 0)
Lw (o()):=Epg(r)
Σ wy (ow(T: 6w (Tay: =log Tai
(T exp (sy;; (s);0)
When TB and Fr are at their optima, the KI. term in the expression of W is zero. Consequently, F(x) R(x) and (x(x) = F(x)/ Σ, F(x) R(x)/Z", thereby fulfilling the goal of GFlowNet training. The workflow of our offline policy-based method for GFlowNet training is presented in
Algorithm 2 where (0)
DETEDLETTO wy_rōw (Teg; 0)] and (oErp(e))Wii (2;
R EEEDle 2 (7 V log

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
Value-based GFlowNet Training Existing works on value-based GFlowNet training can be cat- egorized into two directions. The first one focuses designing training objectives to character- ize target flow values, thereby improving the estimation of flow imbalance. The Detailed Balance (DB) objective (Bengio et al, 2023) aims to reduce the logarithmic mismatch between the for- ward edge flow (F(8 (8) (s)) and the hackward target edge flow TB ( ) ((s",, respectively. The target term F(s")PB((s)
provides biased but low-variance feedback, as F encodes the learned partial knowledge about the task. The Trajectory Balance (TB) objective (Malkin et al. 2022a)
optimizes the logarithmic mismatch between the forward trajectory flow (T( and the true back- ward trajectory flow R(() = F (T), which yields unbiased but high-variane, feedback.
The Sub-TB objective (Madan et al. 2023> generalizes both DB and TB by minimizing the flow logarithm mismatch of subtrajectories of varying lengths. By reweighting subtrajectories according to their lengths, it enables a bias-variana trade-off, thereby achieving better performance. Build- ing on Sub-TB objectives. various improvements are also proposed. Kim al. (2023a) introduced cemperature-condiiee
objectives, whose goal turns to make P=(x) x R (z) with positive scalar 3 < 1. Taking R the exponent /3 reduces its sharpness, making it easier match. As (rr(x: does not match R(x) in this case, this representation is specifically useful when the focus is solely on mode seeking. For all the objectives mentioned above, B (s(s') can be chosen freely for any intermediate edges (s + s') and only π T )) is fixed to R(xx). Therefore, target flow values of intermediate edges or subtrajectories do not directly reflect the ground-truth knowledge about the reward function R. However, under the special cases where the covered object space of R can be extended from X' to S. Pann et al. (2023) and Jang et al. (2023) improved the formulation of the DB and Sub-TB objectives, propagating partial knowledge of R directly to intermediate edges. Due to the similarity of the Sub-TB objective and our Sub-EB objective, these improvements can also be easily adapted to Sub-EB to facilitate learning V.
A key characteristic of value-based methods is that Tp can be off-policy, that is, different from IF. leading to numerous efforts at designing TD (Kim et al.) 2023b Rector-Brooks et al. 2023). The goal is to effectively identify edges that precede highly-rewarde while allowing revisiting the edges already found to yield high reward (exploitation). The most widely used approach is a-greedy design that mixes Fe with a uniform policy by a factor a. These efforts, however, fail achieve deep exploration, which requires considering not only immediate information gain but also the long-term consequences of a transition (edge) in future learning (Os-) band et al., 2019). Although there have been many theoretical advances in efficient exploration design from an RL perspective (Azar et al. 20177] Jin et al.) 2018: Ménard et al. 2021), their expen- sive computational cost limits their applicability in practical problems.
terminating states (exploration)
Policy-based GFlowNet Training As mentioned above, designing data-collection policy that both computationally efficient and capable of deep exploration remains a significant chal- lenge. Moreover, what is truly needed for training efficiency is identifying the edges of high flow-imbalance rather than the edges that lead to high terminating rewards. However, the flow imbalance is closely related to F and changes whenever TTp is updated. Empirical evidence shows that on-policy training, meaning TD = TF. can result in faster convergence under many condi- tions (Atanackovic & Bengio 2024). Accordingly, policy-based GFlowNet methods (Malkin al. 20226, Niu et al.) 2024) Zimmermann et al., 2022) conduct on-policy training. which typically corresponds optimizing the KL divergence between PP(T) and the unnormalized distribution Pn(r)=)((x)( (*), which has gradient equivalence to the divergence between Pp and (T((). Removing the need to design data-collection policy, the main challenge of posicy-based methods shifts to robust estimation of the divergence and its gradients. that is, balancing the trade-off between variance and bias of the estimators. Malkin al. (2022b) and Silva et al. (2024) construct estimators empirically and require fixed kB. During gradient- based optimization, these estimators are computed solely from the training data sampled in the current iteration. These estimators typi- cally exhibit low bias but high variance. From the theory of policy gradients in RL. et al. (2024) proposed estimators on a parameterized evaluation function V. which enables leveraging sampled data from all previous iterations. These estimators generally have high bias but low variance. Com- bining empirical and parameterized approaches, NNu et al. (2024) introduced a hyperparameter to control variance-bias trade-off explicitly, resulting in significant performance gains. As the ef- fectiveness of these policy-based methods critically depends on how V is learned, our paper is a subsequent work towards this challenge.
7

---

Published as a conference paper at ICLR 2026
GFlowNet training and RL Value-based methods such as DB and Sub-TB, well as RL RL-inspired variants of DB, including Munchausen DQN (denoted as Q-Much) (Tiapkin et al.. 2024) and Rec- tified Flow Iteration (RFI) (He et al. 2025). follow the Q-learning framework (Haamojj et al. 2017]. In contrast, policy-based methods, based on policy gradients, operate within the actor-critic framework (Sutton & Barto [2018] HHaarojj al., 2018). A detailed discussion of the theoretical distinctions between these two frameworks is provided in Appendix [A.6] To further illustrate how the Sub-EB objective and policy gradients operate within the actor-critic framework, we derive the basic soft actor critic algorithm (Alg. 3) for GFlowNet, together with a modified variant (Alg. in Appendix A.7
EXPERIMENTS
We consider tree policy- based methods: the empirical method of Silva et al. (2024), which uses the Control Variate (CV) technique for variance reduction during gradient estimation, and two policy- gradient methods (Niu et al. 2024) that learn V by either the X-TD objective or the proposed Sub-EB objective. We refer to these methods as CV, RL, and Sub-EB, respectively. Since Sub-TB (Madan al. 2023 (Tapkin et al., 2024) is closely related Sub-EB, we also include as represen- tative baselines for value- based methods. Among RI-inspired malue-based variants, we consider Q-Much (Tiapkin et al., 2024) and RFI (He et al.) 2025). However, due to potential computational scalability issues with RFI (see Appendix(A.6) we include only Q-Much in our benchmarking exper- iments. To design the data collection policy TD in the Sub-TB method, we follow a common choice, where TD is equal to VF with probability (1 a)) and a uniform policy with probability a (Shen ct al. 2023 Rector-Brooks et al. 2023). The common settings of ny can be either a uniform policy or parameterized policy. By default, adopt the uniform policy as the parameterized policy does not carry ground-truth information about the reward function. We also augmented Sub-TT T'B and the offiine Sub-EB (Algorithm{) with the local search technique (Kim et al. 2023b) for designing PD, to explicitly promote the exploration of high-reward terminating states during trajectory sampling. These variants are denoted Sub-TB-B and Sub-EB-B. As explained in Section 4 and confirmed by the following experiment results, off-policy techniques that explicitly encourage exploration may not benefit distribution modeling. However, when the focus is on discovering modes of terminating states, these techniques can be valuable. For Sub-EB, Sub- TB and their variants, the weight / for i < j E [ + 1] is set to / Σi<jE[H+]]] X following Madan et al. (2023).
Instead of the average I diddtttee adopted in the literature, we chose total variation Drv and Sansen-Shannow divergence DasD between (z) and P*(x)) as the metrics for performance com- parison between competing methods. The rationale and formal definitions are detailed in Ap- pendix B We have conducted three sets of experiments. The first set conducted in simulated environments, referred to as 'Hypergrids'. The second set focuses on biological and molecular sequence design tasks using real-world datasets. The third set involves real-world applications of GelowNet in Bayesian Network (BN) structure learning. More experimental details, such as model configurations and hyperparameter choices, can be found in Appendix B Parts of our implementa- tion code are adapted from the torchgfn package (Lahlou al. 2023).
Hypergrids Hypergrid experiments are widely used for testing GFlowNet performance (Malkin al. 2202bb] Niu al. 2024). Here, states are the coordinate tuples of an D-dimensional hyper- cubic grid of height H. The detailed description of the generative process is provided in Ap- pendix B. We perform experiments on 256 256, 128 128 x 128 and 64 x 64 x 64 grids. We use dynamic programming (Malkin et al. 2022a) to explicitly compute (z) of learned TT, and compute the exact Dry and DISD between (r(z) and P=xx). Experimental results are depicted in Fig. 2 for DTV and Fig. 4 in Appendix B.1 for DaSD respectively.
On the 256 x 256 grid, it can be observed that replacing the X-TD objective for learning V with the proposed Sub-EB objective significantly improves the stability and convergence rate of the policy- gradient method. While the final performances of the two policy-gradicn close, they both outperform Sub-TB and CV. These experimental results strongly support the effectiveness of the Sub-EB objective in enabling more reliable learning of the evaluation function V, leading to improved stability and faster convergence during policy-based based GFlowNet training. The empirical gradient estimator constructed solely from the current training batch is not adequate for reliably guiding policy-based based training. On the 128 128 x 128 grid, the stability and convergence
methods (RL and Sub-EB)
8

---

Published as a conference paper at ICLR 2026
44
04
04
00
Figure 2: Plots of the means and standard deviations (represented by the shaded area) of Drv for different training methods with parameterized n and uniform TB on the 256 x 256 (left) and 128 x 128 (middle) and 64 x 64 x 64 (right) grids, based five randomly started runs for each method. By default, metric values are recorded every 20 iterations over N = 2000 training iterations and smoothed by sliding window of length 5 for all plotted curves in this paper.
rate of Sub-EB are again much better than those of RL. While CV, RI., Sub-EB and Sub-TB achieve similar final performance, both RL and Sub-EB outperform Sub-TB TBb and CV in terms of convergence rate. These findings further validate the effectiveness of our Sub-EB objective. Finally, on the 64 x 64 x 64 grid, both RL and Sub-EB outperform Sub-TB and CV, but the behavior RL and Sub-EB is very close. This can be ascribed to the fact that the stability of RL good enough for this experiment, so the advantages brought by the Sub-TB objectives is not obvious. Besides, the grid height N have more effect on the modeling difficulty than grid dimension D since hypergrids are homogeneous w.r.t. each dimension, and the minimum distance between modes only depends on grid height N (Nuu et al. 2024). The training batch size is set 128 by default, whereas a batch size of 16 and grid height of 20 are used for hypergrid experiments in Tiapkin et al. (2024). To ensure a fair comparison and assess the impact of these differences, we additionally evaluate the performance of Sub-T'B and Q-Much on 128 x 128 and 20 x 20 grids. Detailed discussions and results are provided in Appendix B.1
Ablation study on TB and A To demonstrate that the Sub-EB objective naturally accommodates parameterized π-- We compare the performance of the different methods with parameterized and uniform Ty on 256 x 256, 128 x 128, and 64 x 64 x 64 grids. We use Sub-TB-P and Sub- EB-P to denote the corresponding methods with parameterized Tg- Furthermore, RL-P and RL- MLE denote two variants of RL that adopt the two-phase algorithm by NNu et al. (2024) and the approach by Gritsaev al. for parameterized æa respectively. As CV requires fixed π it does not support a parameterized T and is therefore excluded from this ablation study. As shown in Fig. 2 and Fig. 4 in Appendix B.1 Sub-EB-P achieves the best performance and training stability among all the considered methods. This confirms that the Sub-EB objective well accommodates backward policies, which are parameterized and updated jointly with evaluation functions. For the 128 x 128 grid, we also conduct an ablation study on the hyperparameter À of the Sub-EB weights W w- x--=(=x-:00
X --))). Detailed discussions and results are provided in Appendix B.
Sequence design In this set of experiments, use GFlowNets generate biological and molec- ular sequences of length D, which are composed of M building blocks (Shen et al. (023). We use nucleotide sequence datasets (SIX6 and PHO4), and molecular sequence datasets (QM9 and sEH ) from [hen et al.] (2023). The detailed description of the generative process and experimental re- sults provided in Appendix [B.2] Experimental results of RI, Sub-TB, Sub-EB, Sub-TB-B and Sub-EB-B on mode discovery and distribution modeling tasks validate that the proposed Sub-EB objective enables the integration of offline sampling techniques into policy-based methods.
BN structure learning this experiment set, we focus on real-world studies of Bayesian Net- work (BN) structure learning (Malkin et al., 2022b] Niu et al. 2024). Here, the object space X' corresponds to the space BN structures. The detailed description
the generative process is provided in Appendix B.3 We consider three cases with 5, 10, and 15 nodes, where the sizes of X are approximately 2.95 x 10 4.18 x 10 and 2.38 x 10a5 respectively. For the large-scale cases, either (z P (zx is computationally infeasible. Instead, we report the average reward of the top 100 unique graphs that are discovered during the training process. Since effective distribution modeling performance implies not only optimality but also diversity of generated candidates,

---

Published as a conference paper at ICLR 2026
Avg 110
200
Figure 3: Plots of the mean and standard deviation values (represented by the shaded area) of average reward (left), diversity (right) and FCS (right) of the top 100 unique candidate graphs over 10 nodes, based on five randomly started runs for each method.
also compute the mean pairwise Hamming distance among these 100 graphs as a measure of diver- sity. It should be noted that neither excessively high nor excessively low diversity is desirable: the former corresponds to near-random generation, while the latter indicates that generation gets stuck in limited set of structures. As exact computation of Drv is infeasible, use Flow Consistency in Sub-graphs (FCS) (Silva et al. 2025) as the unbiased estimation Drv. For FCS, we randomly sampled 32 batches of terminal states, each of size 128, to construct the Monte Carlo estimator.
We present the discussion for the two large-scale cases below and defer the small-scale case to Ap- The corresponding experimental results are presented in Fig. and Fig. 12 in the Ap- pendix B.3 Among RI., Sub-EB, Sub-TB, and Q-Much, the results indicate that Sub-EB achieves the highest average reward. The four methods attain similar diversity. Notably, RI. and Sub-EB con- verge faster than Sub-TB and Q-Much, and only RI. and Sub-EB obtain strong distribution-modellee performance as measured by the FCS metric. Taking together all these findings, we can conclude that all methods achieve appropriate distribution modeling, and Sub-EB performs the best. This supports that Sub-EB not only enables reliable policy-baser training but also scales effectively large com- binatorial spaces, providing both high-quality and diverse solutions. For the two variants, Sub-TB-B and Sub-EB-B, it can be observed that Sub-EBB- achieves the highest average reward among all six methods, accompanied by a more noticeable drop in diversity. Given that the local search compo- nent explicitly prioritizes high-reward regions of the solution space, such a trade-off trade-off-significan reward improvement at the expense of reduced diversity-is expected. In contrast, Sub-TB-B does achieve a higher average reward compared to its non-augmented counterpart (Sub-TB) with a mod- erate decrease in diversity. However, the trade-off becomes much less pronounced. Without the local search technique, Sub-EB already achieves a comparable average reward and higher diversity than Sub-TB-B, Overall, Sub-EB-B proves to be more effective than Sub-TB-B, aligning well with our expectations of the optimality-diversiyy trade-off. This finding further supports the superiority of the policy-basee methods and validates that the Sub-EB objective enables the integration of offline techniques within policy-based frameworks.
pendix B.3
Molecular graph design In this of experiments, we consider the molecular graph design task (with |Aa)| = 1) described in Bengio et al. (2021). The sequence design task based on the sEH dataset (with |X| Rs 3.4 107) (Shen et al. 2023) is simplified from these tasks. A detailed descrip- tion of the generative process and the corresponding experimental results for these graph design tasks is provided in the Appendix(B.. The experimental results demonstrate that Sub-EB achieves strong overall performance large-scale molecular graph design, providing higher average rewards, faster convergence, and competitive diversity compared to RI, Sub-TB and Q-Much.
DISCUSSION AND CONCLUSION
In this work, we have established the connection between the state flow function F(s) and the evaluation function V(s). Built upon that, new objective, called Sub-EB, is proposed for learning the evaluation function (s). Through four sets of experiments, we provide empirical evidence that the new Sub-EB objective enables more stable and flexible learning of V than the X-TD objective, thereby improving the performance of the policy-basce methods for GFlowNet training. In principle, the Sub-EB objective allows flexible choices of weight coefficients and goes beyond policy-grn colicy-gradient methods. Further investigation into designing optimal weight coefficients and integrating the Sub- EB objective into more advanced policy-based methods is left for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the U.S. National Science Foundation (NSF) grants SHF- 2215573 and IIS-2212419. Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing.
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Lazar Atanackovic and Emmanuel Bengio. Investigating generalization behaviours of generative flow networks. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In International conference on machine learning, pp. 263-272. PMLR, 2017.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 14:27381-27399 2021.
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1 2023.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. In Uncertainty in Artificial Intelligence, pp. 518-528. PMLR, 2022.
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete proba- bilistic inference as control in multi-path environments. In Uncertainty in Artificial Intelligence, pp. 997-1021. PMLR, 2024.
Timofei Gritsaev, Morozov Nikita, Samsonov Sergey, and Daniil Tiapkin. Optimizing backward policies in gflownets via trajectory likelihood maximization. In The Thirteenth International Conference on Learning Representations.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
Haoran He, Emmanuel Bengio, Qingpeng Cai, and Ling Pan. Random policy evaluation uncov- ers policies of generative flow networks. In Forty-second International Conference on Machine Learning. pp. 22351-22364 PMLR, 2025.
Hyosoon Jang, Minsu Kim, and Sungsoo Ahn. Learning energy decompositions for partial inference of gflownets. In The Twelfth International Conference on Learning Representations, 2023.
Hyosoon Jang, Yunhui Jang, Minsu Kim, Jinkyoo Park, and Sungsoo Ahn. Pessimistic backward policy for gflownets. Advances in Neural Information Processing Systems, 2024.
Chi Jin, Zeyuan Allen-Zha Sebastien Bubeck, and Michael Jordan. Is q-learning provably effi-
cient? Advances in neural information processing systems. 31, 2018.
Minsu Kim, Joohwan Ko, Dinghuai Zhang, Ling Pan, Tacyoung Yun, Woo Chang Kim, Jinkyoo Park, and Yoshua Bengio. Learning to scale logits for temperatur--condiionnl gflownets. In NeurlPS 2023 Al for Science Workshop, 2023a.

---

Published as a conference paper at ICLR 2026
Minsu Kim, Taeyoung Yun, Emmanuel Bengio, Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. In The Twelfth International Conference on Learning Representations, 2023b.
Jack Kuipers, Giusi Moffa, and David Heckerman. Addendum on the scoring of gaussian directed acyclic graphical models. 2014.
Salem Lahlou, Joseph D Viviano, and Victor Schmidt. torchgfn: A pytorch gflownet library. arXiv preprint arXiv:2305. 145.4 2023.
Kanika Madan, Jarrid Bector-Brooks Maksym Korablyov, Emmanuel Bengio, Moksh Jain, An- drei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning GFlowNets from partial episodes for improved convergence and stability. In International Conference on Machine Learning. pp. 23467-23483. PMLR, 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment GFlowNets. Advances Neural Information Processing Systems, 35:5955-555 5967,, 2022a.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. Gflownets and variational inference. In The Eleventh International Conference on Learning Representations, 2022b.
Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Inter- national Conference on Machine Learning, pp. 7599-7608. PMLR, 2021.
Puhua Niu, Shili Wu, Mingzhou Fan, and Xiaoning Qian. Gflownet training by policy gradients, In Forty-first International Conference on Machine Learning. 2024.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-66 62, 2019.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference on Machine Learning, pp- 2887--68890. PMLR, 2023.
Jarrid Rector-Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng-Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Model- ing, 2023.
Robert W Robinson. Counting unlabeled acyclic digraphs. In Combinatorial Mathematics V: Pro- ceedings of the Fifth Australian Conference, Held at the Royal Melbourne Institute of Technology, August 24-26, 1976, pp. 28-43 43. Springer, 2006.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. Proceedings of the International Conference on Learning Representations (ICLR), 2016.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning, arXiv preprint
2017.
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International Conference Machine Learning, pp. 30956-30975. PMLR, 2023.
Tiago Silva, de Souza da Silva Eliezer, and Mesquita Diego. On divergence measures for training gflownets. Advances in Neural Information Processing Systems, 2024.
Tiago Silva, Alves Rodrigo Barreto, da Silva Eliezer de Souza, Souza Amauri H. Garg Vikas, Kaski Samuel, and Mesquita Diego When do gflownets learn the right distribution? In The Thirteenth International Conference Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Daniil Tiapkin, Morozov Nikita, Naumov Alexey, and Vetrov P Dmitry. Generative flow networks as entropy-regularized rl. In International Conference on Artificial Intelligence and Statistics. PMLR, 2024.
Heiko Zimmermann, Fredrik Lindsten, Jan- Willem van de Meent, and Christian A Naesseth. A variational perspective on generative flow networks. Transactions on Machine Learning Research, 2022.
13

---

Published as a conference paper at ICLR 2026
LLM USAGE DISCLOSURE
LLMs were used only for text refinement (grammar and style). All scientific content was developed and verified by the authors.
A THEORETICAL ANALYSES
A. PROOF OF THEOREM
Proof. We first prove the sufticiency of the Sub-EB condition (5). Assume that V is an evaluation function over S that satisfies the Sub-EB condition for a given forward policy TF. Then, for any [
E (s (B + ((s++1) - log (fg [(5)]== =0
(14)
Bmg(snsilow) [logg((nlln+t)) + ((N++1) - 10g V(sn]] =0
(15)
V(sn) E p
log TB(sh ($
+ V(s++1)
(16)
Here, the second equation holds by our assumption that any valid Thp introduces trajectory distribu- tion PF(T) that is strictly positive for any E T. Consequently, the corresponding state probability (s(s) strictly positive for any S.
Based on 16 the previous definition (g(())))))))))))))))))))({)))( exp V(sy) =- R(x), we have:
log R(x)
((x) = (sp(s)
(s)s((2)
log fx + s5)
(egla))
(F(5)(x)
log PB(x + 8z)
Pp(x 88 8)(zz)
log Pn(r + 85(za)
Epp(e-00)12)
log Pa (
Pp(x - +))(z)
llog F* (x)
DKL (PI(PP(TTH) (8H) P
log (S + (xp(x- (s)
PB(x - 2²) log Pp(x 85(x))
((SH-1) (alsnni
log Pn(TH-1:)2*
11)
PP(TH-119H-1)
log Pn((n-1:188 Pp(TH-11sH-11
log PB(SH-1)
log F* F*(HH-1)= P
(17)
TB(SA|8h=1) EEp((t/alenn)) log Ps(TT-1:)**
V(sn) (mr(eniilen
log TT 84)
Pr(Th-1:15441)
log Pn(TT.)
(18)
Pp Pp shh)
(p(T
lEpp(n..lu))
log Pr (n))
log (
log F" (sn)
DKI. Drr
(19)
14

---

Published as a conference paper at ICLR 2026
log PB( Pr(T)
((0))= (pr(rlan)
(T(
(PF(T)
log (F(T)
+ log Z
log FM (s0) DKL(PP(T)]IP((T))
(20)
Now, prove the necessity of the Sub-EB condition 5 associated with a forward policy TF. Then, Vh [M]]
Assume that V is the evaluation function
(B(8)18(41) Σ log (8(4411(8))
V(sh. (p..
+ log Z
log (B( (1+1)))4+1)
log Pb(T++1))*
(x mn)
+ (pp(tnl
#F(# (8h)
TB(s sh+11)
+ ((snt1)
Exp(sat1 sn)
log
(21)
TF #F($511|8h)
Therefore,
Ep (11 (og + ((8++1) - log ( V(sh)] =0
(22)
BP((a + ((s+1) V(si) log (8 =
(23)
1=1
Epp(Tes)
log ( V(si)
Σ log o F (81+1/88) =0
(24)
Epp(ne)) [log PB(Tagl3)) + V(s,) - log (T(Trg())) - V(s;)] =0
(25)
for any i <jj [H + 1).
A.2 PROOF OF THEOREM 3.2
Assume that F' is a state flow function over S, and my is forward policy, such that they satisfy the Sub-TB condition. Then, for any h E [H]:
Proof. We first prove the sufficiency of the Sub-TB condition ①
(po( [logTn(salss F(s++)) - log TF (ox.))
(26)
() - log (0g =0
(27)
log F(sh) = loog TB TB(81|8t+1) + log F(sh+1).
(28)
TF TT(Sst1|5)) (s;)
Here, the second equation holds due to the following two reasons. First, by assumption that the trajectory distribution Pp. which is induced by Tp, assigns positive probability to all trajectories in T. Thus, the marginal probability PD(s - s') is strictly positive for any (s + s') E. Second, ap arbitrarily constructed and may differ from πF. Let
TT(s)S)F(s))
t T = Σ. (
(29)
Then, taking the expectation of (28
w.r.t. TF, we have for any [H]:
TB (45n)s)41))).mm))))))))()))))( + log F(8h+1)
log F(*)) = ( (8w)
(9h41
log IT ( (p))
(30)
15

---

Published as a conference paper at ICLR 2026
ा (
log Tr (s (8h))
+ log Σ
1 ((NN41))
Sh11
DKL (+ss) (-(sa)) + log TB(SA
(31)
Note that the second term the last equality is independent of xF sn). Then, given (T(*s)) and F(8++1), F(s)) is maximized at when Tr(-(sa) = (-1ss)) Accordingly, given Th (88++11
and (8))22), F(8)+1) maximized at 84+1 when - = (-(-/1--1) Keeping doing this
recursion from h = to h H, we get the conclusion that F is maximized when OF n! for any (s (5+ E
Taking the exponential of 27 and summing over 84+1 yields Eea., log F(s)). Inserting this back into (27. we arrive at TF nt. Achieving the maximum, we have log F(x) = log De, (x)( F = log R(x) log Fx(x). log 1() log Eg (B(s)-1|z)F (x)
Eme, log ((s))TT(()+111)
log F* (S log F(s)) (*o)))- Therefore, F(s) = (*)s) for any s E S. Combining this with 270. we have = Th.. where (s(ssss) xn(al=")F"(s') is the optimal forward pol- (*(.)
log E., (1
icy induced by лв. Morcover, based (30) and derivation analogous to that of V, we have, for Markovian Pg and any h E [
log (sh) log F* (sn) DaaeaDaeeeeetellllaaalaaaaaaaaa
(32)
Therefore, when AF = Tp. we have:
log F (sh) log F" (sh) DKL (
(33)
Now we prove the necessity the Sub-TV condition (T). Assume that F and mp satisfy the equa- tion 33 above. Then, the equation (32 must also hold for F and TT, and we have, for any h E [H]:
PB(Th:)Z*
log Fshh = (rp(aa))
log
Pp PP(Th:
log T (8(8)(8411)
+ log Z
#F($411|6 (s:))
imh
log (s) (8h+1)
PB PB(TT+1:)
E (Pr
log Pr (7
Tr (8h)
log (s (sh+1) TF($5+1|ss)
(1
log F(sht1)
(34)
We can rewrite the above equation as follows:
log F(SN) == - -DK((FF(()s)))) (-1sx)) + log Σ I=o(snl9ht)))(mnni Vh [H].
(35)
( :)) and F(s=-1), the assumption that log F((s)) is maxi- mized at 8n, indicates ((((sa)) = (-)ss)) Therefore, we recover vh. E [H]:
Since TF (-((s)) is independent of TB
log n (s)
(og
(36)
and the Sub-TB condition can be easily derived from it.
A.3 PROOF OF THBBREM 3.3
Proof. We first prove the sufficiency of the backward Sub-EB condition [1 evaluation function over S Si{s } that satisfies the backward Sub-EB condition for given backward policy TR. Then, for any h E H 1]:
Assume that W an
Ex () PP log FB (8 (84+1) W(8++1) log Tr(s6+1/5)) W(sh)) (37)
16

---

Published as a conference paper at ICLR 2026
(o (oggT((H|ssstt (1+11) + ((s++1) - log 1o W(sn)] 0
(38)
log TF(sh+1|sh) ( s8+1)
(s$++1) (snlona)))))
+ ((ss))
(39)
Here, the second equation holds by our assumption that any valid and R should introduce trajectory distribution PT(T) that strictly positive for any r E T. Consequently, the corresponding state probability (s()) is strictly positive for any E S.
Based on (9). we have:
(s1) = (an(so)ss) log (1 (0)
+ ((0))
(5 (1)
-pp((0-01(41) log Ps( + s1)
+ log F(5))
(8( + 81 s1]
Pr(so - 81 s1
-BPo(so-silan)
log (3 + s1
+ log PF(s))
log F(s)) DKL (P (p
(40)
TT (sh++11 E PPn(lll
PF(T.))
((s++1) = B
(sn B (m)+1)
log TB (1hh/111)) (8h+1)
log PA(TA (9h)
+ (so)
PP(Ts++1)
log PB(T.N+1|5M (/411)
+ log F(so)
(41)
log PF(Ta++ (1944+119941)))
+ log (F
(9((T441(5441)
log F(Sh+1) DKL
(42)
TF (x((//H-1)
log PF(T.--1)
W(x) = Eng (sw
1|m))
log
TB (s
+ Epo(TH aleH
PH(TH-IlsH-i)
+ W(o)
log PF(T)
= Ero(T(z)
+ log F(so)
PB(T(x)
log PF(T(x)]
= EPe(T(z)
+ log PFso))((z)
PB(T(x)
= log F(x) Det((PP(tts)((((((tttt))
(43)
In particular, when the backward Sub-EB condition satisfied for H, we have
Epo(s) (sylaz) + ((x) log (z(s)))) (s,))
(44)
E Er (x) - log R(x)] 0.
(45)
Since OF is arbitrarily chosen provided that PD(T) > 0 for any E T, it follows that Po(x) for any x e and (xx) = log R(x).
Now, we prove the necessity of backward the Sub-EB condition evaluation function of given forward policy иB. Then, Yh E [H] 1:
Assume that W is the
h log (F(
((s++1) = Bla
+ W
(3(3)18541)
log TF (p(5 + Epp(rale))
(T(T.))((s))
= -Bggten(ene))
log
(
P
17

---

Published as a conference paper at ICLR 2026
=B log (s(h++1|s)) + W((o)) o (=(0(88n+1)
(46)
and W (a) = log R(x). Therefore,
E Epp (e log TB (sn + ((s+11) log (ogmm))))1))(((( W(sh]] [on)]=1 =
(47)
$
E (s (2-)))))))) 1og ((1++1) - ((:)) log logg
(48)
1=1
E pg
log ((ggT(())s41)++WW)) - W(ss) Σ log कF (8141(51)
=0
(49)
Brg(ru)) (Tus)
log Pn(Tupl8g) + (83) - log Pr(Teglos) W(s:)] =0
(50)
for any i < [H +1]
L
A.4 EVALUATION FUNCTION WITH TOTAL FLOW ESTIMATOR
When incorporating the total-flow estimator Z, the true evaluation function Vt still takes the form
of (3), only differing by replacing Z" with Z* Z7/Z. In this case, it be verified that V1(o: 0)
log Z(0) DKL (p))))))))))(T))) Then, the gradient of [B)() [V 0)] is equal to that of
2-
DKL PF (T)) + (log Z(0) log Z )² where p(so; 0) 2(0)/Z as is unique (Niu
et al. 2024).
The Sub-EB condition and objective remain unchanged, except that Pa (x)ss) exp V(sy) is refined as RZ.
Corollary A.1 (Corollary to Theorem 3.1 Suppose V is an arbitrary evaluation function over S, and F* is the optimal flow induced by a backward policy TB. Given a forward policy ny,
F* (*N)))
Yh [H] V(sn) log
(s)).
(51)
and only V satisfies the Sub-EB condition
5
Proof. The proof can be done by replacing R(x) in the proof of Theorem 3.1 by R(x)/Z
Finally, in Algorithm VoLn (so: (sp; 0)]
0(0) + V. (so; 0)
approximated by Vo Ba (V
1 K ETED DE R(sh 8/ log Z(0)
E ( - sh+1) Ve log Te ( 0)].
A.S COMPARISON BETWEEN A-TD AND SUB-EB OBJECTIVES
The traditional X-TD objective for V(: () can be expressed as follows:
H
Epe(T)
Σ
h=
(*())) = V(sk) +
Σ x (v sit1),
(52)
4=h
where V is considered as constant when computing gradients w.r.t. Φ, and dy - 8 equal to Dv-(TT,441 as defined in (8). Without consideration of gradient computation, the expression of the objective value can simplified as:
H
H x x - 8141))
Epp(T)
ΣΣ
(53)
18

---

Published as a conference paper at ICLR 2026
In comparison, the expression of the Sub-EB objective value is:
(r)
w ()-((6((Ts5))) [
Σ
(54)
It can be observed that the A-TD objective only considers the events that start step h for learning (s)). and edges-wise mismatch by (s; - 1. In contrast, the Sub-EB objective incorporates information from both events that start at h and those that end at h by considering subtrajectory- wiie mismatches dy (Ta()) that start at i(2 h) and (T())))))) that end at j(s h). This results in a more balanced and reliable learning of (sx). Besides, the form of w can be freely chosen, while must be X in the X-TD objective (Schulman et al. 2016).
A.6 GFLOWNET AND RL
RL methods can be roughly categorized into two main frameworks (Sutton & Barto 2018): the first the Q-learnin framework, and the second the actor-crii critic framework.
Q-learning Under the Q-learning framework, the core idea, exemplified by the soft Q-learning algorithm, is to learn a parameterized function Q that minimizes the offline Bellman objective for the transition environment G with edge reward log (s(s)s') and log (ggls()))))) + V(sy) == log Rxx) (Haarnoja et al. 2017). This objective function can be written as
E (Pp( (6Q {)
(s +)) = log + V(8) Q(8,
(55)
with V(s) == log E. exp Q( s') and IT (s'[s) == exp Q(s.s")
exp V(s) Any function Q that achieves zero V and
mismatch is guaranteed equal the optimal soft Q-function Q* with the corresponding
=.pp. Tiapkin et al. (0224) showed that if we treat Q(s, s') as log F(s - 5') such that Vss) = log (3). then the Bellman objective transforms into the DB objective. The
log F(- 8-8 distinction lies only in parameterization: instead of parameterizing Q(s, directly and deriving V and Fe from it, one may parameterize (IF, V) jointly and define Q(8, 8') == = [s)) + log ((s F. The authors further showed that the optimal solutions of the Bellman objective coincide with those of the DB objective from the perspective of Soft Q-learning. However, as acknowledged by the authors, this result applies only to the DB objective with fixed TB. To address this limitation, Deleu et al. (2024) established an equivalence between path-consistency learning (which incorporates soft Q- learning as a special case) and the Sub-TB objective (which incorporates DB as special case) from a gradient-based perspective. Compared to Deleu et al. (2024), our Theorem 3.2 offers a more direct and explicit connection along this RL direction.
In analogy to soft Q-learning, we may instead parameterize V, define Q(s, = log (s('l1s) + Tn (slsss") (oslss)exp exp Vis)
(s) and ==(('1s)= En mo(d)ee) eep (or.. and learn V via the state-level Bellman objective:
(p))))(00()))), (x(()= = log Σ TT(s)s") exp V(5') - logg Σ exp ((s,
(56)
where log E., exp (s, 8¹) = log E. (s'(s) exp (s)) = (s)). Notably, RFI He et al. 2025
follows this formulation, differing in that it parameterizes F(s) as exp (s) and multplies F (s) by a rectified scalar g(s). However, this definition of ग. can lead to computational scalability issues when sampling trajectories. Let Ch(s) {s' (s s') E} denote the set of child states of state 8. As TD is equal to or design based on TF, making a transition from TD( {)) at state requires [Ch(s)] separate passes through V (a neural network with scalar outputs) to obtain the value of (s') for every s' E Ch(ss.. In contrast, (- ()) can be readily obtained from Q (a neural network with vector outputs), where single pass at yields Q(s, s') for every s' Ch(s). In tasks like BN structure learning, where [Ch(s) can be larger than 102, methods that only parameterize V can be much slower than those that jointly parameterize (V, TF) or directly parameterize Q.
To be specific, we have da(s' s') log (s)ss ) exp V(s') log (s) exp V(s) log (og))(s)s') log (s(ss)s) F()), coinciding with the formulation in the DB objective.
19

---

Published as a conference paper at ICLR 2026
Actor-Critic Our works presented here and in Niu et al. (2024) are based on the theory of policy- gradient (Agarwal et al. 2021), which operates under the actor- critic framework (Sutton & Barto, 2018 Haarnoja et al. 2018). In this framework, the central idea of the soft actor-critic algorithm
is to learn a parameterized evaluation function V that minimizes (but not necessarily) the online
Bellman objective:
(p (6 s'))9, 65(s - log F + V(s) Q(s,
(57)
with ((s == log + V(s') and Q(x, sy) := log R(x),, where (p-(s--a))-- [...] above can be generalized into (pp(s) [xr( .]. but the inner online expectation must still be maintained. Notably, the online Bellman objective can be viewed as special case of the proposed Sub-EB objective. At this point, the two RL directions begin diverge (Schulman et al. 2017). When V achieves the optimal solution of the objective (denoted by VT), we have:
((5) (s) (+')) (Q(s)) log (s8)(s)))s)]
exp s')
LogE. exp ((s (og log Σ exp Q(s.
(r(((s)
exp Q(s,s') [=(s((s===] E.
=-DKL (TP()(//)) log Σ exps(ssss), no((s|ss
exp En
Q(100((ss 5')
Defining F(s - -> exp Q(s, 8') yields expressions that coincide with those presented in the main text. This clarifies why Vt (and its approximation V) serves critie: it evaluates how far the actor TF is from the local optimal policy TQ. Here ιo is only locally optimal as ((s, 8') may still deviate from the globally optimal point (s))s)) The divergence is then minimized w.r.t. TF using V, so that Vt of TF moves closer to the optimal one V This can be achieved either by simply setting ( to (- for all sampled states or by applying a policy-gradies update along sampled trajectories.
A.7 SOFT ACTOR- ACTOR-CRITIC FOR GELOWNET
The soft actor-critic algorithm tailored to GFlowNet training is presented in Alg. 3 When all states are visited through sampling in Alg. 3 and the online Bellman objective of V reaches zero, meaning ss E SS DRL (Tr( (s), . (s)) = 0, and V = Vt, we show below that policy or and V will converge to optimal quantities Tp and V Starting at terminating states, we have:
V(x) -Dxi (=((((((())) + (z)
Qzx, = log R(x)
(58)
Q(s x) log π (8 V(a)
F F'1 x)
log
+ log R(x) = log F" (sH-1 +x),
(59)
Fx)
where we use the definition of TB. Next, we have:
V(sH-1) = -DL (TPT(T((
+ log Σ exp ((5H-11))
= log Σ
F* (*HH-11 = log FoogF((a (sH--),
(60)
exp((sH-1522)
( no (2((81111) Σ. CxpQ(sH-1,2)
F (5H +2) = (
(61)
(*1
Continuing this recursion, we will arrive at V(s) = log F* Fs), Q(s + 5') = log F + s'), and Tr(s'(s) = (s'ls) for all states and edges.
The online expectation over Tr makes Alg. 3 computationally expensive and precludes improve-
ments the edge-wise formulation (s 8'). To address this, one may modify it into Alg. 4
20

---

Published as a conference paper at ICLR 2026
where the key difference is online trajectories sampled from Pp(r). The definition of os(TT.) is identical to that of óy (Ta.)).
Algorithm 3 Soft Actor-Critie Workflow
Require: TF (- 0))0), T.( V(:;)), batch size K, number of total iterations N for 1, I,..., N....No do
D < [
Pb(T)]
Based on D. update & by its gradients w.r.t t ErDD
Exer (= Based on D and V, setting Tr( (s) to mo(: ( (s) for any s(F 81) ED.
end for
return TF T=(:1:30). ((:
Algorithm 4 Modified Soft Actor-Critic Workflow
Require: ( 0). T for 1, I,...u N do
(-1--1.).. 1.), V):;)), batch size K, number of total iterations N
D < [ Pr(T))
Based on D, update @ by its gradients w.r.t. * &EEED ōs(8 - 8';)) ( (rr
EveD Enger W/ (j,=d((Ta.;
Based on D and V, setting np(. (s) to no( (s) for any s( 81) ED.
end for return TF ())
Compared to Alg. Alg. [ immproes the basic policy-optimization operator using policy-gradien theory for practical implementation. Since original policy-gradient methods operate strictly in an on-policy manner, Alg. Dis introduced to enable the use of an offline sampler PD.
B EXPERIMENTAL SETTINGS AND RESULTS
Hyperparameters For both the original policy-based method and the proposed one with the Sub- TB objective (RL and Sub-EB), we set the hyperparameter 7 0.99 based on the ablation study results reported in Niu et al. (2024). For the data collection policy TD of Sub-TB, the hyperparame- ter starts at 1.0 and decays exponentially a rate 0.99, where the decay rate is also determined based on the results of the ablation study in Niu et al. (2024). In the Sub-TB objective, the hyperpa- rameter À set to 0.9 following the ablation study by Madan al. (2023). For the Sub-EB objective, A is set to be 0.9 selected from (0.1, 0.2, 2,... ..... 0.9, 0.99 based on the ablation study results shown in Fig. 6 All experiments are conducted on the Grace cluster, one of the main Texas A&M High Per- formance Research Computing systems. Each node is equipped with an Intel Xeon Cascade Lake CPU, For our experiments, we requested a single node, 20 GB of RAM, and no GPU was used. The running times of the considered methods across experiments are reported in Appendix B.5
Optimization The Adam optimizer is used for optimization. The sample batch size is set to 128 for each optimization iteration following Niu et al. (2024). The learning rates of FF 0) and
log F(:;0) ;) are equal to 1 x 10-3 which is selected from {5 x 10 10-3, 3, 3,003,33 5 x 10 1 x 10 *) based on the performance of Sub-TB on the 256 x 256 grid. The learning rate of () is set to 5 x 10 which is selected from (10-2 5.5% 10-3 10-3. 5 10 10-4. 10 1004) based on the performance
of RL on the 256 x 256 grid. In all experiments, each training method is run five times, initialized from five different random seeds.
Model architecture The forward policy TF(- ()))) and evaluation function V(+: Φ) are each pa- rameterized by neural network with four hidden layers, with hidden dimension of 256 in each layer. The backward policy Tz (-) is uniform distribution over valid transitions (edges). In hypergrid, sequence design, and molecular graph design experiments, coordinate tuples, integer se- quences, and integer matrices transformed using K-hot encoding before entering the neural net-
21

---

Published as a conference paper at ICLR 2026
DOYA CHALL
02
02
02
Figure 4: Plots of the means and standard deviations (represented by the shaded area) of DasD for different training methods on 256 256 (left) 128 128 (middle) and x 64 64 (right) grids, based on five randomly started runs for each method.
79-22
08
07
Figure 5: Plots of the means and standard deviations (represented by the shaded area) of Drv (right) for different training methods on the 128 x 128 (left) 20 x 20 (right) grids, based on five randomly started runs for Sub-TB-16 Sub-TB-128, Q-much- 16 and Q-much 128. Here, "16" and "128" denote the training batch sizes.
works. BN structure learning, adjacency matrices are used directly as input into neural networks without encoding.
Peformance metrics The first one is the total variation Drv between Pr(x) and P" (x). which is defined Drv(Pr(x). P* (=))) = Ezex (Pr(x) (1:)). An alternative performance metric adopted in literature is the average 1 -diitaanc,, which is defined -x PP* reason that Σn (x Pr(()) < 2, resulting in the average diiitttneee being heavily scaled down by | We also evaluate different methods using the Jensen-Shannon divergence DJsD the second metric, which can be written as: DASD DISD(PF(I), (P((T) PM (r)) ( (PF((())), PM (P (x (x))),
(x)). The Dry instead is as follows. The design space | is usually large (> 10 and
where PM = (Pp PP).
B. HYPERGRIDS
The generative process of hypergrid experiments defined as follows. For grid with height H and width D, the state space excluding the final state, S) (){ss) consists of all D-dimensional coor-
dinate vectors the form {s = ([s] [[s]...... [s] d: .... [s] () [s a E (0...., H 1}}. The generating process begins at the initial state 8o (O...., ....... 0) and ends in the final state sy, which we de- note as (-1,..., -1). From any state a E S there are D + valid transitions (edges): (1) for each d (O...., D). the d-th coordinate can be incremented by one, leading s' = ([s] [s] 1, 1s) (2) if X, the process can be stopped by taking transition (s->8)), returning s as the terminating coordinate tuple By the definitions above, the GFlowNet DAG G is not graded, meaning every state (excluding 8y) can be returned as a terminating state. The reward function associated with ferminating states defined as:
new state
{g N
[g N
R(x) Rn Rí I
0.5 E (0.25, 0.5]
+ Rs II
0.5 (0.3, 0.4
d=1
d=1
where Rg 10-2, Rí 0.5 and R₂ 2 in experiments.
22

---

Published as a conference paper at ICLR 2026
011
10
Figure 6: Plots of the means and standard deviations (represented by the shaded area) of Drv (left) and DIS (right) for Sub-EB runs on the 128 x 128 grid. The hyperparameter À of Sub-EB ranges from 0.1 0.99. The results are based on five Sub-EB runs for each setup.
Additional experiment results As discussed in our Appendix A.6 Q-much may prioritize ex- ploitation compared the Sub-)trraectory level methods and perform less effectively on challeng- ing tasks with well-separated modes. To verify this, we conducted hypergrid tasks with batch sizes of 16 and 128 (our default setting) on both a 20 20 grid and 128 128 grid. The rationale is as follows. For a grid with height N and dimension D, the minimum distance between reward modes depends only on the height N. With moderate D and small N, even a uniform policy can still reach the modes with considerable probability. Therefore, N plays more significant role in determining task difficulty. The experimental results are presented Fig. [ On the 20 20 grid with batch size 16, Q-much performs better than Sub-TB. However, this advantage disappears when using batch size 128. For 128 x 128 grid with batch size 128, the performance of Q-much considerably worse than Sub-TB. When the batch size is reduced to 16, neither method converges to satisfactory solution; nevertheless, Q-Much still underperforms Sub-TB.
Ablation study on For the 128 128 grid, we conduct an ablation study on the hyperparameter
À of the Sub-EB weights (- X / [i<je(++)) X=+) to investigate its effect policy-based GFlowNet training. We run Sub-EB methods with À equal to experimental results are depicted in Fig. 6 It can observed that the Sub-EB method with À = 0.9 achieves the best performance. Although convergence rates and final performances differ, Sub-EB methods under configurations exhibit good stability, demonstrating that the proposed Sub-EB objective enables reliable learning of the evaluation function V.
0.90 and 0.99. The
B.2 SEQUENCE DESIGN
The generative process of this set of experiments defined as follows. The state space excluding the final state (' } is equal to U (0, ..,
13', where each state a sequence com- posed of integers ranging from 0 to N I. I. The set {0, ,0...., 1} corresponds to the N types of building blocks. The process begins at the initial state s. = that is, the empty sequence. For any intermediate state 8/2 Sh. it contains h elements drawn from {0....., N 1). There are N possible transitions for & with f < D, corresponding to either appending or prepending one element from (0. N - the current sequence. The procedure continues until sequences reach length D, yielding terminal states in X' = {0.... N I} and transitioning to 8 == (N, (N.....A .N)........ NN.. Ac- cording the definition of the generative process, the GelowNet DAG G is graded and Sp X. In practice, each state $, is represented as sequence of fixed length D. where the first elements are integers from (O....,A) represented as (-1,... ......-.. 1). We use nucleotide sequence datasets (SIX6 and PHO4) and molecular sequence datasets (QM9 and sEH from [Shen et al. (2023). Rewards are defined as the exponents of raw scores from the datasets, R(x)= score" (x). The hyperparameter 3 is set to 3, 5, 3 and 6 for SIX6, QM9, PHO4 and sEH, respectively, and the rewards are normalized to [10- 10]. [10-3 10), [0, 10) and [10-3 10). respectively. In this experimental setting, consider additional metric introduced by Shen et al. (023). It Mode Accuracy (MA) Pr(x) w.r.t. P (z), defined as:
1}. and the others are equal
In particular, the initial state 8g
MA(P)()), (*))) min E [(x)] E Ep-(w) [[xx]]
(62)
23

---

Published as a conference paper at ICLR 2026
140
125
180
1000
380
1250
130
2005
Figure 7: Plots of the mean and standard deviation values (represented by the shaded area) of MN for the SIX6 (top left), QM9 (top right), PHO4 (bottom left), and sEH (bottom right) dataset, based on five randomly started runs for each method.
0.39
6- 130
032
Figure 8: Plots of the mean and standard deviation values (represented by the shaded area) of Drv (left), D DID (middle), and MA (right) for the SIX6 dataset, based on five randomly started runs for each method.
We dynamic programming (Malkin et al. 2022a compute Pr(z) and the exact MA, Dry and DISD between Pr(x:) and (*))).
In this of experiments, where the goal is to uncover high-reward terminating states. addition to RL, Sub-EB, and Sub- TB methods, consider Sub-TB-B and Sub-EB-B, which explicitly promote the exploration of high-reward states during trajectory sampling.
focus not only distribution modeling but also mode discovery,
Distribution modeling Results We compare different training methods by their performance mea- sured by MA, DTV and DISD, as shown Figs. 8 10 and II for SIX, QM9, PHO4, and sEH datasets respectively. It can be seen that Sub-EB performs slightly better than RL. This be cribed the sufficient stability of RL these experiments. rendering the advantages brought by the Sub-TB objective less obvious. Nevertheless. Sub-EB and RL outperform Sub-TB in terms of both convergence rate and final performance. For the offline variants, can be observed that Sub-EB-B performs slightly better than Sub-TB-B, but performs PHO4, The experiment results validate that the proposed Sub-EB objective can be applied to the polecy-based training workflow for learning the valuation function V effectively.
than Sub-EB for all datasets except
Mode discovery Results We measure performance using Mode Number (MN), defined as the number of unique high-reward terminating states discovered during training. A terminating state is regarded as highly rewarded if its reward falls within the top 0.5% for the SIX6, QM9, and PHO4 datasets, and the top 0.01% % for the sEH dataset. As depicted in Fig. 7 Sub-EB-B and Sub-TB-B can find a fixed number of unique modes faster and discover more unique modes within a fixed number of optimization iterations, despite decline in distribution modeling performance all datasets
24

---

Published as a conference paper at ICLR 2026
114
150
380
2230
40.25
0.10
2.15
1000
Figure 9: Plots of the mean and standard deviation values (represented by the shaded area of Dry (left), DIsD (middle), and MA (right) for the QM9 dataset, based on five randomly started runs for each method.
8-
0.14
Figure 10: Plots of the mean and standard deviation values (represented by the shaded area) of Drv (left), DASD (middle), and MA (right) for the PHO4 dataset, based on five randomly started runs for each method.
1.2
8
0.55
0.43
Figure 11: Plots of the mean and standard deviation values (represented by the shaded area) of Drv (left), DIsD (middle), and MA (right) for the sEH dataset, based on five randomly started runs for each method.
except PHO4. These results validate our offline policy-based training workflow and support our claim that the proposed Sub-EB objective enables the integration of offline sampling techniques.
Combined with the mode discovery results in Fig. this indicates that offline techniques that encour- age high-reward terminating states are helpful for mode discovery, while they may hinder accurate distribution modeling.
B.3 BN STRUCTURE LEARNING
A Bayesian Network is probabilistic model, representing the joint distribution of N random variables, whose factorization is determined by the network structure I, which is DAG graph. Accordingly, the distribution can be written as P(y....., UN) II,. PP (v,(),) denotes the parent nodes of Yn graph x. Since any graph structure can be en- coded adjacency matrix, the state space excluding the final state is defined Skssy} = {s|C(s) 0, (0, 1[NNN]. where C corresponds by Deleu et al. (202)). should be noted that each BN DAG E X' corresponds to a state s E S in the GelowNet DAG G. The generative process begins from the initial state su 0 representing a graph without edges, and ends at s == -1 XNN For any s E S',sy, transition (s + s') corre- entry in the adjacency matrix to one, provided that the
where
the acyclic graph constraint introduced
sponds adding edge by flipping
25

---

Published as a conference paper at ICLR 2026
200
110
606
17
Figure 12: Plots of the mean and standard deviation values (represented by the shaded area) of average reward (left), diversity (middle) and FCS (right) of the top 100 unique candidate graphs over 15 nodes, based on five randomly started runs for each method.
514
118
1300
1.31
Figure 13: Plots of the mean and standard deviation values (represented by the shaded area) of DTv (left) and DISD (right) for the BN learning structure learning task over 5 nodes, based on five randomly started runs for each method.
resulting state s' remains acyclic. Alternatively, the generative process is stopped by transitioning to sy and returning as the terminating graph structure. According to the definition of the generative process, G is not graded for this set of experiments. Given an observed sample set D, of y1.N. the goal of structure learning is to approximate the posterior distribution P(x[Dy) x P(D y (x) P(x). In the absence of prior knowledge about x, the prior distribution P(x) is often assumed to be uniform, reducing the task to maximizing the likelihood, ((D,|=z) P(x[D,)))
The ground-trume graph structure x² and the corresponding dataset Dy of size [Dyl 10 are simulated from Erdōs-Rény: model (Deleu et al. 2022). We use BGe score (Kuipers et al. 2014) to
assess generated graph structures, and define the reward R(x) (A(r:
where A (x; D,)
C
BGe( x; Dy) BGe(su; Dy), 8 = 10 sharpens the reward function toward high-score structures, and A C=
[ D2) normalizes the reward so that R(x")= = 1. The exact number of DAGs on nodes, denoted as a (n) satisfies (Robinson) 2006):
a(n) =
Σ 1- k+1 2((a k) a(n - k),
(63)
with a()) = 1. For the of accessing distribution modeling performance, Malkin et al. 2022b set the number of nodes to 5, resulting in about 2.92 x 10a possible DAGs. In this setup, the ground- truth DAG contains edges. In addition this small-scale case, also consider two much larger cases with 10 and 15 nodes, corresponding to about 4.18 x 1018 and 2.38 x 1 possible DAGs, respectively. We set the ground-truth DAGS contain 10 and 15 edges in the two respective cases.
Additional experimental results For the small-scale case, we use dynamic programming (Malkin et al. 2 to explicitly compute (r(x) for learned Tr, and compute the exact Drv and DISD between (F( and Pd P*(x). (z)) In Fig. 13 the mean and standard deviation values of DTV and DISD are plotted for five runs of Sub-TB, RL. Sub-EB, and Q-Much, respectively. It can be seen that Sub- EB performs better than the RL, Sub-TB and Q-Much. These results confirm our conclusion that the Sub-EB objective enables more reliable learning of the evaluation function V compared the X-TD objective, thereby improving the performance of policy- based methods. We also include results for Sub-EB-B and Sub-TB-B. can be seen that they both perform worse than their original versions.
26

---

Published as a conference paper at ICLR 2026
Figure 14:
Four exemplary building blocks. Red and blue dots indicate the available atoms for bonding, where each bonding edge originates from red dot and terminates at blue dot.
These results further support conclusion that explicitly encouraging exploration of high-reward regions may not beneficial for distributional modeling performance.
B.4 MOLECULAR GRAPH DESIGN
Each molecule (graph) is a composition of at most 8 building blocks, selected from N 105 predefined molccular substructures provided by [Bengio et al. (021)) Each block (graph node) has list that contains at most M available blocks. Each block's s list contains exactly one target atom and ( to M source atoms. A bond is formed by connecting a source atom one block to target atom another. A graphical illustration of 4 exemplary blocks is shown in Fig. 14 Following Bengio al. (202T), we restrict the maximum is represented as a H
indices for forming bonds (graph edges) with other
number of blocks molecular graph be H = 8. Each state matrix. The generative process begins from the initial state so -1 " representing empty graph, and ends at $y = 105 x2, For any state sn. E Sy:
1. The first column contains h integers drawn from {O,..., N 1}, representing the indices of the building blocks present in the molecule.
2. The second column contains - integers drawn from {o..... ... H x X - 11, representing the indices of bonding sites that encode the connectivity structures. Specifically, if an integer is equal to (i 1) x H + j. this indicates that the target atom of the newly added block will be connected to the j-th source atom of the i-th block.
For any E S (s,, so). a transition (s + s') corresponds to a two-level action:
1. Selecting building block to add, represented by an integer {0, ......, N-1}.
2. Selecting bonding site represented by integer i {0, .0..... H M 1}.
The generative process stops when we make the transition (x: $1). when available source atoms remain for forming bonds, or when the state reaches the limit of H blocks. According to the definition of the generative process, the GFlowNet DAG G is not graded, meaning that any state except 5 can be a terminating state x X. Since there may be multiple types of bonds between a given pair of blocks, the size of the terminating state space greater than 105!
(105-8) Rs 10 The Oater Partition Coefficient (LogP) and c-Jun N-terminal Kinase 3 (JNK3) scores provided in the PyTDC package ative process involve two levels, we use separate neural networks to represent the policy for each action component. The log probability of action is computed the sum of the log-probabilities produced by the corresponding networks for the two action levels. Finally, following [Bngiio et al. (022), ), we choose parameterized n and use a batch size of emulate scenarios where querying the real-world molecule oracle is computationally expensive.
directly used the reward functions R(x). Since actions in this gener-
The experimental results are presented Figs. 15 and 16 We report the average reward and diver- sity of the top 100 unique graphs discovered during the training process. The diversity of set of molecules is computed as the average pairwise dissimilarity based on Tanimoto distance of Morgan fingerprints via the PyTDC package. For the LogP task, RL, Sub-EB, and Q-Much achieve similar average rewards, all outperforming Sub-TB. Moreover, RL and Sub-EB converge much faster than Sub-TB and Q-Much. All four methods exhibit comparable diversity. For the JNK3 task, Sub-EB
27

---

Published as a conference paper at ICLR 2026
Logp
Legs
2.50
320
0.71
1000
Figure 15: Plots the mean and standard deviation values of average reward (left) and diversity (right, Tanimoto diversity of molecular Morgan fingerprints) of top 100 unique molecules for the LogP task, based on five randomly started runs for each method.
0.00
Figure 16: Plots of the mean and standard deviation values of average reward (left) and diversity (right, Tanimoto diversity of molecular Morgan fingerprints) of top 100 unique molecules for the JNK3 task, based on five randomly started runs for each method.
achieves the highest average reward and demonstrates the fastest convergence among all methods. Its diversity is comparable to that of Sub-TB and higher than that of RL. Although Q-Much achieves the highest diversity, its average reward is almost the lowest. Overall, these results indicate that Sub-EB achieves the best performance in terms of average reward and convergence rate while main- taining reasonable diversity, confirming its effectiveness for large-scale tasks.
B.5
RUNTIME REPORT
Method
Mean
Std
Method
Mean
Std
Method
Mean
Std
Sub-TP TB
Ih 08m
1.09
Sub-TB
Oh 52m
0.27
Sub-TB
Ih 59m
1.53
Sub-TB-P
Ih 14m
10.36
Sub-TB-P
Oh 58m
7.92
Sub-TB-P
2h 19m
4.40
RL
1h 03m
0.92
RL
Oh 44m
1.43
RL
1h 27m
7.37
RL-P
1h 27m
0.57
RL-P
06m
2.21
RL-P
2h 03m
10.68
RL-MLE
1h 09m
1.78
RL-MLE
Oh 58m
3.27
RL-MLE
2h 28m
9.58
Sub-EB
17m
2.31
Sub-EB
54m
2.66
Sub-EB
2h 04m
4.11
Sub-EB-P
1h 17m
1.49
Sub-EB-P
1h 11m
0.82
Sub-EB-P
2h 41m
15.46
Table 1: The mean and standard deviation (std) of the total runtimes for each method the 64 x 64 64 (left), 128 128 x 128 (middle), and 256 256 x 256 (right) grids. Here, "h" denotes hours and 'm' denotes minutes, and all std values reported in minutes.
28

---

Published as a conference paper at ICLR 2026
Method
Mean
Std
Method
Mean
Std
Sub-TB TB
Oh 05m
0.21
Sub-TB
Oh 04m
1.10
Sub-TB-B
Oh 16m
0.67
Sub-TB-B
Oh 12m
0.57
RL
Oh 04m
0.06
RL
Oh 04m
0.12
Sub-EB
Oh 04m
0.34
Sub-EB
Oh 04m
0.06
Sub-EB-B
Oh 17m
1.63
Sub-EB-B
Oh 13m
0.71
Table 2: The mean and standard deviation (std) the total runtimes for each method on the SIX6
and QM9 datasets. Here, 'h' denotes hours and 'm' denotes minutes, and all std values are reported in minutes.
Method
Mean Time
Std (min)
Method
Mean Time
Std (min)
Sub-TB
19m
2.05
Sub-TB
3h 30m
10.23
Sub-TB-B
Oh 27m
0.42
Sub-TB-B
3h 46m
11.59
RL
18m
0.10
RL
3h 42m
13.78
Sub-EB
Oh 18m
0.15
Sub-EB
3h 41m
13.97
Sub-EB-B
Oh 30m
1.09
Sub-EB-B
3h 58m
7.57
Table 3: The mean and standard deviation (std) of the total runtimes for each method on the PHO4 and sEH datasets. Here, 'h' denotes hours and 'm' denotes minutes, and all std values reported in minutes.
Method
Mean
Std
Method
Mean
Std
Method
Mean
Std
Sub-TB
57m
1.11
Sub-TB
Oh 10m
0.80
Sub-TB
Oh 16m
1.28
Sub-TB-B
1h 23m
4.27
Sub-TB-B
Oh 36m
1.82
Sub-TB-B
Oh 42m
3.82
RL
Oh 56m
0.46
RL
11m
3,58
RL
Oh 12m
1.79
Sub-EB
Oh 49m
0.87
Sub-EB
Oh 10m
0.91
Sub-EB
Oh 14m
0.28
Sub-EB-B
1h 07m
1.43
Sub-EB-B
Oh 31m
2.27
Sub-EB-B
Oh 39m
1.13
Q-Much
Oh 59m
3.57
Q-Much
Oh 12m
1.34
Q-Much
Oh 19
0.62
Table 4: The mean and standard deviation (std) of the total runtimes for each method on the 5- node, 10-node and 15-node BN tasks. Here, 'h' denotes hours and 'm' denotes minutes, and all std values reported in minutes. The runtimes for the 5-node cases are the largest due to the explicit computation of Drv for performance comparison during training.
Method
Mean Time
Std (min)
Method
Mean Time
Std (min)
RL
Oh 9m
0.36
RL
Oh 12m
1.33
RLEval
Oh 13m
0.17
RLEval
Oh 9m
0.08
Sub-TB
Oh 13m
0.93
Sub-TB
Oh 9m
0.29
Q-Much
Oh 19 m
3.50
Q-Much
0 10 m
0.26
Table 5: The mean and standard deviation (std) of the total runtimes for each method on the LogP and JNK3 tasks. Here, 'h' denotes hours and 'm' denotes minutes, and all std values reported in minutes.
29


Reviews:
[{"review_id": "1EhLtsYwlX", "reviewer": "Reviewer_hiRW", "summary": "This paper proposes Subtrajectory Evaluation Balance (Sub-EB), a new training principle for GFlowNets that enables learning a value function V(s) from partial trajectories, serving as a critic for forward policy improvement. The authors derive Sub-EB as a sufficient condition for flow consistency, connecting GFlowNets to reinforcement learning style. Empirical results across synthetic and real-world tasks demonstrate improved training stability, flexibility in backward policy design, and effectiveness in off-policy learning scenarios.", "questions": "1. Is it possible to prove that Sub-EB condition imply that the learned forward policy $\\pi_F$ is optimal?\n\n2. Is it possible to perform ablation studies on the update frequency of the evaluator V?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Sub-EB gives an alternative way for optimization of value-based and policy-based GFlowNet;\n\n2. Both on-policy and off-policy training regimes are considered, which broadens the method's applicability.", "weaknesses": "1. The result analogous to theorems 1-2 in current submission are well-known from the point of view of entropy-regularized (soft) RL, see [Tiapkin et al, 2024]. In particular, Proposition 1 in [Tiapkin et al, 2024] is precisely the result of Theorem 1 for complete trajectories. In light of this result, novelty and technical originality of theorems 1 and 2 is limited.\n\n2. It’s unclear how much faster the algorithm is in wall-clock time, rather than in terms of the number of steps;\n\n3. The paper does not provide a theoretical guarantee that optimizing the Sub-EB objective leads to convergence of the target distribution, even under ideal conditions; \n\n4. The paper does not provide sufficient comparisons with other approaches that use learnable backward policies, e.g. [Gritsaev et al, 2024], [Jang et al, 2024]. Also the paper does not include Sub-TB-B or Sub-EB-B experiments on the Bayesian network structure learning task.\n\nReferences: \n\n[Tiapkin et al, 2024] Tiapkin, D., Morozov, N., Naumov, A., and Vetrov, D. P. (2024). Generative flow networks as entropy-regularized RL. In AISTATS-2024.\n\n[Gritsaev et al, 2024] Gritsaev, T., Morozov, N., Samsonov, S., and Tiapkin, D. (2025). Optimizing backward policies in GFlownets via trajectory likelihood maximization. In ICLR-2025.\n\n[Jang et al, 2024] Jang, H., Jang, Y., Kim, M., Park, J., and Ahn, S. (2024a). Pessimistic backward policy for GFlowNets. In Neurips-2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTYlzfaZ82", "reviewer": "Reviewer_wVfJ", "summary": "The authors build upon prior work on GFlowNet training with policy gradients where an evaluation function is defined using rewards based on the log-ratio of forward and backward transitions. This formulation establishes a direct connection between minimizing the value function and minimizing the KL divergence between the forward and backward policies.\n\nIn this work, the authors propose a novel objective called Subtrajectory Evaluation Balance (Sub-EB), which enables more balanced learning compared to the conventional RL λ-TD objective. Sub-EB incorporates information from subtrajectories that both start and end at a given step whereas the conventional $\\lambda$-TD objective focuses on events starting at a step and edge-wise mismatches only while also allowing for arbitrary weighting schemes, unlike λ-TD, which is constrained by an exponential λ-decay. It additionally establishes a theoretical connection between the flow function and their evaluation function. \n\nFurthermore, the authors extend their approach to off-policy learning, in contrast to Niu et al. (2024). They introduce an evaluation function for backward trajectories, whose minimization likewise reduces the KL divergence. Because the objective is defined over backward trajectories, this formulation allows the use of a behavior policy distinct from the forward policy to generate states from which the backward rollouts are performed.\n\nTheir results demonstrate strong performance compared to the Sub-TB objective as well as the $\\lambda$-TD one. Their objective allows for faster convergence with less variance when looking at total variation. Likewise, they outperform Sub-TB in mode discovery with their best performance coming from Sub-EB-p that parametrizes the backward policy.", "questions": "- Figure 3 is too small and hard to read. It would be better if the font is made bigger.\n- It would be good to compare against LED-GFN since they learn energy decompositions that can be seen as a proxy for evaluating state-transitions.\n- In theorem 3.2, it would be good to clarify in the main text what the minimum is taken over exactly.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Strong theoretical analysis and results\n- I appreciated the theoretical connections established between the evaluation function and the flow function.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yp2xv0aQu7", "reviewer": "Reviewer_P5HE", "summary": "The authors proposed a novel algorithm for GFlowNet training, based on a policy-iteration-style framework, where the SubTB/PCL-style loss is used only to optimize the value and backward policy. In contrast, a policy gradient objective is applied to optimize the forward (generative) policy with an additional GAE-style variance reduction. The authors provided experimental validation of their method across several tasks, including hypergrids, sequence design, and Bayesian network learning.", "questions": "- Is it possible to provide a simple toy experiment that shows that indeed using the SubTB-style loss provides a faster convergence of the value function to an actual regularized value of the current policy?\n- Is it possible to provide a direct ablation on the variance reduction quality for the gradient of the forward policy?\n- Do you use a shared backbone for your policy and flow estimators? \n- How is it possible to introduce additional exploration into this algorithm?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The first (to my knowledge) policy-iteration style approach for GFlowNet training, which shows a consistent performance on various benchmarks, including a high-dimensional one such as 10-vertex BNs and SEH;", "weaknesses": "- The theoretical results automatically seem to follow from the GFlowNet-RL equivalence, described in the works (Deleu et al. 2024, Tiapkin et al. 2024). In particular, thanks to the graded DAG structure, Theorems 3.1 and 3.2 follow from Proposition 1 of Tiapkin et al. (2024), applied to a sub-DAG that is rooted at the vertex $s_h$, as well as the interpretation of a value function as negative KL-divergence up to a log-normalizing constant. Also, the paper does not cite a related work by Tiapkin et al. (2024).\n- The value loss itself is equal to on-policy SubTB loss, which also does not bring any novelty there: in many practical cases, the SubTB loss is applied in an on-policy manner, which is exactly the same as SubEB loss up to the lack of backpropagation through the forward policy.\n- Lack of RL-inspired baseline comparison, such as Rectified Policy Evaluation (He et al. 2025) and Muchnausen DQN (see Vieillard et al. (2020) and its application to GFlowNets in Tiapkin et al. 2024). \n- No discussion or comparison with the gradient variance reduction techniques of da Silva et al. (2024); if I understand correctly, the main difference between Sub-EB loss and Sub-TB loss is the usage of GAE-style variance-reduced objective for the forward policy, whereas the on-policy optimization through Sub-TB will result in a weaker variance reduction; thus, the comparison with other gradient variance reduction techniques applied in GFlowNets seems to be important.\n- On the large-scale task of 10-vertex Bayesian Network generation, the presented metrics do not show how the method actually solves the sampling problem. I would recommend computing the reward correlation using a test set of randomly generated graphs, or using the FCS metric proposed by da Silva et al. (2025). \n\n- Minor remark: Due to the on-policy nature of the algorithm, it's impossible to use a simple exploration technique such as $\\varepsilon$-greedy exploration or replay buffers;\n\n### References\n\nDeleu, T., Nouri, P., Malkin, N., Precup, D., & Bengio, Y. (2024). Discrete probabilistic inference as control in multi-path environments. arXiv preprint UAI-2024;\n\nTiapkin, D., Morozov, N., Naumov, A., & Vetrov, D. P. (2024, April). Generative flow networks as entropy-regularized RL. In International Conference on Artificial Intelligence and Statistics (pp. 4213-4221). PMLR.\n\nHe, H., Bengio, E., Cai, Q., & Pan, L. (2024). Random policy evaluation uncovers policies of generative flow networks. ICML-2025\n\nVieillard, N., Pietquin, O., & Geist, M. (2020). Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33, 4235-4246.\n\nSilva, T., de Souza da Silva, E., & Mesquita, D. (2024). On divergence measures for training gflownets. Advances in Neural Information Processing Systems, 37, 75883-75913.\n\nSilva, T., Alves, R. B., da Silva, E. D. S., Souza, A. H., Garg, V., Kaski, S., & Mesquita, D. (2025). When do GFlowNets learn the right distribution?. In The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vLqoOWaWAc", "reviewer": "Reviewer_C8bA", "summary": "The focus of this paper is policy gradient based learning of GFlowNets. The paper presents a sub-trajectory based loss for learning the evaluation function of a GFlowNet forward policy. The evaluation function is also theoretically connected to flow functions of GFlowNets. The learnable evaluation function is then utilized in the policy gradient algorithm to train the GFlowNet forward policy. The algorithm also allows for learning the backward policy. The approach is experimentally evaluated on hypergrids, biological sequence environments and bayesian structure learning tasks.", "questions": "0) See Weaknesses.\n\n1) Why are there expected values in Equations 1 and 5? Subtrajectory balance conditions mean that the forward and backward subtrajectory flows should be equal for all possible subtrajectories, not only on average (see [1]).\n\nReferences:\\\n[1] Madan et al. Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper studies an interesting and important direction of applying policy gradient methods to GFlowNets. I also found the presented connections between evaluation functions and flows to be interesting and insightful (although not entirely new, see Weaknesses).", "weaknesses": "The main weakness of this work in my opinion is its limited novelty. The definition of evaluation function (Equation 3) coincides with the negative entropy-regularized value in the construction that frames GFlowNet learning as an entropy-regularized RL problem from [1]. The connections between entropy-regularized values and flows were also previosly presented in [1] (see Theorem 1 and Proposition 1 of [1]). Theorem 3.1 in the present paper can be seen as a generalization of results connecting flows and values from Theorem 1 and Proposition 1 of [1], however, from what I understand, this can be shown by a simple modification of the proof of Proposition 1 from [1]. In addition, such contributions and their novelty must be correctly framed with respect to the previous literature, while the work [1] is neither discussed nor cited in this paper.\n\nThe proposed SubEB objective (Equation 8) exactly coincides with SubTB loss [2], which is already well-known and widely studied in the GFlowNet literature. The only difference is that this loss is used to learn only flows/values in the proposed algorithm for the current policy, while the policy uses a different expression for gradient update. The policy gradient training of GFlowNets itself is also not new and was previously studied in [3]. From what I understand, the difference of the proposed algorithm from the algorithm in [3] is a different way to evaluate states/trajectories (by learning the evaluation function via SubEB/SubTB).\n\nThe writing in this work seems unclear in a number of places. In line 46 it is stated that the direct optimization of distributional divergences between forward and backward trajectory distribution is not possible since the normalizing constant is unkown. However, for example, the normalizing constant does not affect the optimization of KL divergence, and directly optimizing it is possble (see [4]). Moreover, in line 345 the authors themselves acknowledge that the are works that directly optimize this KL divergence. I also found the writing regarding backward policy optimization in Section 3.2 and Section 3.3 to be a bit jumbled up and hard to get through (e.g. $W$ is used in the text prior to its definition), so the clarity could be improved here in my opinion. However, since this is mostly a retelling of the results from the prior work [3], I think this is a minor point. In addition, Algorithm 1 does not specify how the gradients wrt forward policy are calculated. Does it use the expression from Equation 4?\n\nAnother minor point, but since the paper partially focuses on the backward policy optimization, it would be good to cite and discuss prior works on this topic in the text [5, 6].\n\nNext, the presented empirical results on sequence design tasks, which are the only non-synthetic experiments in the main text, are unconvincing. The improvements are very marginal in comparison to SubTB in Figure 3 in the case when the local search is used. When it is not used, the improvements are notable only in the first two plots. \n\nIn addition, all the experiments in the main text are on environments of a very small scale: 256 × 256 grid (65 thousand terminal states), 64 x 64 x 64 grid (262 thousand terminal states), SIX6 (65 thousand terminal states), QM9 (58 thousand terminal states), PHO4 (1 million terminal states), sEH (34 million terminal states). All of these environments can be considered toy since the space of objects we work with is small enough that it is possible to iterate over all objects in reasonable time, thus the task of training a GFlowNet policy for sampling is artificial and redundant. There are experiments on a larger-scale structure learning task in Appendix, so I would suggest moving them to the main text and putting more emphasis on them.\n\nOverall, I found this to be an interesting paper, but the presented contributions in my opinion are not enough to recommend acceptance to a conference of this level. \n\nReferences:\\\n[1] Tiapkin et al. Generative Flow Networks as Entropy-Regularized RL. AISTATS 2024\\\n[2] Madan et al. Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023\\\n[3] Niu et al. GFlowNet Training by Policy Gradients. ICML 2024\\\n[4] Malkin et al. GFlowNets and variational inference. ICLR 2023\\\n[5] Jang et al. Pessimistic Backward Policy for GFlowNets. NeurIPS 2024\\\n[6] Gritsaev et al. Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization. ICLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)