QGFN: Controllable Greediness with Action Values
Authors: Elaine Lau
Stephen Zhewen Lu
Ling Pan
Doina Precup
Emmanuel Bengio
Venue: NeurIPS 2024 poster (2024)
Keywords: GFlowNets
generative models
molecule design
Primary area: generative_models
TL;DR: We combine a GFlowNet policy and an action-value estimate, $Q$ into mixture policies that get better rewards without losing diversity

Abstract: Generative Flow Networks (GFlowNets; GFNs) are a family of energy-based generative methods for combinatorial objects, capable of generating diverse and high-utility samples. However, consistently biasing GFNs towards producing high-utility samples is non-trivial. In this work, we leverage connections between GFNs and reinforcement learning (RL) and propose to combine the GFN policy with an action-value estimate, $Q$, to create greedier sampling policies which can be controlled by a mixing parameter. We show that several variants of the proposed method, QGFN, are able to improve on the number of high-reward samples generated in a variety of tasks without sacrificing diversity.

QGFN: Controllable Greediness with Action Values
Elaine Lau
Stephen Zhewen Lu
Ling Pan
Doina Precup 23
Emmanuel Bengio*
Mila Québec Al Institute Mcciil University Google Deepmind
Valence Labs
Honn Kong University of Science and Technology
Abstract
Generative Flow Networks (GFlowNets; GFNs) are a family of energy-based generative methods for combinatorial objects, capable of generating diverse and high-utility samples. However, consistently biasing GFNs towards producing high-utility samples non-trivial. In this work, we leverage connections between GFNs and reinforcement learning (RI.) and propose to combine the GFN policy with action-value estimate, Q, create greedier sampling policies which be controlled by mixing parameter. We show that several variants of the pro- posed method, QGFN, are able to improve on the number of high-reward samples generated in variety of tasks without sacrificing diversity.
Introduction
Generative Flow Networks [Bengio et al., 2021a,b), also known as GFlowNets, or GFNs, were recently introduced as novel generative framework in the family of energy-based models [Malkin et al., 2022b, Zhang et al., 2022]. Given some energy function f fa:) over objects X, the promise of GFNs is to train sampler pa such that at convergence pa (x) x exp(-f(x)); exp(-f(x)) is also referred to the reward R(xx in GFN literature, inheriting terminology from Reinforcement Learning (RL). GFNs achieve this sampling via a constructive approach, treating the creation of some sequential additive process (rather than an iterative local process à la Markov chain Monte Carlo (MCMC) that can suffer from mode-mixin issues). The main advantage GFN is its ability generate a greater diversity of low- www--enggghhhghreweeeee
object
objects compared to approaches based on MCMC or RL [Bengio et al., 202 la, Jain et al., 2022, 2023], or even -hhich, while related to GFNs, accomplishes something different by default [Tiapkin al., 2023, Mohammadpour et al., 2024, Deleu et al., 2024].
To generate more interesting samples and avoid oversampling from low-reward regions, it is common to train a model to sample proportion to R(x) 8; |3 is an inverse temperature, typically >> 1, which pushes the model to become greedier. The use of this temperature (hyper)parameter is important control knob in GFNs. However, tuning this hyperparameter is non-trivial, which complicates training certain GFNs; for example, trajectory balance [Malkin et al., 2022a] is sensitive to the "peakiness" of the reward landscape [Madan et al., 2023]. Although it is possible to train temperature-condee conttiiioon models [Kim et al., 2023a], doing essentially requires learning whole family of GFNs-no easy task, albeit doable, e.g., in multiobjective settings [Jain et al., 2023].
In this work, we propose approach that allows selecting arbitrary greediness at inference time, which preserves the generality of temperature-conditionall while simplifying the training process. We do so without the cost of learning a complicated family of functions and without conditionals, instead only training two models: a GFlowNet and an action-value function Q [Watkins and Dayan, 1992, Mnih et al., 2013].
38th Conference on Neural Information Processing Systems (NeurIPS 2024). Correspondence to: Elaine Lau <tsoi lau@mail mogill.cas ca> Source code available at: https://gg //ttt////btbbb com/ caam//yggaaa//PP "Thi work was done during an internship at Valence Labs.

---

Armed with the forward policy of a GFN, Pp (which decides probability distribution over actions given the current state, i.e. in RL), and the action-valus value, Q, we show that it is possible create a variety of controllably greedy sampling policies, controlled by parameters that require no retraining. We show that it is possible to simultaneously learn Pr and Q, and in doing so, to generate more high-reward yet diverse object sets. In particular, we introduce and benchmark three specific variants of our approach, which we call QGFN: p-greedy, p-quantile, and p-of-max.
We evaluate the proposed methods on 5 standard tasks used in prior GFN works: the fragment-based molecular design task introduced by Bengio et al. [2021a] J RNA design tasks introduced by Sinai et al. [2020], a small molecule design task based on QM9 [Jain et al., 2023], as well as a bit sequence task from Malkin et al. [2022a], Shen et al. [2023]. The proposed method outperforms strong baselines, achieving high average rewards and discovering modes more efficiently, sometimes by large margin. We conduct an analysis of the proposed methods, investigate key design choices, and probe the methods to understand why they work. We also investigate other possible combinations of Q and Praagain, entirely possible at inference time, by combining trained Q and Pp models.
2 Background and Related Work
We follow the general setting of previous GFN literature and consider the generation of discrete finite objects, but in principle our method could be extended the continuous case [Lahlou et al., 2023].
GFlowNets GFNs [Bengio et al., 2021b] sample objects by decomposing their generation process (8o, .., ST) of constructive steps. The space can be described by pointed directed acyclic graph (DAG) g = AS, where s E S is a partially constructed object, and (s + 5') A C S a unique initial state 80.
in a sequence
a valid additive step (e.g., adding a fragment molecule). is rooted at
GFNs are trained by pushing a model satisfy so-called balance conditions of flow, whereby flows F(s) going through states are conserved such that terminal states (corresponding to fully constructed objects) are sinks that absorb R(s) (non-negative) units of flow, and intermediate states have much flow coming into them (from parents) as flow coming out of them (to children). This can be described succinctly follows, for any partial trajectory (Sn:-1 8m):
F(s.) II PP(s441(s)) = (sm)
II Ps(s:(5441)
where Pp and PB. the forward and backward policies, are distributions over children and parents respectively, representing the fraction of flow emanating forward and backward from a state. By construction for terminal (leaf) states F(s) R(s).
Balance conditions lead to learning objectives such as Trajectory Balance [TB; Malkin et al., 2022a]. where n = 0 and m is the trajectory length, and Sub-trajectory Balance [SubTB; Madan et al., 2023]. where all combinations of (n, m) are used. While variety of GFN objectives exist, we use these two they are considered standard. If those objectives are fully satisfied, i.e. 0-loss everywhere, terminal states are guaranteed to be sampled with probability x R(s) [Bengio et al., 2021a].
Action values the action-value function QT(ss, a). which estimates the expected "reward-to-go when following a policy π starting in some state and taking action a; for some discount 0 S S 1,
For broad overview of RL, see Sutton and Barto [2018]. A central object in RL is
("((s, a) EE (,s))
Σ 7' (' a0
While a(s,a) can be a stochastic transition operator, in a GFN context objects are constructed in a deterministic way (although there are stochastic GFN extensions; Pan et al. [2023b]). Because rewards are only available for finished objects, R(s) = 0 unless is terminal state, and we use 7 = to avoid arbitrarily penalizing "larger" objects. Finally, as there are several possible choices for we will simply refer to Q" as Q when statements apply to large number of such choices.

---

X
sf
Sb
si
2
R(x) 100
GFNs
F(C() = 100
F(>) = 100
Xg9
RL
(() =
((s,,>)) = 100
R(x) =
edge the DAG
$ intermediate
x
terminal state
Rxx) reward
Figure I: Solely relying on flow functions F in GFNs can be insufficient. While GFNs capture how much stuff there is, they spend time sampling from lots of small rewards.
2.1 Related Work
RL and GFlowNets There are clear connections between the GFN framework and RL frame- work [Tiapkin et al., 2023, Mohammadpour et al., 2024, Deleu et al., 2024]. Notably, Tiapkin et al. [2023] show that is possible to reformulate fixed-Pp GFlowNets as a Soft-RL problem within a specific class of coward-modifee MDPs. While they show that this reformulated problem then be tackled with any Soft-RL method, this still essentially solves the original GFlowNet problem, i.c. interested in greedier-yet-divess methods.
learn (1( R(z). Instead,
commonly used tool GFNs (and QGFN) to increase the average reward, aka "greediness" the drawn samples, is to adapt the reward distribution by using an altered reward function R(x)) = R(x)) and adjusting the exponent parameter 3: the higher the 3, the greedier the model should be [Jain et al., 2023]. However, increasing 3 often induces greater numerical instability (even on a log scale), and reduces diversity because the model is less incentivized to explore "middle-reward" regions. This can lead to mode collapse. Kim et al. [2023a] show that is possible to train models that are conditioned on 3, which somewhat alleviates these issues, but at the cost of training more complex model.
Again, while we could leverage the equivalence between the GFN framework and the Soft-RL framework [TTaakin et al., 2023], this approach would produce soft policy. We propose a different approach that increases greediness of the policy via "Hard-RL".
Improving GelowNet sampling number of works have also made contributions towards im- proving utility in GFlowNets, via local search [Kim et al., 2023b], utilizing intermediate signals (Pan et al., 2023a], or favoring high-potential-uriitty intermediate states [Shen et al., 2023], as well as the use of RL tools such as replay buffers [Vemgal et al., 2023], target networks [Lau al., 2023], or Thompson sampling [Rector-Brow Brooks et al., 2023].
Motivation
Consider the following scenario, illustrated in Figure an agent faced with two doors. Behind the left door, there are 100 other doors, each hiding reward of 1. Behind the right door, there is single door hiding a reward of 100. The flow will be such that Flefft) = F(right) = 100, meaning that a GFN agent will pick either door with probability 1/2. The action value function is Q(so. left) 1, Q(50, right) = 100, so an agent basing its decisions on Q will reach for the door with reward 100.
This example shows that relying solely on flows is not always sufficient to provide high-value samples frequently and illustrative of real-world scenarios. Consider molecular design very large search space) with some reward in [0. 1]; there may be 10 molecules with reward .9, but just dozen with reward Since .9 x 10 is much bigger than 12 1, the probability of sampling a reward I molecule will be low if one uses this reward naively. While using temperature parameter is a useful way to increase the probability of the reward molecules, we propose a complementary, inference-time adjustable method.
3

---

Note that relying solely on Q is also insufficient. If Q were estimated very well for the optimal policy (which extremely hard), it would be (somewhat) easy to find the reward molecules via some kind of tree search following Q values. However, in practice, RI. algorithms easily collapse to non-diverse solutions, only discovering a few high reward outcomes. This is where flows are useful: because they capture how much stuff there is in particular branch (rather than an expectation), it is useful to follow flows to find regions where there is potential for reward. In this paper, we propose a method that can be greedier (by following Q) while still being exploratory and diverse (by following F through Pp).
QGEN: controllable greediness through Q
Leveraging the intuition from the example above, we now propose and investigate several ways in which we can use Q-functions to achieve our goal; we call this general idea QGEN. In particular, we present three variants of this idea, which are easy implement and effective: p-greedy QGFNs, p-quantile QGFNs, and y-of-max QGFNs. In $5 and $6, we show that these approaches provide a favourable trade-off between reward and diversity, during both training and inference.
As is common in GFlowNets, we train QGFN by sampling data from some behavior policy μ. We train F' and PF (and use a uniform PB) to minimize flow balance loss on the minibatch of sampled data, using a temperature parameter B3. Additionally, we train a Q-network to predict action values on the same minibatch (the choice of loss will be detailed later). Training the GFN and Q variety of behaviors 1s is possible because both are off- off-policy methods. Indeed, instead of choosing μ to be a noisy version of Pp is usual for GFNs, we combine the predictions of PF and Q to form greedier behavior policy. In all proposed variants, this combination is modulated by a factor p E [0, 1]. where p means that jr depends only on Pr, and p 1 means μ is greediest, as reflected by Q. The variants differ in the details of this combination.
p-greedy QGEN Here, we define μs mixture between Pp and the Q-greedy policy, controlled by factor p:
((5'1s) (1 (1 Pr('sss)))) + pIls' argmax, Q(s, i)]
In other words, we follow Pr. but with probability p. the greedy action according to Q is picked. All states reachable by Pp are still reachable by μa. Note that p can be changed to produce very different without having to retrain anything.
p-quantile QGEN Here, we define μ as masked version of Pr, where actions below the p-quantile of Q, denoted qp,Q, 8), have probability 0 (so are discarded):
p (s) x P=(5')s)
2 op (Q,s]
This can be implemented by sorting Q and masking the logits of PF accordingly. This method is more aggressive, since it prunes the search space, potentially making some states unreachable. Again, p is changeable.
p-of-maa QGEN Here, we define μ as a masked version of Pp, where actions with Q-values less than max, Q(s, a) have probability O:
((s'(s) x Pp(ss(s)[(Q(s s') 2 p pmax ((s, i)]
(5)
This is similar to p-quantile pruning, but the number of pruned actions changes as a function of Q. If all actions are estimated as good enough, it may be that no action is pruned, and vice versa, only the best action may be retained is none of the others are good. This method also prunes the search space, and remains changeable. Note that in a GFN context, rewards are strictly positive, so Q is also positive.
Policy evaluation, or optimal control? In the design of the proposed method, we were faced with an interesting choice: what policy should Q" evaluate? The first obvious choice is to perform Q- learning [Mnih et al., 2013], and estimate the optimal value function Q", with a 1-step TD objective. As we detail in $6, this proved to be fairly hard, and 1-step Qa ended up being poor approximation.
commonly used trick to improve the performance of bootstrapping algorithms is to use n-step returns [Hessel et al., 2018]. This proved essential our work, and also revealed something curious:

---

Average Reward
Number of Modes
Top-1000 Tanimoto similarity
1.2
Average Reward
Number of Modes
Sinilarity score
1410
0.7
0.4-
6.3
2008
MurCON AZC. DABTE, DON
02
5
Trajectories Sampled
Trajectories Sampled
Trajectories Sampled
p-greedy QGFN
p-of-max QGFN
SubTB
LSL-GFN
MunDON
p-quantile QGFN
TB
DQN
A2C
Figure 2: Fragment-based molecule task. Left: Average rewards over the training trajectories. Center:
Number of unique modes with reward threshold exceeding 0.97 and pairwise Tanimoto similarity score less than 0.65. Right: Average pairwise Tanimoto similarity score for the top 1000 molecules sampled by reward. Lines are the interquartile mean and standard error calculated over 5 seeds.
we've consistently found that, while results started improving at 2 5, consistently good value of n was close the maximum trajectory length. This has interesting interpretation, as beyond certain value of n. Qo becomes closer to Q" and further from Q- In other words, using an "on-policy" estimate Q" rather than an estimate of the optimal policy seems beneficial, at least easier to learn as whole. In hindsight, this makes sense because on on-policy evaluation is easier than learning Q', and since we are combining the Q-values with Pr, any method which biases μ correctly towards better actions is sufficient (we do not need to know exactly the best action, its exact value).
Selecting greediness In the methods proposed above, can be changed arbitrarily. We first note that we train with a constant or annealed value of p and treat as standard hyperparameter in all the results reported in $5.
Second, as discussed in $6, after training, p can be changed with a predictable effect: the closer p is to 1, the greedier μ becomes. Presumably, this is because the model generalizes, and Q-value estimates for "off-policy" actions are still a reliable guess of the reward obtainable down some particular branch. When making p higher, may remain a good lower bound of the expected reward (after all, μ is becoming greedier), which is still helpful. Generally, such a policy will have reasonable outcomes, regardless of the specific μ and p used during training. Finally, it may be possible and desirable use more complex schedules for p. or to sample during training from some (adaptive) distribution, but we leave this for future work.
Main results
Average Reward
Number of Modes
We experiment on 5 standard tasks used in prior GFlowNet literature. As baselines, use Trajectory Balance, Sub-Trajectory Balance, LSL-GFN [Kim al., 2023a] i.e. learning to scale logits which controls greediness through cemperatture-condiiionnng and as RL baselines A2C [Mnih et al., 2016], DQN [Mnih et al., 2013] (which on its own systematically un- derperforms in these tasks), and Tiapkin et al. [2023]'s MunDQN//FlooNee.
Number Modes
Averaye
DON
02 t0
Trajectories Sampled
is
26 25
10
15 26 Pajectories Sampled
p-greedy QGFN
T8
LSL-GFN
puquatticc QGFN
SubTB
A2C
MunDON
Figure 3: QM9 task. Left: Average rewards over training trajectories. Right: Number of modes with a reward above 1.10 and pairwise Tanimoto similarity less than 0.70.
We report the average reward obtained by the agents, as well the total number of modes of the distribution of interest found during training. By mode, mean high-reward object that is separated from previously found modes by some distance threshold. The distance function and
2Specifically for p-quantile QGFN and p-of-mar max. we found training to be more stable if we started with p and annealed p towards its final value with single 1/- perioo cosine schedule over 1500 steps.
5

---

L244 ANAL Average Reward
114RNA1 Number of Modes
114 RNA1++ Average Roward
L14 R2 Number of Modes
Normgee tewenr
Mollss
to ' 34
Averoge Rewaee
Numce Mecees
0.5
50-
24
Tayeciores Sampleed
Sampled
Sampleed LSL-GFN
p-greedy QGFN p-quantiee QGFN
p-of-max QGFN
SubTB DQN
MunDON
Figure 4: RNA- tasks, Average reward and modes. Left: L14RNAI task. Right: L14RNAI+2
task, based on 5 seeds (interquartile mean and standard error shown).
threshold we use, as well as the minimum reward threshold for an object be considered a mode, depend on the task.
trggennt-based molecule generation task: Generate graph of up to 9 fragments, where the reward is based on prediction of the binding affinity to the sEH protein, using a model provided by Bengio et al. [2021a]. |X] 10200 there are 72 available fragments, some with many possible attachment points. We use Tanimoto similarity [Bender and Glen, 2004], with a threshold of 0.65, and reward threshold of 0.97. Results are shown in Fig. 2.
Atom Based QM9 task: Generate small molecules of up to atoms following the QM9 dataset [Ra- makrishnan al., 2014]. | =s 10 the action space includes adding atoms bonds, setting node or bond properties and stop. A MXMNet proxy model [Zhang et al., 2020], trained on QM9, predicts the HOMO-LUMO gap, key indicator of molecular properties including stability and reactivity, and used as the reward. Rewards are in the [0, 2] range, with a 1.10 threshold and a minimum Tanimoto similarity of 0.70 to define modes. Results are shown in Fig. 3.
RNA-binding task: Generate string 14 nu- cleobases. The reward is a predicted binding affinity to the target transcription factor, pro- vided by the ViennaRNA [Lorenz et al., 2011 1] package for the binding landscapes; we exper-
Number of Modes
50
Number Modes
40
30
iment with two RNA binding tasks; L14-RNA1,
20
and LI4-RNAI+2 (two binding targets) with
optima computed from Sinai et al. [2020]. | is 414 se 10", there are tokens: adenine (A), cytosine (C), guanine (G), uracil (U). Results are shown Fig. 4.
puuuntile QGFN, DON
Trajectories Sampled
165
F-greedy OGFN
QEEN
SUSTD
1SL-GEN
Prepend-Append bit sequences: Generate a bit sequence of length 120 in a prepend-
Figure 5: Bit sequence task, 1: 1. Interquartile append MDP, where [x'. limited to {0, mean and standard error over seeds.
220 x 30 For a sequence of length n.
R(z) exp (1 minye d(z,, v)/n). A sequence is considered mode if it is within edit distance from M, where M is defined as per Malkin et al. [2022a] (although the task we consider here is a more complex version, introduced by Shen et al. [2023], since prepend actions induce DAG instead of a simpler tree). In our experiment, |M] 60, = 120, & = 1,6 28, where is the bit width of actions. Results shown in Fig.
5.1 Analysis of results
Across tasks, QGFN variants produce high rewards and find higher number modes, i.e. high- reward dissimilar objects. The latter could seem surprising, because a priori, increasing the greediness of method likely reduces its diversity. This fundamental trade-off is known in RL as the exploration- exploitation dilemma [Sutton, 1988, Sutton and Barto, 2018]. However, methods and combining their strengths to reap the best from both worlds: GFlowNets are able to cover the state space, because they attempt to model all of it, by learning (x)) x R(x),, while Q approximates the expected reward of a particular action, which can help guide the agent by selecting
are leveraging two
We note that there exist number of differing implementations of this task in the GFlowNet literature, we that of httpp https:// //ts::///ttttlt www/recursioneeoooccccoooooooceo www.rarraaaaaaaaaaaaaa gf ownet

---

Table 1: Fragment-basee molecule task: Reward and Diversity inference after training.
METHOD
REWARD(T)
SIMILARITY (4)
GFN-TB
0.780=+0.003
0.545=0.002
GFN- SUBTS
0.716:10.006
0.513=0.003
LSL-GFN
0.717=0.020
0.689=0.062
P-GREEDY QGEN
0.95011.000
0.551=0.015
P-OF-MAX QGEN
0.969=0.003
0.514#0.001
P-QUANTILE QGEN
0.955=000003
0.509=0.008
high-eexpetee-reewadd branches. Another way to think about this: GFNs are able to estimate how many high-reward objects there are in different parts of the state space. The agent thus ends up going in all important regions of the state space, but by being bit more greedy through Q. it focuses on higher reward objects, so it is more likely to find objects with reward past the mode threshold. To further understand the performance of QGEN, we formally analyse bandit setting, and include derivations to illustrate the general case, in Appendix SA.
We also report the average reward and pairwise similarity for the fragment task based on 1000 samples over seeds taken after training in Table 1. Again, QGENs outperform GFNs in reward, while retaining low levels of inter-sample similarity. We again note that at inference, we are able to use a different (and better) p value than the one used at training time. We expand on this in $6, and show that it is easy to tune p to achieve different neward-diversee p values used for Table 1 are provided in Appendix $E. Also note that in LSL-GFN B is tunable at inference, and in Table 1 we choose the 3 value such that average similarity is near the 0.65 mode threshold we use (choosing a greedier /3 induces a collapse in diversity).
trade-offs at inference. The exact
QGEN variants matter We point the reader to an interesting result, which is consistent with our understanding of the proposed method. the fragment task, the number of actions available to the agent is quite large, ranging from about 100 to 1000 actions depending on the state, and the best performing QGEN variant is one that consistently masks most actions: p-quantile QGEN, is likely indeed that most actions are harmful, combining two fragments that do not go together may be irreversibly bad, and masking helps the agent avoid undesirable regions of the state space. However, masking a fixed ratio of actions can provide more stable training.
On the other hand, in the RNA design task, there are only 5 actions (4 nucleotides ACGU & stop). We find that masking constant number of actions is harmful-it is likely that in some states all of them are relevant. So, in that task, p-greedy and p-of-max QGFN work best. This also the case in the bit sequence task, for the same reasons (see Fig. 5). To confirm this, we repeat the bit sequence task but with an expanded action space consisting not just {0, 1}, but of all 16 (24) sequences of 4 bits, i.e. {0000, 0001, .., 1111}. We find, as shown in Fig 18, that p-quantile indeed no longer underperforms.
6 Method Analysis
We now analyze the key design choices in QGFN. We start by investigating the impact of (the number of bootstrapping steps in Q-Learning) and p (the mixture parameter) on training. We then look at trained models, and reuse the learned Q and Pr show that it is possible to use variety of sampling strategies, and to change the mixture factor to obtain a spectrum of greediness at test time. Finally, we empirically probe models provide evidence to why QGFN is helpful.
Impact of 3 in QGEN Fig. shows the effect of training with different 3 values on the average reward and number of modes when taking 1000 samples after training is done in the fragment task (over 5 seeds). As predicted, increasing 3 increases the average reward of the agent, but at some point, causes it to become dramatically less diverse. As discussed earlier, this is typical of GFNs with a too high B, and is caused by a collapse around high-reward points and an inability for the model to further explore. While QGFN is also affected by this, it does not require drastic values of B to obtain a high average reward and discover a high number of modes.
Impact of in QGEN As mentioned in $4, training Q with 1-step returns is ineffective and produces less useful approximations of the action value. Fig. shows the number of modes within 1000 post- teattrraiiig samples in the fragment tasks, for models trained with variety of n-step values.
7

---

Numbor
Monbbr 1en0le
2.00
Number of modes
-30
Number
338
252
Loast Grendy
Greddut
Effect
78
P-g. antlle DGFN
p-at-max OGTN
p-greedy OGTN
P-paattee QUTN
p-tmaax OGTN
pigreedy GGTN
Figure 6: Fragment task. Left: Effect of 3: Increasing greediness through 8 increases the average reward but may lead diversity collapse. QGFN maintains diversity with a lower B. while GFN collapses. Modes are counted from 1000 samples at inference, using an inference-adjusted Effect of training parameters p. and n: Changing p can control greediness, while increasing n is generally beneficial. Modes are counted from 1000 samples generated using the training p.
Right:
Models start being consistently good at and values close to the maximum length of a trajectory tend to work well too.
Impact of the training p in QGEN While our method allows changing p more or less at will, we still require some value during training. Fig. 6 shows that there are clear trade-offs between choices of p, some yielding significantly better diversity than others. For example, p-of-max is fairly sensitive to the chosen value during training, and for the fragment task doesn't I seem to perform particularly well during training (especially when not annealed). On the other hand, as we will see in the next paragraph (and is also seen in Fig. 6), p-of-max excels at inference. and is able to generate diverse and high-reward samples by adjusting p.
Changing strategy after training We now look at the impact of changing the mixture parameter p and the sampling strategy for already trained models average reward and average pairwise similarity. We use the parameters
model trained with p-greedy QGEN, = 0.4.
With this model, we sample 512 new trajectories for a series of different, values. For p-greedy and p-quantile, we vary p between and I; for p-of-max, we vary p between .9 and (values below .9 have minor effects). We visualize the effect of reward and similarity statistics in Fig. 9.
First, we note that increasing p has the effect we would hope. increasing the average reward. Second. we note that this works without any retraining; even though we (a) use values of different than those used during training, and (b) use QGFN variants different than those used during training, the behavior is consistent: p controls greediness. Let emphasize (b): even though we trained this Q with p-greedy QGFN, we are able use the Q(s, a) predictions just fine with entirely different sampling strategies. This has some interesting implications; most importantly, it can be undesirable to train with too high values of p (because may reduce the diversity to which the model is exposed), but what is learned transfers well to sampling new, high-reward objects with different values of p and sampling strategies.
Finally, these results suggest that we should be able prototype new QGFN variants, including expensive ones (e.g. MCTS) without having to retrain anything. We illustrate the performance of few other variants in $B.2, Fig. 12.
1.00
Predicted
0.95
Is calibrated? For intuition of why QGFN works to really pan out, Q has to be accurate enough to provide useful guidance to- wards high-reward objects. We verify that this the case with the following experiment. We take a trained QGFN model (p-greedy, 0.4, fragment task, maximum n-step) and sample trajectories. For each of those trajectories, we take random state within the trajectory as starting point, thereafter generating 512 new tra-
0.90
0.85
0.8
0.850
0.875 0.900 0.925 0.950 0.975 1.000 1.025 1.050
Empirically estimated O"
Figure 7: Comparing Q(s, a; 0) predictions with empirical estimates obtained by rollouts. Bars are standard error. Q relatively capable estimate the returns of the corresponding policy.
jectories. We then use the reward of those 512 trajectories empirical estimate Q of the expected

---

return, which Q should roughly be predicting. Fig. 7 shows that this is indeed the case. Although Q
is not perfect, and appears to be underestimating Q, it is making reasonable predictions.
Is Q really helpful? In this experiment, we verify our intuition that pruning based on Q- values is helpful. We again take a trained model for the fragment task, and sample 512 trajecto- ries. We use p-of-max QGFN (p 0.95), and compare it to a strategy estimating best prumed actions: for each trajectory, after some random number of steps ~ U14, 20] (the max is 27), we start deterministically selecting the action that is the most probable according to PF but would be masked according to Q. To ensure that this is valid thing to do, we also simply look at Best actions, i.e. after ~ U [4, 20) steps, de- terministically select the action that is the most probable according to PF. regardless of Q.
p-ofmax QGFN (0.95)
pruned actions
Best actions
actions
Density
0.10
0.05
0.00
1.0
Re Rewd
Figure 8: Pruning helps avoid low-reward parts of the state space. Reward distributions when (a) sampling with p-of-max; (b) greedily according to PF selecting actions that p-of-max would prune, Best pruned actions; (c) selecting most likely Pf actions regardless of Q, Best actions; and (d) nor- mal sampling from Pp (without using Q).
Fig. 8 shows that our sanity check, Best actions, receives reasonable rewards, while selecting ac- tions that would have been pruned leads to much lower rewards. The average likelihood from Pr of these pruned actions was .035, while the av- erage number of total actions was R 382 (and 1/382 0.0026). This confirms our hypothesis that Q indeed masks actions that are likely ac- cording to Pe but that do not consistently lead to high rewards.
greediest
Average reward
0.95
Why does changing p work? Recall that for QGFN to be successful, we rely on n-step TD, and therefore on somewhat "on-policy" es- timates of Q*. μ is really μp. a function of p, meaning that if we change p, say to p', during inference, Q"P
0.85
p-of-max greedy p-quantile
least preedy
least greedy
0.5
0.6
0.7
0.8
0.9
1.0
Average similanty
not an accurate estimate of
Q If this is the case, then there must be reason why it is still helpful to prune based on Qt while using Mp'. In Fig. 10,we perform the same measurement as in Fig. 7, but we change the p value used to measure Quyy We find that, while the rank correlation drastically goes down (although it stays well above 0), Q' remains helpful because it lower bounds Que' If we prune based on Q"',, then we would want it to not lead us astray, and at least make us greedier as we increase p. This means that if an action not pruned, then we expect samples coming from it to be at least good as what Q (s, a) predicts (in expectation). This is indeed the case.
Figure 9: Varying at inference time induces
reward-diversity trade-offs; fragment task.
Note that reducing simply leads to behave more like Pr, which is still good sampler, and to rely less on Q, whose imperfections will then have less effect anyways.
7 Conclusion and Discussion
In this paper, we showed that by jointly training GFNs and Q-functions, predictions to form behavior policies that are able to sample larger numbers of diverse and high- reward objects. These policies' mixture parameter is adjustable, even after training, to modulate the greediness of the resulting policy. We implement multiple ways of combining GFNs and Qs, referring to the general idea QGFN: taking greedy with probability (p-greedy QGFN), restricting the agent to the top p% of actions (p-quantile QGFN), and restricting the agent to actions whose estimated value is least fraction p of the best possible value (n-of-max QGFN).
combine their
We chose to show several variants of QGFN in this paper, because they all rely on the same principle, learning Q, but have different properties, which lead to better or worse behavior in different tasks.

---

For example, pruning too aggressively on task with a small number of actions is harmful. We also hope that by showing such a diversity of combinations of Pp and Q, we encourage future work that combines GFNs and RI methods in novel and creative ways.
We also analyzed why method works. We showed that the learned action-value is predictive and helpful in avoiding actions that have high probability under PF but lower expected reward. Even when predictions are not accurate, e.g. because we sample from different policy than the one which Q models, they provide a helpful lower bound that facilitates controllable greediness.
Our analysis suggests that training time, QGEN works because it helps the agent "waste" less time and capacity modeling low-reward objects, and that conversely the policy family that QGEN learns is able to sample more distinct nigh-reward objects given the same budget. In this sense, QGFN benefits from the advantages of both the GelowNet and "Hard"- -RI frameworks.
What didn't work The initial stages of this project were quite different. Instead of combining RL and GFNs into one sampling policy, we instead trained two agents, a GFN and DQN. Since both are off-policy methods and make GFN better on high-reward trajectories. This was not the case, instead, the DQN agent simply slowed down the whole method-despite trying wide variety of tricks, see &C.
were hoping that sharing "greedy" DQN data with a GFN would be fine
Limitations Because we train two models, our method requires more memory and FLOPs, and consequently takes more time to train compared to TB (as shown in Table 3). QGFN is also sensitive to how well Q is learned, and as we 'v shown n-step returns are crucial for our method to work. In addition, although the problems we tackle are non-trivial, we do not explore the parameter and compute scaling behaviors of the benchmarked methods.
Future work We highlight two straightforward avenues of future work. First, there probably exist more interesting combinations of Q and PF (and perhaps F),, with different properties and benefits. Second, it may be interesting to further leverage the idea of pruning the action space based on Q, forming the basis for some sort of constrained combinatorial optimization. By using Q to predict some expected property or constraint, rather than reward, we could prune some of the action space to avoid violating constraints, or to keep some other properties below some threshold (e.g. synthesizability or toxicity in molecules).
Finally, we hope that this work helps highlight the differences between RL and GFlowNet, while adding to the literature showing that these approaches complement each other well. It is likely that we are only scratching the surface of what possible in combining these frameworks.
10

---

Acknowledgements
The bulk of this research was done Valence Labs as part of an internship, using computational resources there. This research was also enabled in part by computational resources provided by Calcul Québec, Compute Canada and Mila. Academic authors are funded by their respective academic institution, Fonds Recherche Quebec through the FACSAcquity grant, the National Research Council of Canada and the DeepMind Fellowships Scholarship.
The authors are grateful to Yoshua Bengio, Moksh Jain, Minsu Kim, and the Valence Labs team for their feedback, discussions, and help with baselines.
Author Contributions
The majority of the experimental work, code, plotting, and scientific contributions were by EI., with support from EB. SI. helped run some experiments, baselines and plots. The project was supervised by EB, and DP and LP provided additional scientific guidance. Most of the paper was written by EB. DP, LP, and EL contributed to editing the paper.
References
Andreas Bender and Robert C Glen. Molecular similarity: a key technique in molecular informatics. Organic & biomolecular chemistry, 2(22),3204-3218, 2004.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network hased generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 14:27381-273944 2021a. URL https:
org/ abs//166........ 043999
Yoshua Bengio, Tristan Deleu, Edward Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. arXiv preprint arXiv:2111.099 100026,,, 2021b. URL https: https: :ttt:ii////aai//aaaaeeaeeaasaava org/ abs//11.. 09266.
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, and Yoshua Bengio. Discrete probabilistic inference control in multi-path environments. arXiv preprint arXiv:2402.10309 2024.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pages 1861-1870. PMLR, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Moksh Jain, Emmanuel Bengio. Alex Hernandez- Garciaa Jarrid Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pages 9786-9801. PMLR, 2022.
URL https: http/axxivvv.r/... org/ abaa 2203. 04115.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernández Garcia, Jarrid Rector-Brook Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. International Conference on Machine Learning, pages 14631-14653 PMLR, 2023. URL
wwwrriiiv../reeeee ab//aaas///// 12765.
Minsu Kim, Joohwan Ko, Dinghuai Zhang, Ling Pan, Taeyoung Yun, Woochang Kim, Jinkyoo Park, and Yoshua Bengio. Learning to scale logits for cemperature-condition gflownets. arXiv preprint arXiv:2310.02823,
2023a. URL https::t//// / /twpiv..ai////////////////aaaaa. org/ aa/aab////// 2t1...31..0000 028223.
Minsu Kim, Taeyoung Yun, Emmanuel Bengio. Dinghuai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets. arXiv preprint urXiv:2310.02710 2023b.
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Hernández-Garcua, Iéna Néhale Ezzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pages 18269- 18300. PMLR, 2023.
Greg Landrum. Rdkit documentation. Release, 1(1-79):4, 2013.
Elaine Lau, Nikhil Vemgal, Doina Precup, and Emmanuel Bengio. Dgfn: Double generative flow networks. arXiv preprint 211v:23100.16855 2023.
11

---

Ronny Lorenz, Stephan H Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo L Hofacker. Viennarna package 2.0. Algorithms for molecular biology, 6:1-14, 2011.
Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning, pages 23467-23483. PMLR, 2023.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. Advances in Neural Information Processing Systems, 35:5955-5990 2022a.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward Hu, Katie Everett, Dinghuai Zhang, and Yoshua Bengio. Gflownets and variational inference. arXiv preprint arXiv:2210.00580, 2022b. URL https: //arxiv.orgg/.eeee httpi//.rrg/b/////////// abs/2210.....22 0008001..
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou. Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu, Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
Sobhan Mohammadpour, Emmanuel Bengio. Emma Frejinger, and Puerre-Luc Bacon. Maximum entropy gflownets with soft q-learning. In International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2024.
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflownets with local credit and incomplete trajectories. In International Conference on Machine Learning, pages 26878 26890. PMLR, 2023a.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. networks. arXiv preprint arXiv:2302.09460 2023b.
Stochastic generative flow
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
Jarrid Rector Brooks, Kanika Madan, Moksh Jain, Maksym Korablyov, Cheng- Hao Liu, Sarath Chandar, Nikolay Malkin, and Yoshua Bengio. Thompson sampling for improved exploration in gflownets. arXiv preprint MiXiv:2306.16623 2023.
Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training, 2023.
Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141, 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Daniil Tiapkin, Nikita Morozov, Alexey Naumov, and Dmitry Vetrov. Generative flow networks as entropy- regularized rl. arXiv preprint arXiv:2310.12233 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1710.1003, 2017.
Nikhil Vemgal, Elaine Lau, and Doina Precup. An empirical study the effectiveness of using a replay buffer on mode discovery in gflownets. arXiv preprint arXiv:2307.07644 2023.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. arXiv preprint arXiv:2209.00 02606, 2022, URL attpp:://arxiv wrw.lw.rrr/bbbaee /ttp
12

---

Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C Courville. Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gflownets. Advances in neural information processing xystems, 36:1 1952-11969, 2023.
Shuo Zhang, Yang Liu, and Lei Xie. Molecular mechanics-driver graph neural network with multiplex graph for molecular structures, arXiv preprint urX1v:2011.07457, 2020.
13

---

A Analysing p-greedy
Consider the bandit setting where trajectories are I step and just consist in choosing terminal
state. Let (G(s) = R(s)/Z. Let 0 < P < 1, then with ju('(ss) = (1 11-pp Pp + pTls' =
arg max ((s.s)),, assuming there is only single argmax s", then P,((s R(s)) + pl[s
arg max R(s)]. This means that for every non- non-argmax state, Pu(s) = (1 - -p)pa(s) < pa((s) We
get that E [R] [f]::
[ [a[R]
Σ Pa(())((s)) - E [p((s)R(s))
(6)
=(+ + - -6--))(3)/2--(020)0) + E(-DMM93/2-R697
-pR(s") - 1/2 Σ (()))
(8)
=p/Z
R(s")z -(s))
ΣR(s)
== E R(s)
(9)
-p/Z
R(s') Σ R(s)] R(00)2-
Σ R(s)?
(10)
-p/Z
A('') R(s)) - R({))
(11)
-p/Z
Σ R(s R(()) R(s)? -R(s)
(12)
(13)
since R(s*)= > R(s) and both are positive then R(s))R(s) > R(s)2 thus the last sum is positive. All other terms are positive, therefore F,, [R] Eg R 0.
In the more general case, we are not aware of a satisfying closed form, but consider the following exercise.
Let m(s, 5¹) = I[s' arg max ((s, Let F" be the "QGEN flow" which we'll decompose as
F' = FG + FQ where we think of Fo and FiQ as the GFN and Q-greedy contributions to the flows. Then:
F'(s)=
Σ zEPar(s)
+ pm(2,s)
(14)
Σ - ( + Σ F' F'(s))m((zz))
(15)
Σ - P)) Σ F'(2)pm(zz)) s)
(16)
((-))) + > FQ(2))(21s) + (c(2)mm(.s))
(17)
Recall that p(s) x F(s). Intuitively then, the probability of being in a state is reduced by a factor (1 - p). but possibly increased by this extra flow that has two origins. First, flow FQ carried over by μ4, and second, new flow being "stolen" " from FG from parents when m(2, s) = 1, i.c. when s is the argmax child.
This suggests that flow (probability mass) in p-greedy QGFN is smoothly redirected towards states with locally highest reward from ancestors of such states. Conversely, states which have
14

---

many ancestors for which they are not the highest rewarding descendant will have their probability diminished.
B Additional experiments and analyses
In this section, we provide additional experiments to support our main findings. We explore the use of Q functions from different behavior policies, assess various QGEN inference variants, examine QGFN variants trained with alternative objectíves, and investigate the effects of weight sharing in QGEN models.
0.9
1.000
0.8
0.975
Predicted QU
0.950
0.925
0.875
Mule
0.850
0.825
0.3
0.80
0.85
0.90
0.95
1.00
1.05
Emplrically
Q"
ratio of wwwerbbouddedeeeeeeee estimates
0.80
0.75
0.75
0.70
0.70
0.65
0.60
0.65
0.55
0.60
Figure 10: What happens to Q8,, a) when changing p? We show here that while the rank correlation between Q and the empirically estimated expected reward QMp goes down when changing Q remains a useful estimate in that it mostly lower bounds Q", This means that, at worst, pruning based on the "wrong" Q & p combination drops some high-reward objects, but does introduce more lower-reward objects.
Peedies
1.0
greediest
Average reward
least greedy
0.6
0.4
jeast greedy
p-greedy p-greedy random P:
0.2
0.2
0.4
0.6
0.8
1.0
Average similarity
Figure 11: Comparison of trained Pp vs. untrained Pp with trained Q during inference on fragment- based molecule task
15

---

Table 2: Fragment- based molecule task: reward and diversity of independently trained baseline models using a trained Q The values for p- g-geedyy p-of-max, and p-quantile QGFN are set at 0.4, 0.9858, and 0.93, respectively.
TB
SUBTB
VARIANT
REWARD
DIVERSITY
REWARD
DIVERSITY
BASELINE
0.780+
0.545+0.002
0.716=00006
0.513:1.003
P-GREEDY
0.936=t0.009
0.589=0.018
0.921=0.008
0.589=0.001
P-OF-MAX
0.953:10.004
0.54510.026
0.939-100003
0.536:10.023
P-QUANTILE
0.935++0.007
0.545+0.010
0.911#0.009
0.526+00.006
B.1 Using Q from different behavior policies
Another approach we explored involves using trained Q function during inference that was trained entirely different behavior policy. Similarly, apply the QGFN algorithm at each state of sampling trajectories during inference, but with a key difference: is directly applied to a baseline model that has been trained independently. This approach aims to examine if a previously trained Q, when used in a different training setup but the same task, can guide independently trained models that may perform well during training but, with this assistance, can achieve significantly better results at inference. For instance, as shown in Figure 2, the samples generated by SubTB average around 0.7 rewards throughout training. However, using the trained Q as a greediness signal during inference allows us to discover samples with significantly higher rewards. Table 2 details the effects of applying Q during inference on independently trained baseline models for 1000 samples post-training of 5 seeds.
B.2 Trying other QGEN variants at inference
Since the only cost trying different "inference" variants of QGEN is code them, we do so of curiosity. We show the reward/diversity trade-off curves of these variants in Fig. 12, and include p-of-max baseline variant. As in Fig. 9 take 512 samples for each point in the curves (except for MCTS which is more expensive). We try the following:
p-thresh, mask all actions where Q(s,a) p:
soft-Q, not really QGFN, but as a baseline simply taking softmax(Q/T) for some tempera- ture T, which is varied the greediness parameter;
soft-Q [0.5], as above but mixed with Pp with a factor = 0.5 (i.e. p-greedy, but instead of being greedy, use the soft-Q policy);
GFN-then--. for the first Np steps, sample from PF. then sample greedily (where N is the maximum trajectory length);
MCTS, a Monte Carlo Tree Search where PF is used as the expansion prior and max, (s, a) the value of state. Since this is different sampling method, we run MCTS for a comparable amount time to other variants, getting about 350 samples, and report the average reward and diversity.
We note that these are all tried using parameters from a pretrained p-greedy QGFN model. may be possible for these variants to be much better at inference if the Q used corresponded to the sampling policy.
B.3 QGEN variants with different objective
To demonstrate the robustness of QGFN variants, we explore QGEN with different learning objectives such as SubTB in addition to the TB objective used throughout our experiments. We use the same hyperparameters, except the values (p-greedy 0.4, p-of-max 0.7, p-quantile 0.7), listed in Table 4 and run the experiments on fragment-based molecule generation. The results are shown in Figure 13 and Figure 14.
16

---

groedlesss
1.00
greediest
Averace reward
0.95
p-of-max
pithresh
soft-Qq
[0.5]
GFN thenthhnnd Q
Heast groedh
0.80
greedy
0.6
0.7
Average similarity
Figure 12: Trying other possible QGFN variants.
Average Reward
Number of Modes
Top-1000 Tanimoto similarity
1.2
6.2
Number of Modes
Similarity score
5400
4000
Average
3000
0.5
Trajectories Sampled
1+5
Trajectories Sampled
Trajectories Sampled
1es
p-greedy QGFN-SUbTB
p-quantile QGFN-SubTB
p-of-max QGFN-SubTB
SubTB
Figure 13: QGFN variants on learning objectives SubTB on Fragment-based molecule task
B.4 Exploring weight sharing in QGEN
We explore the impact of weight sharing between PF and Q, as they learn from the same environments and training samples. This sharing learning approach could improve in efficiency and performance. Fig. 15 shows the impact of weight sharing p-greedy QGFN, specifically focusing on sharing parameters across various layers of the graph attention transformer the fragment- task.
Unfortunately, naively summing the GFlowNet loss and the Q-Learning loss does not yield any improvements, and instead slows down learning. This may be due several factors; most likely, interference between Q and Pr. and a very different scales of the loss function may induce undesirable during training. A natural
stop would be adjust the relative magnitude of the gradients coming from each loss, and to consider different hyperparameters (perhaps a higher model capacity is necessary), but we leave this future work. Further exploration in this area could provide additional insights and potentially reduce training complexity.
B.5 Training Time and Inference Time Comparison
In addition performance, we investigate the training time and inference time for TB and p-greedy QGFN. All experiments for this comparison were conducted on NVIDIA V100 GPUs. The results reported in Table 3.
Table 3: Training and inference time comparison between TB and p-greedy QGFN.
TB
QGEN (p-greedy)
Training time (10,000 training iterations) Inference time (1,000 samples)
3 hours, 25 minutes
6 hours, 20 minutes min, 21 sec
min, 44 sec

---

Average Reward
Number of Modes
Top-1000 Tanimoto similarity
1.2
Number of Modes
1606
Sinilarity score
0.7
Averaage
0.4
400
Trajectories Sampled
Trajectories Sampled
Trajectories Sampled
p-greedy QGFN-FM
P-quantile QGFN-FM
p-of-max QGFN-FM
FM
Figure 14: QGEN variants on learning objectives FM (Flow Matching) on Fragment-based molecule task
Average Reward
Number Modes
Top-1000 Tanimoto similarity
1.2
Number of Modes
1000
Similarity score
4006
Average
0.1
1000
2000
2.40
1000
0.00
Trajectories Sampled
Trajectories Sampled
Trajectories Sampled
layer
layer
layers
laaerss
layers
Figure 15: Weight sharing in p p-greedy QGFN (p=0.4) with different layers
C Experiments that did not work
Several approaches attempted prior developing QGFN. These approaches involved sampling from both GFN and DQN independently and learning from the shared data. The key strategies explored include:
This method stores batches of trajectories in the replay buffer and samples them based on pairwise Tanimoto similarity threshold. aims to diversify the experience replay during training.
Diverse-Based Replay Buffer:
This strategy stores batches of trajectories in the replay buffer based on the rewards of the samples. In addition, we dynamically adjusts the sampling proportion between GFN and DQN based on the reward performance of the trajectories.
Adaptive Reward Prioritized Replay Buffer:
Weight Sharing: This involves sharing weights between GFN and DQN to potentially enhance the learning and convergence of the models.
Pretrained-Q for Greedier Actions: This method uses pretrained DQN model for sampling trajectories, helping the GFN to be biased towards greedier actions in the early learning stages.
n-step returns: As per QGFN, using more than 1-step temporal differences accelerate temporal credit assignment. This on its own is not enough to solve the tasks used this work.
Fig 16 shows the performance of these approaches evaluated on fragment based molecule generation task.
18

---

Average Reward
Number of Modes
Average Reward
Number of Modes
5000
Trajectories Sampled
Trajectories Sampled
Bascline
Weight
Replay
Replay
Figure 16: Fragment-based molecule generation task; we showcase the performance of QGFN's
predecessor, which failed to beat baselines regardless of our attempts to improve it.
Pairwise Tanimito Simi Similarity
Painwise Tanimito Similarity
Pairwise Tanimito Similanity
Pairwise se Tanimito Similarity
Close Greediest parieaan Parrwise Tanimito Similarity
Pairwise Tanimito Similarity
0.550
0.70
0.525
0.60
0.55
0.500
0.475
0.55
0.55
0.50
0.50
100
300
Greedy
10
Effect
Effect
n-step
TB
p-quantile QGFN
p-of-max QGFN
p-greedy QGFN
Figure 17: Pairwise Tanimoto Similarity scores assessing the impacts 3, training parameters p and in the fragment task. Left: An increase in 3 initially decreases sample similarities, followed by
gradual increase in similarity as the models get greedier through 8. Center: Increase greediness does not always correlate with sample similarity trade-offs with QGFN, but at peak greediness, similarity scores rebound. Right: Increasing n increases similarity among models.
D Full Algorithm
In this section, we show the detailed implementation of the QGFN algorithm with different variants in Algorithm 1. For inference, the trained models Pr and Q can be loaded to sample trajectories.
19

---

Algorithm QGFN: Full training algorithm details
Require: Reward function R: X' - 1 batch size M, Initialize models Pp with parameters 0,
Q(s, a) with parameters 8', greediness parameter p [0, 1]. training iterations I
1: For p-greedy QGFN:
((s'(s) = (1 (1-))) + pI[s' = arg max ((s, i)]
(18)
2: For p-quantile QGEN:
pu('(s) x (s((s(s)I[Q(s,, 2 (Q( s)]
(19)
where 4p (Q,)) is the p-quantile of Q over actions at state 8.
3: For p-of-max QGFN:
pldlss) x Pr(s'(s)[[((sss pmax Q(s, 1)
(20)
4: for for training iteration in I do
5: for each new trajectory Ty from 71 to TM do
6: Start 7j at 8o
7: while 8 is not terminal do
8:
Sample 8t+-1 from (s(st+1|s)) based current policy
9: Update + +
Compute trajectory balance loss for PF: Es (T)(
11:
Compute MSE n-step loss for Q-network:
(0(0)) - c(
Lo (s)
where the n-step return (1) is defined as:
G
Teek 7" max Q( d')
k=0
12:
Update 0 using VALTR; Update 0' using V Loi
13:
E
Experiment details: Fragment-based molecule generation
In this section, we give the hyperparameters used for each of our experiments in Tables 4, and Table
Parameter
Value
Batch size
64
Number of steps
10,000
Optimizer
Adam
Number of Layers
Hidden Dim. Size
128
Number of Heads
Positional Embeddings
Rotary
Reward scaling ,3 in R
32
Learning rate
x 10 -4
Z Learning rate
x 10-3
Table 4: Hyperparameters and specifications of the Graph Attention Transformer used across all models in Fragment-basee molecule generation.
20

---

Parameter
Value
Objective function
TB
p-greedy
0.4
p-quantile
0.8
p-of-max
0.91
cosine scheduler for p
1500 steps
Model architecture
Graph Attention Transformer
n-step
25
dqn 7
0.95
Table 5: Model-specific parameters for QGFN in Fragment- Frasdent-based molecule generation.
In Table 1, the values for p-greedy, p-of-max, and p-quantile QGFN are set to 0.4, 0.9858, and 0.93, respectively. These values are selected based on Figure 9. The p for p-of-mat is chosen from np. wwisssaacceeeee (0.9, 0.999, 16), with 0.9858 being one of these values. Similarly. for p-quantile. 0.93 corresponds to the second-to-le for p-greedy is selected from np. hap.linsge 1iinspacee 1, 11). For LSL-GFN, the chosen / is 78, selected from mp. www.pacccceeeecc (64, 128, 65).
value from np. linspaccceee.... 1, 16). Meanwhile, the 0.4
In our experimental setup, we follow the exact environment specifications and implementations detailed in Malkin et al. [2022a] with the proxy model, used for evaluating molecules, provided by Bengio et al. [2021b]. The architecture of the GFlow models is based on a graph attention transformer [VVličković et al., 2017]. We a reward threshold of 0.97 define modes, with a pairwise Tanimoto similarity criterion of less than 0.65. RDKit [Landrum, 2013] is used to compare pairwise Tanimoto similarity.
To follow closely the original implementation of LSL-GFN described in Kim et al. [2023a], we use ~ U11,64) where U' denotes a uniform distribution. Additionally, we define a simple Multi-layer Perceptron with hidden size of 256 the learnable scalar-to- to-scalar function for the LSL-GFN. For A2C, we use a learning rate of 1 x 10 a training epsilon of 1 x 10 2, and an entropy regularization coefficient of 1 x 10 For our SubTB baseline with equal weight.
SubTB(1), i.c. all trajectories are averaged
To maintain consistency, the graph attention transformer was used as the model for MunDQN. We sampled 64 trajectories and stored them as transitions in prioritized replay buffer of size 1,000,000. We then sampled 4096 transitions from the replay buffer to calculate the loss. The Munchausen parameters of 0. is selected from {0.10, 0. 15), an la of -2500 and a soft update = 0.1 is used in our experiments. All other parameters are same as the original MunDQN paper in Tiapkin et al. [2023].
We also ran an SAC [Haarnoja et al., 2018] baseline with different a values of 0.5, 0.7, 0.2, along with autotuning, and 7 value of 0.99, but we were unable to get discover more than 50 modes for the same amount of training iterations and mini-batch sizes.
E.1 QGEN hyperparameters:
For all variants of QGEN, we employed a grid-search approach for hyperparameter tuning, with focus on the parameters and n. Similarly, graph attention transformer is initialized as the Q. In the p-greedy QGFN variant, we selected value of 0.4 for p from the set {0.2, 0.4, 0.6, 0.8), and chose an n of 25 from the set 21,2,44 4, 8, 16, 24, 25, 26). For the p-of-max QGFN variant, p of 0.91 was chosen from {0.2, 0.4, 0.6, 0.8, 0.9, 0.91, 0.93, 0.95). with again stability throughout the training process, we applied cosine annealing with annle-period cosine schedule over 1500 steps. An additional threshold parameter of x 10-0 was applied to the condition max; ((s i) > threshold, to prevent the initial training phase from masking actions with very small to prevent instability during
25. To ensure
values. We also introduced a clipping of the Q-value to a minimum initial training. For the y-quantile QGFN, p of 0.8 was selected from (0.6, 0.7, 0.8. 0.9. 0.95), with n 25. For all variants of QGEN, the DQN employed had a 0.95, and random action probability of e was set to 0.
21

---

Experiment details: QM9
Parameter
Value
Batch size
64
Number of steps
5,000
Optimizer
Adam
Number of Layers
Hidden Dim. Size
128
Number of Heads
Positional Embeddings
Rotary
Reward scaling 3 in R
32
Learning rate
10-4
Z Learning rate
1 x 10
Table 6: Hyperparameters and specifications of the Graph Attention Transformer used across all models QM9.
Parameter
Value
Objective function
TB
p-greedy
0.4
p-quantile
0.6
p-of-max
0.6
cosine scheduler for p
1500 steps
Model architecture n-step dqn r
Graph Attention Transformer
29
0.95
Table 7: Model-specific parameters for QGFN in QM9,
In this experiment, we follow the setup described by Jain et al. [2023], but only use the HOMO-LUMO gap as reward signal. The rewards are normalized to fall between [0, IJ, although the gap proxy may range from [1,2]. As mentioned in Section 5, the modes are computed with a reward threshold of 1.10 and pairwise Tanimoto similarity threshold of 0.70. We employ the same architecture for all models as used in the fragment-based experiments. The training models are 5,000 iterations with a mini- batch size of 64, and / is set 32. RDKit [Landrum, 2013] 3] is used to compare pairwise Tanimoto similarity. We train A2C with random action probability 0.01 chosen from {0.1, 0.01, 0.001} and D.001).Similar .imiilar to the fragment- based molecule task, we initialized the graph attention transformer with the Munchausen parameter set to 0.15, a prioritized replay buffer size of 1,000,000, and soft update coefficient r 0.1. We sample 64 trajectories and store them in the replay buffer, subsequently sampling 4096 transitions from this buffer. We used the other hyperparameters mentioned in original MunDQN paper (Tiapkin et al., 2023].
entropy regularization coefficient 0.001 chosen from (0, 0.1, 0.01, 0.001).
F1 QGEN hyperparameters:
For all variants of QGFN, employed an exhaustive grid-search approach for hyperparame- ter tuning, focusing on parameters and For p-greedy QGFN, we selected 0.4 for p from (0.2, 0.4, 0.6] and 29 for from 291,29,30 30). For p-of-max QGEN, we chose 0.6 for p from (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9} and 29 for n from {28, 29, 30). :. For p-qunatile QGFN, we selected for from (0.5, 0.7, 0.8} and 29 for n from (11, 27, 28, 29). Similarly to the fragment task, we implemented an additional threshold of 10-3 to p max, is) threshold and clipped Q-values to a minimum of for stability during initial training. Cosine annealing over 1500 steps used for all variants.
22

---

G Experiment details: RNA-binding task
Parameter
Value
Batch size
64
Number of steps
5,000
Optimizer
Adam
Number of Layers
Hidden Dim. Size
64
Number of Heads
Positional Embeddings
Rotary
Reward scaling R
Learning rate
1 x 10
Z Learning rate
x 10-2
Table 8: Hyperparameters and specifications of the Sequence Transformer used across all models in RNA-binding task.
Parameter
Value
Objective function
TB
P-greedy
0.4
p-quantile
0.25
p-of-max
0.9
stepwise scheduler for
500 steps
Model architecture
Sequence Transformer
n-step
13
dqn
0.95
Table Model-specific parameters for QGEN in RNA- task.
We follow the setup of Jain et al. [2022] but using the task introduced Sinai et al. [2020]. We use sequence transformer [Vaswani al., 2017] architecture with 4 layers, 64-dimensional embeddings, and 2 attention heads. The training for this task is 5000 iterations over mini-batch sizes of 64. The reward scaling parameter 3 is to 8 and learning rate of 1 14 and 1 10 for log Z. In this task, 3 ~ (66 is used for LSL-GFN. Following the approach described Sinai al. [2020], the modes are predefined from enumerating the entire RNA landscape for LI4RÑAI and L14RNAI 112 to identify local optimal through exhaustive search. ViennaRNA [Lorenz al., 2011] is used to provide the RNA binding landscape. For MunDQN, Munchausen parameter a set to 0. 0515, a prioritized replay buffer size of 800,000, and soft update coefficient 0.1. We sample 16 trajectories and store them in the replay buffer, subsequently sampling 1024 transitions from this buffer. We used the other hyperparameters mentioned in the original MunDQN paper [Tiapkin al., 2023].
G.1 QGFN hyperparameters:
For all QGFN variants, we used grid-search for tuning hyperparameters p and We used n = 13 from the set 12, 13, 14. For p-greedy QGEN, = 0.4 was chosen from (0.2, 0.4, 0.6, For = 0.9 was selected from {0.6, 0.7, 0.8, 0.9). In p-quantile QGFN, we tried p = 0.25 and 0.50, but neither performed well due to small action spaces. stepwise scheduler set at 500 steps is applied to p-of-max QGFN.
p-of-max QGFN,
23

---

H Experiment details: Bit sequence generation
Number
60
0-
Trajetoriis Sampled
p-grocdy QGFN
p-of-max QGFN
SubTB
LFL-GFN
poquantile QGNN
DON
AZC
Figure 18: Bit sequence generation, 4.
Parameter
Value
Batch size
16
Number of steps
50,000
Optimizer
Adam
Number of Layers
Hidden Dim. Size
64
Number of Heads
Positional Embeddings
Rotary
Reward scaling 33 in R
Learning rate
x 10 -4
Z Learning rate
x 10
Table 10: Hyperparameters and specifications of the Sequence Transformer used across all models in bit sequence generation.
Parameter
Value
Objective function
TB
p-greedy
0.4
p-quantile
0.25
p-of-max
0.3
stepwise scheduler for p
500 steps
Model architecture
Sequence Transformer
n-step
120
dqn
0.95
Table 11: Model-specific parameters for QGFN in bit sequence generation.
The bit sequence generation task follows the same environmental setup as Malkin et al. [2022a] with 3 value as 3. We generate M 60 reference sequences by randomly combining 15 symbols from an initial vocabulary H = {00000000, 11111111, 11110000, 00001111, 00111100).
Beyond the original auto-regressive generative framework in Malkin et al. [2022a], we generate sequences in prepend-apppend fashion motivated by Shen et al. [2023]. We use sequence transformer [Vaswani et al., 2017] architecture for all experiments with rotary position embeddings, hidden layers with hidden dimension 64 across attention heads. All methods were trained for 50,000 iterations with minibatch size of 16. For trajectory balance, we use learning rate of of 1 x 10-1

---

for the policy parameters and 1 x 10 for log Z. For SubTB, we use the same hyperparameters as in Madan et al. [2023]. For LSL-GFN, we use 8 ~ U11,6,, where U denotes uniform distribution.
H.1 QGEN hyperparameters:
For all variants of QGFN, did grid-search approach for hyperparameter tuning for p and For k=1 where actions are limited to {0,1}, we set n at 120, selected from (30, 60, 90, 120), across alla chose p = 0.4; for the p-of-max QGEN, p was to 0.3,
QGEN variants. For the p-greedy QGEN, with cosine annealing applied at 500 steps. In the p-qunatile QGFN, we tested values of 0.25 and 0.50, but neither achieved good performance due the binary nature of the action space.
I Experiment details: Graph combinatorial optimization problems- maximum independent set (MIS)
As an additional task, we explore graph combinatorial optimization, specifically the maximum independent set (MIS) problem mentioned in Zhang et al. [2023]. We directly used the codebase shared by Zhang et al. [2023] and report the performance at test time in Table 12.
Method
Small Metric Size
Small Top 20
FL
18.20
18.72
FL QGFN (p-greedy)
18.21
19.06
FL QGFN (p-quantile)
18.20
18.75
FL QGFN (p-of-max)
18.26
18.74
Table 12: Comparison of different methods on small graphs: metric size and top 20 metrics.
We used the same parameters in the fragment- based molecule generation experiments. At test time, we applied the p-greedy, p-quantile, and p-of max sampling strategies respectively methods. Note that the reported performance might not reflect the best achievable results on this task, as we did not explore different hyperparameter settings.
for different
Compute Resources
All of our experiments were conducted using A100 and V100 GPUs. For the fragment- bassed task. we used 8 workers on A100 GPUs, and it ran in less than 4 hours. For RNA, we used 4 workers, and completed less than hours. For QM9, we used 0 workers, and it finished in less than 9 hours. For Bit sequence, we used 8 workers, and it ran in less than 24 hours.
K Broader Impacts
The research conducted in this work is not far removed from practical applications of generative models. As such, acknowledge the importance of considering safety and alignment in applications closely related to this work such as drug discovery, material design, and industrial optimization. We believe that research GFlowNets may lead to models that better generalize, and in the long may be easier to align. Another important application of GFlowNets is the causal and reasoning domains; believe that improving in those fields may lead easier to understand and safer models.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made the abstract and introduction are supported by the results presented in Section 5. Additionally, Section 4 provides thorough analysis of the proposed method, further strengthens the paper's 's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 7 includes subsection on limitations. Additionally, the discussion on QGFN variants in Section 5 provides a more thorough analysis of the strengths and weaknesses of each variant.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix $E, we outline all of the hyperparameters needed to reproduce our results, with references to other papers that also used the same environment. In addition, we will be sharing all of our code run our QGFN algorithm in all tasks presented this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
27

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All of the environments open access with sufficient instructions provided in Appendix $E.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nips.. cc/
cosssiicyyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n httpsscccppbbb//guuuu guwde// /CodeSubmis ss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All of the environments setting/details provided in Appendix $E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All of our experiments are run with 5 seeds, with interquartile mean and standard error calculated over these seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In appendix $J, we specified the compute resources used to for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /wwrrpuu..../// cc
Answer: [Yes].
Justification:
We also consider and discussion societal implications of our work in ŞK.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive socictal impacts SK. We do not see any negative social impacts of this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset throughout the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cwwccoo//aaoo/// datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "pnbaYYaOYn", "reviewer": "Reviewer_bgQq", "summary": "The paper proposes jointly learning a $Q$ function and a policy network $P_F$ to improve the search for high-valued states when training GFlowNets. To achieve this, the authors develop three sampling strategies for composing $Q$ and $P_F$, $p$-greedy, $p$-of-max, and $p$-quantile, and show that the resulting algorithm, termed QGFN, often leads to faster discovery of modes relative to GFlowNet baselines.\n\nIn spite of QGFN’s notable performance, I believe the paper would greatly benefit from a more clearly described solution and from the inclusion of stronger, optimization-focused baselines. I will be happy to increase my score if the enumerated weaknesses and questions below are properly addressed during the rebuttal period.", "questions": "* Why does QGFN provide more diverse samples than, e.g., GFN-TB in Table 1 even when $p$ is set as high as $p = 0.9858$?\nDo we need to train a GFN? Given the large values of $p$ (above 0.9), it seems that the GFN is not very helpful. I wonder whether an untrained $P_F$ would produce similar results.\n\n* What is $k$ in line 230?\n\n* Figures 9 and 11 are quite difficult to grasp. Please consider visually encoding the greediness of the method as the size or opacity of points in a scatter plot.\n\n* Also, the convergence speed of GFlowNets via TB minimization is known to be dramatically affected by the learning rate of $\\log Z$, which the authors set to $10^{-3}$. I wonder whether the relative results would remain the same if such learning rate was increased to $10^{-2}$ or $10^{-1}$.", "limitations": "Authors claim that the computational overhead induced by learning a Q network and the sensitivity of sampling to the accuracy of $Q$ are the main limitations of their work.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Well-described problem**. Section 3 highlights the issues of an exclusively GFlowNet-oriented approach to the search of high-valued states with clarity, namely, the large probability mass associated with low-reward states.\n\n2. **Intuitively sound method with strong empirical results**. When the $Q$ function is accurately learned, QGFN — which interpolates between a GFlowNet (p = 0) and a DQN (p = 1) — should yield samples with higher rewards for larger $p$. This behavior is experimentally confirmed. However, the distribution from which QGFN samples, even when $Q$ is perfectly estimated, is mostly unclear. See weakness 4 below.\n\n3. **Extensive experimental campaign**. (however, important baselines are missing) Experiments include four commonly used benchmark tasks for GFlowNets, and QGFN often outperforms alternative methods by a large margin.", "weaknesses": "1. **Missingness of important baselines**. FL-GFlowNets (Pan et al., 2023) and LED-GFlowNets (Jang et al., 2024) exhibited strong performance for the optimization-focused tasks of molecule and set generation. Notably, contrarily to the greediness $p$ of QGFN, these methods do not require tuning of a hyperparameter that may unpredictably modify the sampling distribution. Authors should include FL-GFlowNets and LED-GFlowNets in Figures 2, 3, and 4. Have the authors considered increasing the sharpness of the distribution by modifying the temperature in LSL-GFN?\n\n2. **Training of the $Q$-network is insufficiently described**. Lines 163-175 are hard to follow and Algorithm 1 in Section B does not provide sufficient details for understanding the training of QGFN. If I understand correctly, the adoption of multi-step returns with the horizon size set as the maximum trajectory length implies that the each $R(s_{t})$ in Equation (2) corresponds to the expected reward among $s_{t}$’s children under the chosen sampling policy. In any case, an unambiguous equation representing the learning objective for $Q$ should be included in Section 4.\n\n3. **Difference between each sampling strategy is unclear**. In practical applications, a sampling strategy would have to be chosen, as selecting among the proposed approaches can be excessively time-consuming — even without retraining the model. However, $p$-greedy, $p$-of-max, and $p$-quantile seem to perform relatively different when the benchmark task is modified. A general approach for choosing a sampling strategy should, then, be proposed by the authors. Otherwise, the method is hardly usable in real applications.\n\n4. **QGFNs lack theoretical guarantees**. Important questions are left unanswered. What if $Q$ is not properly trained? How do we know that $Q$ is inaccurately learned? What is the sampling for a given $p$ and sampling strategy? To improve QGFN’s reliability, a theoretical analysis should be included. Does a sufficiently trained QGFN finds, in average, more rewarding than standard GFlowNets?\n\n5. **Rationale for choosing $p$ is not well-defined**. Experiments are filled with cryptically chosen numbers. In Table 1, the $p$ value for $p$-of-max and $p$-quantile are respectively set to $0.9858$ and $0.93$; in Figure 3, a “mode” is defined as a molecule with Tanimoto similarity smaller than $0.70$ and a reward larger than $1.10$. What is the rationale behind such numbers? How can a practitioner choose $p$? Also, if the choice of $p$ relied on a grid search over $N$ values in $[0, 1]$, a fair comparison with GFN-TB and GFN-SubTB in Table 1 should allow these methods to sample $N$x times more trajectories than QGFN. However, it is unclear whether this observation was considered for the experiments defining Table 1.\n\n6. [Minor] Typos. Algorithm 1 refers to $\\mathcal{L}\\_{flow}$ and $\\mathcal{L}\\_{huber}$, which are undefined. I assume $\\mathcal{L}\\_{flow}$ is $\\mathcal{L}\\_{TB}$, but I could not find a definition for $\\mathcal{L}\\_{huber}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the time and effort you put into reviewing our work.\n\nOur work aims to provide an empirical analysis of the method we introduce, a novel idea that can be applied to any GFN. While we understand the reviewer's emphasis on theoretical guarantees, it is worth noting that many significant contributions in the field do not include such guarantees. We believe that our approach, with thorough empirical analysis, is a strong contribution to the field and aligns with the standards of many accepted papers in major ML conferences.\n\nOverall, we hope to have addressed the reviewer's concerns in our response, and hope they would reconsider their score. We are committed to improving the presentation of the paper, particularly to dispel any sense of choices and (hyper)parameters being arbitrary.\n\n> baselines, temperature in LSL-GFN\n\nWe compared against six baselines typically used in the GFN literature, covering characteristics like greediness and diversity. Further, the QGFN idea can be applied to any GFN, including FL-QGFNs and LED-QGFNs. In this spirit, we trained FL-QGFNs for Table A of Reviewer eVPz's rebuttal. Yes, if a base GFN algorithm is already optimal, QGFN may not improve it, but this is unlikely to be the case in general. Our contribution is to show that mixtures of action values and GFN policies have desirable behaviors; we did not intend to show that a specific QGFN implementation is SoTA.\n\nFor LSL-GFN, we tested the effects of increasing sharpness during inference using a range of $\\beta$ values (training $\\beta\\sim U[1,256]$), and chose $\\beta = 78$, with rewards closest to SubTB for comparison (larger $\\beta$ greatly increased similarity). We will include these results in our revision in Table 1 (see attached pdf).\n\n> Training of the $Q$-network\n\nWe will include the exact objective of $Q$ in $\\S$ 4 and Alg. 1. The reviewer is correct that in the max horizon case $Q_\\theta(s,a)$ converges to the expected reward. This is the crux of lines 163-175, where we attempt to convey this unexpected reliance on max horizon returns.\n\n> Difference between strategies\n\nSection 5.1 highlights that the best strategy depends on the environment, particularly the number of relevant actions. Although not foolproof, this is a reasonable starting point. That QGFN is adjustable post-hoc is a strength of the method which can be used here.\n\n> QGFNs lack theoretical guarantees\n\n- the distribution from which a perfect QGFN samples is hard to define theoretically. We would be happy to find such a result, but we would ask the reviewer to not hold us to impossible standards. For example, Schulman et al. did not know what distribution PPO sampled from, but produced an empirically well understood method. It took two years for proper analysis using two-time-scale methods to provide us with strong theory\n- In specific cases one can show that the expected reward of QGFN is at least as much as GFN's. We include analysis for the bandit case and intuition-building for the general case in the global response. We would like to emphasize that we've performed extensive empirical work beyond measuring performance to show that our method indeed works according to its hypothesized mechanism\n- _What if $Q$ is not properly trained?_ Early in training we observe that QGFNs tend to outperform baselines. At that stage it is unlikely that $Q$ is precise.\n- _How do we know if $Q$ is accurate?_ This can be measured, e.g. in Fig 7 showing an imperfect $Q$. Note that this $Q$ is still improving the sampling of high-reward diverse objects\n \nOn $p$ and other values:\n\n- as explained in $\\S5.1$ 244-288 and $\\S6$ 280-286, and shown in Figure 6, the $p$s of Table 1 are chosen based on Figure 6.\n- _Why this Tanimoto threshold?_ 0.7 has been used in every GFlowNet paper with this task, we simply reuse this choice from the literature.\n- _How can one choose $p$?_ We are unsure what the reviewer means here. There are standard ways to choose (hyper)parameters, but we show in Fig 6 and 9 that e.g. sampling from the model with different $p$s and measuring the diversity-reward trade-off works\n- _baselines should be allowed to sample $N$x times._ This is incorrect, because we are reporting averages. As a thought experiment, if we sampled infinitely many samples for them, the _average_ reward and similarity would converge to some values. Table 1 uses 1000 samples. \n    - A better test would be to log the time of training QGFN + tuning $p$. The cost of training $Q$ is small (thank parallelism), and taking ~1000 samples with different $p$s takes a minute. That QGFN is adjustable without any retraining is a strength of the method\n \nQuestions:\n- _Why is QGFN [with $p=0.9858$] more diverse than GFN-TB?_ Note that this is for **$p$-of-max QGFN**: if many actions have a $Q$ value that is at least 98.58% of the max $Q$, then they can all be sampled by $P_F$, retaining the diversity of $P_F$, but sampling more high-reward objects. Note QGFN isn't strictly more diverse than GFN, but for $n$ samples its _number of modes_ will be higher because less time is spent sampling low-rewards\n- _with large $p$, is QGFN helpful?_ As above, a good $P_F$ is still needed, and a $p$ close to but less than 1 does **not** entail the greedy policy, except for the $p$-greedy mixture (where $p\\approx 0.5$ are best; see Fig 9 & l287-292)\n- _Would an untrained $P_F$ be similar?_ This is an interesting question! We compared a trained $P_F$ to an untrained $P_F$ (trained $Q$ for both). The latter has an ok performance for high $p$, but is still far from what QGFN can reach (see attached pdf Fig. 1)\n- $k$ in line 230 refers to the bit width of actions that we later use (lines 259-261) as per Malkin et al. We will clarify this\n- Style of Fig. 9&11, we were uncertain about this, and use a revised style in Fig. 1 of the attached pdf. We would welcome feedback\n- For us large learning rates for $Z$ were not helpful, and too often caused runs to diverge", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0zbQk5h55", "reviewer": "Reviewer_Fo8k", "summary": "The paper focuses on improving high-reward sample collection, i.e., exploitation, in training GFlowNets. The motivation stems from the fact that the flow may pursue states that lead to many low-reward states rather than focusing on states that lead to high-reward states. To this end, the authors propose incorporating Q-value estimation, i.e., expected future reward, and interpolating between flow and Q to make a transition. Experimentally, the proposed method shows promising performance in fragment and atom-wise molecule generation and RNA-sequence generation.", "questions": "- Are there failure scenarios where QGFN shows poor performance compared to GFN? (opposite of Figure 1)\n- I am curious if QGFN performs better than a policy interpolated between GFlowNets trained with low temperature and high temperature (with $p$).\n- Is there an analytic form of the sampling distribution of terminal states for the proposed methods?\n- It would be interesting to illustrate the sample efficiency for discovering modes in a toy example, e.g., hyper-grid.\n- Line 160: \"good..\" => \"good.\"", "limitations": "The authors have acknowledged the limitations of their approach.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow, with a well-developed motivation and idea.\n- It seems interesting that the compositional nature of the generation model can be considered to improve the performance of GFlowNets for practical applications.\n- This paper discusses various design choices for incorporating Q-value estimation, which offer different perspectives on \"greediness.\"\n- Overall, the experiments are well-done, and the reported results demonstrate the effectiveness of the proposed model.", "weaknesses": "No major weaknesses. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper. We appreciate your questions and have provided detailed answers below. \n\n> Are there failure scenarios where QGFN shows poor performance compared to GFN? (opposite of Figure 1)\n\nWe haven't encountered such scenarios in practice other than $p$ being set too high. In such cases, the generated samples tend to be overly similar to each other, reducing the diversity and generalization of the model. \n\nAdditionally, it is possible that QGFN could show poor performance if the underlying GFN is improperly trained. If the GFN is suboptimal, the combination of $Q$ could amplify this problem. \n\n> I am curious if QGFN performs better than a policy interpolated between GFlowNets trained with low temperature and high temperature (with $p$).\n\nWhile we haven't performed this exact experiment, we can extrapolate from Figure 6b that this interpolation would still be less diverse than QGFN, stemming from the loss in diversity in low-temperature (high $\\beta$) GFN.\n\n> Is there an analytic form of the sampling distribution of terminal states for the proposed methods?\n\nWe haven't been able to derive any interesting results in this respect. In the limits of $p$ one simply retrieves either the GFN behavior or the greedy $Q$ policy behavior. Note that in some restricted settings (think of the bandit case for example), sampling from QGFN is provably at least as good as sampling from GFN. We include such an analysis as well as intuition-building for the general case in the global response.\n\n> It would be interesting to illustrate the sample efficiency for discovering modes in a toy example, e.g., hyper-grid.\n\nWhile the hyper-grid example has been used in many GFlowNet papers, we do not believe it to be particularly interesting other than as a sanity check of convergence; i.e. it should not be used to rank algorithms. The main reason is that the setting is too simple to get much generalization out of the DNN approximating $P_F$ and/or $F$ (c.f. [1]). We can nonetheless add such an experiment to the appendix as an additional sanity check.\n\n[1] Investigating Generalization Behaviours of Generative Flow Networks, Lazar Atanackovic, Emmanuel Bengio, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4gQP4ADvGR", "reviewer": "Reviewer_eVPz", "summary": "The paper \"QGFN: Controllable Greediness with Action Values\" introduces a novel approach to enhance Generative Flow Networks (GFNs) by incorporating action-value estimates (Q-values) to control the greediness of sampling policies. This method, called QGFN, includes three variants—p-greedy, p-quantile, and p-of-max—each designed to balance the generation of high-reward samples with the maintenance of diversity. Through comprehensive experiments on tasks like molecule generation and RNA design, the authors demonstrate that QGFN significantly improves the generation of high-utility samples while preserving diversity, providing a practical and effective solution for deploying safe and helpful LLMs.", "questions": "See **Weaknesses**", "limitations": "See **Weaknesses**", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Innovative Approach**: The introduction of Q-values to control the greediness of sampling policies in Generative Flow Networks (GFNs) is a novel and creative solution to the challenge of balancing high-reward sample generation with diversity.\n- **Comprehensive Evaluation**: The paper includes thorough experiments across various tasks, such as molecule generation and RNA design, providing strong empirical evidence for the effectiveness of QGFN.\n- **Ablation Study**: The paper provides in-depth ablation studies on the hyperparameters of GFNs.", "weaknesses": "- **QGFN variants matter**. This paper does not provide a method to select different QGFN variants. Different variants of QGFN have very different performances in different environments. In Section 5, the authors claim that p-of-max is suitable for small action spaces, and p-quantile is suitable for large action spaces. What if different states have different action spaces? For example, in the graph combinatorial optimization problems [1], earlier states have a much larger action space than later states, and the action space size will change (decrease) during the sampling. I think compared with a fixed action space, this setting probably requires the use of different QGFN variants on different states during the sampling. \n- **Lack of more complex environment**. Therefore, I wonder about the QGFN's performance on graph combinatorial optimization problems, such as MIA. [1]\n- There are some missing recent works that deal with sampling high-reward candidates with diversity, such as [2, 3] \n- I am happy to raise my scores if my concerns are resolved. \n\n\n[1] Zhang, D., Dai, H., Malkin, N., Courville, A., Bengio, Y., & Pan, L. (2023). Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint arXiv:2305.17010.\n\n[2] Chen, Y., & Mauch, L. (2023). Order-Preserving GFlowNets. arXiv preprint arXiv:2310.00386.\n\n[3] Jang, H., Kim, M., & Ahn, S. (2023). Learning Energy Decompositions for Partial Inference of GFlowNets. arXiv preprint arXiv:2310.03301.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the time and effort you put into reviewing our work. Below, we address your questions and comments in detail. We hope this clarifies our approach and findings. \n\n> QGFN variants matter. This paper does not provide a method to select different QGFN variants. Different variants of QGFN have very different performances in different environments. In Section 5, the authors claim that p-of-max is suitable for small action spaces, and p-quantile is suitable for large action spaces. What if different states have different action spaces? For example, in the graph combinatorial optimization problems [1], earlier states have a much larger action space than later states, and the action space size will change (decrease) during the sampling...\n\nThis is a great question, and as the reviewer points out, we find that the optimal variant tends to be a function of the action space. Although not always optimal, the $p$-greedy variant consistently outperforms its GFN counterpart in finding modes and is a safe choice to default to that is less dependent on the action space. \n\nWe also would like to highlight that a variant such as $p$-of-max QGFN will select a variable number of actions (any action whose $Q$ value is at least a fraction $p$ of the best action's $Q$ value), and might be a suitable choice for varying action sizes. It is also likely that more appropriate mixtures exist. Additionally, note that in both the fragment-based molecule generation task and atom-based molecule generation task (QM9), the size of the action space is highly variable as it depends on the number of generated fragments/atoms at each timestep.\n\n> Lack of more complex environment. Therefore, I wonder about the QGFN's performance on graph combinatorial optimization problems, such as MIA. [1]\n\nWhile we feel that the fragment-based molecule task is complex (Figure 2; $|\\mathcal{X}|\\approx 10^{100}$), we address the concern by training QGFN on the MIS codebase [1] with default hyperparameters. From superficial experimentation over the last few days, we show we can improve performance upon the baseline model. \n\n| METHOD               | Small - Metric Size | Small - Top 20 |  \n|----------------------|---------------------|----------------|\n| FL                   | 18.20              | 18.72         | \n| FL - QGFN (p-greedy) | 18.21              | **19.06**         | \n| FL - QGFN (top-p)    | 18.20              | 18.75         | \n| FL - QGFN (high-p)   | **18.26**              | 18.74         | \n\nTable A: Max independent set experimental results on small dataset (200 to 300 vertices) using FL-GFlowNets (Pan et al., 2023). We sample 20 solutions for each graph configuration and get the average and top results. \n\nDue to the limited time for training during the rebuttal period, we anticipate that experimenting with different hyperparameters could yield even better performance. We will include these extended results in the revised paper for a more thorough evaluation.\n\n> There are some missing recent works that deal with sampling high-reward candidates with diversity, such as [2, 3]\n\nWe appreciate that other GFN methods exist that increase the diversity-reward quality of sampling. However, our experimentation focuses on showcasing how QGFN can be easily integrated into the GFN framework across various tasks with improved performance. In other words, one _could_ train OP-QGFNs and LED-QGFNs. As shown in the above experimentation on MIS, we successfully used QGFN variants on FL-GFlowNets (Pan et al., 2023) and showed that they can adapt to varying action spaces dynamically and provide a boost in performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all feedback from the reviewers. To provide more analysis on QGFN, we have conducted an analysis using a bandit example. We also use some derivations to illustrate the general case. We hope these examples illustrate the behavior of QGFN. We will include this analysis in our revised manuscript.

### Analysing $p$-greedy

Consider the bandit setting where trajectories are 1 step and just consist in choosing a terminal state. Let $p_G(s) = R(s)/Z$. Let $0<p<1$, then with $\mu(s'|s) = (1-p)P_F(s'|s) + p\mathbb{I}[s'=\arg\max Q(s,s')]$, assuming there is only a single argmax $s^*$, then $p_\mu(s) = (1-p)R(s)/Z + p\mathbb{I}[s=\arg\max R(s)]$. This means that for every non-argmax state, $p_\mu(s) = (1-p) p_G(s) < p_G(s)$. We get that $\mathbb{E}_\mu[R] > \mathbb{E}_G[R]$:

$$\\begin{align}
\\mathbb{E}_ \\mu[R] - \\mathbb{E}_ G[R]=&\\sum_ s p_ \\mu(s) R(s) - \\sum_s p_G(s) R(s)\\\\
=& (p + (1-p)R(s^*)/Z - R(s^*)/Z) R(s^*) + \\sum_{s\\neq s^*} (1-p)R(s)^2/Z-R(s)^2/Z\\\\
=& pR(s^*) - pR(s^*)^2/Z  + \\sum_{s\\neq s^*} (-p)R(s)^2/Z\\\\
=& p/Z \\left(R(s^*)Z - R(s^*)^2  - \\sum_{s\\neq s^*} R(s)^2\\right),\\;\\;\\;Z=\\sum_s R(s)\\\\
=& p/Z \\left(R(s^*)[\\sum_s R(s)] - R(s^*)^2  - \\sum_{s\\neq s^*} R(s)^2\\right)\\\\
=& p/Z \\left(R(s^*)[\\sum_s R(s)] - \\sum_{s} R(s)^2\\right)\\\\
=& p/Z \\left(\\sum_{s} R(s^*)R(s) - R(s)^2\\right)\\\\
\\end{align}$$

since $R(s^*) > R(s)$ and both are positive then $R(s^*)R(s) > R(s)^2$ thus the last sum is positive. All other terms are positive, therefore $\mathbb{E}_\mu[R] - \mathbb{E}_G[R] > 0$.

In the more general case, we are not aware of a satisfying closed form, but consider the following exercise. 

Let $m(s,s') = \mathbb{I}[s'=\arg\max Q(s,s')]$. Let $F'$ be the "QGFN flow" which we'll decompose as $F'=F_G + F_Q$ where we think of $F_G$ and $F_Q$ as the GFN and Q-greedy contributions to the flows. Then:

$$\begin{align}
F'(s) &= \\sum_{z\\in\\mathrm{Par}(s)} F'(z)((1-p)P_F(s|z) + pm(z,s))\\\\
&=\\sum_z F'(z)(1-p)P_F(s|z) + \\sum_z F'(z) pm(z,s)\\\\
&=\\sum_z F_G(z)(1-p)P_F(s|z) + F_Q(z)(1-p)P_F(s|z)+ \\sum_z F'(z) pm(z,s)\\\\
&=(1-p)F_G(s) + \\sum_z F_Q(z)\\mu(z|s)+ F_G(z) pm(z,s)\\\\
\end{align}$$

Recall that $p(s)\propto F(s)$. Intuitively then, the probability of being in a state is reduced by a factor $(1-p)$, but possibly increased by this extra flow that has two origins. First, flow $F_Q$ carried over by $\mu$, and second, new flow being "stolen" from $F_G$ from parents when $m(z, s)=1$, i.e. when $s$ is the argmax child. 

This suggests that flow (probability mass) in a $p$-greedy QGFN is smoothly redirected towards states with locally highest reward from ancestors of such states. Conversely, states which have many ancestors for which they are not the highest rewarding descendant will have their probability diminished.

Meta-review: This paper proposes QGFN, which jointly trains G-flownets and Q-functions and combines their predictions with the goal of sampling diverse and high-reward objects.

Several important points were raised by the reviewers, which were partially addressed during the rebuttal phase:

- Training details for the Q-network are not sufficiently described (reviewer ``bgQq``).

- Additional complex environment experiments are needed on MIA [1].

Overall, my recommendation is to accept the paper. I suggest the authors incorporate the feedback in the final submission.

[1] Zhang, D., Dai, H., Malkin, N., Courville, A., Bengio, Y., & Pan, L. (2023). Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with G-Flownets. arXiv preprint arXiv:2305.17010.
Decision: Accept (poster)