Distributional GFlowNets with Quantile Flows
Authors: Dinghuai Zhang
Ling Pan
Ricky T. Q. Chen
Aaron Courville
Yoshua Bengio
Venue: Accepted by TMLR ()

Abstract: Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a stochastic policy for generating complex combinatorial structure through a series of decision-making steps. 
There have been recent successes in applying GFlowNets to a number of practical domains where diversity of the solutions is crucial, while reinforcement learning aims to learn an optimal solution based on the given reward function only and fails to discover diverse and high-quality solutions.
However, the current GFlowNet framework is relatively limited in its applicability and cannot handle stochasticity in the reward function. 
In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training.
By parameterizing each edge flow through their quantile functions, our proposed \textit{quantile matching} GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty.
Moreover, we find that the distributional approach can achieve substantial improvement on existing benchmarks compared to prior methods due to our enhanced training algorithm, even in settings with deterministic rewards.

Published in Transactions on Machine Learning Research (01/2024)
Distributional GFlowNets with Quantile Flows
Dinghuai Zhang'
Mila, University of Montreal
Ling Pan' Hong Kong University of Science and Technology
Ricky T. Q. Chen
Meta AI, Fundamental Al Reseorch
Aaron Courville
Mila, University of Montreal
Yoshua Bengio
Mila, University of Montreal
Reviewed on OpenReview: https: // openr/viewww.... net/ forum? id= ed=
Abstract
Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns stochastic policy for generating complex combinatorial structure through series of decision-making steps. There have been recent successes in applying GFlowNets to number of practical domains where diversity of the solutions is crucial, while reinforcement learning aims to learn an optimal solution based on the given reward function only and fails to discover diverse and high-quality solutions. However, the current GFlowNet framework the reward
relatively limited in its applicability and cannot handle stochasticity function. In this work, we adopt distributional paradigm for GFlowNets, turning each flow function into distribution, thus providing more informative learning signals during training. By parameterizing each edge flow through their quantile functions, proposed quantile matching GFlowNet learning algorithm is able to learn risk-sensitive policy, an essential component for handling scenarios with risk uncertainty. Morcover, we find that the distributional approach can achieve substantial improvement on existing benchmarks compared to prior methods due to our enhanced training algorithm, even in settings with deterministic rewards.
1 Introduction
The success of reinforcement learning (RL) (Sutton & Barto 2005 has been built learning intelligent agents that are capable of making long-horizon sequential decisions MMih et al. 2015 Vinyals et al. 2019 These strategies are often learned through maximizing rewards with the aim of finding a single optimal solution. That being said, practitioners have also found that being able to generate diverse solutions rather than just a single optimum can have many real-world applications, such as exploration in RL (Hazan et al, 2018: Zhang et al. 2022b), drug-discovery (Huang et al. 2016 Zhang al. 2021 Jumper al.) 2021) ), and material design (Zakeri & Syri 2015 Zitnick et. al. 2020 One promising approach search for a diverse set. of high-quality candidates to sample proportionally to the reward function (Bengio et al.) 2021a).
Recently, GFlowNet (Bengio et al. 2021a by has been proposed as novel probabilistic framework to tackle this problem. Taking inspiration from RL, a GFlowNet policy takes a series of decision-making steps to
OThhe asterisk mark denotes equal contributions. Correspondence to: Dinghuai Zhang <dinghuai. haainggnnii.hanngamii..uuuu

---

parent of s' if (s + s') an edge in G. We call states Published in Transactions on Machine Learning Research (01/2024)
generate composite objects x, with probability proportional to its return R(x). The number of particles in each "flow" intuitively denotes the scale of probability along the corresponding path. The use of parametric polices enables GFlowNets to generalize to unseen states and trajectories, making it more desirable than traditional Markov chain Monte Carlo (MCMC) methods (Zhang et al.. 2022c) which are known to suffer from mode mixing issues (Desjardins et al. 2010 Bengio et al. 2012). With its unique ability to support. off-policy training, GFlowNet has been demonstrated superior to variational inference methods (Malkin et al. 2022b
Yet the current GFlowNet frameworks only learn from deterministic reward oracle, which is too stringent assump- tion for realistic scenarios. Realistic environments are often stochastic (e.g. due to noisy observations), where the need for uncertainty modeling (Kendall & Gal, 2017 Guo et al. 2017 Teng et al. 2 emerges. In this work, we propose adopt- ing probabilistic bilistic approach to model the flow function (see
$2
R(s²)
So
R()
Figure 1 in order to account for this stochasticity. Analogous to distributional RL (Bellemare et al.] 2017) approaches, we think of each edge flow as a random variable, and parameter- ize its quantile function. We then use quantile regression to train the GFlowNet model based on temporal-differenee- flow constraint. The proposed GFlowNet learning algorithm, dubbed quantile matching (QM), is able to match stochastic reward functions. QM can also output risk-sensitive policies under uscr-provided distortion risk mcasures, which allow it to behave more similarly to human decision-making: The proposed method also provides a stronger learning signal during training, which additionally allows it to outperform existing GFlowNet training approaches on standard benchmarks with just deterministic environments. Our code is openly available at httpp::/giitae http://hhw...oeelllaaaaa/aaaaaaa
R(-
Figure 1: Illustration of distributional
GFlowNet with stochastic edge flows
To summarize, the contributions of this work are:
We propose quantile matching (QM), novel distributional GFlowNet training algorithm, for handling stochastic reward settings.
A risk-sensitive policy can be obtained from QM, under provided distortion risk measures.
The proposed method outperforms existing GelowNet methods even on deterministic benchmarks.
2 Preliminaries
2.1
GFlowNets
Generative Flow Networks (Bengio et al. 2021a b GFlowNets) are a family of probabilistic models to generate composite objects with a sequence of decision decision-maling steps. The stochastic policies are trained to generate complex objects x in combinatorial space X with probability proportional to a given reward function R(x), where R: + R,, The sequential nature of GelowNets stands upon the fact that its learned policy incrementally modifies partially constructed state E S 2 X' with some action (s - s') E A x S. To be more specific, let g (S,A) be directed acyclic graph (DAG), and GFlowNet sequentially samples trajectory r = (So - S1 ....) within that DAG with forward policy (s Here the state s and action (s -+5] s') are respectively a vertex and an edge in the GFlowNet trajectory DAG G. We also typically assume that in such DAGs, the relationship between action and future state is a one-to-one correspondence. This is unlike typical RI. setups (where the environment generally stochastic) and more appropriate for internal actions like attention, thinking, reasoning or gencrating answers to question, or candidate solutions to problem. We say s' child of and
Each circle denotes state; concentric circkes on the right side denote terminal states which rewards assigned. 8o + 81 + 82 is complete trajectory which starts from the initial state so and ends terminal state 82. order cope with stochastic reward, we represent, every edge flow as random variahle, denoted by gray probahility curve icons.
2

---

Published in Transactions on Machine Learning Research (01/2024)
without children terminal states. Notice that any object x E A' is terminal state. We also define special inilial state 8o (which has no parent) as abstraction for the first state of any object generating path.
A trajectory = (so + S,) is complete if it starts at the initial state So and ends at a terminal state Sa E A. We define the trajectory flow on the set of all complete trajectories to be a non-negative function ::T- + 4. is called flow, as F(r) could be thought of the amount of particles flowing from the initial state a terminal state along the trajectory T, similar to the classical notion of network flows (Ford & Fulkerson 1956). The flow function is unnormalized measure over T and could define distribution complete trajectories (F(r)= F where Z E R4 is the partition function. The flow is Markovian if there exists forward policy that satisfies the following factorization:
n-1
Pp(T = (so + > s.,) = II (s t=
(1)
Any trajectory distribution arising from a forward policy satisfies the Markov property. On the other hand, Bengio et al. (2021b) show the less obvious fact that any Markovian trajectory distribution arises from a unique forward policy.
We use Pr(x) = E PF(T) to denote the termånating probability, namely the marginal likelihood of generating the object x, where the summation enumerales over all trajectories that terminate at x. The learning problem considered by GFlowNets fitting the flow such that it could sample objects with probability proportionally to given reward function, i.e., Pr(x) x R(x). This could be represented by the reward matching constraint:
R(x) = 7= B8
Σ F(T).
(2)
It is easy to see that the normalizing factor should satisfy Z= Exex R(x). Nonetheless, such computation is non-trivial as it comes down to summation enumeration GelowNets therefore provide way to approximate intractable computations, namely sampling given unnormalized probability function, like MCMC methods and marginalizing - in the simplest scenario, estimating the partition function 2, but this can be extended to estimate general marginal probabilities in joint distributions [Bengio et al. 2021b
exponentially large combinatorial space.
2.2 Distributional modeling in control
Distributional reinforcement learning Bellemare et al. 2023 an RL approach that models the distribution of returns instcad of their expected value. Mathematically, it considers the Bellman equation of a policy æ
2 (x 1 r(x, + (x', a'),
(3)
where 4 denotes the equality between two distributions, > E [0 is the discount factor, x, the state and action RL, (r(x, x)) are the reward and next state after interacting with the environment, a' is the next. action selected by policy at x', and z denotes random variable for the distribution of the Q-function value.
The key idea behind distributional RL is that it allows the agent to represent its uncertainty about the returns it can expect from different actions. In traditional RL, the agent only knows the expected reward of taking certain action in certain state, but it docsn't have any information about how likely different rewards are. In contrast, distributional RL methods estimate the entire return distribution, and use it make more informed decisions.
3

---

Published in Transactions on Machine Learning Research (01/2024)
3 Formulation
3.1 GFlowNet learning criteria
Flow matching algorithm. It is not computationally efficient to directly model the trajectory flow function, as it would require learning function with high-dimensional input (i.e., the trajectory). Instead, and taking advantage of the Markovian property, we define the state flow and edge flow functions F(s) = Erss Grra=4' F(T) and (r)). The edge flow is proportional to the marginal likelihood of trajectory
F(s + ((s+s))= -(. sampled from the GelooNeet including the edge transition. By the conservation law of the flow particles, it is natural see the flow matching constraint of GelowNets
F(s 5() 8:88-8))
Σ
Σ 8"=(8'"8")AA
F(s' s"),
(4)
for any s' E S. This indicates that for every vertex, the in-flow (left-hand side) equals the out-flow (right-hand side). Furthermore, both equals the state flow F(s'). When s' in Equation is terminal state, this reduces to the special case of aforementioned reward matching.
Leveraging the generalization power of modern machine learning models, one could learn a parametric model s') to represent the edge flow. The general idea of GFlowNet training objectives to turn constraints like Equation | into losses that when globally minimized enforce these constraints. To approximately satisfy the flow-matching constraint (Equation 4 the parameter A can be trained to minimize the following flow matching (FM) objective for all intermediate states s':
CFM(S':0) log E(e-S)E Po(s,s') 12
(5)
"(sr+ Pols' ("")
In practice, the model is trained with trajectories from some training distribution π(r) with full support Erwa(r) |Eger (s where π()) could be the trajectory distribution sampled by the GFlowNet (i.e., PF(T), which indicates an on-policy training), or (for off-policy training and better exploration) a tempered version of (F(T) or a mixture between PF(T) and a uniform policy.
Trajectory balance algorithm. With the knowledge of edge flow, the corresponding forward policy is given by
F(s ss) F(s -ss).
PP(s'(s
(6)
F(s)
Similarly, the backward policy PB(s]s') defined to be F(s + s')P((ss)) x F's -(ss+s'), distribution over the parents of state s'. The backward policy will not be directly used by the GFlowNet when generating objects, but its benefit could be seen in the following learning paradigm.
Equivalent with the decomposition in Equation a complete trajectory could also be decomposed into products backward policy probabilities
PB(T = (So > ..... 8n/8n = x)) = II
Pn(
(7)
t=0
as shown in Bengio et al. (2021b). Tn order to construct the balance between forward and backward model in (2022a) propose the following trajectory balance (TB) constraint,
the trajectory level, Malkin et al.
Z
Pr (s R(x) II 1=
Pa(s)(5141).
(8)
1=0
where (So + ... + sn)) a complete trajectory and Sn =x. Suppose we have a parameterization with 0 consisting of the estimated forward policy Pr( 0), backward policy Pn 0), and the learnable global
4

---

Published in Transactions on Machine Learning Research (01/2024)
scalar Zo for estimating the real partition function. Then we turn Equation into the LTB objective to optimize the parameters:
Zo I==o Pr se: 0) ]²
(TB(T: log
(9)
R(x) Ii=o Ps(Ss|SL+11:
where 5 (Sg > Sn =). The model is then trained with stochastic gradient Eyor(s) Vo 0)] Trajectory balance (Malkin al., 2022a) is an extension of detailed balance (Bengio et al. 2021b) to the trajectory level, that aims to improve credit assignment, but may incur large variance as demonstrated in standard benchmarks (Madan ct al. 2022 TB is categorized as Monte Carlo, while other GFlowNets (e.g.,
flow matching, detailed balance, and sub-trajectory balance) objectives are temporal-difeeren (that leverages the benefits of both Monte Carlo and dynamic programming).
3.2 Quantile flows
For learning with a deterministic reward function, the GFlowNet policy is stochastic while the edge flow function is deterministic as per Equation [ Nonetheless, such modeling behavior cannot capture the potential uncortainty in the environment with stochastic reward function. See the bchavior analysis in the following proposition.
Proposition 1 (informal). Consider the reward R(x) for object x to be a stochastic random variable, then given sufficiently large capacity and computation resource, the obtained GFlowNet after training would generate objects with probahility proportional to exp (E [og R(x))).
The proof is deferred to Section in a probabilistic manner: we see the state and edge flow as probability distributions rather than scalar values. Following the notation of Bellemare el al. (2017), we use (s) and Z(s - s') to represent the random variable for state / edge flow values. Still, the marginalization property flow matching holds, but on a distributional level:
B.1 In this work, we propose to treat the modeling of the flow function
Z(s')
Σ Z(s (B-+s")EA
Σ Z(s' +s"),
(10)
(E
where L denotes the equality between the distributions of two random variables. We thus aim to extend the flow matching constraint to distributional one.
Among the different parametric modeling approaches for scalar random variables, it is effective to model its quantile function (Müller 1997p. The quantile function (2) [0,
- R is the generalized inverse function of cumulative distribution function (CDF), where Z is the random variable being represented and B is a scalar in [0, 1]. Without ambiguity, we also denote the S-quantile of Z's distribution by Zg. For simplicity, we assume all quantile functions are continuous in this work. The quantile function fully characterizes a distribution For instance, the expectation could be calculated with a uniform quadrature of the quantile function [Z] Jo' Qz) dB.
Provided that we neural networks to parameterize the edge flow quantiles (similar to the flow matching parameterization). we able to represent the distribution of the marginal state flows? Luckily, the following quantile mixture proposition from Dhaene et al. (2006) provides an affirmative answer.
Proposition 2 (quantile additivity). For any set of M one dimensional random variables { jss Jm=1 which
shore the same randomness through a common 3 E [0, 1] in the way that zm = (B), Vm, where (-) is the quantile function for the m-th rondom variable, there erists random variable z such that z " z and its quantile function satisfies (-)-= (M)) (.)..
We relegate the proof to Section B.2
Remark 3. Such additive properly of the quantile function is essential to an efficient implementattion of the distributional matching algorithm. On the other hand, other distribution representation methods may need considcrable amount of computation deal with the summation over a series of distributions. For example, for the discrete categorical representation (Bellemare et al. 2017 Barth-Maron et al.) 20188), the summation between M distributions would need M convolutionn operations, which is highly time-consuming.
5

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm GFlowNet quantile matching (QM) algorithm
Require: Gel quantile flow (s - s'; 0) with parameters 0, larget reward oracle.
repeat
Sample trajectory with the forward policy Pr(())) estimated by Equation 13
0 - (s 0) (as per Equation 12:
Update 0 with some optimizer; until some convergence condition
Quantile matching algorithm. We propose to model the A-quantile of the edge flow of - s' ( + s': 0) with network parameter 0 on the log scale for better learning stability. A temporal-ddiffrence- like [Sutonn & Barto 2005) error 5 is then constructed following Bengio et al. (2021a) 's Flow Matching loss (Equation but across all quantiles:
8 (
(s': 0) log Σ (s' EA
exp L :"; log Σ
exp 2 +s;;0),
(11)
(s-1')EE
where 3,3 E [0, 1] and 0 is the model parameter. This calculation is still valid as both log and exp are monotonic operations, thus do not affect quantiles.
Notice that we aim to learn the quantile rather than average, thus we resort to quantile regression (Kenner 2005) to minimize the pinball error pa( 1 1(8 < (() where E(-) is usually & norm or its smooth alternative. In summary, we propose the following quantile matching (QM) objective for GFlowNet learning:
LQM 0)
Pa. Pa. (8...8 (s: 0)),
(12)
N
==
where 3., 3, are sampled f. from the uniform distribution 00 1], Vi,j. Here N, N are two integer value hyperparameters. The average over Bj makes the distribution matching valid; see an analysis in Section B.3 During the inference (i.e., sampling for generation) phase, the forward policy is estimated through numerical integration:
((s + s')
N exp (z + -s)
Pp(s'|s)= P=(s)s)==E
E Es-AS E ( + +]]
& E + +s']]
~
N
Σ
(13)
where 85 L0, Vi. We summarize the algorithmic details in Algorithm 1 inequality, this approximation is smaller than its true value.
We remark that due Jensen's 's
The proposed GFlowNet learning algorithm is independent of the specific quantile function modeling method. In the literature, both explicit (Dabney et al. 2017) and implicit (Dabney et al. 2018)) methods have been investigated. In practice we choose the implicit quantile network (TQN) implementation due to its light- property and powerful expressiveness. We discuss the details about implementation and efficiency, and conduct an ablation study in Section C.T
4 Risk Sensitive Flows
The real world is full of uncertainty. To cope with the stochasticity, financial mathematicians use various kinds of risk measures value the amount of assets to reserve. Concretely speaking, risk measure is a mapping from random variable to real number with certain properties (Artzner et al. 1999. Commonly adopted risk measures such mean or median do not impute the risk / stochasticity information well. In this work, we consider a special family of risk measures, namely the distortion risk measure (Hardy 2002
Balbás et al. 2009).
E [2] = [ Qz(o(8))
(14)
6

---

Published in Transactions on Machine Learning Research (01/2024)
where g [0, - [0,1] is monotone distortion function +
Different distortion functions indicate different risk-sensitive effects. In this work, we focus on the following categories of distortion classes. The first cumulative probability weighting (CPW) function (Terrsk &
Kahneman 1992
Gonzalez Wu 1999 reads
((:n)) (8n (1 (1-3)) 3)
(15)
where > 0 is a scalar parameter controlling its performative behaviour. We then consider another distortion risk measure family proposed by Wang (2000) as follows,
( m)) 1b (4 (B) +n)
(16)
where (-)) is the CDF of standard normal distribution. When 17 > O, this distortion function is convex and thus produces risk-seeking behaviours and vice-versa for < 0. Last but not least, we consider the conditional value-at-risk [Rockafllaar et al.) 2000 CVaR): g(3; n)) = n7, where 7 E 1]. CVaR measures the mean the lowest. 100 X 1) percentage data and is proper for risk-averse modeling.
Provided distortion risk measure 9, Equation 13 now reads (s E9 [Z(s - s'), which be
estimated by Equation 17 where B, 2100 1i.
exp
Z20 g(8))
(s
(17)
Risk-averse quantile matching. We now consider a risk-sensitive task adapted from the hypergrid domain. The risky hypergrid is variant of the hypergrid lask studied in Bengio et al. (2021a) (a more detailed description can be found in Section 5.1 whose goal is to discover diverse modes while avoiding risky regions in D-dimensional map with size Н. We illustrate 2-dimensional hypergrid in Figure 2 as an example, where yellow denotes a high reward region (í.e., non-risky modes) and green denotes a region with stochasticity (i.e., risky modes). Entering the risky regions incurs very low reward with small probability. Beyond that, the risky mode behaves the same as the normal modes (up-left. and bottom-right mode in Figure 2) As investigated by Deming al. (1945) Arrow [1958) Hellwig (2004), human tends to be conservative during decision-making. Therefore, in this task we propose to combine QM algorithm together with risk-averse distortion risk measure to avoid getting into risky regions while maintaining performance.
risky regions
Figure 2: A risky hy- pergrid environment.
We compare the risk-sensitive quantile matching with risk-neutral variant and the standard GFlowNet (flow matching). We quantify their performance by the violation rate, i.e., the probability ratio of entering the risky regions, with different dimensions of the task including small and large. We also evaluate each method in terms of the number of standard (non-risky) modes discovered by each method during the course of training. As shown in Figure 3 aa-b), CV.R(0.1) and Wang(-0.755 leads to smaller violation rate; CVaR(0.11) performs the most conservative and achieves the smallest level of violation rate, as it only focuses on the lowest 10% percentile data. Notice that CPW(0.71)) performing similar to risk-neutral QM and FM (i.e., baseline) matches the theory, since its distortion function is neither concave nor convex. This indicates that the risk-sensitive flows could effectively capture the uncertainty in this environment, and prevent the agent from going into dangerous areas. Figure 3 c-d) demonstrates the number of non-risky modes discovered by each algorithm, where they can all discover all the modes to have competitive performance. Results validate that the risk-averse CVaR quantile matching algorithu is able to discover high-quality and diverse solutions while avoiding entering the risky regions. We relegate more details to Section C.2
2In some literature, different but equivalent notation used: So Qz((0) dg(3) Se (1))

---

Published in Transactions on Machine Learning Research (01/2024)
2.0
L8
Velsdoon rade
Methed
Cranti
Wengl-b.75)
CPM10-711
14
14
0.5
30
23
25
10 34
56
3.4
1.8
State united (x10")
State visited DX10M)
State visited (x110))
State visited (x110))
small
(b) large
(c) small
(d) large
Figure 3: Experiment results on stochastic risky hypergrid problems with different risk-sensitive policies. Up: CV.)( and Wan5(-0.75) induce risk-averse policies, thus achieving smaller violation rates. Bottom: Risk- sensitive methods achieve similar performance with other baselines with regard to the number non-risky modes captured, indicating that the proposed conservative method do not hurt the standard performance.
3.080
16x16x16
1675
16
20 x 20
3.003>
1003)
Tregrta
Fiow Harthing
30805
MCRC
2.8
20020
20335
22300
State vistted (x10%)
State visted (a70h)
State sished ciihee
Shate sistted (x100)
8
16 x6
16 16x1x1
20 20
State sesttd (x20%)
State viaited (420%)
State vaited (x10f)
vasttd (x10f)
Figure 4: Experiment results on the hypergrid tasks for different scale levels. learned distribution density and the true target density, Bottom: the number of discovered modes across the training process. The proposed quantile matching algorithm achieves the best results across different. hypergrid scales under both quantitative metrics.
Up: the & error between the
5 Benchmarking Experiments
The proposed method has been demonstrated to be able to capture the uncertainty in stochastic environments. On the other hand, in this section evaluate its performance deterministic structured generation benchmarks. These tasks are challenging due to their exponentially large combinatorial search space, thus requiring efficient exploration and good generalization ability from past experience. In this work, all experimental results are run on NVIDIA Tesla V100 Volta GPUs, and are averaged across 4 random seeds.
5.1 Hypergrid
We investigate the hypergrid task from Bengio et al. (0211a). The space of states is a D-dimensional hypergrid cube with size H x ... x H with H being the size of the grid, and the agent is desired to plan in long horizon and learn from given sparse reward signals. The agent initiated at one corner, and needs to navigate by taking increments in one of the coordinates by for each step. A special termination action also available for each state. The agent receives reward defined by Equation 18 when it decides to stop. The reward
8

---

Published in Transactions on Machine Learning Research (01/2024)
function is defined by
D
0.5 E (0.25, 0.5] + R2 II (xx) Ro + R1 X
X H 0.5 (0.300.4)
(18)
where 1 is the indicator function and Ro = 0,001, R1 = 0.5, R2 = 2. From the formula, we could see that there are 2D modes for each task, where mode refers to a local region (which could contain one or multiple states) that achieves the maximum reward value.
The environment is designed to test the GFlowNet's ability of discovering diverse modes and generalizing from past experience. We use & error between the learned distribution probability density function and the ground truth probability density function as an evaluation metric. Besides, the number of modes discovered during the training phase is also used for quantifying the exploration ability. In this task there are 2 modes for each task. We compare the QM algorithm with previous FM and TB methods, plus we also involve other non- Non-GFlowNet basclines such as MCMC.
Figure 4 demonstrates the efficacy of QM on tasks with different scale levels, from 8 8x8888 to 20 x 20 20 x 20. We notice that TB has advantage over FM for small scale problems (8 x 8 x 8) in the sense of lower error, but is not as good as FM and QM on larger scale tasks. Regarding the speed of mode discovering, QM is the fastest algorithm with regard to the time used to reach all the diverse modes. We also test PPO (Schulman ct al. (017) in this problem, but find it hardly converges on our scale level in the sense of the measured error, thus we do not plot its curve. We also examined the exploration ability under extremely sparse scenarios in Figure See Section C.2 for details. To demonstrate the superiority against distributional RL baselines, we conduct experiments between distributional RL method and QM on hypergrid in Figure g
5.2 Sequence generation
In this task, we aim to generate binary bit sequences in an autoregressive way. The length of the sequence fixed to 120 and the vocabulary for each token is as simple as the binary set, making the space be {0, 1) The reward function is defined to be the exponential of negative distance to predefined multimodal sequence set, whose definition can be seen in the Appendix. Each action appends one token to
Method
Quantic Matching
Trajectory Salance
25
Flow Manching
strpyy
20
SoftActor-Cri Critc
MCNC
Number of
45
the existing partial sequence. Therefore, it is an autoregressive generation modeling, and each GFlowNet state only corresponds to one generation path. We compare the proposed quantile matching algorithm with previous GFlowNet algorithm (TB and FM), together with RL-based (A2C (Mnih al. 2016) Soft
20000
20000
30000
40200
50000
Step
Actor-Critic (Haarnoja et al.) 2018) and MCMC-based (Xie
Figure The number of modes reached by each algorithm across the whole training process for the sequence generation task. QM outperforms other baselines in terms of sample efficiency.
et al. 2021) method. In this problem setup, it is intractable to enumerate all possible states to calculate the error between learned and target density functions, thus we only report the number of discovered modes. We define finding a mode by a edit. distance less than 28. Figure 5 shows the number modes that each method finds with regard exploration steps, demonstrating that QM achieves the best sample efficiency in the sense of finding diverse modos. We take the bascline results from Malkin et al. (2022a). We relegate other information to Section C.3 including average reward for top candidates in Figure 10
5.3 Molecule optimization
We then investigate a more realistic molecule synthesis setting. The goal in this task is to synthesize diverse molecules with desired chemical propertics (Figure a))). Each state denotes a molccule graph structure, and the action space vocabulary of building blocks specified by junction tree modeling (Kumar al. 2012 Jin et. al.] 2018 We follow the experimental setups including the reward specification and episode
9

---

Published in Transactions on Machine Learning Research (01/202))
Mecked
130
Tloy Hatthng
Tregchay bolence
1.
PARS
egent
230
88
next state Rex3
State utabted (x207)
Stare wiitted (c107)
Stare viited (x10f)
(a)
(c)
(d)
Figure 6: Molecule synthesis experiment. (a) Illustration of the GFlowNet policy. Figure adapted from Pan
et al. 2022 (b) The number of modes captured by algorithms. (c) Tanimoto similarity (lower is better). (d)
Average reward across the top-100 molecules.
constraints in Bengio et al. (2221a. We compare the proposed algorithm with FM, TB, as well as MARS (MCMC-based method) and PPO (RL-based method).
In realistic drug-design industry, many other properties such as drug-likeness (Bickerton et al. 2012 synthesizability (Shi et al., 20200, or toxicity should be taken into account. To this end, a promising method should be able to find diverse candidates for post selection. Consequently, for diverse molecules by the number of modes discovered conditioning on reward being larger than 7.5 (see details in Section C.4p, and show the result Figure G(b, where our QM surpasses other baselines by large margin. Further, we also evaluate the diversity by measuring the Tanimoto similarity in Figure ((c), which demonstrates that QM able to find the most diverse molecules. Figure G(d displays the average reward of the top-100 candiclates, assuring that the proposed QM method manages find high-quality drug structures.
quantify the ability of searching
6 Related Work
GFlowNets. Since the proposal of Bengio et al. (2021a b), the field has witnessod an incrcasing number of work about GFlowNets on different aspects. Malkin et al. (2022b); Zimmermann et al. (2022) analyze the connection with variational methods with expected gradients coinciding in the on-policy case (πT PF), and show that GFlowNets outperform variational inference with off-policy training samples; Pan et al. 2002 2023b) develop frameworks to enable the usage of intermediate signal in GFlowNets to improve credit assignment efficiency; Jain et al. (2022b) investigate the possibility of doing multi-objective generation with GFlowNets; Pan et al. (2023c) introduce world modeling into GFlowNets; Pan et al. (2023a) propose an unsupervised learning method for training GFlowNets; Ma et al.) (2023) study how utilize isomorphism tests Lo reduce the flow bias in GFlowNet training. Regarding probabilistic modeling point. of view, Zhang al. 2022ch jointly learn energy-based model and GFlowNet for generative modeling, and testify its efficacy on a series of discrete data modeling tasks. It also proposes back-and-forth proposal, which is adopted by Kim et al. (2023) for doing local search. Further, Zhang et al. (022aa) analyze the modeling of different generative models, and theoretically point out that many of them are actually special cases of GFlowNets; this work also builds up the connection between diffusson modeling and GFlowNets, which is adopted by conitnuous sampling works (Lahlou et al., 2023 Zhang et al.] 2023bb. Different from the above efforts, this work aims at the opening problem of learning GFlowNet with stochastic reward function. GFlowNet also expresses promising potential in many object generation application arcas. [Jain et al. (2022a use it in biological sequence design; Deleu et al. 20022) Nishikawa-Tooner el. al. (2022) leverage it for causa
structure learning: Liu et al. (2022) employ it to sample structured subnetwork module for better predictive
robustness; Zhang et al. (2023a c) utilize it for combinatorial optimization problems; Zhou et al. 2023) learn
a GFlowNet for Phylogenetic inference problems.
Distributional modeling. The whole distribution contains much more information beyond the first order moment (Yang al.) 2013 Imani & White 2018). Thus, learning from distribution would bring benefit from more informative signals. Interestingly, similar mechanism is also turned out exist in human
10

---

Published in Transactions on Machine Learning Research (01/2024)
brains Dabney al.
2020 Specifically, in distributional RL (Bellemare et al. 2023) literature, people
minimize distributional Bellman error in order to achieve Equation Many different implementations are proposed: categorical DQN (Bellemare et al.) 2017), quantile regression DQN (Dabney et al., 2017),
implicit quantile network based DQN (Dabney et al. 2018: Yang et al.
2019 and expectile regression based DQN (Rowlad et al. 2019). Distributional modeling methods can be used with different types of methods,
such as Q-lcarning (above-mentioned works), actor-critic (Ma et al. 2020) or policy gradient (Barth-Maron Maron jet al., 2018). The well-recognized Rainbow algorithm (Hessel al., 2017) also adopts categorical DQN important component. In model-base RL methods, people have also found that distributional modeling could boost the performance (Hafner et al. 2023). In the domain of generative modeling, the idea of TQN is integrated into autoregressive models by Ostrovski et al. (2018).
7 Conclusion and Discussion
In summary, we would like to highlight that our proposed method, Quantile Matching, is not a straight forward combination of FM and distributional RI, but has considcrable technical novelty.
Importance of the problem. We investigate an important problem in GFlowNets we discover an important limitation of current formulations of GFlowNets in that they fail to tackle stochastic rewards well, which is gencrally in a wide range of real-world tasks and may limit its application. As a consoquence, it fails to take the risks associated with actions into consideration, which is important in real-world applications (e.g., healthcare). Novelty of the algorithm. Different from the Bellman equation in RL, we need to consider all possible parents and children of a state s in the flow consistency constraint, and directly employing techniques in Bellemare et al. (2017) does not permit efficient computation as detailed in Remark 3 matching based with the justification of Proposition 2 with flexible computation.
We propose quantile
What makes quantile matching prominent? Apart from the risk-sensitive modeling advantage brought by the implicit quantile modeling, it is an exciting surprise to see that the proposed QM also surpasses previous methods deterministic reward settings. We hypothesize the following rationales why QM brings benefits into GelowNet training. (a) More informative learning signals for better gencralization. As more complex models, the extra non-linear quantile flows encourage the capture additional information besides the expected values, acting as regularization with auxiliary tasks gLyle et al.) 2019). In practical setups where it is intractable for GFlowNet learner to see all possible trajectorics, the issue of generalization matters very much. Therefore, it is important to extract as much useful generalizable information possible from small (b) Regularization effects. It has been previously observed that GFlowNets can overfit to past trajectories and thus have estimation bias to some flow values (Bengio et al. 2021a). However, since we maintain a distribution of flows, this helps propagate useful information and improve the prediction of flow values, thus regularizing this overestimation issue and benefiting the optimization process (Imani & White, 2018). (c) Pseudo uncertainty. In settings where we have uncertainty in the actual rewards (e.g., they are estimated from data), it would make sense to propagate reward distributions. However, even in deterministic environments, due to the complex representation of states, two different states may be incorrectly represented as the same by the network. This results in pseudo uncertainty in the environment and is similar to the partial observability (McCallum & Ballard 1996), which leads to the so-called state aliasing in control.
number of training samples.
Acknowledgments
The authors would like to thank Marc Bellemare, Moksh Jain, and Emannuel Bengio. Furthermore, Dinghuai Zhang would like to thank Sagrada Familia to remind him of the existence beauty in this world.
References
Kenneth J. Arrow. Essays in the theory of risk-bearing. 1958.
Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical Finance, 9, 1999.
11

---

Published in Transactions on Machine Learning Research (01/2024)
Alejandro Balbás, José Garrido, and Silvia Mayoral. Properties of distortion risk measures. Methodology and Compuling in Applied Probabilily, 11:385-300 399, 2009.
Gabriel Sarth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Manfred Otto Heess, and Timothy P. Lillicrap. Distributed distributional deterministic policy gradients. ArXiv, 0bbs/1804.0805 2018.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, 2017.
Mare G. Bellemaaee Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023. http://www. http://www..tttrrrtttoooooll--/-
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Newral Information Processing Systems (NeurIPS), 2021a.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. International Conference on Machine Learning. 2012.
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Net foundations. Xiv preprint 2111. 2111.09266 092266, 2021b.
G. Richard J. Bickerton, Gaia Valeria Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4 2:90-8, 2012.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In A AAT Conference on Artificial Intelligence, 2017.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. ArXiv, abs/1806.06922 2018.
Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara. Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew M. Botvinick. A distributional code for value in dopamine-based reinforcement learning. Nature, 577:671-675, 2020.
Tristan Deleu, Ant onio G'ois, Chris C. Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2022.
William Edwards Deming, John von Neumann, and Oscar Morgenstern. Theory of games and economic behavior. Journal of the American Statistical Association, 40:263, 1945.
Guillaume Desjardins, Aaron C. Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau. Tempered markov chain monte carlo for training of restricted boltzmann machines. In International Conference on Artificial Intelligence and Statistics, 2010.
Jan Dhacne, Steven Vanduffel, Marc J. Goovaerts, Rob Kaas, Q. Tang, and David Vyncke. Risk measures and comonotonicity: A review. Stochaslic Models, 22:573 606, 2006. URL sttps::/app..seeatticcchhoo semanticschollar
org/CorpusID: 11340721
Lester Randolph Ford and Delbert Ray Fulkerson. Maximal flow through a network. Canodian Journal of Mathematics, 8:399 404, 1956.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. ArXiv, abs/1704.01212, 2017.
Richard Gonzalez and George Wu. On the shape of the probability weighting function. Cognitive Psychology, 38:12 1666166, 1999.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, (bs/1700.04599) 2017.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
Danijar Hafner, Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. ArXiv, abs/2300 2300.04104, 2023.
Mary R. Hardy. Distortion risk mcasures: Coherence and stochastic dominance. 2002.
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. ArXiv, abs/ 1112/.282.000 2018.
Martin F. Hellwig. Risk aversion in the small and the large. Mox Planck Institute for Research on Collective
Goods Research Paper Series, 2004.
Matteo Hessel, Joseph Modayil, H. V. Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. ArXiv, abs/1710.0229s 2017.
Po-Ssu Huang, Scott E. Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537: 320-327, 2016.
Ehsan Imani and Martha White. Improving regression performance with distributional lossos. In International Conference on Machine Learning, 2018.
Moksh Jain, Emmanuel Bengio, Alex García, Jarrid Rector-Brooks, Bonaventure F. P. Dosson, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Micheal Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. International Conference on Machine Learning (ICML), 2022a.
Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Gareía, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. ArXiv, abs/2210. 12765, 2022b.
Wengong Jin, Regina Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. ArXiv, abs/1802.04388 2018.
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Ziclinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021.
Juha Karvanen. Estimation of quantile mixtures via l-moments and trimmed I-moments. Comput. Stat. Data Anal., 51:947-959, 2006.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NIPS, 2017.
Minsu Kim, Tacyoung Yun, Emmamuel Bengio, Dinghnai Zhang, Yoshua Bengio, Sungsoo Ahn, and Jinkyoo Park. Local search gflownets, 2023.
Roger W. Koenker. Quantile regression. 2005.
A. Kumar, Arnout R. D. Voet, and K Y J Zhang. Fragment based drug design: from experimental computational approaches. Current medicinal chemistry, 19 30:5128 47, 2012.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Salem Lahlou, Tristan Deleu, Pablo Lemos, Dinghuai Zhang, Alexandra Volokhova, Alex Heraandez-Garca,, Téna Néhale Exzine, Yoshua Bengio, and Nikolay Malkin. A theory of continuous generative flow networks. In International Conference on Machine Learning, pp. 18269 18300. PMLR, 2023.
Dianbo Liu, Moksh Jain, Bonaventure F. P. Dossou, Qianli Shen, Salem Lahlou, Anirudh Goyal, Nikolay Malkin, Chris C. Emezue, Dinghuai Zhang, Nadhir Hassen, Xu Ji, Kenji Kawaguchi, and Yoshua Bengio. Gflowout: Dropout with generalive flow networks. Arx,v, abs/2210.12925, 2022.
Clare Lyle, Pablo Samuel Castro, and Marc G. Bellemare. A comparative analysis of expected and distribu- tional reinforcement learning. ArXio, abs/ abs/1901.1100 2019.
Jiangyan Ma, Emmanuel Bengio, Yoshua Bengio, and Dinghuai Zhang. Baking symmelry into gllowness, 2023.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv: Learning, 2020.
Kanika Madan, Jarrid Rector-Brooks. Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bose, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. ArXiv, abs/2209. 2bs22209.2788,, 2022.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. arXiv preprint 2201.13259, 2022a.
Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J. Hu, Katie Elizabeth Everett, Dinghuai Zhang, and Yoshua Bengio. Gflownets and variational inference. ArXiv, abs/2210.00588 2022b.
Andrew McCallum and Dana H. Ballard. Reinforcement learning with selective perception and hidden state. 1996.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement. learning. Nature, 518:529-55 533, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillierap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ArXio, abs/1602.01188, 2016.
Alfred Miller. Integral probability metrics and their generating classes of functions. Advances in Applied Probabilily, 29:429 443, 1997.
Mizu Nismiaa Tristan Delen, Jithendaraa Subramanian, Yoshua Bengio, and Laurent Charlin. Bayesian learning of causal structure and mechanisms with gflownets and variational bayes. ArXie, 002663, 2022.
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. ArXio, 055515,06.05755 2018.
L. Pan, Dinghuai Zhang, Aaron C. Courville, Longbo Huang, and Yoshua Bengio. Generative augmented flow networks. ArXiv, abs/2220.033008, 2022.
Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pro-training and fine-tuning generative flow not- 2023a. URL
works. ArXie, abs/
org/CoorpsID::2331777 263671952
Ling Pan, Nikolay Malkin, Dinghuai Zhang, and Yoshua Bengio. Better training of gflowntss with local credit and incomplete trajectories. arXiv preprint NXXi:2302.01687, 2023b.
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. Xiv preprint ar Xix:2302.00466 2023c.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2: 21 42, 2000.
Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Marc G. Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In International Conference on Machine Learning, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/ 0bs/1707.06347, 2017.
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph to graphs framework for retrosyuthesis prediction. ArXiv, abs/2003.12725, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions Newral Nelworks, 16:285-280 2005.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv, abs/2006.10739, 2020.
Jiaye Teng, Chuan Wen, Dinghuai Zhang, Yoshua Bengio, Yang Gao, and Yang Yuan. Predictive inference with feature conformal prediction. ArXiv, /os/2200.00173, 2022.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty.
Journal of Risk and Uncertainty, 5:297 323, 1992.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevels, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pp. 1-5, 2019.
Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. Journal of Risk and Insurance, 67:15, 2000.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. ArXiv, abs/2103.10423 10432, 2021.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. In Neural Information Processing Systems, 2019.
Jiyan Yang, Xiangrui Meng, and Michael W. Mahoney, Quantile regression for large-scale applications. SIAM J. Sci. Comput., 36, 2013.
Behnam Zakeri and Sanna Syri. Electrical energy storage systems: A comparative life cycle cost analysis. Renewable and sustainable energy reviews, 42:569 596, 2015.
David Zhang, Corrado Rainone, Markus Peschl, and Roberto Bondesan. Robust scheduling with GFlowNets. International Conference on Learning Represenlations (ICLR), 2023a. To appear.
Dinghuai Zhang, Jie Fu, Yoshua Bengio, and Aaron C. Courville. Unifying likelihood free inference with black-box optimization and beyond. In International Conference on Learning Representations, 2021.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Dinghuai Zhang, Ricky T. Q. Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. ArXiv, abs/2209.02002 02606, 2022a.
Dinghuai Zhang, Aaron C. Courville, Yoshua Bengio, Qinqing Zheng, Amy Zhang, and Ricky T. Q. Chen. Latent state marginalization as a low-cost approach for improving exploration. ArXiv, abs/2210.00990 2022b.
Dinghuai Zhang, Nikolay Malkin, Z. Liu, Alexandra Volokhova, Aaron C. Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modcling. International Conference on Machine Learning (ICML), 2022c.
Dinghuai Zhang, Ricky T. Q. Chen, Cheng-Hao Liu, Aaron C. Courville, and Yoshua Bengio. Diffusion generative flow samplers: Improving learning signals through partial trajectory optimization. ArXie, abs/2310.026699, 2023b.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint ar arXiv:2305.110 17010, 2023c.
Mingyang Zhou, Zichao Yan, Elliot Layne, Nikolay Malkin, Dinghuai Zhang, Moksh Jain, Mathieu Blanchette, and Yoshua Bengio. Phylogfn: Phylogenctic inference with generative flow networks. orXiv preprint arXiv:2310.0871,, 2023.
Heiko Zimmermann, Fredrik Lindsten, J.-W. de Meent, and Christian A. Naesseth. A variational perspective generative flow networks. ArXiv, abs/2210.07992, 2022.
C Lawrence Zitnick, Lowik Chanussot, Abhishek Das, Siddharth Goyal, Javier Heras- Deraas-omnngo,, Caleb Ho, Weihua Hu, Thibaut Lavril, Aini Palizhati, Morgane Riviere, et al. An introduction to electrocatalyst design using machine learning for renewable energy storage. arXiv preprint ar Xiix:20100094333 2020.
A Summary of Notation
Symbol
Description
S
state space
X
object (lerminal state) space, subset of S action / transition space (edges + s')
A
directed acyclic graph (S,
T
set of complete trajectorics state S
8
So
initial state, element of S
terminal state in X trajectory in T Markovian flow
F F:T + R
F: S + R
state flow
F:A A edge flow
PF
forward policy (distribution over children)
PB
backward policy (distribution over parents)
Z
scalar, equal to Σrer F(T) for Markovian flow
16

---

Published in Transactions on Machine Learning Research (01/2024)
B Missing Details about Methodology
B.1 Proposition
1
Proof. For flow matching algorithm, the reward matching training loss in practice is on the log scale:
log
Σ
x() -lg R((x)
(s-X)E
By assuming sufficiently large capacity and sufficiently much computation resource, assume the obtained GFlowNet can sample correctly with probability proportional to the given reward. Now that the reward function is stochastic, and since the assumption presumes that we have infinite compute resource and neural network capacity, the property of square loss would let the log of in-flow to learn to fit the expectation of the log reward Ellog R(x)] (it is the optimumm this minimization problem). This makes the optimization
to have the same optimal solution minimizing (log E(s-xjed F(s x) E [log R(x)] According to the property of flow matching algorithm (Bengio et al. 2021a| Proposition 3), the GFlowNet would learn to sample with probability proportional to the reward defined by exp (Eilog R(x))).
For trajectory balance algorithm, since the loss could be written as
2 log PF(T) log R(x)) PB(T(x)
with the same reasoning we know that is equivalent minimizing (log PE(r)
Pn(F|x] log R(x))) According
to the property of TB algorithm (Malkin et al. 2022a Proposition 1), the GFlowNet would learn to sample with probability proportional to the reward defined by exp (E[log R(x)]). The same ratiocination could be
made for the detailed balance algorithm (Bengio et al. 2021b
B.2 Proposition 2
We first rephrase the proposition as follows.
Proposition. For any set of M + 1 quantile functions {Qm{ m= that satisfies Qo(-) = ΣM there erists a set of random variables { that satisfies Qm is the quantile function of Zmm Vm, and - d >M Zm.
Qm(). ())
The following proof inspired by Karvanen 20006
Proof. For Vz e R,
M
( BE [0, 1]:
M Σ m=1
> (m) $²) sz) ))
M
Zm (B) S x0)) =P
P Σ Zm S 2) =P
B [0, 1]:
m=1
m=1
suu 8 [0,1]: 12 Σ Qm
inf
8 [0. 1]
Qm 2m)
inf {3 E [0, 1] 2 (o(3)) P (Zo 22)
This indicates that Zo d DM1 Zm.
For the statement Proposition 2 as we assume all quantile functions are continuous in this work, the summation of several continuous monotonic functions (EM (m)+)) is also continuous monotonic function, thus could be quantile function of random variable. Then the above argument.
17

---

Published in Transactions on Machine Learning Research (01/2024)
B.3 Regarding the Quantile Regression Objective
Say Z is random variable and we want to get its 8-quantile. To achieve this we should find æ solve
min Ez loo(Z - -x]]
Pa(Zz, - x)), ~10 11,
where Zz denotes the B By-qunntile of Z. Note that B does not overlap with (3,)
Let us then look at Equation 12 According to the above analysis, 1 Σ Pa. (60..) (s: 0)) will guide the in-flow to learn the 3 quantile of the out-flow. When we sum over different Br ~ u10 1]. i= I...., N, this helps us match the in-flow (as a distribution) the out-flow (as a distribution), as two distributions with the same quantile function are the same distribution. Note that {8,] do not overlap with (8,)
N
C More about Experiments
C.1 Details and Ablation about Quantile Modeling
We explain the modeling of quantile function in this subsection. For explicit modeling, the neural network directly output of i/M x 100 percentage quantile of some distribution, S iis S M. This means the neural network output head should output M-dimensional vectors. On the contrary, for implicit modeling, the neural network takes additional input 3 E [0, 1], and outputs the 8-quantile value (f.e., scalar output). The latter modeling choice could provide more flexibility, distribution. Both modeling methods share the same quantile regression based algorithm, as described in
could get arbitrary quantile of the modelled
Section 3.2 For the implicit modeling, we use the Fourier feature (Dabney et al. 2018 Tancik et al. 2020
where DF is the dimensionality of the Fourier feature, which is a hyperparameter. We then use glement-wise multiplication to combine the Fourier feature and the processed state representation for downstream processing; see the following subscctions for specific modeling details for cach different task. We compare their performance in Figure (b) on a 16 x 16 x 16 hypergrid, where M = 200 for explicit modeling and N = 8, 256 dimensional Fourier features for implicit modeling. The implicit way is shown to have better performance, thus we use it in all the other experiments in this work. Regarding computation consideration, we remark that although the implicit way seems to need more number of network evaluation, however the actual runtime stays similar since we could parallel the multiple calls of the implicit. quantile network through batch-level operation, thanks to the efficient implementation of batch network inference in PyTorch (Paszke et al. 2019).
to augment the scalar 3 input: (B3, = ReLU EEP cos(Tri)ww + bb) bb))) : ......i......D
We also conduct an ablation study about the implicit quantile network implementation. For the N and Ñ described in A Amgorithm D we try different values in a 16 x 16 x 16 hypergrid with 256 dimensional Fourier feature. Figure c indicates that the performance of QM algorithm robust to the choice of N (we always set N = Ñ for simplicity. For the number of dimension of the Fourier feature, we conduct an ablation study shown in Figure dd) still with 16 16 x 16 hypergrid and N 8. The result also shows that QM is robust to the selection of the Fourier feature dimension.
C.2 Hypergrid
An illustration of the reward landscape when D H 88 is shown in Figure aa). The probability density function of the learned model is empirically estimated from the past visited 200000 states. The Get uses networks that are three layer MLPs with 256 hidden dimension and Leaky ReLU activation with onc-hot state representation as inputs. FM uses an MLP to model the edge flows. TB uses an MLP to output the logits of the forward and backward policy at the same time. QM takes a similar three layer MLP modeling to FM, in the sense that the state input first goes through one layer, clement-wise multiplied with the Fourier feature, and then goes through two linear layers. All methods are optimized by Adam. Regarding hyperparameters, we do not do much sweeping: QM uses the same learning rate as FM which is 1 x 10-1.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Methos Implic Fxplict
Nethod
Method
2,00003
0.00342
0.20040
Fourer dinuds
Fouter dmmi33
2,00335
0.00235
0.00035
Fourer dm
Fauter dm--112
Foureer dimw0024
2.00025
0.00325
0.00025
1,00000
0.00328
0.00020
2000100
0.00213
0.00010
1.00055
0.03333
0.38005
State visited (x10")
visited (x10")
State visite (x20")
(a)
(b)
(c)
(d)
Figure Hypergrid figures. (a) Illustration of the target reward function for 8 X 8 hypergrid, where a darker colour means higher reward. Figure adapted from Malkin et al. (2022a). (b) The ablation study between implicit and explicit modeling of the quantile function; the implicit way achieves betler sample
efficiency. (c) Ablation study on the number of 3 percentage sampled. (d) Ablation study on the number of Fourier features used.
3.003
C00K
30001
1000
64
(a)
(b)
(c)
(d)
2000
2413"
(c)
(g)
(h)
Figure& 8: Experiment results of IQN on the hypergrid tasks for different scale levels (the red dashed line corresponds to the final result of Quantile Matching). The first row demonstrates the empirical L1 error while the second row illustrates the number of modes. As shown, TQN underperforms Quantile Matching by a large margin in terms of both convergence and diversity.
what's more, QM uses N = = 8 and 256 dimensional Fourier feature. Other baselines like TB, MCMC, PPO use the same configuration as in Malkin et al. (022a).
Risky hypergrid domain. We set = 8, = 2 or 4 for small or large environments, respectively. An illustration of the risky hypergrid with D= 2 is shown in Figure 2 It triggers low reward of 0.1 when reaching the risky regions (located at the bottom-left and top-right corners of the grid with D = 2, and symmetrically for D = 4) with probability 30,, and the agent obtains the original reward of 2.6 or 0.6 otherwise. For risk-neutral agents (flow matching and quantile matching), they may still enter risky regions while risk-averse agents should be aware of avoiding reaching these areas as much as possible. We track the number of modes discovered by each method during training, and also evaluate the violation rate. The latter metric is computed based the number of times the agent entered the risky regions over a number of past.
19

---

Published in Transactions on Machine Learning Research (01/2024)
Medted
2.00775
Ry=1e
0.008
0.310
Ry 16-05
2,00300
Ry-1--06
0.030
0.325
Ry=lc
0.2315
0.200
Rym1o
Rywle
0.0055
0.000
2
3.00300
08200
0.000
0.00225
0:205
2,00000
State vistted (x10")
(x10")
vistted
(a) Quantile matching
(b) Flow matching
(c) Trajectory balance
Figure 9: Hypergrid results with extremely sparse signals for GFlowNeet methods. We find that TB is very casy to be affected by sparse reward sctups and gives highly unstable performances, while QM behaves stably across different levels of reward landscape.
samples. Experiments in risky hypergrid domain follow the same hyperparameter setup as described in the above section.
Exploration with extreme sparse signals. We also investigate the setup with extremely sparse learning signal, where assign very small value (i.e., from 10 to 1 10-9) to Ro in Equation 18] In this part use a 3 dimensional grid with H = 8. When Ro is extremely small, the agent could hardly get any learning signal for most of the time, as the reward is near zero for most areas in the hypergrid (there is high reward near modes, but in high dimensional the area of modes is very little). Our results show that the proposed QM method is much more robust to the change of sparsity in reward landscape, while the exploration ability of both FM and TB are easily affected by sparse rewards.
C.3 Sequence generation
The reward is defined as R(x) maxmeM exp(-dist(x, m]}, where M is a pre-generated set of sequences and the distance the Levenshtein distance. The is constructed by randomly combining symbols from ("00000000'; 11111111', 11110000,, 00001111', 0001110000) We re-generate the set of target sequences with the same generation protocol as Malkin et al. (2022a). We set the forward policy to uniform distribution with 0.5% probability for exploration for all methods. The reward exponential hyperparameter is set to 3. Regarding the network implementation, we use transformer with 3 hidden layers and 8 attention heads. For evaluation, we also plot the curve of top-100 rewards for three GelowNet mot in Figure 10(a) with our own experimental results. We do not use the correlation between log reward and model log likelihood on a given dataset as in Malkin et al. (2022a), as we find that the dataset could not cover the diverse modes appropriately, which causes that the correlation sometimes even reaches the high point at initialization; see Figure 10 b). Although the final rewards similar among all three methods, the proposed QM reaches plateau in the shortest time.
All methods are optimized with Adam optimizer for 50000 training steps, with the minibatch size being 16. We use a fixed random action probability of 0.005. For all the baselines, we take the results from Malkin ct al. 2022a). For quantile matching we use a two-layer MLP to process the Fourier feature of B, and then compute its element-wise product with the transformer encoding feature;about hyperparameters, we use the same learning rate (5 10-1 10-4) as FM, N N 16, and 256 dimensional Fourier feature.
C.4 Molecule synthesis
We train proxy model to predict the normalized negative binding energy to the 4JNC inhibitor of the soluble epoxide hydrolase (sEH) protein to serve as the reward. The number of the action in the range between 100 and 2000, making | As 1016 For the choice of neural network architecture,
message
20

---

Published in Transactions on Machine Learning Research (01/2024)
0.93
0.0
0.6
Peaman Commatann
0.4
11p-100 Horoon
0.73
Mchoad
0.82
Method
Quantie Masciin
Ouantle
7.4
*iow Nakting
talante
MAR3
Hartiig
10.6
Fllva Marnting
6.8
PPD
10000 15333 20000 25330 30000
10000
20333
33928
Stop
State (w110)
(a)
(b)
(c)
Figure 10: (a) Top-100 average reward for the sequence generation task. (b) Instability of the correlation between model log likelihood and log of true reward for the sequence generation task. (c) Top-10 average reward for the molecule synthesis task.
passing neural networks Gilmer et al. 2017 MPNN) for all models. Apart from the results in the main text, we also show top-10 reward plots in Figure c).c).
For evaluation of number of modes, we define a set and add a newly discovered molecule into it if its reward value larger than 7.5 and its Tanimoto similarity with all previous set elements is smaller than 0.7. Note that this criterion is stricter than simply counting the number of different Bemis-Murcko scaffolds that reach the reward threshold Bengio al. (2021a); Pan et al. (2022). Pan al. (2022) is experimented on modified and thus different task, where the signal is sparsified to make the importance of active exploration to be prominent. Therefore, we do not include it in this work due to different setups. The way using Tanimoto separated modes is more applicable for de novo molecule design, while scaffold-based metric is more appropriate for lead optimization. Further, counting Tanimoto separated modes is a more striet metric as can be seen from the Figure 14 and Figure 15 from Bengio et al. (2021a). We mcasure the Tanimoto similarity for the top-1000 molecules as in [Bngii et al. (2021a). TB uses a uniform backward policy as in Malkin al. (2022a) as it provides better results. For baseline setups we simply follow the hyperparameters from [engii al. the same Adam learning rate (5 x 10-4) as FM,
(2021a): Malkin et al. (2022a). For quantile matching N N N 16, and 256 dimensional Fourier feature.
There are two ways for counting the modes of molecules: (i) the first way is to distinguish molecules by their Taninoto similarity; (ii) the second way is to check if the Bemis-Mureko scaffold (i.e., a simplified representation denoting the core structure of molecules) of the molecules are different. For de novo drug discovery, one should use the first way bocause it captures more biological information, while the second way is mostly used in the application of lead optimization. Therefore, we follow the first option for reporting the results. We remark that in Bengio et al. (2021a) the authors chose the second way due to unfamiliarity with the domain at the time of project preparation.
21


Reviews:
[{"review_id": "e3wdwsfmZy", "reviewer": "Reviewer_RuLS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The method combines GFlowNets with distributional RL.\n* The presented results show that the method outperforms the baselines.\n\nWeaknesses:\n* In eq. (7) the left-hand side should be $P_B(\\tau=(s_0\\to\\ldots\\to s_n)|s_n=x)$, according to the cited works.\n* Proposition 1 is written informally in a way that is hard to understand. For example, it is unclear what \"given sufficiently large capacity and computation resource\" or \"the obtained GFlowNet\" refer to.\n* Proposition 2 is a well-known fact, see e.g., [1].\n* The origin of equation 11 and its relation to TD error should be explained. \n* The approximation in Equation (13) is not merely MC approximation of expected value (which would correspond to $\\frac{1}{N}\\sum_{i=1}^N \\exp(Z_{\\beta_i}^{log}(s\\to s';\\theta))$ ). The used approximation is smaller due to the Jensen's inequality. The paper should comment on this.\n* The paper should put the parameters used for the algorithm and the model details in one place. In particular, how often are the parameters updated (once per trajectory, more often?).\n* There is no single section in the paper that clearly describes all the necessary components of the environments (MDP, including state space, action space, transition kernel, and the reward function) and metrics (see the details below).\n* The hypergrid environment is not clearly described:\n\t* It is unclear from the description what the \"mode\" means. After looking in the Appendix and cited papers, one can deduce that it refers to the states that yield the maximum value of $R$.\n\t* The cited papers show that the reward is only awarded when the agent chooses the special terminal action. However, the paper does not mention this (it only says it \"is available for each state\"). Also, what is the episode's length, and what is the reward if that limit is reached?\n\t* In Section 5.1 (and Appendix C.2), it is said that there are exactly $2^D$ modes, which seems not to be true in general. For example, for the defined $R$ and $H=32$, there are $3^D$ modes (corresponding to each coordinate belonging to $\\{4, 5, 6\\}$). \n\t* In Appendix C.2., Figure 7 should be described as $D=2$, $H=8$ (not the other way around).\n* The reward function for the sequence environment is only provided in Appendix C.3.\n* The molecule environment used in Section 5.3 is almost not described, and only some details are provided in Appendix C.4.\n* Experiments:\n\t* One of the main metrics, a \"number of modes\", is not defined. Is it counted during the entire training phase? Is it discovered by the trained model in an evaluation phase by running some simulations? \n\t* There is a small amount of seeds (4) in experiments.\n\t* What is the x-axis in Figures 3, 4, 5? Is it the training steps, the steps of the agent, or something else?\n\n[1] Dhaene, Jan, et al. \"Risk measures and comonotonicity: a review.\" _Stochastic models_ 22.4 (2006): 573-606.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies GFlowNet generative model where the flow function is modeled as a random variable, similar to how it is done in distributional reinforcement learning (RL). A quantile flow matching algorithm is presented and evaluated on three tasks: hypergrid, sequence generation, and molecule optimization. The results suggest that the method outperforms the chosen baseline approaches.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fj4qCAdQIN", "reviewer": "Reviewer_NWKL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As stated above, the strength are that it works out well in a natural manner and is an extension that makes a lot of sense.\nThe only weakness is that it is slightly incremental and an almost expected development, but that is not an important factor for this jornal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors are further developing GFlowNets from 2021 that has quickly gathered substantial attention.\nGFlowNets represents a policy as a flow of a graph such that the total probability depends on the reward as desired while having been formed by a sequence of decisions for tasks that are combinatorial in nature.\n\nHere the authors adds the distributional aspect that has found to be very useful in more standard deep RL, and one moves the quantile matching approach to GFlowNets. Having a distribution allows both for risk-sensitive decisions and to achieve more diversity, and the authors demonstrates that their approach yields this on the hyper-grid tasks from the original paper and on a more realistic task of generating diverse molecules with the goal of finding such with desired chemical properties.\n\nIt is a well written paper and useful development of an approach that is quickly developing momentum and finding great interest at this time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have no specific changes to request.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OPTW58HeGE", "reviewer": "Reviewer_ycWV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method appears sound\n- The experiments confirm the claims\n\nWeaknesses\n- The application to molecule optimization is limited as there is no comparison to SOTA", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of GFlowNet to settings where reward is stochastic. The proposed method models the distribution of rewards and can be used for risk-sensitive generation. Quantile networks are used to model the reward distribution and a method called \"quantile matching\" is introduce to train such distributional GFlownets. Experiments provided on Hypergrid and molecule optimization tasks show that the method improves over prior work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Prior work reports better performance on the 7.5 reward diversity benchmark (6000 in Bengio'21a). The current paper reports a performance of 400. Why the difference? The appendix notes that prior work uses a different less strict metric. Is it possible to compare on both the prior work and the new metric?\n2. It is unclear whether any line in Fig 6 corresponds to Bengio'21a or Pan'22 (or any other prior work). If not, this comparison should be added.", "broader_impact_concerns": "\\-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Suggested changes:
1. Define what a flow is before using that word in the abstract and introduction.
2. Specify the (overall) objective optimized by Algorithm 1. The objective in Eq. 12 is per-state.
3. Make Proposition 1 more precise

Required changes:
1. Make the changes promised to the reviewers
2. Incorporate the clarifications provided to the action editor
3. Describe the tasks used in greater detail in the body of the paper, to make the paper more readable and self-contained
4. Correct the claim in Section 5.1 that the number of modes is always 2^D. Reviewer RuLS has pointed out that e.g. for H=31 there are actually 4^D modes.
 5. Mention Autoregressive Quantile Networks for Generative Modeling, Ostrovski et al., ICML 2018 in Related Work
    
Also please proofread the paper carefully, as it contains a fair number of typos and minor mistakes.
Here are some corrections:
- Remove "in order" from the first contribution in Section 1
- "satisfy" -> "satisfies" above Eq. 1
- p.5 "Among different the" -> "Among the different"
- Figure 3 caption: methods achieves -> methods achieve
- p. 10 mergence -> combination
Decision: Accept (poster)