CFlowNets: Continuous Control with Generative Flow Networks
Authors: Yinchuan Li
Shuang Luo
Haozhi Wang
Jianye HAO
Venue: ICLR 2023 poster (2023)
Keywords: Continuous control tasks
Generative flow networks

Abstract: Generative flow networks (GFlowNets), as an emerging technique, can be used as an alternative to reinforcement learning for exploratory control tasks. GFlowNets aims to sample actions with a probability proportional to the reward, similar to sampling different candidates in an active learning fashion. However, existing GFlowNets cannot adapt to continuous control tasks because GFlowNets need to form a DAG and compute the flow matching loss by traversing the inflows and outflows of each node in the trajectory. In this paper, we propose generative continuous flow networks (CFlowNets) that can be applied to continuous control tasks. First, we present the theoretical formulation of CFlowNets. Then, a training framework for CFlowNets is proposed, including the action selection process, the flow approximation algorithm, and the continuous flow matching loss function. Afterward, we theoretically prove the error bound of the flow approximation. The error decreases rapidly as the number of flow samples increases. Finally, experimental results on continuous control tasks demonstrate the performance advantages of CFlowNets compared to many reinforcement learning methods, especially regarding exploration ability.

Published as a conference paper at ICLR 2023
CFLOWNETS: CONTINUOUS CONTROL WITH GENERATIVE FLOW NETWORKS
Yinchuan Li', Shuang Luo²; Haozhi Wang", Jianye Hao 'Huawei Noah's 's Ark Lab, Beijing, China *Zhejiang University, Huangzhou, China Tianjin University, Tianjin, China
(liyinchuan, hao jianee jianyyJehuaveeii Hehuawei.ccooo com, 1uoshuang@zju. edu. .c..cn wanghaozhi @t edu. on
ABSTRACT
Generative flow networks (GFlowNets), as an emerging technique, can be used as an alternative to reinforcement learning for exploratory control tasks. GFlowNet aims to generate distribution proportional to the rewards over terminating states, and to sample different candidates in an active learning fashion. GFlowNets need to form a DAG and compute the flow matching loss by traversing the inflows and outflows of each node in the trajectory. No experiments have yet concluded that GelowNets can be used to handle continuous tasks. In this paper, we propose gen- erative continuous flow networks (CFlowNets) that be applied continuous control tasks. First, we present the theoretical formulation of CFlowNets. Then, a training framework for CelowNets is proposed, including the action selection process, the flow approximation algorithm, and the continuous flow matching loss function. Afterward, we theoretically prove the error bound of the flow approx- imation. The error decreases rapidly as the number of flow samples increases. Finally, experimental results on continuous control tasks demonstrate the perfor- mance advantages of Celow compared to many reinforcement learning meth- ods, especially regarding exploration ability.
INTRODUCTION
As emerging technology, generative flow networks (GFlowNets) (Bengio et al. 202 b) can make up for the shortcomings of reinforcement learning (Kaelbling et al.) 1996; Sutton & Barto 2018) exploratory tasks. Specifically, based on the Bellman equation (Sutton & Barto 2018)) reinforcement learning is usually trained to maximize the expectation of future rewards; hence the learned policy is more inclined to sample action sequences with higher rewards. In contrast, the training goal of GFlowNets is to define a distribution proportional to the rewards over terminating states, i.c.. the parent states of the final states, rather than generating a single high-reward action sequence (Bengio et al. 20) This is more like sampling different candidates in an active learning setting (Bengio et al.) (021b), thus better suited for exploration tasks.
GFlowNets construct the state transitions of trajectories into directed acyclic graph (DAG) struc- ture. Each node in the graph structure corresponds to different state, and actions correspond to transitions between different states, that is, an edge connecting different nodes in the graph. For discrete tasks, the number of nodes this graph structure limited, and each edge can only correspond to one discrete action. However, in real environments, the state and action spaces are continuous for many tasks, such as quadrupedal locomotion (Kohl & Stone 2004), autonomous driving (Kiran et al. 2 Shalev-Shwartz et al. 2016 Pan et | 2017), or dexterous in-hand ma- nipulation (Andrychowicz et al. 2020). Moreover, the reward distributions corresponding to these environments may be multimodal, requiring more diversity exploration. The needs of these envi- ronments closely match the strengths of GelowNets. (Bengio et al.) 2021b) proposes an idea for adapting GFlowNets to continuous tasks by replacing sums with integrals for continuous variables.
*Corresponding Author: Shuang Luo. This work was completed while Shuang Luo and Haozhi Wang were members of the Huawei Noah's Ark Lab for advanced study.

---

Published as a conference paper at ICLR 2023
and they suggest the of integrable densities and detailed balance (DB) trajectory balance (TB) Malkin et al. (2022) criterion to obtain tractable training objectives, which can avoid some integration operations. However, this idea has not been verified experimentally.
In this paper, we propose generative Continuous Flow Networks, named CFlowNets for short, for continuous control tasks to generate policies that be proportional to continuous reward func- tions. Applying Gets to continuous control tasks is exceptionally challenging. In generative flow networks, the transition probability is defined as the ratio of action flow and state flow. For discrete state and action spaces, we can form a DAG and compute the state flow by traversing a node's 's incoming and outgoing flows. Conversely, is impossible for continuous tasks traverse all state-action pairs and corresponding rewards. To address this issue, we use important sampling to approximate the integrals over inflows and outflows in the flow-matching constraint, where we use deep neural network to predict the parent nodes of each state in the sampled trajectory. The main contributions of this paper are summarized as the following:
Main Contributions: 1) We extend the theoretical formulation and flow matching theorem of pre- vious GFlowNets to continuous scenarios. Based on this, a loss function for training CFlowNets is presented; 2) We propose an efficient way sample actions with probabilities approximately pro- portional to the output of the flow network, and propose a flow sampling approach to approximate continuous inflows and outflows, which allows to construct continuous flow matching loss; 3) We theoretically analyze the error bound between sampled flows and inflows/outflows, and the tail becomes minor the number of flow samples increases; 4) We conduct experiments based tinuous control tasks to demonstrate that CFlowNets can outperform current state-of-the-ar RL algo- rithms, especially in terms of exploration capabilities. To the best of our knowledge, our work the first to empirically demonstrate the effectiveness of flow networks on continuous control tasks. The codes are available at http::/wwooniiioooosteeeeesssess
2
PRELIMINARIES
2.1 MARKOV DECISION PROCESS
stochastic, discrete-time and sequential decision task can be described as Markov Decision Process (MDP), which is canonically formulated by the tuple:
M = (S,A,,
In the process, E S represents the state space of the environment. each time step, agent receives a state and selects an action a on the action space A. This results in a transition to the next state s' according to the state transition function P(s [s, a) S x X S [0. 1J. Then the agent gets x - R. stochastic policy æ maps each state to distribution over actions r--ss) and gives the probability T r[ass) of choosing action a in state s. The agent interacts with the environment by executing the policy т and obtaining the admissible trajectories {(st, Qz T't, Tt, 1 where is the trajectory length. The goal of an agent to maximize the discounted return Eo........ IEe= 7'r 88 8, ag = a, T.. where E is the expectation the distribution of the trajectories and 7 E [0, 1) is the discount factor.
the reward r based on the reward function R(s, a)
2.2 GENERATIVE FLOW NETWORK
GFlowNet sees the MDP as a flow network. Define s' = T(s, a) and F(s) as the node's transition and the total flow going through s. Define an edge/action flow F(s, a) = F(s - s') as the flow through an edge 8 + s'. The training process of vanilla GFlowNets needs to sum the flow of parents and children through nodes (states), which depends on the discrete state space and discrete action space. The framework is optimized by the following flow consistency equations:
Σ
Σ F(s'.a'). d'EA(e")
which means that for any node s, the incoming flow equals the outgoing flow, which is the total flow F(s) of node
2

---

Published as a conference paper at ICLR 2023
3 CFLOWNETS: THEORETICAL FORMULATION
Considering a continuous task with tuple AS, where S denotes the continuous state space and A denotes the continuous action space. Define trajectory ($1,-..,, in this continuous task sequence sampled elements S such that every transition Oę 8t - 8141 E A. Further, we define an acyclic trajectory
(81,,.., 5n) trajectory satisfies the acyclic constraint: Vsm E T, 8h m + k, we have 8m + Sg. Denote and 8 respectively as the initial state and the final state related with the continuous task AS, we define the complete trajectory as any sampled acyelic trajectory from (S. .) starting in So and ending in s.. Correspondingly, a transition s + sy into the final state defined as the terminating transition, and F(s - 84) a terminating flow.
A trajectory flow F(r) T + R+ is defined as any nonnegative function defined on the set of complete trajectories т. For each trajectory т, the associated flow F(r) contains the number of particles (Bengio et al. 2021b) sharing the same path т. In addition, the tuple A, F)) is called a continuous flow network. Let 7 (ss a) = indicate an action that could make transition from state s to attain 8'. Then we make the following assumptions.
Assumption 1. Assume that the continuous take (S,A) is an "acyclic" task, which means that arbitrarily sampled trajectories T are acyclic, i.e.,
8; + 8j. Vs 8 ET ($0. sn); i
Assumption 2. Assume the flow function F(s, a) is Lipschitz continuous, iee,
[F(s, a) F(s, a')l I L aa d'll, a, a, A, (F(s,a) F(s',a) & L s'll, 8,8' ES
(3)
(4)
where L is a constant.
Assumption 3. Assume that for any state pair (st, there is a unique action a, such that
T(s: 04) = 8t-1, Le., taking action a, in 8t is the only way to get to 8t+1- Hence we can define 8, = g( (1)+1, where g(-) is transition function. And assume actions are the translation actions.
The necessity and rationality of Assumptions 3 are analyzed in the appendix. Under Assumption we define the parent set (st) of state 8, as the set that contains all the direct parents of s, that could make a direct transition to St, i.e., P(st) = {s S T(s, E A) = st}. Similarly, define the child set C(x)) direct transition from i.e., C(s.) continuous flow definitions, where Assumptions make these integrals integrable and meaningful.
state 8, as the set contains
the direct children 8, that could make {s E S T(s,, E A) s). Then, we have the following
Definition 1 (Continuous State Flow). The continuous state flow F(s): S + R is the integral of the complete flows passing through the state:
/ F(T)dT.
F(s)
(5)
T:SET
Definition 2 (Continuous Inflows). For any state 8t, inflows are the integral of flows that can reach state 8t, i.e.,
JEST F(s + st / s:T(ss))=s
F(s,a)ss=F((5 = Ja:T(sa)=ss
F(s, als,a)da,
(6)
where 8t and s g(st,a) a) since Assumption holds. Definition (Continuous Outflows). For any state st, the outflows are the integral of flows passing through state with all possible actions A. i.e.,
{(. F(se, a)da.
F(st s)ds F(si)=
JAEA
Based on the above definitions, we can define the transition probability P(s s'(s) of edge s' as a special case of conditional probability introduced in Bengio et al. (2021b). In particular, the forward transition probability is given by
F(s: + s(+1)
PF Pf(s:41|ss) = P(st + S141
(8)
F(st)
3

---

Published as a conference paper at ICLR 2023
Similarly, the backwards transition probability is given by
PB((e|s441) P(s, + +1+115441) F(te + 88+1)
(9)
F(su+1)
For any trajectory sampled from continuous task (S, we have
n-1
VT 8,),
II PF (s(8+41(5))
(10)
47 (8), Pb(TT) (5(Se(s
(11)
and we further have
S Pr(8'sadds' = and Vs E So JSEPCS Pn(s'(s)ds'== =1 (12)
Ys E SM8r),
(sg, ..., Sns 8) that starts in so and ends in Markovian flow (Bengio et al. 202Tb) is defined as the flow that satisfies
Given any trajectory т
P(s + s'(T) = P(ss s'(s) PP(((sss)
and the corresponding flow network AS, F) called a Markovian flow network (Bengio et al,, 2b2 Then, we present Theorem proved in the appendix B.T. which is an extension of Proposi- tion 19 in [Bngiio et al. (2021b) to continuous scenarios.
Theorem (Continuous Flow Matching Condition). Consider non-negative function F(s, a) tak- ing state S and action A as inputs. Then we have F corresponds to flow if and only if the following continuous flow matching conditions are satisfied:
As' so: F(5')= = (SEP(N) F(s + s'dds [SST(Ca)= F(s,a:s s')ds
(13)
AA sss, F(5)) [ Fls' + s''ds" -J a(s'.a)do.
Furthermore, F uniqueey defines a Markovian flow F matching F such that 11 n+1 F( + 8.)
F(T)
(14)
IT,,, F(s))
Theorem T means that long as any non-negative function satisfies the flow matching conditions, a unique flow is determined. Therefore, for sparse reward environments, i.c., R(s) 00 Vs # s,, we can obtain the target flow by training a flow network that satisfies the flow matching conditions. Such learning machines are called CFlowNets, and we have the following continuous loss function:
24 Σ (J F(s - 8t)d8:11 - R(se) -
2
((T)
F(st 81+1 88+1) )dse)))s))))
8:=81
However, obviously, the above continuous loss function cannot be directly applied in practice. Next, propose method to approximate the continuous loss function based the sampled trajectories to obtain the flow model.
4 CFLOOWNET:: TRAINING FRAMEWORK
For continuous tasks, is usually difficult to access all state-action pairs to calculate continuous inflows and outflows. In the following, propose the CFlowNets training framework to address this problem, which includes an action sampling process, a flow matching approximation process. Then, CelowNets can be trained based approximate flow matching loss function.
4

---

Published as a conference paper at ICLR 2023
Interaction
Flow Sampling
Training
Environment
G
CFlowNet
Inflows
Probability Buffer P
d2
Ag-
Ap-1
Inflows Approximation
Randomly Sample Continucus Action
Continuous Matching Lose
Probabilty Distribution
Forward Propagation
Outtlows Approximation
Randomly Sample Continuous Action
flgg
CFlowNet
CFlowNet
Outtiows Reward
Figure Overall framework of CFlowNets. Left: During the environment interaction phase, we sample actions update states with probabilities proportional to the reward according to CFlowNet. Middle: randomly sample actions to approximately calculate the inflows and outflows, where DNN is used to estimate the parent states. Right: Continuous flow matching loss is used to train the CFlowNet based on making inflows equal to outflows or reward.
4.1 OVERALL FRAMEWORK
The overview framework of CFlowNets is shown in Figure including the environment interaction, flow sampling, and training procedures. During the environment interaction phase (Left part of Fig- [. we sample an action prohability buffer based on the forward-propagation of CFlowNets. We name this process the action selection procedure, detailed in Section 4.2] After acquiring the tion, the agent can interact with the environment to update the state, and this process repeats several steps until the complete trajectory sampled. Once a buffer of complete trajectories is available, randomly sample K actions and compute the child states to approximately calculate the outflows. For the inflows, we use these sampled actions together with the current state as the input the deep neural network G to estimate the parent states. Based these, we can approximately determine the inflows. We name this process the flow matching approximation procedure (Middle part of Fig- ure [ detailed in Section 4.3 Finally, based on the approximate inflows and outflows, can train a CFlowNet based on the continuous flow matching loss function (Right part of Figure I as details in Section 4.4 The pscudocode is provided in Appendix C
4.2 ACTION SELECTION PROCEDURE
Starting from an empty set, CFlowNets aim to obtain complete trajectories 7 (So, 81. .1.......)) 85) E ET by iteratively sampling a <(a](s)) F(s..a)) with tuple {(se, 0t, Tt, 8141) However, is difficult to sample trajectories strictly according to the corresponding probability at, since the actions are continuous, we cannot get the exact action probability distribution function based on the flow network F(s;,,)) To solve this problem, each state 8t, we first uniformly sample actions from A and generate an action probability buffer P {Fr(s,, Ad which is used as an approximation to the corresponding probabilities of all actions. Obviously, actions with larger F(st, a,) will be sampled with higher probability. this way, we approximately sample actions from a continuous distribution according to their corresponding probabilities.
F(*.)
action probability distributions. Then we sample an action from P according
Remark 1. After the training process. for tasks that require a larger reward, we can sample actions with the maximun flow output in P during the test process obtain a relatively higher reward. How the output of the flow model is used is flexible, and we can adjust it for different tasks.
4.3 FLOW MATCHING APPROXIMATION
Once a batch of trajectories available, to satisfy flow conditions, we require that for any node St, the inflows Sa.T(sa))=,, F(s, a) equals the outflows SaSA F(se, adda, which is the total flow
5

---

Published as a conference paper at ICLR 2023
F(sr) of node 8t. However, obviously, cannot directly caleulate the continuous inflows and outflows to complete the flow matching condition. An intuitive idea is to discretize the inflows and outflows based reasonable approximation and match the discretized flows. To do this, sample K' actions independently and uniformly from the continuous action space A and calculate corresponding F(st, ak), 1,... K as the outflows, i.e., we the following approximation:
I. (((s,aade x
(A(A)
Σ
K
F(s az),
(15)
K
where p(A) denotes the measure of the continuous action space A.
By contrast, an approximation of inflow is more difficult since we should find the parent states first. To solve this problem, we construct deep neural network G (named "retrieval" neural network) parameterized by e with (#1+1, at) as the input while as the output, and train this network based on B with the MSE loss. That is, we want use to function g(-). The network usually easy to train since consider tasks satisfy Assumption 3 and we can obtain high-precision network G through simple pre-training. As the training progresses, we can also occasionally update G based on the sampled trajectories to ensure accuracy. Then, the inflows can be calculated approximately:
J e)T
F(g(s;, a). m(A) K
K F((Go(s as),
(16)
Next, by assuming that the flow function F(s, a) is Lipschitz continuous in Assumption could provide a non-asymptotic analysis for the error between the sample inflows/outflows and the true inflows/outflows. Theorem _ establishes the error bound between the sample outflows (resp. inflows) and the actual outflows (resp. inflows) in the tail form and shows that the tail is decreasing exponentially. Furthermore, the tail gets much smaller with the increase of K, which means the sample outflows (resp. inflows) are a good estimation of the actual outflows (resp. inflows).
Theorem Let {ax) k== be sampled independently and uniformly from the continuous action space A. Assume Gø can optimally ousput the actual state with (8t+1,0)) 01). For any bounded continuous action a E A and any state St E S. we have
((A) K
F(st, ax) - f.e F(st; 21) 2exp
Kt²
P
(17)
2(Lp((A)diam (A))
and
(MA)
P
F(G (st, ak),
F(s.a)da 2t
K
Ja:T(o,a)=er
Kt²
2eep
(18)
((Lu(A) A) diam(S)))
where is the Lipschitz constant, diam(AA denotes the diameter of the action space A and dSam(S) denotes the diameter of the state space S.
4.4 LOss FUNCTION
Based on
15 and (6), the continuous loss function can be approximated by
84 Σ
Co(T)
Σ ((GGa(s).a)..... -RR(st)
Σ
Fo Fo as)
(19)
where 0 is the parameter of the flow network F(-) and A = K/u(A). Note that in many tasks we cannot obtain exact ((AA). For such tasks, we can directly set 1, and then adjust the reward shaping to ensure the convergence of the algorithm!]
A commonly used reward shaping method is to multiply the reward by a constant and adjust the reward to appropriate range to ensure better convergence. Therefore, after setting À to 1, reasonable reward shaping operation can also compensate for the influence of error.

---

Published as a conference paper at ICLR 2023
It is noteworthy that the magnitude of the state flow at different locations in the trajectory may not match. For example, the initial node flow is likely to be larger than the ending node flow. To solve this problem, inspired the log-scale loss introduced in GFlowNets (Bengio et al. 2021a), we can modify (19) into:
K plog
La(T) =
log
E
+
(G exp
8c=81
k=1
K
}} 2
- log e + AR(ss)
+ exp F (8t, oxl)
Σ k=1
(20)
where é is a numerical problem of taking the logarithm of tiny flows. Note that Theorem 2 cannot be used to guarantee the unbiasedness of (20) because log [E(xx) * log(()). But experiments show that this approximation works well.
that helps to trade off small versus large flows and helps avoid the
RELATED WORKS
Generative Flow Networks. Generative flow networks proposed enhance exploration capa- bilities by generating a distribution proportional to the rewards over terminating states (Bengio et al.) 2021b Since the network only samples actions based on the distribution of the corresponding wards, rather than focusing only on actions that maximize rewards such as reinforcement learning, it can perform well on tasks with more diverse reward distributions, and has been successfully applied to molecule generation (Bengio et al. 2021a] Malkin et al. [022 Jain et al. 2022). discrete proba- bilistic modeling (Zhang 2002b), structure learning (Deleu et al. 2022p. p, causal discovery Li al. (2022) and graph neural network Li al. (2023). The connection between deep generative models and GFlowNets is discussed in Zhang et al. (2022a) through the lens of Markov trajectory learning. In Bengio et al. (2021b), an idea is proposed for adapting GFlowNets to continuous tasks by replacing sums with integrals for continuous variables. Malkin et al. (2022) and Bengio et al. (2021)) propose detailed balance (DB) and trajectory balance (TB) objectives, which use paramet- ric forward and backward policies in the objective function. These new objective functions do not require evaluating the flow model multiple parents of a state, which is more efficient, especially for high-dimensional environments. Malkin et al. (2022) and Bengio et al. (2021b) mentioned that these objective functions can also be used in continuous scenarios by replacing the policy likelihoods in the objective with probability densities. A possible disadvantage is that it is not easy to estimate Pr and PB a continuous environment, since the state space is much larger than in a discrete sce- nario, and small error in modeling probability densities can greatly affect the final performance. How to combine DB and TB with CFlowNets will worthy future work.
Continuous Reinforcement Learning. Policy gradient algorithms are widely used for rein- forcement learning problems with continuous action spaces. The deterministic policy gradient (DPG) (Silver et al. 2014) algorithm is actor-critic (Grondman et al. 2012; Rosenstein et al.) 2004) method that uses an estimate of the learned value Q(s, a) to train a deterministic policy A parameterized by d". Compared with CFlowNets, the policy is updated by apply- ing the chain rule to the expected return J from the start distribution with respect to the policy parameters:
V J R ED
V Q (s (88) [a=p(s]om)
(21)
ED V.Q (s 02) (=)(s)) Vonu (s 0m))
where D is the replay buffer. The policy aims to maximize the expectation of future rewards, which are estimated by Q-learning. In this setting, the trajectories generated by the policy may be relatively homogeneous. However, the training goal of CFlowNets to define distribution proportional to the rewards over terminating states, resulting in more diverse trajectories that are beneficial for exploring the environment.
Later, deep DPG (DDPG) (Liliicrap et al. 2015) improves DPG and has good sample efficiency but suffers from extreme brittleness and hyperparameter sensitivity. Therefore, it is difficult extend
7

---

Published as a conference paper at ICLR 2023
DDPG complex, high-dimensional tasks. To improve DDPG, twin delayed DDPG (TD3) (Fu- jumoto et al.] 2018) adopts an actor-critic framework and considers the interaction between value update and function approximation error and in the policy. There also some policy gradi- ent (Sutoon et al. 1999 Kohi & Stone 2004 Khadka & Tumerr 2018) based algorithms that can be adapted for continuous tasks, such proximal policy optimization (PPO) (Schulman et al., (017) algorithms, asynchronous advantage actor-critic (A3C) (Stooke & Abbeel 2018). and importance weighted actor-learner architecture (IMPALA) (Espeholt al. (018). PPO has the benefits of trust region policy optimization (Schulman et al. 2015), enabling multiple batches of data to be updated together. Therefore, it is simpler to implement, more general, and has lower sample complexity. Recently, phasic policy gradient (PPG) (Cobbe et al. 2021) is proposed to decouple the training between policy and value function while keeping their feature sharing, and PPG optimizes each objective with an appropriate level of sample reuse to improve sample efficiency. Most of these im- proved policy gradient methods can be classified as aiming at maximizing reward, so none of them are better suited for exploration tasks than CFlowNets.
Furthermore, some maximum entropy (Pitis et al. 2020 Haarnoja et al. 2018a| Hazan et al. 2019; Yarats et al., 2021) based reinforcement learning algortthms can also be adapted for continuous tasks, such as soft actor-critic (SAC) (Haarnoja et al. 20188). By maximizing the expected reward and entropy, the actor network of SAC can successfully complete tasks while acting randomly as possible. The difference between CHlowNets and SAC is: 1) SAC selects actions by a Gaussian policy, which is less expressive than using general unnormalized action p.d.f. F(s, a); 2) In the general case, SAC learns to be proportional to the long-term return, which generates the trajectory distribution satisfying p(T) x R(T) with R((T) is the return of r. CFlowNets considers all possible trajectories that lead to terminal state sy, and learn the policy to generate sy with p(sy) x R(sy).
6 EXPERIMENTS
To demonstrate the effectiveness of the proposed CFlowNets, we conduct experiments on several continuous control tasks with sparse rewards, including Point-Robot-Sparsee Reacher-Goal-Sppe Sparse, and Swimmer-Sparse. The visualization of these environments is shown in Figures 7 8 and 9 Then compare CFlowNets with a few state-of-the-art baseline RL algorithms, such as DDPG (Lillierap)
et al. 2015), TD3 (Fujimoto et al. 2018). PPO (Schulman et al. 2017). and SAC (Haarnoja et al.
2018b) More implementation details are provided Appendix
D
Figure 2 illustrates the distributions of learned policies for CelowNets and RL algorithms. All curves are max-min nor- malized. The gray curve is the ground truth of reward distri- bution generated by the agent's different actions when it goes to coordinates (7, 7), which indicates that the optimal action here is to go right or up. The red curve shows the flow net- work output of CFlowNets under different actions, indicating that CelowNets have an excellent fitting ability to the reward. In contrast, other reinforcement learning algorithms have dif- ficulty fitting the actual reward distribution well.
Estimated Reward Value
1.0-
0.8
0.6-
0.4-
DOPO
0.2
BAG
0.0
Chioweete
-1.0
-0.5
0.0
0.5
1.0
Figure 2: Reward distributions on Soint-Robot-Spares Task.
Figures B(a)-(c) show the number of valid-distinetive trajec- tories explored as training progresses in Soint-Robot-Sparsee Reacher-GGoal-Sparee, and Swimmer-Sparse environment, respectively. After certain number of training epochs, 10000 trajectories are collected. A valid-distinctive trajectory is defined as a re- ward above threshold 8, while the MSE between the trajectory and other trajectories is greater than another threshold 6me- That is. if the returns of both trajectories are high, but the two are close and the MSE small, we consider it only one valid-distinetive exploration. 8, in Point- Robot-Sparse, in
and Swimmer-Sparse is set as 0.5, -0.2, 5.0, respectively. &me Reacher-Godd BraccherGGal--paaree -Sarsse and Swimmer-Sparss is set as 0.02, 4.0, 1.0, respec- tively. As can be seen from the figure, DDPG, TD3 and PPO have the worst exploration ability, only one aalid-distinctive trajectory is generated. SAC explores better at the beginning of training, and decreases as the training progresses and gradually converges. In contrast, the exploration ability of CelowNets is very outstanding, the number of trajectories explored far exceeds other algorithms, and the exploration ability has been stable the training progresses.
8

---

Published as a conference paper at ICLR 2023
of Distinctvv Trajectonts
10000
of Distinatvv Trajentries
DDPd T0B
Diatinctive Trajentoriss
8000
T00
6000
3000
3000
4000
2000
2000
1000
1900
2000
4000
6000
8000
10000
2000
1000 6000 8000
10000
0 2000 4000 6000 8000 10000
of Total Sampled Trajectorts
# Total Sampled Trajestories
# of Total Sampled Trajectores
Doint-Robot-So Robot-Sparse
(b) Reacher-Godd Goaet-Goal--aaaee Goal--parse
(c) Swimmer- Sparse
1.6
BAC
DBPS
SeC
DDTG TDD
SAC
ChosNiza
Average Roward
1.4
Averag Reward
-0.10
Average Reward
30
PPO
1.2
1.0
-0.15
20
0.8
0.6
0.4
-0.25
0
100
100
80
Total Timesteps (x10')
Total Timestops (x10P)
Total Timestepa (x10'))
(d) Point- Point-Robot-Se Spaare
(c) Seacher-Goall-ppe
(f) Swimmer-Sparse
Figure 3: Comparison results of CelowNets, DDPG, TD3, SAC and PPO Point-Robot-Sparse Seacher-Goal-Sparee, and Swimmer-Sparse tasks. Top: Number of valid-distinetive trajectories generated under 10000 explorations. Bottom: The average reward of different methods.
Figures 3 df) indicate the rewards during the training process in Point-Robot-Sparse, Reacher- Goal-Sparse, and Swimmer-Sparee environment, respectively. The shaded region represents 95% confidence interval across runs. Figure dd) d) and Figure ( show that CFlowNets has the fastest and more stable upward trend, and the final reward is ahead of that of other algorithms by a large margin. In contrast, CFlowNets do not perform as well as other algorithms in Figure ftf.. Since the rewards in two tasks are more inclined to exploration. CFlowNett has better exploration ability and hence can converge stably. As for Swimmer-Sparee near the maximum reward can achieve faster convergence. is reasonable for CFlowNets to perform worse than RL this task in terms of reward. However, in this environment, CFN can still maintain a good exploration ability.
and Reacher-Goed
are more evenly distributed, so these
its reward distribution is relatively steep, and sampling
CONCLUSION
In this paper, we propose generative continuous flow networks to enhance exploration in continuous control tasks. The theoretical formulation of CFlowNets is first presented. Then, training frame- work for CFlowNets is proposed, including the action selection process, the flow approximation algorithm, and the continuous flow matching loss function. Theoretical analysis shows that the error of the flow approximation decreases rapidly as the number of flow samples increases. Experimental results on continuous control tasks illustrate the performance advantages of CFlowNets compared to many reinforcement learning methods. Especially in the exploration ability, the effect of C'lloNNets far exceeds other vate-of-the-ar reinforcement learning algorithms.
Limitations: Similar to GFlowNets, CelowNets aims to sample actions according to the flow net- work, rather than selecting actions with maximizing rewards. Therefore, CFlowNets are more suit- able for exploration-biased tasks. It does not perform as well reinforcement learning on tasks that aim to maximize reward. Of course, the purpose of CFlowNets is not to completely replace reinforcement learning, but supplement to reinforcement learning, giving option for tinuous control tasks. Future work: Future work will be how to combine CFlowNets with DB and TB objective functions to improve training efficiency.

---

Published as a conference paper at ICLR 2023
REFERENCES
Dan Alistarh, Torsten Hoefler, Mikacl Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. Advances Neural Information Pro- cessing Systems, 31, 2018.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1);3-20, 2020.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation, 2021a.
Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations, 2021b.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Interna- tional Conference on Machine Learning, pp. 2020 2027. PMLR, 2021.
Tristan Deleu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. arXiv preprint arXiv:2202.1399 2022.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference machine learning, pp. 1675- 1685. PMLR, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6)::291-1307, 2012.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp- 1851-1860. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018b.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691. PMLR, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022.
Leslie Pack Kaelbling, Michael Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
10

---

Published as a conference paper at ICLR 2023
Shauharda Khadka and Kagan Tumer. Evolution-guideer policy gradient in reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
Nate Kohl and Peter Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 3, pp. 2614-2624. IEEE, 2004.
Wengian Li, Yinchuan Li, Shengyu Zhu, Yunfeng Shao, Jianye Hao, and Yan Pang. Gflowcausal: Generative flow networks for causal discovery. arXiv preprint arXiv: 10.08185, 2022.
Wenqian Li, Yinchuan Li, Zhigang Li, Jianye HAO, and Yan Pang. Dag matters! gflownets enhanced explainer for graph neural networks. International conference on learning representations, 2023.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint ww0.021509.0277,, 2015.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201. 2022.
Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv: 1704.03999 2017.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learming, pp. 7750-7761. PMLR, 2020.
Michael T Rosenstein, Andrew G Barto, Jennie Si, Andy Barto, Warren Powell, and Donald Wun- sch. Supervised actor-critic reinforcement learning. Learning and Approximate Dynamic Pro- gramming: Scaling Up the Real World, pp. 359-380, 2004.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 2017.
Shai Shalev-Shwarde Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03293 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. arXiv preprint / 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Roman Vershynin. dimeensimmnion probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Dinghuai Zhang, Ricky TQ Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with gflownets. arXiv preprint arXiv:2209.02606, 2022a.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabílistic modeling. arXiv preprint arXiv:2202.01361, 2022b.
12

---

Published as a conference paper at ICLR 2023
A DISCUSSIONS
A.1 WHY IS ASSUMPTION NECESSARY AND REASONABLE?
Necessity: For most environments, it is difficult to generate cycles when sampling trajectory in continuous space. Since vt. bility of st+1 E {so...., ...y 8t) } is very small. However, cycles often arise when certain environments have some special constraints. For example, simple pendulum task (see Figure D the action is control the pendulum
st)) = 0 and (M(A) = (u (su......... (.)) that is, the proba-
certain angle. For this task, it is difficult for pendulum to rotate to exactly the same position in continuous space. However, if wall added the task, the pendulum can easily go to the same position (see Fig- ure 5 i.e. cycle will occur. Therefore, we still need to add an acyclic assumption to make the theory and performance of CelowNets guaranteed.
rotate from the previous position the next position
Rationality: This assumption is reasonable because for many continuous environments it is difficult to form cycles trajectories without special constraints, Even for tasks prone to form cycles, we can directly add time steps in the state space satisfy this assumption.
Figure 4: Pendulum. is difficult for the state to be completely consistent in this continuous space.
Figure 5: Pendulum-with Wall. The state becomes consistent when reaching the wall.
Maximum Lipschitz Constant
0.9
Maximum Lipschitz Constant
0.5
0.8
0.4
0.7
0.3
0.6
0.2
0.5
0.1
0.4
0.0
2000
4000 6000 8000
10000
2000 4000 6000 8000 10000
#
Total Sampled States and Actions
# of Total Sampled States and Actions
(a) for (s, a')
(b) for (s, sa))
Figure Accumulated maximum Lipschitz constant of flow network F(s.a).
A.2 WHY IS ASSUMPTION NECESSARY AND REASONABLE?
Necessity: This assumption is mainly used to guarantee the existence of flow-related integrals, and to ensure that Theorem[2holde
Rationality: We justify this assumption based simulations. As shown in Figure 6 calcu- late [(s.a))((ssa')) and F(s.e)-F(ssee)) of each sample tuple (s, a, a') and (s, s', a) to analysis the
|| Lipschitz constant, respectively. Their accumulated maximum Lipschitz constants are shown in Fig- ures 6 (a) and (b), respectively. Clearly, there exists finite Lipschitz constant for our flow network.
[0-07]
13

---

Published as a conference paper at ICLR 2023
In addition, Lipschitz continuous is common assumption of neural networks, just some quick amples: Du et al. (2019;; Jacot et al. (2018); Allen Zhu et al. (2019); Alistarh et al. (2018) use this assumption prove the convergence of algorithms.
A.3 WHY IS ASSUMPTION NECESSARY AND REASONABLE?
This assumption is used in Definition and enables the retrieval neural network to fit the function g(s, a). While there is one-to-one correspondence between most environment state transitions and actions, there still some special cases where, given state pair (s, s'), there be an infinite number actions. For example: for Pendulum-with-We will in Figure [ after reaching the wall, continuing to increase the action will not continue change the state s In addition, special case of the translation action could be Tss a) + or using the special linear group, such that Definition 2 and 3 hold. The translation action used ensure that there is no Jacobian in the continuous flow definition.
Necessity:
Rationality: This assumption is a property of many environments and therefore reasonable. For environments that do not satisfy this assumption, we can try to satisfy this assumption by modifying the state add more information. For example, we can add the duration of the action the state space of Pendulum-with Wall task in Figure 5] Even if the action increases after reaching the wall, the position information will be changed, but the duration will increase, so that the state transition and the action will correspond one-to-one. The worst case is that cannot change the environment this time, we mainly need to solve the problem that the output of retrieval neural network G cannot be multiple when the input is fixed. One of our conjectures is that maybe we can alleviate this problem by adding some small random noise to the input, but this idea has not been tested.
to satisfy the assumption.
B PROOFS
B. PROOF THEOREM 1
Theorem 1. (Continuous Flow Matching Condition). Consider non-negative function F(s,a)
taking state S and action A inputs. Then we have F corresponds to flow if and only if the following continuous flow matching conditions are satisfidd:
[ )EEP(E)) F(s + s'dds [ ( F(s,a:s s')ds
AS so. F(s')
(22)
J [ F(8' s'' = -
AA < 8f. F(s')=
F(s', a)da.
Furthermore, F. uniquely defines a Markovian flow F matching F such that
11 n+1 F + st) I1 F(s))
F(r)
(23)
Proof. The proof is an extension of that of Proposition 19 in Bengio et al. (0021b) to the continuous case. We first prove the necessity part of the proof. Given flow network, for non-initial and non- final nodes on trajectory, the set of complete trajectories passing through state s' is the union of the sets of trajectories going through s s' for all E P(s'), and also is the union of the sets of trajectories going through s" for all 8" E C(s'),
:ETTSSEEe-
U : sEP(8)
U (TET: -5" ET) s"EC(e)
Then we have
F(r)dr= J J F(r)drds
F(5') = Jnster
[SSP(N))
F(s + s'dds,
Trs-sster
14

---

Published as a conference paper at ICLR 2023
and
F(6) = STadee F(r)dr= Jsrecton I
F(T)drdd" = -Srecto) F(s' + 8")d6".
Then we finish the necessity part. Next we show sufficiency. Let 2 = F(so) the partition function and Pp be the forward probability function, then there exists a unique Markovian flow F with forward transition probability function PF = PF and partition function Z according to Proposition 18 in Bengio al. (2021b), and such that
1- F(s + st) 1==1
F(r) 2 II P=(s)(s(11)=
(24)
IT== ((8))
where 8n+1 = 8y- In addition, according to Lemma 1 we have
J Pn(T)dT=
{
II
Pn((s(oe41)d7==
TET,, t
Lemma 1. Considering a continuous task A), where we have the transition probabilities defined in ( and (9). Define Ta. and To,s the set of trajectories sampled from a continuous task starting and ending in 8fi and starting in 8o and ending s, respectively. Then we have
E E s48th
JEETH.
Pf(r)dr=1
(25)
Vs E s(fool
TETT.
Pb(F)dT=1.
(26)
Thus, we have for + 80:
F(s') 2
JTETOW (o--te--))r PF($
II
7 F(5)
II
Pa((o(oe+1)d7= F(8)).
(27)
F(so) JETo,, (s1+)
Combine (27 with Pr Pr yields Vs - E A, F(s F(s++8)= F(s -(s-8)).
8¹). Finally, according to Proposition 16 in Bengio et al. (2021b), for any Markovian flow F' matching F on states and edges, have Fx (T) F(T), which shows the uniqueness property. Then complete the proof.
B.2 PROOF THEOREM 2
LK Theorem 2. Let {ax ]k=1 be sampled independently and uniformly from the continuous action space A. Assume Gg. can optimally output the actual state St with (St+1, Gt). For any bounded contimuous action E A and any state S. have
((AA K
K
F(st,as) - 1/4
-f F(st, 2t) 2eexp (-
K12
P
Σ
(28)
((Lau(A)
k=1
and
((A) K
(G1g-(s1, (s1,
(a:T(ssa)=m) F(s.a)da 21
K
Kt²
2exp
(29)
([Lu(A) + diam(S)))
where L is the Lipschitz constant, diam(AA denotes the diameter of the action space A and diam(S) denotes the diameter of the state space S.
15

---

Published as a conference paper at ICLR 2023
Proof. First, show that the expectation of sample outflow is the true outflow and the expectation of sample inflow is the true inflow Lemma 2
) Lemma 2. Let {ax k=1 be sampled independently and uniformly from the continuous action space A. Assume Gø. can optimally output the actual state &t with (441,a)). Then for any state 8t E S, have
M ((A)
F(s,, ax))
= =Sec F(st,
E
(30)
and
((A))
K
F(G or (s,, (8))
= (s:((s))=ss
F(s, a)da,
(31)
K
where = g(st,
Then, define the following terms:
p((A) K
1/ / a(s
1
K / [F(s ax) F(s a)] da
1
///oc
Ia
F(sc. ax)
(32)
K
and
As m(A)
F(G as)) 1
J K JacT(s,a)=s F(s,a)da
(33)
K
1
[ (F(Go- ak), F(s, a)] da,
(34)
K a=Ta))))=
where 8 = g(st,a).
Note that the variables {T are independent and E[Tx] 0, 1...., ..... K according to
Lemma 2
So the following equations hold
(((A) K
/ F(s,,a)da 2t PP
K 2
P
F(s
zt
(35)
and
((A)
K
K
(s), Ja:Taaaa=a F(s,a)da 21== =P
A+
21) (36)
K
Since (s) is Lipsehitz function, we have
/ [F(se, F(ss.
Tel S JUEA
L { |lax 11ax-allde allda S Lu(A)diam(A)
(37)
K
uEd
Together with Assumption that is, for any pair of (s, satisfying T(s, a) 8t, unique if fix 8, we have
|Ax S Jo:((aa)=s
F(G (st, ax). ax) F(s,
K I Ja.(..))
F (s:, ax) F(s, ax) + F(s, - F(s, a)
G|| (st, ax) 811 + Lllas - alIda
K Ja:T(s,0)=s:
(d)aam))) + diam(S))
(38)
K
16

---

Published as a conference paper at ICLR 2023
Lemma (Hoeffding's inequality, Vershynin (2018)) Let 21....,.. HK be independent random vari- ables. Assume the variables {a [ are bounded in the interval [T,Tr), Then for any t > 0.we have
ExE) 2t s exp (- K(T, -
21
(39)
Incorporating T, L (())diam(A) and Ti k ((A)ddiam(A) in Lemma with (37p. and
k
incorporating T, (AA) (diam((A) + diam(S)) and Ti - {(A)(diiam A) + diam(S)) in
Lemma with (38), we complete the proof.
B.3 PROOF OF LEMMA I
Lemma 1. Considering continuous task A), where we have the transition probabilities defined in (8 and (9). Define Ta.5 and To.s as the set of trajectories sampled from a continuous task starting and ending sf; and starting so and ending s, respectively. Then we have
DEET., Pr(T)dr=1
As E
(40)
[re- Pb(r)dT=11 11.
vs E s1100)
TETO.
(41)
Proof. We show by strong induction that (40) holds, mainly following the proof of Lemma 5 in Bengio et al. (2021b), and then extending to 41 is trivial. Define d as the maximum trajectory
length Ta,f, + 8f, we have:
Base cases: If d = 1, then
JEETH Pr(r)dr= Pp(s - -88 =
holds by noting To, {(s - $t)}.
Induction steps: Consider > 1, by noting 12 we have
J Pp(T)dr
/secis I TET
Pp(r)drds'
(42)
J J JTETAN
PF(s PF(s(ss)P((r)drds
(43)
-f (sECtS) Pp(s'ss)ds'
Pf(T)dr =
(44)
TET
where the last equality follows by the induction hypotheses.
B.4 PROOF OF LEMMA 2
Lemma 2. Let {a be sampled independently and uniformly from the continuous action space
A. Assume Gø can optimally output the actual state 8t with (8141,0t). Then for any state 8; S, we have
a(A)
F(se, ann)
E
F(se, a)da
(45)
JaEA
and
((A)
K
E
F(G (8t, (x);
JaiT(s,))ms F(s,a)ds,
(46)
K
where a).
17

---

Published as a conference paper at ICLR 2023
Proof. Since {az [k is sampled independently and uniformly from the continuous action space A, then we have
E
F(s,, a)da.
(47)
1e(A)
JUEA
Therefore, we obtain
M(A) K
K
((A)
K
E
F(st: ax)
E
[( az)l
(48)
K
k=1
F(sc, a)da.
(49)
aEA
Since Assumption 3 holdd, for any pair of (s, a) satisfying T(s, 3t, is unique if we fix have
E [F(GGg (8t, (s),aa),, ax)) m(A) /
1
a=T(ss))=
where s = g(st, a).
Therefore, we get
M(A) K
((A) K
E
F(G (st,
E
ax)]
K
K
k=
k=
F(8,
Then we complete the proof.
C
PSEUDOCODE OF CFLOWNETS
For clarity, we show pseudocode for CFlowNers Algorithm
1
Algorithm Generative Continuous Flow Networks (CFlowNets) Algorithm
Initialize: Flow network 0; pretrained retrieval network Gø; and empty buffer D and P
1: repeat
2:
=
3: while s # terminal and t<T do
4:
Uniformly sample M actions M from action space A Compute edge flow Fa(st, a;) for each a, E J to generate P Sample d, ~ and execute a, in the environment obtain Tt+1 and St+1
5:
6:
7:
t=t+1
8:
end while
9:
Store episodes {(8,, (t,, I'z,, 8t4, in replay buffer D [Optional] Fine-tuning retrieval network Go based on D
11:
Sample random minibatch B of episodes from D
12:
Uniformly sample actions {ax Jo from action space A for each state in
13:
Compute parent states according to (Ga(s, ax) [ for each state in
14:
Inflows:
15:
logie + Σx= exp Flos (Go(se, a4), 44)
16:
Outflows reward:
17:
logle + > XR(ss) + Ek= exp Flog St, ax)]
18:
Update flow network Fo according to 20
19:
until convergence
18

---

Published as a conference paper at ICLR 2023
10
10
10
10
10
10
(a)
(b)
(c)
(d)
Figure 7: Visualization of Soint-Robot-Sye Sparse task.
(a)
(b)
(d)
Figure 8: Visualization of Reacher-Goal SparsS task.
(d)
Figure 9: Visualization of Swimmer-Sparse task.
D ADDITIONAL EXPERIMENTS
D. VISUALIZATION OF ENVIRONMENT
As shown in Figures 7 and we provide the visualization of Point-Robot-Sparse. Reacher-Goal- Sparse. and Swimmer-S Sparse tasks. In Point-Robot-Sparse,, the goal of the agent to navigate two different goals. The agent starts at the starting coordinate (0, 0) and moves towards the target coordinate one step at time. The environment has two target coordinates (5, 10) and (10.5) with maximum episode length of 12, and the environment returns reward only when the last step is reached. Rewards are issued by measuring the distance between the agent's current position and the target node, and the closer the distance, the greater the reward. Each time the agent take step from any angle to the upper right.
Both Reacher--GallSpaasee and Swimmer-Sparse are adapted from OpenAI Gym's 's MuJoCo envi- ronment. the Reacher- Reacher-Goal- Goal- Spaase, "Reacher" a two-jointed robotic arm. The goal of the agent is to reach randomly generated target by moving the robots end effector. Figure 8 shows the movement process of the robotic arm. By adjusting the torque applied at the hinge joint, the end effector gradually approach the target. In the Swimmer-Spars, the "swimmer" suspended in
19

---

Published as a conference paper at ICLR 2023
two-dimensional pool, and the goal is to move fast possible towards the right left. Figure shows the shape change process of the robot during motion. By taking the action that applies torque on the rotors and using the fluids friction, the robot swim faster. We set the maximum number of steps to 50 for these two environments. For Reacher- Reacher-Grall-Spasse when the last step is reached, the environment returns reward that measures how far the agent is from the randomly generated larget. The closer the agent is to the target, the greater the reward. For Swimmer-Spane the farther the left or right from the starting point, the greater the reward returned.
D.2 ADDITIONAL ANALYSIS
Figure shows that average reward and reward distribution of different algorithms on the Point- Sapott-OnnGoal-ppaaee task, where agent needs to navigate to specific location. Figure TO (a) indicates that CFLowNets can obtain the highest average return compared to other RL-based algorithms. In Figure I0 (b), all algorithms goal setting, while CelowNets can achieve better. Note that RL algorithms can also learn the reward distribution in this task, since maximizing the reward the optimal policy in the case of single objective, and the policy is not difficult to learn.
able to fit the reward distribution well under the one
In Figure 11 provide the action reward distribution of different algorithms with 2e4 total timesteps on Point-Robot-Sparee with Point (4,8), Point (8,4) and Point (7.7), respectively, Note that unlike Figure where the total number of timesteps is le5, here show the result with 2e4 total timesteps since we found DDPG is overfit after le5 timesteps in this task. Therefore we show the results without overfitting for fairer comparison. We can see that no matter which point, the policy of CFlowNets can better match the real reward distribution. For example, at points (4,8) and (8,4), CFlowNet tends to choose actions that guide the agent towards (5, 10) and (10, 5), respec- tively. For a location between two goals (point (7,7)), there are two directions that allow the agent to reach goals with high rewards. contrast, the policy learned by RL algorithms can only occasion-
1.2
Estimated Reward Value
1.0
DDPG
SAC
CFlowNots
True Reward
Average Reward
0.9
TD3
0.8
TD3
DDPG
SAC
CFiwwNels
0.6
0.4
0.3
0.2
0.0
0.0
4
12
16
20
-1.0
-0.5
0.0
0.5
1.0
Total Timesteps (x (x10))
Action
(a) Point-Robot-OOneGal--Sppip
(b) Point-Robot Soabott-OneGoal-Spaass
Figure 10: The average reward and reward distributions of CFlowNets, DDPG, TD3, SAC and PPO on Soint-Rooot-Oncee Sparse task.
Esliatedd Reward Value
1.0
Estimtted Roward Value
1.0
Estimated Reward Value
1.0
0.8
0.8
0.8
0.6
0.6
0.6-
0.4
Trao Nowar
0.4
Poward
0.4
1H0 Romar
DDPG
0.2
0.2
0.2
0.0
0.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
Action
Action
Action
(a) Point (4,8)
(b) Point (8,4)
(c) Point (7,7)
Figure 11: The reward distributions of different points Point- Point-Robot-Sparse task.
20

---

Published as a conference paper at ICLR 2023
(a) DDPG
(b) TD3
(c) SAC
(d) PPO
(c) CFlowNeets
Figure 12: Sampled trajectories on Www.RoootOOnGGGa------ task.
(a) DDPG
(b) TD3
(c) SAC
(d) PPO
(e) CFlowNets
Figure 13: Sampled trajectories on Soint-Robot-Spee task.
ally match the true reward distribution of certain point, and cannot stably match every point. This also shows that the policies learned by RL algorithms is relatively simple. CFlowNets learn more diverse policies for agents to reach different goals with high rewards, while other methods usually find one goal instead of all potentially high reward locations.
Figure 12 and Figure T3 show the results trajec- tories visualization produced by different algorithms. In the Point-Robot-OneGaal-Spee Sparsee task, the trajecto- ries of DDPG, TD3, and PPO are single, while SAC can select actions from the policy probability distribu- tion, different trajectories can be obtained. In con- trast, CFlowNets found more diverse trajectories and also found the highest reward goal (thickened red tra- jectory), which means that CFlowNets can better ex- plore the region near the goal. In the Point-Robot Sparse task, the RL-based algorithms seek only one goal. However, CFlowNets can find all goals.
0.9
0.8
2
0
2
6
8
10
Figure 14: Reward distributions on Point-
Robot-Sparse Task.
It is worth noting that in Figure 13 (e), the density of CelowNets sampling trajectories not as dense in Figure T2 (e) near the maximum reward. Rather, it is denser the diagonal. This is because in most positions, the action probability of choosing to go up and to the right is relatively high, so casier to go the diagonal direction in combination. In addition, the reward on the line between two goals is not small. When sampling according to the output of the flow model as a probability, many trajectories themselves more likely to reach the diagonal. Figure 14 shows the true reward distribution of Point-Ro Natt--oooottttt Sparse, where the reward is higher in the area near two goals and the line between two goals.
D.3 EXPERIMENT RESULTS ON PEEHWAYYPARKIGG-SPAAAE
which is an ego-vehicle control task. As shown Figure 1 the goal is to make the ego-vehicle park a given space with the appropriate orientation by adjusting its controller. The dimension of the vehicle observation 18, consisting of the distance between the vehicle and parking, the vehicle speed, the triangular heading information, the goal the agent should attempt to achieve, and the goal that it currently
We evaluate the performance of CFLowNets on Highway-Parking-SSae
21

---

Published as a conference paper at ICLR 2023
Figure 15: Visualization of Highway-Parking Paaaiaa--aaninnnaaaaraaaee task.
0.0
# of Distinctive Trajectories
1000
Average Reward
-0.3
TD3
PPO
800
TD3
PPO
600
CFlowNets
-0.6
400
-0.9
200
-1.2
20
40
60
80
100
0
2000 4000
6000
8000
10000
Total Timesteps (x10²)
# of Total Sampled Trajectories
(a) Average Reward
(b) Number Distinctive Trajectories
Figure 16:
The average reward and number of valid-distinetive trajectories generated under 10000 explorations CFlowNets, DDPG, TD3, and SAC on Highway--Prking-Spaae -Saars..
achieves. The action space includes control over the throttle and steering angle, and the reward function is set as the distance between the ego-vehicle and parking. Figure 16 shows the average reward and the number of didiiddittinttve trajectories explored as training progresses of different algorithms, which illustrates that the performance of CFlowNets is more promising than other RL- based algorithms. Even for higher-dimensional continuous tasks, CFlowNets have very competitive reward results (outperforming DDPG, TD3, and SAC), while achieving much better exploration performance than RL-based algorithms.
D.4 BASELINES
We compare our proposed CFlowNets the following baselines:
Deep Deterministic Policy Gradient (DDPG) Lillicrap et al. 2015).
https:
com/ http/////m//////////aaaaaee/ee//
py
Twin Delayed Deep Deterministic Policy Gradient (TD3)
FFujimoto et al.
2018
https:
//git//ub.ccm//ffjjjim//
Soft Actor-Critic (SAC) (Haarnoja et 2018b
https://gittie com/
denisyaracssppyoorch.co
Proximal Policy Optimizationn (PPO) (Schulman et al. 2017 attps://github.com
DLR-RM/ hute:s//hcccbbleaassessses//ssar
stabbl https//eesstaabbeeaaaaaeeeeeee/s baselines3/ppo/
pps.py
D.5
HYPER- HYPER-PARAMETER
We provide the hyper- parem--prrmettrs of all compared methods under different environments in Table ] Table 2 Table 3 Table 4 and Table
As for "Total Timesteps", "Start Traning Timestep". "Max Episode Length", "Actor Network Hid- den Layers", "Critic Network Hidden Layers", "Optimizer", "Learning Rate", and "Discount Fac-
22

---

Published as a conference paper at ICLR 2023
tor". we set them the same for all algorithms for fair comparison. As for these specific parameters for baseline algorithms, we remain them the same as those in the original code to achieve good per- formance. As for these specific parameters of our CFlowNets, to 100 and the action probability buffer size to 1000 to tradeoff the performance and computational load. Note that CFlow dose require as large replay buffer size other RI. algorithms, since the exploration ability of CelowNets is better than that of others. And good policy can al- ready be learned from a small replay buffer. This is also an advantage CFlowNets compared to RL based algorithms.
set the number of sample flows
Table 1: parpeeersr of CFlowNets under different environments.
Soott--soot
GealSSae
Swimmer-Sparse
Total Timesteps
100,000
100,000
100,000
Start Traning Timestep
4,000
7,500
7,500
Max Episode Length
12
50
50
Flow Network Hidden Layers
[256,256]
[256,256]
[256,256]
Retrieval Network Hidden Layers
[256,256,256]
[256,256,256]
[256,256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0003
0.0003
0.0003
Batchsize
128
128
128
Number of Sample Flows
100
100
100
Action Probability Buffer Size
1,000
1,0000
10,000
Replay Buffer Size
8,000
2,000
2,000
1.0
1.0
1.0
Table 2: Hyper-parametee of DDPG under different environments.
Sonnt-Rooot-pparss
Reacher- Goal- Sooal-Sparse
Swimmer- Sparre
Total Timesteps
100,000
100,000
100,000
Start Traning Timestep
4,000
7,500
7,500
Max Episode Length
12
50
50
Actor Network Hidden Layers
[256,256]
[256,256]
[256,256]
Critic Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0003
0.0003
0.0003
Batchsize
256
256
256
Discount Factor
0.99
0.99
0.99
Replay Buffer Size
100,000
100,000
100,000
Target Network Update Rate
0.005
0.005
0.005
Table 3: Hyper-parameter of TD3 under different environments.
Point Spott-
Reacher-Gool--paree
Swimmer-Sparse
Total Timesteps
100,000
100,000
100,000
Start Traning Timestep
4,000
7,500
7,500
Max Episode Length
12
50
50
Actor Network Hidden Layers
[256,256]
[256,256]
[256,256]
Critic Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0003
0.0003
0.0003
Batchsize
128
128
128
Discount Factor
0.99
0.99
0.99
Replay Buffer Size
100,000
100,000
100,000
Gaussian Exploration Noise
0.1
0.1
0.1
Target Network Update Rate
0.005
0.005
0.005
23

---

Published as a conference paper at ICLR 2023
Table 4:
of SAC under different environments.
Point-Roe Soant-RoottSparre
Swimmer-Sparse
Total Timesteps
100,000
100,000
100,000
Start Traning Timestep
4,000
7,500
7,500
Max Episode Length
12
50
50
Actor Network Hidden Layers
[256,256]
[256,256]
[256,256]
Critic Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0003
0.0003
0.0003
Batchsize
1024
1024
1024
Discount Factor
0.99
0.99
0.99
Replay Buffer Size
100,000
100,000
100,000
Target Update Interval
Table Hyper-paramett of PPO under different environments.
Point-Robot-Syee
Reacher-Goal Saachhrr-oal-Spaarrs
Swimmer-Sparee
Total Timesteps
100,000
100,000
100,000
Max Episode Length
12
50
50
Policy Network Hidden Layers
[64.64]
[64.64]
[64,64]
Value Network Hidden Layers
[64,64]
[64,64]
[64,64]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0003
0.0003
0.0003
Batchsize
64
64
64
Discount Factor
0.99
0.99
0.99
GAE Parameter
0.95
0.95
0.95
Timesteps per Update
2048
2048
2048
Number of Epochs
10
10
10
Clipping Parameter
0.2
0.2
0.2
Value Loss Coefficient
0.5
0.5
0.5
24


Reviews:
[{"review_id": "vLAzJtg_1lf", "reviewer": "Reviewer_Rdez", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A variants of GFlowNets with continuous state and action spaces is proposed, based the flow-matching conditions and importance sampling to approximate the sums over children and parents of each state. Low-dimenional control experiments are performed showing that a greater diversity of trajectories are obtained compared to a number of standard RL baselines, as would have been expected.", "summary_of_the_review": "If the authors fix the false claims about previous work on GFlowNets with continuous valued states and actions (which is localized in a few sentences and should be easy to do without changing the main messages of the paper), the paper should be published. \n\n** update: ** given the changes promised by the authors, I have increased my recommendation from 6 to 8.\n\nThe experiments on control tasks are interesting, and a first for continuous GFlowNets, so quite interesting as experimental proof that GFlowNets can be applied in continuous domains.\n\n", "strength_and_weaknesses": "The main strength is that this is the first published paper exploring a continuous state-and-action with GFlowNets. A fixable weakness is that they claim that GFlowNets as previously published cannot handle continuous states and actions, which is not true (they were proposed in the Bengio et al 2021b paper, section 6). The authors also seem to have misunderstood the proposal in that paper. Nonetheless, Bengio et al 2021b was a purely theoretical paper and did not report experiments with continuous GFlowNets, so this is an important validation. In addition, the proposed approach to handle continuous variables is different from the one proposed by Bengio et al 2021b.\n\nThe mathematical derivations in page 3 and Theorem 1 are not really novel: they are identical or continuous versions of the equations in the GFlowNet papers, but replacing sums by integrals. This should be clarified to give a false impression of novelty. The real novelty is in how the authors propose to approximate the integrals and that is worth explaining in more detail. One concern I have is that importance sampling (used to approximate the integrals) is notoriously problematic in high dimension, due to rapidly increase variance of the estimator. Experiments where different numbers of dimensions are compared could be useful in this regard.\n\nThe authors present a series of experiments on control tasks in a low-dimensional state-space and a terminal reward, where the proposed formalism can be directly applied. This is a first and very interesting, along with the results showing greater diversity with GFlowNets (like earlier GFlowNet papers, but now in a control task with continuous variables), and even better rewards in two of the three tasks.\n\nDetailed comments:\n\n* In several places, e.g. par. 2 of sec. 1, 1st two lines of page 2, 1st par. of sec 2.2, the authors make false claims about GFlowNets for continuous states and actions. Bengio et al 2021b state very clearly that all the math shown in earlier sections can be applied to continous variables by replacing sums by integrals, and they propose using integrable densities and the detailed balance criterion to obtain a tractable training objective (with no need to approximate integrals by sums).\n\n* The statement linking DAGs and MDPs (first sentence of 2.2) seems wrong: in general, the seequences of states associated with an MDP do not form a DAG (we can visit a state several times, i.e., form a cycle, in an MDP).\n\n* The term \"particle\" is used on page 3 but not defined (I understand it comes from the GFlowNet paper but the readers of this paper may not have read them).\n\n* The statement made on page 7 (end of 1st par. of sec. 5) about the proposal regarding continuous GFlowNets in Bengio et al 2021b is incorrect. That proposal does NOT require decomposing the continuous state into bins or discretizing it with continuous residuals. The math that is described can be applied with a purely continuous state. Hence there is no issue of exponential explosion as incorrectly claimed. I copy here some relevant sentences from that paper:\n\"However, for the most part one can replace these sums by integrals in case the states or actions are either continuous or hybrid (with some discrete components and some continuous components).\"  \n\"The challenge is to represent continuous densities on the output, with the need to both being able to compute the density of a particular value (say P (sxt+1 | sit+1, st)) and to be able to sample from it. Computing categorical probabilities and sampling from a conditional categorical is standard fare, so we only discuss the continuous conditional. One possibility is to parametrize sxt+1 | sit+1,st with a density for which the normalization constant is a known tractable integral, like the Gaussian.\"\n\"Other approaches include modeling the conditional density with an autogressive or normalizing flow model\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rTiVd6--XMK", "reviewer": "Reviewer_y8C2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a generalization of GFlowNets to environments with continuous action spaces. The continuous version of the flow matching condition, which involves integrals rather than sums, is approximately enforced by using a Monte Carlo estimator of the in- and out-flows. Experiments are performed on three continuous control domains, where the proposed algorithms show strong performance.", "summary_of_the_review": "My main reason for leaning positive for this paper is the novelty of the problem studied in two respects: using GFlowNets for RL tasks to encourage exploration and providing the first experimental validation of continuous-time GFlowNets, albeit with a different approach that what has been suggested -- but not empirically tested -- in past work. ~There are some issues with the mathematical foundations, but I believe they are fixable by introducing some mild assumptions.~ Post-rebuttal update: The response have adequately addressed my questions and concerns. I would like to see this paper published because it contributes several important GFlowNet firsts (continuous action spaces, application to RL problems, learnable inversion of actions).", "strength_and_weaknesses": "Strengths:\n- (+++) This is the first paper to consider GFlowNets with continuous action spaces, which is an important methodological step.\n- (+) It is also the first time that GFlowNets are applied to reinforcement learning environments (discrete or continuous).\n- (+) Clear and (mostly) theoretically sound exposition.\n  - But see my suggestions / comments below.\n- (+) **Update, 10 November:** A strength that I neglected to point out in the initial review is that this paper considers environments where inversion of actions cannot be done analytically -- the \"transaction network\" has to be learned to find the parent (state,action) pairs. This is an interesting idea and the first time such a method is combined with GFlowNets.\n\nWeaknesses / questions:\n- (--) It is hard to interpret and contextualize the experiments, at least for a reader who is closely familiar with GFlowNets but not with these particular RL environments. It would be very helpful to have some figures showing some sampled trajectories in each environment.\n  - Even better would be a demonstration on a toy environment, like a continuous generalization of the 2D hypergrid from [Bengio et al., 2021a].\n- (-) The proposed approach to GFlowNets with continuous action spaces has some weaknesses compared with alternatives. (Ideally, these alternatives would be compared with in the experiments, but at least they should be discussed in the text.)\n  - [Bengio et al., 2021b] writes that the detailed balance (DB) objective, which requires estimates of a state flow and parametric forward and backward policies, can be applied to continuous action spaces: the policy likelihoods in this objective is simply replaced by probability densities. The trajectory balance (TB) objective from [Malkin et al., 2022] can be applied in this setting as well.\n  - DB and TB have an advantage over flow matching in that they do not require evaluating the flow model on multiple parents of a state, and this advantage becomes more prominent in terms of computation cost when the number of parents is large (for discrete spaces). In the setting of this paper, the computation cost grows with the number of samples used for the estimation of inflow, and thus DB or TB could have a strong computational advantage, in addition to the known benefits of TB for faster convergence.\n  - The issue of the variance of the in- and out-flow estimators should receive more attention. (Is the bound involving the Lipschitz constant practically useful?) Also, it should be mentioned that even if the the in-flow and out-flow estimators are unbiased, the estimators of **log**-in- and out-flows, which are used in the loss of equation (20), are not unbiased.\n- (-) On the mathematical formulation: The definition of child set on p.3 uses the trajectory history, which makes it impossible for the Markovian condition on the density of the forward policy to hold (though in many settings it holds almost everywhere).\n  - In addition, this definition looks tailored to the case when the sets of states reachable in any given number of steps are open sets in a fixed \"universe\", which is natural for modeling motion in space, but is not very general. For instance, it does not make sense to talk about new states equalling past states in settings where the dimensionality of the reachable state space grows with the number of steps, such as the obvious continuous generalization of [Zhang et al., 2022b].\n  - I wonder if a better approach to acyclicity would be to remove the constraint in the definition of child set and instead to say something like: \"in our experiment domains, **the set of trajectories in which states repeat has zero measure under the induced trajectory flow** if all policies are absolutely continuous with respect to the standard measure on A, so such trajectories do not affect the analysis\", similar to current Remark 2.\n  - On a related note, I did not find in the text the answer to whether the time step (number of actions from the initial state) was used as an input to the policy model. If the time step is part of the state, then acyclicity is guaranteed. It does not change the math in this case, but could be necessary in RL problems with discrete action spaces, where not appending the time step would result in cycles having nonzero measure.\n  - (A **very** simple way to deal with the cycle issue without having to talk about zero-measure sets: say that the state includes the information about the time step, so cycles are impossible, but the policy ignores the time step part of the state.)\n- (-) Related work could use some improvements:\n  - \"In Malkin et al. (2022), the trajectory balance loss is proposed for GFlowNets to explore the capabilities of previously used objectives\" -- this isn't accurate. In fact, that paper proposed and tested a new objective (TB) and tested another objective that had not been used before (DB).\n  - \"In Bengio et al. (2021b), an idea based on hybrid state is pre\u0002sented to make GFlowNets suitable for continuous tasks\" -- this is also inaccurate (see the comment on DB and TB for continuous spaces above -- the proposal in that paper does not require hybrid states).\n  - The differences between CFlowNets and continuous SAC (pp.7-8) are hard to understand and could be reworded. To my understanding, (1) means that a Gaussian policy in SAC is less expressive than one that uses a general unnormalized action p.d.f. F(s,a), and (2) means that SAC wants each trajectory leading to a state to have the same likelihood (which should be proportional to the reward), while CFlowNets want the sum of likelihoods of trajectories leading to a state to be proportional to its reward.\n\nQuestions and minor comments:\n- How were hyperparameters chosen for CFlowNets and baselines?\n- Equation (3) seems to have a bug. As written, the integral on the right is usually 0, since it is taken over both $s$ and $a$, but restricted to a lower-dimensional subspace of ${\\cal S}\\times{\\cal A}$. I think it should instead be this: $\\int_s\\left(\\sum_{a:T(s,a)=s_t}F(s,a)\\right) ds$.\n  - Should there also be a compactness or similar assumption on $A$, so that any function on it defined by a neural net is integrable (and, less importantly, has finite Lipschitz constant)?\n- Please use curve markers or line styles, and not just colours, to distinguish algorithms in Figure 3.\n- Some of the citations to GFlowNet work are incomplete.\n  - [Bengio et al., 2021a] was published in NeurIPS 2021.\n  - [Zhang et al., 2022b] was published in ICML 2022.\n  - [Deleu et al., 2022] was published in UAI 2022.\n  - [Malkin et al., 2022] will be published in NeurIPS 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B-h_Pw9TS61", "reviewer": "Reviewer_Y6ub", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the work of (Bengio et al., 2021a) on Generative Flow Networks (GFlowNets), which was limited to discrete state and actions spaces, to continuous control tasks. The authors introduce the notion of continuous flow $F$, and they formulate the conditions required by this function $F$ to correspond to a properly defined Markovian flow, via a generalization of the flow-matching condition of (Bengio et al., 2021b) to continuous states and actions. This new framework is called CFlowNets. Similar to GFlowNets, CFlowNets  While this provides theoretical guarantees, this condition involves integrating over the whole action space, which is typically impractical; fortunately, the authors devise a practical approximation of the flow-matching condition that only involves sample actions only to estimate the outflow, as well as a separate neural network, called the transaction network, that is responsible for estimating the inflow. The authors also provide theoretical guarantees on the soundness of their approximation. Finally, CFlowNets have been applied successfully to 3 different continuous control problems, both in terms of estimating the reward distribution accurately, and in terms of raw performance and exploration on sparse reward environments.\n\n---\n\n*Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation, 2021a.*\n\n*Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations, 2021b.*", "summary_of_the_review": "I am very hopeful with this submission, and I believe this could be a very important step toward applying flow networks to continuous control problems. The fact that the authors showed empirically that CFlowNets offer an advantage on exploratory tasks is also an important evidence. I have unfortunately not been able to thoroughly review the proofs in the Appendix, as well as the source code provided, due to the short period of time for the reviews.\n\nDespite my enthusiasm, I strongly believe that in its current state the paper cannot be accepted. The paper is making too many approximations on the theory, and those missing details are crucial. The list I provided is certainly not exhaustive, and most of the theory part of this submission must be reworked with care. That's why I can't say that the paper \"has minor issues that only require small changes\" (in **Correctness**), because this requires major changes, albeit technical.\n\nBecause of this, I am currently recommending borderline reject.", "strength_and_weaknesses": "**Strengths**: The generalization to continuous state and action spaces proposed here is a very important step towards making GFlowNets applicable to a broader setting beyond discrete states. These generalizations are for the most part natural extensions of prior results, replacing summation by integration anywhere applicable, which makes this new formulation very appealing. And while some extensions to hybrid states and actions have been proposed in the past (Bengio et al., 2021b), CFlowNet is the first work to show empirical evidence of the effectiveness of flow networks in continuous settings. The practical implementation of CFlowNets via sample actions and the transaction network is also well thought, and the theoretical guarantees about the approximation error (Theorem 2) is appreciated to ensure that these approximations are valid.\n\n**Weaknesses**: Unfortunately, this paper falls short in big ways theoretically, with many approximations in their formulation of continuous flows. The authors are too often ignoring some technical aspects, as well as making implicit assumptions that are crucially missing.\n\n 1. The very first definition (Definition 1) of the continuous state flow as $F(s) = \\int_{\\tau:s\\in\\tau} F(\\tau)d\\tau$ is ill defined: what guarantees do we have that this integral is properly defined and finite? I guess you would have to assume that $F$ defines a finite measure over the space of complete trajectories $\\tau$. This is only one example, but none of the integrals used throughout the paper have any guarantee to exist.\n 2. Related to 1., the forward transition probability and backward transition probability in equations (5) & (6) are only properly defined if all the terms involved are well defined and finite.\n 3. The formulation of CFlowNet requires a generalization of the notion of parents and children to trajectories over continuous states (as opposed to parents and children in a DAG for (discrete) GFlowNets). However the definition of parents of a state, preserving the acyclicity constraint, is incorrect (and similarly for children). In the paper (p.3), the parents $\\mathcal{P}(s_{t})$ of $s_{t}$ is defined as the set of states $s$ such that (1) $T(s, a\\in A) = s_{t}$ an action $a$ could make a transition from $s$ to $s_{t}$, and (2) $T(s, a\\in A) \\notin \\mathrm{set}(s_{0}, \\ldots, s_{t-1})$ the state cannot be transferred to a preexisting state, otherwise a cyclic would occur. This implicitly assumes that we are taking a specific (partial) trajectory from $s_{0}$ to $s_{t}$, and we don't want to \"wrap around\" that particular trajectory with any action. However, this condition should be satisfied **for any** trajectory from $s_{0}$ to $s_{t}$, not just a single trajectory. The definition of children is also incorrect and should be defined **for any** trajectory from $s_{0}$ to $s_{t}$, and this has very serious implications later in the paper. I am including an **Example** below to explain why this criterion is incorrect.\n 4. A big consequence of 3. is that Remark 2 in the paper is incorrect, and you must handle the acyclicity carefully, even in practice (this is not simply for \"theoretical convenience\" as claimed in this Remark. See the **Example** below (point 2). A similar claim is made in Section 4.4 too (the measure of $\\mathcal{A}$ is constant).\n 5. In Assumption 1, you assume that $a \\mapsto F(s, a)$ is a Lipschitz continuous, with Lipschitz constant $L_{s}$. However since the state space is itself continuous, how can we guarantee that $\\sup_{s\\in\\mathcal{S}}L_{s}$ is finite?\n 6. You implicitly make the assumption that the state $\\mathcal{A}$ has finite measure (using $\\mu(\\mathcal{A})$), and has finite support (using $\\mathcal{diam}(\\mathcal{A})$). Moreover, the action space $\\mathcal{A}$ must be a measured space defined ahead of time, so that $\\mu(\\mathcal{A})$ is a fixed constant, and therefore it is incorrect to claim that $K/\\mu(\\mathcal{A})$ may be considered as a hyperparameter $\\lambda$ (Section 4.4). Otherwise, if $\\lambda$ is truly a free hyperparameter, what prevents you to set it to $1$?\n 7. According to (Bengio et al., 2021b), a GFlowNet defines a distribution proportional to the rewards **over terminating states** (to borrow their naming conventions, i.e. the parent states of the terminal states $s_{f}$). This is not what is claimed in this paper. For example Section 1: \"*In contrast, the training goal of GFlowNets is to approximately sample candidate actions with probability proportional to a given reward function*\", Section 4.1: \"*we can sample an action with probability proportional to the reward*\", Section 5: \"*generating policies that sample objects through discrete action sequences with probabilities proportional to a predefined reward function*\", Section 5: \"*for CFlowNets [...] the probability of an action being sampled is proportional to the corresponding reward*\". There seems to be some confusion between the distribution proportional to the rewards found by GFlowNets/CFlowNets and the policy (which, according to equation (5), is proportional to the continuous flow). While the flow itself depends on the reward, it is misleading to claim that the policy samples actions proportionally to the rewards.\n\n---\n\n**Example**: Consider the 2D point-mass environment, where the agent moves on a square of size 2 around it with a certain angle. The state space is $\\mathcal{S} = \\mathbb{R}^{2}$ is any point on the plane, and the action space $\\mathcal{A} = [0, 2\\pi]$ (the direction in which the agent moves). Given a state $s = [x, y]$, and an action $a = \\theta$, the environment transitions to a new state $T(s, a) = s' = [x', y']$, where $x' = x + \\mathrm{clip}(\\sqrt{2}\\cos \\theta, -1, +1)$ and $y' = y + \\mathrm{clip}(\\sqrt{2}\\sin \\theta, -1, +1)$. We assume that we start at the initial state $s_{0} = [0, 0]$.\n\n - Since the actions are completely reversible (we can apply action $\\theta$ and then $\\theta + \\pi \\mod 2\\pi$ to get back to the same state), the set of parents $\\mathcal{P}(s_{t}) = \\emptyset$ for any state $s_{t}$ is empty (except all the states on the square of size 2 around $s_{0}$, whose only parent would be $s_{0}$), because if $s\\in\\mathcal{P}(s_{t})$ such that $T(s, a) = s_{t}$ and there exists $s'$ such that $T(s', \\theta) = s$ ($s' \\rightarrow s$ along the trajectory), then we have $T(s, \\theta + \\pi \\mod 2\\pi) = s'$, which violates the condition of $s\\in\\mathcal{P}(s_{t})$. This is a very restrictive notion on parents; but let's imagine for the remainder of this example that we allow situations where the set of such states is of measure $0$ instead.\n\n - Consider the state $s_{t} = [2, 0]$. We have many ways to arrive in $s_{t}$ with a trajectory of length 2; for example $[0, 0] \\rightarrow [1, 0] \\rightarrow [2, 0]$, or $[0, 0] \\rightarrow [1, 1] \\rightarrow [2, 0]$, etc... Effectively, this means that the set of parents $\\mathcal{P}(s_{t}) \\subset \\mathrm{set}([1, y] : y \\in [-1, 1])$. In particular, since for obvious acyclicity reasons the set of children of $s_{t}$ must be distinct from the set of its parents, the set of valid actions satisfies $[3\\pi/4, 5\\pi/4] \\not\\subset \\mathrm{set}(a\\in\\mathcal{A} : T(s_{t}, a) \\in \\mathcal{C}(s_{t}))$. In other words, the whole interval $[3\\pi/4, 5\\pi/4]$ are invalid actions from $s_{t}$, and this does not have measure $0$ in $\\mathcal{A}$, and therefore we can't ignore invalid actions contrary to what is claimed in Remark 2. The situation here is even worse, because we only considered trajectories of length 2 so far; in fact, we can find trajectories so that the parents of $s_{t}$ are all the states on the square of size 2 around $s_{t}$, meaning that $\\mathcal{C}(s_{t}) = \\emptyset$.\n\n - Consider the state $s_{3} = [3, 0]$. If we follow the definition of $P(s_{3})$ in the paper, we need to find states $s_{2}$ such that $T(s_{2}, a\\in \\mathcal{A}) = s_{3}$ and $T(s_{2}, a\\in\\mathcal{A}) = \\mathrm{set}(s_{0}, s_{1}, s_{2})$. For similar reasons as above, a parent state will have the form $s_{2} = [2, y]$, with $y \\in [-1, 1]$. If we only consider a specific trajectory $(s_{0}, s_{1}, s_{2})$ (as is again implied by the argument on the set of measure 0 in Remark 2), then we can take the trajectory $[0, 0] \\rightarrow [1, y] \\rightarrow [2, y]$; modulo the first remark above about sets of measure $0$, state $s_{2}$ satisfies the criterion for being a parent of $s_{3}$. However, we have other trajectories $[0, 0] \\rightarrow [1, y'] \\rightarrow [2, y]$, for any $y' \\in [0, y]$ that are also trajectories, and are distinct from the single trajectory we had to consider before to satisfy the criterion for $\\mathcal{P}(s_{3})$, and in those case, we can reach $[1, y']$ from $[2, y]$. This would violate the acyclicity condition, because we can find for example a cycle of the form $[1, y] \\rightarrow [2, y] \\rightarrow [1, 0] \\rightarrow [2, 0] \\rightarrow [1, y]$. Moreover, the set of such cycles is not of measure 0. That's why the condition for parents should be at least over **all possible trajectories** leading to $s_{3}$.\n\nNote that this example probably suggests that some conditions on the MDP are necessary for the framework of CFlowNets to be valid (similar to how the DAG assumption was necessary in GFlowNets).\n\n---\n\n*Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations, 2021b.*", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3PspzcEJsIl", "reviewer": "Reviewer_T9R9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a continuous formulation of GFlowNet, for both the action and state spaces, by converting those spaces into continuous spaces and converting the sums of flow-matching into integrals. This yields a continuous flow matching objective which in practice is estimated by Monte Carlo integration.\nThe authors show an error bound on the accuracy of the MC flow estimation which decreases loglinearly in the number of samples, provided the real flow function is Lipschitz continuous.\nThis method is then benchmarked against three standard continuous control tasks, where it beats standard RL methods on 2 of them, and in all 3 cases appears to cover much more of the state space than any baseline.\n", "summary_of_the_review": "I am mitigated, on one hand this is a cool and necessary extension of the GFlowNet framework towards continuous domains, on the other hand there is little evidence that the method does what is suggested. Perhaps it does fit flows and learn the right thing, but since relevant quantities are not measured empirically it is impossible to know.\n\n~As is I think the paper needs improvements in order to be accepted, improvements that I think could be addressed in a rebuttal time-length.~\n\nUpdate: The updated revision of the paper is an improvement and makes many things clearer, and so I am changing my score from 5 to 8.", "strength_and_weaknesses": "### Strengths\n- The paper proposes a sensible, if expensive, approach to making GFlowNets compatible with continuous state spaces\n- The experimental results are encouraging, and compatible with previous work suggesting that GFNs are capable of covering states spaces much better than RL\n- The paper is generally well written, and although some sections could be improved, the overall message is clear\n\n### Weaknesses\n- GFlowNet is a new framework, and isn't deeply understood, mathematically or empirically. There isn't much, if anything, in the paper which suggests that the agents are doing better _because of GFN_.\n  - It's not demonstrated that the learned flow distributions $F_\\theta$ really model the theoretically proposed flows\n  - It's not demonstrated that the current formulation converges\n  - The proposed bound is an interesting result but little is known of Lipschitz constants of flow functions (and that's without going into how large DNN Lipschitz constants can get), which makes it hard to relate the bound to empirical quantities.\n- Some of the evaluations done in the paper seem partially incorrect (see below)\n\n\n### Specifics\n\nEq (11) is a bit weird, the theorem prompt asks us to consider $\\hat F (s,a)$, yet the equations refer to $\\hat F(s \\to s')$ and $\\hat F(s)$. It would be good to clarify exactly what is what.\n\nThe indicator function in Eq (13) seems unnecessary. $R(s_t)$ is by definition 0 for all $s_t \\neq s_f$, and ${\\cal C}(s_f)$ is the empty set.\n\n> For continuous tasks, it impossible to access all state-action pairs to calculate the continuous inflows and outflows.\n\nIt is impossible to enumerate all the children of a state, but it is possible in some cases to take integrals over well defined things like Gaussian distributions. It would be nice if this were addressed in the paper.\n\n> we sample an action probability buffer based on the forward-propagation of CFlowNets, from which we can sample an action with probability proportional to the reward.\n\nThis isn't quite the correct wording, and this formulation is repeated multiple times in the paper. GFlowNets (and actually CFlowNets as well, presumably) sample _terminal states_ with probability proportional to their reward. They do not sample _actions_ with probability proportional to any reward (unless in the bandit setting).\n\nSec 4.3 is a bit confusing. \n\n>  we should find the parent state first.\n\nparent _states_? There should be many.\n\n> a transaction deep neural network\n\nI'm not sure what a _transaction_ DNN is. Do you mean a transition DNN? Transient?\n\n> [$G$ has] $(s_{t+1}, a_t)$ as the input while $s_t$ as the output, and train this network based on $B$ with the MSE loss.\n\nThis assumes that taking action $a_t$ in $s_t$ is the _only_ way to get to $s_{t+1}$. This is not a totally unreasonable assumption, but it seems relatively easy to break: imagine a single joint at $\\theta\\in[0,180]$ degrees which moves an arm by $a\\in[-180,180]$ degrees, now imagine that there is a wall such that moving the joint past $\\theta>110$ degrees makes the arm block against the wall. Then for the state $s_{t+1}=\\theta=110$, there are many possible parents with the same action, e.g. $(\\theta=90, a=20)$ and $(\\theta=100, a=20)$ are both valid parents with the same $a_t$.\n\nEither way this assumption should be clearly stated.\n\n\n> All of these improved policy gradient methods can be classified as aiming at maximizing reward\n\nYes, although to be fair most modern PG implementations include some form of entropy regularization (inducing entropy on the trajectory distribution). This is in some sense the basis of control-as-inference/MaxEnt methods, by maximizing the entropy some cover of the state space is induced.\n\n> what SAC learns is not the true distribution of strictly proportional return, [..] This is different from being directly proportional to reward.\n\nThe wording here isn't quite correct. In the gaussian approximation, no, SAC doesn't exactly learn to be proportional to return, but in the general case, yes. But that is not what differentiates SAC and GFN: SAC (or control as inference) learns something like $p(\\tau)\\propto G(\\tau)$ while GFN learns $p(x) \\propto G(\\tau)$ when $\\tau=(s_1,...,s_T)$ ends at $x$, or to relate the two, learns $\\sum_{\\tau:s_T=x} p(\\tau) = G(\\tau_{s_T=x})$. GFN considers all possible trajectories that lead to a terminal state, while SAC (and PG algorithms in general) are trajectory and return-centric: they do not \"care\" about the specific terminal state which the agent reaches.\n\n\n> Figure 2: Reward distributions on Point-Robot-Sparse Task\n\nI'm not sure I understand Figure 2. If I understand correctly, in Point-Robot, an agent is contained within a 2D surface and must navigate using continuous actions to reach some goal, here (10,5) or (5,10). \"Each time the agent can choose to take a step from any angle to the upper right.\" This suggests to me that the action is the angle, or more accurately, the agent moves in the direction of the vector (1,1) rotated by $a \\pi/4$ radians. After checking the supplementary material this seems roughly correct, the agent moves by $(\\cos(\\theta), \\sin(\\theta))$ where $\\theta = (a + 1)\\pi / 4$.\n\nWhat Figure 2 seems to show is $F(s,a)$, or $V(s')$ for the RL methods, and $R(s')$ for the \"True Reward\" _if the agent were to terminate_ (since we are in the sparse setting, I'm assuming this means the agents only get terminal rewards). This seems incorrect, and I'm not sure why the y axis is called \"Estimated Reward Value\".\n\nI see two problems here:\n- I see no reason for flow matching (even continuous) to predict $F(s,a) = R(s')$  when $s'$ is not terminal, this is because with FM there is no notion of preference over paths (see Figure 10 of Bengio et al. 2021a). If this plot was obtained by setting `cnt_step` to 11, this should be explicit.\n   - Similarly for the RL baselines, I see no reason for RL agents to prefer going up or right at (7,7) proportionally to $R(s')$, in fact, the solution found by TD3 might be just fine, it wants to go up to get closer to (5, 10).\n   - Again for the RL baselines, it's not clear why $V(s')$ should be equal to $R(s')$ unless we're terminating.\n   - The straight line for DDPG is honestly suspect. Looks like a bug?\n- As far as I can tell in the current code the agent never sees the current timestep. This is a problem since, as I point out, the plot of Figure 2 only makes sense if (7,7,`cnt_step=11`) is the current state, and more generally, any state has multiple paths which lead to it. Depending on the current timestep the flow predictions for a state _must_ be different (think about the (7,7) example, if there is only 1 action left, then the flow F(s,a) is equal to the reward of the next state, but if there are 2 actions left, then there are many more accessible states and rewards, and so F(s,a) will be larger). As suggested by Bengio et al 2021b S3.3.1, augmenting the state with the current timestep automatically induces a DAG and would make much more sense here.\n\nIf this was truly a terminal reward plot, then it would be a bit reassuring, but it's not obvious to me that this implies that the flow function is fit well in the rest of the state space.\n\n\n\nSome typos: theoritical -> theoretical, \"a cyclic will occur\" -> a cycle will occur, \"need to sum\" -> needs to sum, \"the set contains all\" -> the set that contains all, \"that starting in s0 and ending in s\" -> that starts in s0 and ends in s, \"to form a cyclic\" -> to form a cycle, \"we can modified equation 19\" -> we can modify equation 19, \"has good sample-efficient property\" -> has good sample efficiency.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster