Generative Multi-Flow Networks: Centralized, Independent and Conservation
Authors: Yinchuan Li
Haozhi Wang
Shuang Luo
yunfeng shao
Jianye HAO
Venue: Submitted to ICLR 2023 (2023)
Keywords: GFlowNets
Multi-Flow

Abstract: Generative flow networks utilize the flow matching loss to learn a stochastic policy for generating objects from a sequence of actions, such that the probability of generating a pattern can be proportional to the corresponding given reward. However, existing works can only handle single flow model tasks and cannot directly generalize to multi-agent flow networks due to limitations such as flow estimation complexity and independent sampling. In this paper, we propose the framework of generative multi-flow networks (GMFlowNets) that can be applied to multiple agents to generate objects collaboratively through a series of joint actions. Then, the centralized flow network algorithm is proposed for centralized training GMFlowNets, while the independent flow network algorithm is proposed to achieve decentralized execution of GMFlowNets. Based on the independent global conservation condition, the flow conservation network algorithm is then proposed to realize centralized training with decentralized execution paradigm. Theoretical analysis proves that using the multi-flow matching loss function can train a unique Markovian flow, and the flow conservation network can ensure independent policies can generate samples with probability proportional to the reward function. Experimental results demonstrate the performance superiority of the proposed algorithms compared to reinforcement learning and MCMC-based methods.

Under review as a conference paper at ICLR 2023
GENERATIVE MULTI-FLOW NETWORKS: IZED, INDEPENDENT AND CONSERVATION
CENTRAL-
Anonymous authors Paper under double-blind review
ABSTRACT
Generative flow networks utilize the flow matching loss learn stochastic pol- icy for generating objects from sequence of actions, such that the probability of generating a pattern can be proportional to the corresponding given reward. How- ever, existing works can only handle single flow model tasks and cannot directly generalize to multi-agent flow networks due to limitations such as flow estimation complexity and independent sampling. In this paper, we propose the framework of generative multi-flor flow networks (GMFlowNets) that can be applied to multiple agents generate objects collaboratively through a series of joint actions. Then, the centralized flow network algorithm is proposed for centralized training GM- FlowNets, while the independent flow network algorithm is proposed to achieve decentralized execution of GMFlowNets. Based on the independent global con- servation condition, the flow conservation network algorithm is then proposed to realize centralized training with decentralized execution paradigm. Theoretical analysis proves that using the multi-flow matching loss function can train unique Markovian flow, and the flow conservation network can ensure independent poli- cies can generate samples with probability proportional to the reward function. Experimental results demonstrate the performance superiority of the proposed al- gorithms compared to reinforcement learning and MCMC-based methods.
INTRODUCTION
Generative flow networks (GFlowNets) Bengio et al. (2021b) can sample diverse set of candidates an active learning setting, where the training objective is to approximate sample them proportion- ally to a given reward function. Compared to reinforcement learning (RL), where the learned policy more inclined to sample action sequences with higher rewards, GFlowNets can perform better on exploration tasks. Since the goal of GFlowNets is not to generate single highest-reward action sequence, but to sample a sequence actions from the leading modes of the reward function Bengio let al. (2021a). Unfortunately, currently GFlowNets cannot support multi- agget systems.
multi-agent system a set of autonomous, interacting entities that share a typical environment, perceive through sensors and act in conjunction with actuators Busoniu et al.) (2008). Multi-agent reinforcement learning (MARL), especially cooperative MARL., are widely used in robotics teams, distributed control, resource management, data mining, etc Zhang et al. (202 I Canese al. (202 ));
Feriani & Hossain (2021) Two major challenges for cooperative MARL are scalability and partial observability Yang et 2019); Spaan 2012). Since the joint state-action space grows exponen- tially with the number of agents, coupled with the environment's 's partial observability and commu- nication constraints, each agent needs to make individual decisions based on local action observa- tion history with guaranteed performance Sunehag et al. (2017): : Wang et al. (2020); Rashid et al. 2018). In MARL,, to address these challenges, a popular centralized training with decentralized ex- ecution (CTDE) Olichock et al. (2008): Olichock & Amato (2016) paradigm is proposed, in which the agent's policy is trained in decentralized manner based only local history. However, extending these techniques to GFlowNets is not straightforward, especially in constructing CrDE-architecture flow networks and finding IGM conditions for flow networks worth investigating.
centralized manner by accessing global information and executed
In this paper, we propose Generative Multi-Flow Networks (GMFlowNets) framework for cooper- ative decision-making tasks, which can generate more diverse patterns through sequential joint

---

Under review as a conference paper at ICLR 2023
tions with probabilities proportional to the reward function. Unlike vanilla GFlowNets, our method analyzes the interaction of multiple agent actions and shows how to sample actions from multi-flow functions. We propose the Centralized Flow Networks (CFN), Independent Flow Networks (IFN) and Flow Conservation Networks (FCN) algorithms based on the flow matching condition to solve GMFlowNets. CFN regards multi-agent dynamics of combinatorial complexity and the demand for independent execution, while IFN suffers from the flow non-stationary problem. In contrast, FCN takes full advantage CFN and IFN, which is trained based on the independent global conservation (IGC) condition. Since FCN has the CTDE paradigm, can reduce the complexity of flow estimation and support decentralized execution, which is beneficial to solving practical cooperative decision-making problems.
whole for policy optimization, regardless
1) We are the first to propose the concept of generative multi-flow networks for cooperative decision-making tasks; 2) We propose three algorithms, CFN, IFN, and FCN, for training GessooNeets which are respectively based on centralized training, independent execution, and CTDE paradigm; 3) We propose the IGC condition and then prove that the joint state-action flow function can be decomposed into the product form of multiple independent flows, and that a unique Markovian flow be trained based the flow matching condition; 4) We conduct experiments based on cooperative control tasks to demonstrate that the proposed algorithms can outperform current cooperative MARL algorithms, especially in terms of exploration capabilities.
Main Contributions:
2 GMFLOWNETS: PROBLEM FORMULATION
2.1 PRELIMINARY
Let F : T + R* be trajectory flow Bengio et al. 2021b)) such that F(T) can be interpreted as
the probability mass associated with trajectory т. Then, we have the corresponding defined edge flow F (s + 8') Ex-wer F(T) and state flow (s) Exe- F(r). The forward transition probabilities PF for each step of a trajectory can then be defined as Bengio et al. (2021b)
Pp (s (5) F (( (85)
F(s)
GFlowNets aims to train a neural network to upproximate the trajectory flow function with the output proportional to the reward function based on the flow matching condition (Bnggio et al. (20215):
F(
Σ Fs++
s'EParent(s)
('EChildss)
where Parent(s) and Child(s) denote the parent set and child set of state s, respectively. In this way, for any consistent flow F with the terminating flow as the reward, i.e., F (s > 8y) = R(s) with sy being the final state and s being the terminating state (can be transferred directly to the final state), a policy π defined by the forward transition probability satisfies π (s' [s) = Pr (s' (8) x R(x).
2.2 PROBLEM FORMULATION
A multi-agent directed graph is defined as tuple (S, A) with S being set of state and A A x x... x A* denoting the set of joint edges (also called actions or transitions). which consists of all possible combinations of the actions available each agent. A trajectory in such graph is defined as a sequence (si...., sn)) of elements of S. A corresponding multi-agent directed acyclic graph (MADAG) defined as multi-agent directed graph with unequal pairs of states in the trajectory. Given an initial state and final state sy. we name a trajectory = (Su...., ..... E T starting from 80 and ending 8 as the complete trajectory, where T denotes the set of complete trajectories.
We consider a partially observable scenario, where the state E S is shared by all agents, but it is not necessarily fully observed. Hence, each agent E selects an action a² E AA based only on local observations o¹ made of shared state S. In this way, define the individual edge/action flow F(oj. af) = F(o + o1+1) as the flow through an edge of + oft1 and the joint edge/action flow is defined by F(st, a.) = F(s, + 1141) with a, = [a..... afIT. The state flow F(s) S + R is defined as F(s) ΣTET 1.er Frr). Based on the flow matching condition CanditonnBengioo et al. (2021b).
2

---

Under review as a conference paper at ICLR 2023
have the state flow equal to the inflows or outflows, i.e..
F(s) = Ev a( F(s',a'==
E
F -s)
(1)
d'PParent(e)
F(8) Eaes a(s,a)=
E
F ((s>>")) 8")
s"EChil((s)
where T(s', ((ss,aa)= = denotes an action a' that can transfer state s' to attain s. To this end, generative multi-flow networks (GMFlowNets) are defined as learning machines that can approximate trajec- tory flow functions in MADAG, with outputs proportional to the predefined reward function, trained based on flow matching conditions in equation] and equation]
GMFLOWNETS: ALGORITHMS
3.1 CENTRALIZED FLOW NETWORK
Given such a MADAG, to train GMFlowNet, straightforward approach is to use centralized training approach to estimate joint-flon fiott-fows, named Centralized Flow Network (CFN) algorithm, where multiple flows trained together based the flow matching conditions. In particular, for any state 8 in the trajectory, we require that the inflows equal the outflows. In addition, the boundary condition given by the flow passing through the terminating state based on the reward R(s). Assuming have a sparse reward setting, i.e., the internal states satisfy R(s) = 0 while the final state satisfies A then we have the flow consistency equation:
Σ (ss.a)=((s)))
E as', d'EA(s))
(3)
=5
Lemma Define a joint policy that generates trajectories starting in state so by sampling actions A(s) according to
x {aass) = F(s, F(s))
where F(s, a. > O is the flow through allowed edge (s, a), which satisfies the flow consistency
equation in equation 3 Let (s)) be the probability of visiting state s when starting at so and
following Then we have (a) (())= F( (b) F(so) = EE, R(sy); (c) x(;)) -
P(a)
R(s,)
Σ.g R(s,)*
Proof:
The proof is trivial by following the proof of Proposition 2 in Bengio et al.
(2021a).
We have Lemma 1 which shows that a joint flow function can produce π (8y) = R(s)) correctly when the flow consistency equation is satisfied. Then we can use a TD-like objective to optimize the joint flow function parameter 0:
Lo(T)
Σ 3'Ersto0
Σ (s,))(s
((s,a)-R(s') R(s)) Σ Fo a²)
(5)
a'EA(s')
Note that optimizing equation not straightforward. On the one hand, each iteration, we need to estimate the flow in the order of O(IA, which leads exponential complexity. The joint flow estimation method may get stuck in local optima and can hardly scale beyond dozens of agents. On the other hand, joint flow networks require all agents to sample jointly, which is impractical since in many applications the agents only have access to their own observations.
3.2 INDEPENDENT FLOW NETWORK
To reduce the complexity and achieve the independent sampling of each agent, a simple way is to treat each agent an independent agent, so that each agent can learn its flow function in the order of
For simplicity, here consider homogeneous agents, i.c., A; A,, Vi, E N. Moreover, heterogeneous agents also face the problem of combinatorial complexity.
3

---

Under review as a conference paper at ICLR 2023
root
$1
S3
46
Terminal State
35
87
F(80) = Z
94
Observation
Σ II Filos Σ =
F(ss,a) a)
01
8,7
MADAG
Flow Match
Sampling Process
Training Process
Figure 2: Framework of GMFlowNets. For each state, each agent obtains its own observation and computes its independent flow to sample actions. During training, the agent seeks the parent nodes for computing inflows and outflows, and performs policy optimization through flow matching.
O(IA:)). We call this approach the Independent Flow Network (IFN) algorithm, which reduces exponential complexity linear. However, due to the non-stationarity of the flow (see Definition it is difficult for this algorithm to train a high-perrformade GMFlowNet.
Definition (Flow Non-Stationary) Define the independent policy n, as
Fi di)
m:(adlo:)
F(o:)
where E A.(o,) and Fe(o;, a.) is the independent flow of agent i. The flow consistent equation can rewritten as
Σ
Filo,, a.) R(o,,
Σ
Fild,,
D..o.T (o.
ofEA(O))
where -i represents other agents except agent i, and R(o;, 0:) represents the reward with respect to stute s and action O;,
Note that the transition function T(o;, ai, a-:) of in equation 7 is also related to the actions of other agents, which makes estimating parent nodes difficult. In addition, the reward of many multi- agent systems is the node reward R(s). that is, we cannot accurately estimate the action reward R(o,,a) a.) of each node. This transition uncertainty and spurious rewards can cause the flow non- stationary property. This makes difficult to assign accurate rewards to each action, and thus, is difficult train independent flow network with TD-like objective function.
As shown in Figure [ compared to the centralized training method, it is almost difficult for the independent method to leam a better sampling policy. One way to improve the per- formance of independent flow networks is design individual reward functions that are more directly related to the behavior of individual agents. However, this approach is difficult to im- plement in many environments because it is difficult to deter- mine the direct relationship between individual performance and overall system performance. Even in the case of sin- gle agent, only a small fraction of the shaped reward function aligns with the true objective.
100
Mode Found
80
10
15
20
Epochs
Figure 1: The performance of the centralized training and indepen- dent learning on Hyper-grid task.
3.3 FLOW CONSERVATION NETWORK
In this subsection, we propose the Flow Conservation Network (FCN) algorithm to reduce the com- plexity and simultaneously solve the flow non-stationary problem. FCN aims to learn the optimal
4

---

Under review as a conference paper at ICLR 2023
value decomposition from the final reward by propppropgg the gradients of the joint flow fune- tion F' through deep neural networks representing the individual flow function Fú, Vi E N. The specific motivation for FCN is avoid flow non-stationarity problems and reduce complexity. To start with, we have the following Definition 2 which shows the Individual Global Conservation (IGC) condition between joint and individual edge flows.
Definition 2 (Individual Global Conservation) The joint edge flow is product of individual edge flows, i.e,
F(st, a.) II F[ a.).
Then, we propose the following flow decomposition theorem.
Theorem Let the joint policy be the product of the individual policy {Tilizı where Te with respect the individual flow function Fi(os; 0i), i.e,
T((a:los) = Fi a.) Vi 1i k. F((o;)
8
Assume that the individual flow Fi(o;, a,) satisfies the condition in Definition Define a flow function F, all agents generate trajectories using independent policies nh i I,... k and the matching conditions
as so. (()==
Σ (s and <s. F(5')= sEP(s)
E s0) (9)
s"EC(x")
are satisfied. Then, we have: 1) n(85) R(sy):
2) F uniquely defines a Markovian flow matching F such that
F(r) 11:-1 F (8-11 - 86)
(10)
P(.)
Theorem states two facts. First, the joint state-action flow function F(s, a) can be decomposed into the product form of multiple independent flows. Second, if any non-negative function satisfies the flow matching conditions, unique flow is determined. On this basis, we can design algorithms for flow decomposition based on conservation properties. Each agent maintains neural network to estimate the flow of its actions, then calculates the joint flow function through the flow conservation condition, and trains the model with the relevant reward function. In this case, each agent maintains flow estimation network with the above architecture, which only estimates ([A,l) flows. Compared with the centralized flow estimation network, combining (o, as), we can get an unbiased estimate of F(ss, a) to calculate a TD-like objective function. Next, we illustrate the overall training process.
reduce the complexity ((N()A))) By
During the individual sampling process, each agent samples trajectories using its own policy and composes batch of data for joint training. During the joint training process, the system allows to call of the independent flow functions of each agent and uses the joint reward function to train the flow network. After training, each agent gets a trained independent flow network to the needs of independent sampling. In particular, for each sampled state, we first seek their parent nodes and corresponding observations and independent actions. Then, we compute the estimated joint flow F(s, a) by the flow consistency equation:
((s a) = exp Σ log (o,,,,;;)))
(11)
where 6; the model parameter of the i-th agent, which can trained based on equation
as:
F(5),)))
[(T:0) = Σ
Σ
f(s. R(ss)
Σ ((s'
(12)
3'E7730
a'EA(S))
5

---

Under review as a conference paper at ICLR 2023
Note that the above loss may encounter the problem that the magnitude of the flow each node in the trajectory does not match, for example, the flow of the root node is large, while the flow of the leaf node is very small. To solve this problem, we here adopt the idea of log-scale loss introduced in Bengio et al. (021a), and modify equation 12
[(T,E;))=
Σ S
(log Inflows] log |e + Outflows)),
(13)
where
Inflows
E
exp log (ss
E exp Σ log (o0
s.a:T(s,a)
(s,aa
Outflows = Outflows:=R R(')+
Σ
exp log F (d', R((s))
Σ exp
Σ log
a' a'EA(s')
and é is hyper-parameter that helps to trade-off large and small flows, which also avoids the nu- merical problem of taking the logarithm of tiny flows.
3.4 DISCUSSION: RELATIONSIIIP WITH MARL
Interestingly, there similar independent execution algorithms in the multi-agent reinforcement learning scheme. Therefore, in this subsection, we discuss the relationship between flow conser- vation networks and multi-agent RL. The value decomposition approach has been widely used in multi-agent RL based on IGM conditions, such as VDN and QMIX. For a given global state s and joint action a. the IGM condition asserts the consistency between joint and local greedy action selections in the joint action-value Qui(s, a) and individual action values (Qi(0;, (((os,a;)]:: a;)]
arg max Qu)(s. arg max Q1(01, arg max Qk(Ok, (x3) 44 E. a, EAE
(14)
aEA
a.EAi
Assumption For any complete trajectory in an MADAG (500,., 88), we assume that Qih (8 R(ss))((sss)) with f(s.) Ie= p(als:)
Remark 1 Although Assumption [ is strong assumption that does not always hold in practical environments. Here we only use this assumption for discussion analysis, which does not affect the performance of the proposed algorithms. A scenario where the assumption directly holds is that sample actions according to a uniform distribution in a tree structure, i.e., p(als) A(s)) The uniform policy is also used assumption in Bengio al. (2021a).
Lemma 2 Suppose Assumption I holds and the environment has tree structure, based the IGC and IGM conditions we have:
1) (s, a) F(s, a) (s);
2) (arg max,, Q,(o, a.)) = (arg max,, Fi(o;, az))
Based on Assumptior] we have Lemma 2 which shows the connection between the IGC condition and the IGM condition. This vction-value function equivalence property helps us better understand the multi-flow network algorithms, especially showing the rationality of the IGC condition.
4 RELATED WORKS
Generative Flow Networks: GFlowNets is an emerging generative model that could learn pol- icy to generate the objects with a probability proportional to given reward function. Nowadays, GFlowNets has achieved promising performance in many fields, such molecule generation Ben- gio et al. (2021a); Malkin et al. (202)); Jain et al. (2022), discrete probabilistic modeling Zhang et al. (2022 and structure learning Deleu et al. (2022). This network could sample the distribution of trajectories with high rewards and can be useful in tasks when the reward distribution more diverse. This leamning method is similar to reinforcement learning (RI.) Sutton & Barto (20T8), but

---

Under review as a conference paper at ICLR 2023
Algorithm Flow Conservation Network (FCN) Algorithm
Input: MADAG AS, R. N). Number of iteration T, Sample size B, Initial flow function FP, Vi 1, k; Parameters.
1: for itcration t = 1,... .T do
2: \A Individual sampling process
3: Sample observations {o,, af"", R=) based on the individual flow function F, for agents
4: \A Joint training process
Seek all parent nodes {y²} of the global state {ar] and calculate the inflow F(s", a²)
6: Calculate the outflow y = R EacA(.) F(s, a) by the flow conservation condition
7: Update individual flow function: {Fi} <- arg min (R)).. (an
D F( a²
8: end for
9: Define the joint sampling policy as the product of the individual policies { w.r.t {Fi)
Output: flow function Pr and individual sampling policy {m.)
RL aims to maximize the expected reward and usually only generates the single action sequence with the highest reward. Conversely, the learned policies of GFlowNets can achieve that the sam- pled actions are proportional to the reward and are more suitable for exploration. This exploration ability makes GFNs promising as a new paradigm for policy optimization in the RL field, but there are many problems, such as solving multi-agent collaborative tasks.
Cooperative Multi-agent Reinforcement Learning: There are already many MARL algorithms to solve collaborative tasks, two extreme algorithms are independent learning Tan (1993) and cen- tralized training. Independent training methods regard the influence of other agents as part of the environment, but the team reward function is usually difficult to measure the contribution of each agent, resulting in the agent facing a non-stationary environment Sunnhaa et al.] (017) : Yang et al. (2020) On the contrary, centralized training treats the multi-agent problem as single- eegeet coun- terpart. Unfortunately, this method exhibits combinatorial complexity and is difficult to scale beyond dozens of agents Yang et al. (2019). Therefore, the most popular paradigm is centralized training and decentralized execution (CTDE), including value-based Sunehag et al. (2017); Rashid et al. (2018); Son et al. al., (2019): Wang al. (2020) and policy-based Lowe et al. (2017) x Yu et al. (2021); Kuba et al. (2021) methods. The goal of value-based methods is to decompose the joint value function among agents for decentralized execution, which requires satisfying the condition that the local maximum of each agent's 's value function should be equal to the global maximum of the joint value function. VDN [Sunehag al. (2017) and QMIX Rashid al. (2018) propose two classic and efficient factorization structures, additivity and monotonicity, respectively, despite the strict factor- ization method. QTRAN [onn et al. (2019) and QPLEX Wang et al. (2020) introduce extra design for descomposition, such as factorization structure and advantage function. The policy-based meth- ods extend the aiget- TRPO Schulman et al. (2015) and PPO Schulman et al. (2017) into the multi-agent setting, such MAPPO Yu et al. (2021). ) which has shown the surprising effectiveness in cooperative, multi-agent games. The goal of these algorithms is to find the policy that maximizes the long-term reward, however, it is difficult for them learn more diverse policies, which can generate more promising results.
5 EXPERIMENTS
We first verify the performance of CFN on a multi-agent hyper-grid domain where partition func- tions can be accurately computed. We then compare the performance of IFN and FCN with standard MCMC and some RL methods to show that their sampling distributions better match normalized rewards. All our code implemented by the PyTorch Paszke al. (2019) library. We reimplement the multi-agent RL algorithms and other baselines.
5.1 HYPER-GRID ENVIRONMENT
We consider a multi-agent MDP where states are the cells of a N-dimensional hypercubic grid of side length H. In this environment, all agents start from the initialization point x = (0, -,-.. ), and is only allowed to increase coordinate with action a;, In addition, each agent has stop action. When all agents choose the stop action reach the maximum H of the episode length, the entire
7

---

Under review as a conference paper at ICLR 2023
200
MAPIO MC3CC EURr2
MARTO
350
200
FERELL
Mode: Found
250
250
Modia
Mode
40
150
50
2.5
1.45
1.94
1.43
3.66
1.80
155
1.75
20
20
Epochs
Epochs
Epochs
Figure 3: LI error and Mode Found performance of different algorithms various Hyper-grid environments. Top and bottom are respectively Mode Found (higher is better) and Error (lower is better). Left: Hyper-Grid vl, Middle: Hyper-Grie v2, Right: Hyper-Grid v3.
system resets for the round of sampling. The reward function is designed
R(x) Ro + R II I (0.25 J 0.5)) + R& II I [ 1 - 0.5| 0.4), (15)
where x = [211 [ xM. includes all agent states, the reward term 0 < Ro < R1 < R2 leads distribution modes. By changing Ro and setting closer to 0, this environment becomes harder to solve, creating unexplored region of state space due to the sparse reward setting. We conducted experiments in Hyper- grid environments with different numbers of agents and different dimensions, and we different version numbers to differentiate these environments, the higher the number, the more the number of dimensions and proxies. Moreover, the specific details about the environments and experiments
be found in the appendix.
We compare CFN and FCN with a modified MCMC and RL methods. In the modified MCMC method Xie et al. 20211. we allow iterative reduction of coordinates the basis of joint action space, and cancel the setting of stop actions to form a ergodic chain. As for RI. methods, we con- sider the maximum entropy algorithm, i.e., multi-agent SAC Haarnoja al. (2018), and previous cooperative multi-agent algorithm, i.e., MAPPO. Yu et al. (2021). Note that the maximum entropy method uses the Sofimax policy of the value function to make decision, so as explore the state of other reward, which is related to our proposed algorithm. To measure the performance of these methods, we define the empirical LI error as EE(((s)) π(s)) with p(sy) R(sy)/ being the sample distribution computed by the true reward function. Moreover, we can consider the mode found theme to demonstrate the superiority of the algorithm.
Environment
MAPPO
MASAC
MCMC
CFN
FCN vl
FCN
Hyper-Grid vl
2.0
1.84
1.78
2.0
2.0
2.0
Hyper-Grid v2
1.90
1.76
1.70
1.85
1.85
1.82
Hyper-Gris v3
1.84
1.66
1.62
1.82
1.82
1.78
Table 1: The best reward found of different methods.
Figure illustraates the performance superiority of our proposed algorithm compared to other meth- ods in the L1 error and mode found. For FCN, we consider different decision-making methods,
8

---

Under review as a conference paper at ICLR 2023
the first is sample actions independently, called FCN v1, and the other is combine these poli- cies for sampling, named FCN v2. We find that on small-scale environments shown in Figure BigureBLLen Left, CFN can achieve the best performance, because CFN can accurately estimate the flow of joint tions when the joint action space dimension small. However, as the complexity of the joint action flow that needs be estimated increases, we find that the performance of CFN degrades, but the independently executed method still achieves good estimation and maintains the speed of conver- gence, shown in Figure Middde.. Note that RL-based methods do not achieve the expected performance, their performance curves first rise and then fall, because as training progresses, these methods tend to find the highest rewarding nodes rather than finding more patterns. In addition, as shown in Table [ both the reinforcement learning method and our proposed method can achieve the highest reward, but the average reward of reinforcement learning slightly better for all found modes. Our algorithms do not always have higher rewards than RI., which is reasonable since the goal of GMFlowNets is not to maximize rewards.
5.2 SMALL MOLECULES GENERATION
Similar to Jin et al. (2018) Bengio al. 202 la): Xie
WAPP
et al. 2021), we consider the task of molecular la generation
evaluate the performance of FCN. For any given molec- ular and chemical validity constraints, we can choose an atom attach block. The action space is choose the location of the additional block and selecting the ad- ditional block. And the reward function calculated by pretrained model. We modify the environment to meets the multi-agent demand, where the task allows two agents to perform actions simultaneously depending on the state. Although this approach is not as refined as single-agent decision making. we only use it to verify the performance of FCN. Figure 4 shows that the number of molecules 8
40
Mode Found
30
20
10
0
100
200
300
400
Epochs
Performance of FCN and MAPPO molecules generation task. found by different algorithms, we can see that FCN can generate more molecules with high reward functions over three independent runs.
Figure 4:
with the reward value greater than threshold
CONCLUSION
In this paper, we discuss the policy optimization problem when GFlowNets meet the multi-agent systems. Different from RL, the goal of GMFlowNets is to find diverse samples with probability proportional to the reward function. Since the joint flow is equivalent to the product of independent flow of each agent, we design a CTDE method to avoid the flow estimation complexity problem in fully centralized algorithm and the non-stationary environment in the independent learning process, simultaneously. Experimental results on Hyper-grid environments and small molecules generation task demonstrate the performance superiority of the proposed algorithms.
Limitation and Future Work: Unlike multi-agent RL algorithms that typically use RNNs as the
value estimation network Hochreiter & Schmidhuber (1997); Rashid et al. (2018), RNNs are suitable for our algorithms for flow estimation. The reason is that the need to compute the par- ent nodes of each historical state introduces additional overhead. Another limitation that, like the original GFlowNets, GMFlowNets are constrained by DAGs and discrete environments, which makes GMFlowNNts temporarily unavailable for multi-agent continuous control tasks. Therefore, our future work is to design multi-agent continuous algorithms to overcome the above problems.
REFERENCES
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup. and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, $4:27381-273/) 2021a.
Yoshua Bengio, Tristan Deleu, Edward Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. arXiv preprint 1 2021b.

---

Under review as a conference paper at ICLR 2023
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey of multiagent rein- forcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172 2008.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanó. Multi-agent reinforcement learning: A review of challenges and applica- tions. Applied Sciences, 11(1) (111):4948, 2021.
Tristan Delcu, António Góis, Chris Emezue, Mansi Rankawat, Simon Lacoste-Julien, Stefan Bauer, and Yoshua Bengio. Bayesian structure learning with generative flow networks. arXiv preprint arXiv:2202.13903 2022.
Amal Feriani and Ekram Hossain. Single and multi-agent deep reinforcement learning for ai-enabled wireless networks: A tutorial. IEEE Communications Surveys & Tutorials, 23(2):1226-112 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learming, pp. 1861-1870. PMLR, 2018.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Moksh Jain, Emmanuel Bengio, Alex Garran, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning, pp. 9786-9801. PMLR, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pp. 2323-2332. PMLR, 2018.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation multi-agent reinforcement learning. arXiv preprint ar arXiv:2109.112::
2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compettiiee environments. Advances in neural informa- tion processing systems, 30. 2017.
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in gflownets. arXiv preprint arXiv:2201.132.3 2022.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate valuee func- tions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International conference on machine learning, pp. 4295 4304. PMLR, 2018.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. International conference on machine learning, pp. 4344-4353.
PMLR, 2018
10

---

Under review as a conference paper at ICLR 2023
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-18888888 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Interna- tional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement Learning, pp- 384-414. Springer, 2012.
Peter Sunehag, Guy Lever, Audrunas Gruslys. Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Yalue-decomposition networks for cooperative multi-agent learning, arXiv preprint arXiv:1706.0000 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ming Tan. Multi-agen reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards, Advances in Neural Information Processing Systems, 32, 2019.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: multi-agent q-learning. arXiv preprint arXiv:2008. 01062, 2020.
Duplex dueling
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. arXiv preprint 103. 2021.
Yaodong Yang, Rasul Tutunov, Phu Sakulwongtana, Haitham Bou Ammar, and Jun Wang. a -ank: Scalable multi-agent evaluation through evolution. 2019.
Yaodong Yang, Ying Wen, Jun Wang, Liheng Chen, Kun Shao, David Mguni, and Weinan Zhang- Multi-agent determinantal q-learning. In International Conference on Machine Learning, pp- 10757-10766. PMLR, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint 2003:22003.01955, 2021.
Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. arXiv preprint arXiv:2202.01361. 2022.
Kaiqing Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.

---

Under review as a conference paper at ICLR 2023
A PROOF OF MAIN RESULTS
A.1 PROOF OF THEOREM 1
Theorem 1. Let the joint policy the product of the individual policy {Ni where a, with respect to the individual flow function Fo(o,, a4), i.e.,
T((ailo:) Fi(o,, a.) 1i k.
(16)
F((o;)
Assume that the individual flow Fi a,) satisfies the condition in Definition function F, if all agents generate trajectories using independent policies Wit matching conditions
Define a flow
1,,. k and the
4A 80. F())= = Σ
( and AA <8; P(5)= = E
f - 8")
(17)
SEP(N)
S"EC(()
are satisfied. Then, have: 1) T(ss) & R(s;);
2) F uniquely defines Markovian flow F matching F such that
1==1 F (8-11 - 8t)
F(r)
(18)
I P (s:)
Proof: We first prove the part 1). Since
F(st, a.) II at).
then we have the global state flow as
F(si) = Σ F(s), = Σ a.E.A
II Fi af).
(19)
a,EA
According to the flow definitions, the observation flow Fi(o)) and individual observation flows have the relationship:
F((oi)= = Σ Pilof, af).
(20)
ofEA
Hence, we have
II (o( II
Σ
at(obaat)
(21)
i=1
aFEA'
Σ
Bi(o, ...
Σ Pilot, af)
(22)
atEA
atEAK
Σ
al(ol,a))... F(l,.,, a((o",t))
(23)
ale
x
(24)
a.Aii=
yielding F(s.) II, F.(o)). Therefore, the joint policy
F(s F(s,)
II. Filof. af)
<als))
F(se)
(25)
II. Fo(of,
II Te(a)(os).
=
T.. Fi(ot)
12

---

Under review as a conference paper at ICLR 2023
indicates that if the conditions in Definition 2 is satisfied, tency of joint and individual policies. Based on Lemma we can conclude that the reward of the generated state satisfies ((s)) R(sy) using the individual policy (a;ooo;; of each agent.
Equation 25
establish the consis-
Next, we prove the part 2). We first prove the necessity part. According to Definition and Bengio
al. (202bb) have
F(d) II BiOd = II
E
Pild' -0'1= E
PIO - 0')),
o'EP(ott))
OEP(O')
F(s) II Fi( = II
E
Rloo + b0")=
E d'ECio") i
Il Filo" o'"
Then we prove the sufficiency part. We first present Lemma
which shows that
Σ
PB(T)= E II
(=(()(s(41)= =.
TET0.
TETo, 81-781417
Lemma (Independent Transition Probability) Define the independent forward and backward transition respectively as
Fi (o) - +
Pf (oi+1l0t) = P. (o) + 004110))
(26)
F₁ (0:)
and
F. (0l+1 + of) Pi (o(())1)
Pa (o.l0441) Ps (0+41 of1o++1)=
(27)
Then we have
Σ 1,
TETA,
(28)
Pa(T) = 1, L, E
To..
where T the set of trajectories starting in s and ending in 8y and To,a is the set of trajectories starting in 80 and ending in
Define 2 F(so) the partition function and Pp as the forward probability function. Then, according to Proposition 18 in Bengio et al. (2021b), we have there exists a unique Markovian flow F with forward transition probability function PF = Pp and partition function Z, and such that
F(r)= = 2 n+1
1 f( + 8x)
II 1=1
Pr(3
(29)
II:- F(st)
where 8n+1 = Sj. Thus, we have for s' + Sg:
F(s'= 2 Σ II
Pp(s-1(si)
rETow! (or46))))
2 F(s)) Σ
II
Pr((e(S(+1) F(s)).
(30)
F(00) TET (a (8))r()))
Combining equation 30 with Pp Pr. we have + 5 A, F(s + s'). Finally, for any Marko- vian flow F' matching F on states and edges, we have ('(T) F(T) according to Proposition 16 in Bengio et al. (021b)) which shows the uniqueness property. Then complete the proof.
13

---

Under review as a conference paper at ICLR 2023
A.2 PROOF OF LEMMA 2
Lemma 2. Suppose Assumption I holds and the environment has tree structure, based on the IGC and IGM conditions we have:
1) (m (8, a) = F(8, ())(a))();
2) (arg max., Q,(o,, a))) (arg max,, Fo a=))
Proof: The proof is an extension of that of Proposition in Bengio et al. (2021a). For any (s, a) satisfies sf Tss, a), have ( a) {) and F(s, a) ss)) Therefore, we have (s, a) F(s, (s). Then, for each non con-final node s', based on the action-value function in terms of the action-valuu the next step, we have by induction:
as) = R(s') +
Σ a'EA(s))
Quils','' a';
(31)
0+ (0+u(als))
Σ
(()
a'EA(S))
where f(('') is the reward of Vor (8, a) and (a) is due to that f((')) if s' is not a final state. Since the environment has tree structure, have
F(s.a)= E F(s',a'),
(32)
a'EA(e)
which yields
(s, pualss)) (((l):a)))((()===()))))))=)))))) u(als') ((s)))((s).
According to the IGC condition we have F(st, a.) = II, Fi a)), yielding
arg max (so a) (a)
arg F(s
arg max
log Fi(o a:)
(33)
arg max F1 a1) ... . arg max Fo as)) BiEÁ;
agEAr
where (a) is based on the fact F and f(s) positive, (b) is due the IGC condition. Combining with the IGM condition
arg max (s, arg max Q1)0),a1). arg max Qu(ok, at) Vs E S. aeAA
(34)
E.A1
a
we can conclude that
arg max Fi(o;; at) .i)) a,EA,
arg max Q, ((o,,a)) aiEAi
) &
i=1
im1
Then we complete the proof.
A.3 PROOF OF LEMMA
Lemma 3 [Independent Transition Probability). Define the independent forward and backward transition rexpectively as
F3 (ot +b+1)
Pp (of41 (f)) = P. (of + of+11oo)
(35)
F. (0:)
and
F; (o + of)
PB (oblof+1) (oi41 oflo(tn)
(36)
F (o(11)
14

---

Under review as a conference paper at ICLR 2023
Then have
Σ 1
TET..1
(37)
Σ 1,
TETo.
where To.5 the set of trajectories starting in and ending in sy and Tu.s is the set of trajectories starting in so and ending in s.
Proof: When the maximum length trajectories is not more than have
Σ
PF(T)= =1
(38)
TET.
Then we have the following results by induction:
Σ PF(T)
E E
PF(T)=
E
PF(o'lo) E PF(T)
TET..J
('E0())
d'EC(O)
TET
(39)
E dEE((o))
E Po(dfoo)
Σ Pp(T)=11
TET
where C()) is the children set of the current state observation and the last equation based on the fact EojeC(o.) Pr(o/oo:) = Since the proof process of PB is similar to that of Pr, it is omitted here.
B EXPERIMENTAL DETAILS
B. HYPER-GRID ENVIRONMENT
Here we present the experimental details on the Hyper-Grid environments. Figure the curve of the flow matching loss function with the number of training steps. The loss of our proposed algorithm gradually decreases, ensuring the stability of the learning process. For some RL algo- rithms based on the state-action value function estimation, the loss usually oscillates. This may be because RL-based methods experience replay buffer and the transition data distribution is not stable enough. The method we propose uses an on-policy based optimization method, and the data distribution changes with the current sampling policy, hence the loss function is relatively stable. We set the same number of training steps for all algorithms for a fair comparison. Moreover, we list the key hyperparameters of the different algorithms Tables
Loss
Loss
Vhlyyh M 2 Mo Veegnn Aww.n Anttraadd
2
50
100
150
200
250
300
100
150
200
250
300
Training Steps
Training Steps
Figure 5: The flow matching loss of different algorithm.
We study the effect of different reward Figure In particular, we set Rg = 10002,10-2,000 210-1,10-2,1001 10 for different task challenge. A smaller value of Ro makes the reward function distribution more
15

---

Under review as a conference paper at ICLR 2023
80
(P4--15-
TCVR 10
1,8
FEN -
Mode Found
TCUR TCVR-12" 11
60
Error
RDMR 10-1
RC 10
KC 10
1.4
Epachs
Epochs
Figure 6: The effect of different reward RS different algorithm according to L.1 emror and mode found.
sparse, which makes policy optimization more difficult [Bngiio et al. (2021a): Riedmiller et al. 2018); Trott et al. 2019). As shown in Figure 6 we found that our proposed method is robust with the cases Rs = 10t and Rs = 10 When the reward distribution becomes sparse, the performance of the proposed algorithm degrades slightly.
Table 2: Hyper-parameter of MAPPO under different environments
Hyper-Grid-vl
Hyper-Grid-v2
Hyper-Grid-v3
Train Steps
20000
20000
20000
Agent
2
2
3
Grid Dim
3
3
Grid Size
[8,8]
[8,8]
[8,8]
Actor Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0001
0.0001
0.0001
Batchsize
64
64
64
Discount Factor
0.99
0.99
0.99
PPO Entropy
lc-1
le-1
le-1
Table 3:
of MASAC under different environments
Hyper-Grid-vi
Hyper-Grid-v2
Hyper-Grid-v3
Train Steps
20000
20000
20000
Grid Dim
3
3
Grid Size
[8,8]
[8,8]
[8,8]
Actor Network Hidden Layers
[256,256]
[256,256]
[256,256]
Critie Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0001
0.0001
0.0001
Batchsize
64
64
64
Discount Factor
0.99
0.99
0.99
SAC Alpha
0.98
0.98
0.98
Target Network Update
0.001
0.001
0.001
16

---

Under review as a conference paper at ICLR 2023
Table 4: Hyper-parameter of FCN under different environments
Hyper-Grid-vl
Hyper-Grid-v2
Hyper-Grid-v3
Train Steps
20000
20000
20000
R2
2
2
2
R1
0.5
0.5
0.5
Grid Dim
2
3
3
Grid Size
[8,8]
[8,8]
[8,8]
Trajectories per steps
16
16
16
Flow Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0001
0.0001
0.0001
0.0005
0.0005
0.0005
Table 5: Hyper-parameter of CFN under different environments
Hyper-Grid-vi
Hyper-Grid-v2
Hyper-Grid-v3
Train Steps
20000
20000
20000
Trajectories per steps
16
16
16
R2
2
2
2
R1
0.5
0.5
0.5
Grid Dim
2
3
Grid Size
[8.8]
[8,8]
[8.8]
Flow Network Hidden Layers
[256,256]
[256,256]
[256,256]
Optimizer
Adam
Adam
Adam
Learning Rate
0.0001
0.0001
0.0001
0.0005
0.0005
0.0005
17


Reviews:
[{"review_id": "r3KeAzvg_sI", "reviewer": "Reviewer_45Kq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to adapt the GFlowNet framework to a multi-agent setting. It studies the typical MARL parameterizations, and in line with past work proposes a Centralized Training Decentralized Execution formulation of a GFlowNet parameterization where each agent produces an independent flow, and where the joint flow is their product.\n\nThe authors show that such a formulation is compatible with the GFlowNet framework, and produce some empirical validation of their approach on multi-agent versions of a grid world task and a molecular generation task.", "summary_of_the_review": "It is nice to see more research on GFlowNets, and there is certainly value in the potential diversity that they could bring to something like MARL methods. Unfortunately, the empirical validation of the approach needs to be strengthened and the paper made more clear. I do not think that the claims made by the paper are correctly validated.", "strength_and_weaknesses": "### Strengths\n\n- The GFlowNet framework is very recent and there is value in exploring how it is compatible with other aspects of ML\n- The proposed parameterizations make sense\n\n### Weaknesses\n\n- There seems to be something off with the evaluation. It would be good for the authors to clarify what is going on and if possible provide additional empirical validations of their approach\n- There is no evaluation of the proposed method on standard multi-agent problems\n- There are many lacking details and comments that aren't quite correct\n- The proposed approaches are not particularly novel ideas but rather combinations of past ideas. This is not necessarily a bad thing, but in some sense the results and ideas presented in this work are expected and unsurprising, which naturally sets the bar higher.\n\n\n### Details\n\n> However, existing works can only handle single flow model tasks and cannot directly generalize to multi-agent flow networks due to limitations such as flow estimation complexity and independent sampling  \n> Unfortunately, currently GFlowNets cannot support multi-agent systems.\n\nInteresting way of putting it, considering prior GFN work had no such ambitions. Saying something like, \"because of its exploration capabilities, we extend the GFlowNet framework to support multi-agent systems\" would make more sense.\n\nThe acronym IGM is used but not defined (I suppose it's Individual Global Maximum?).\n\n> a policy $\\pi$ defined by the forward transition probability satisfies $\\pi(s'|s)=P_F(s'|s) \\propto R(x)$\n\nThis is incorrect. A valid flow induces $p(x) = p(x\\equiv s \\to s_f) \\propto R(x)$. In other words, when following policy $P_F(a|s)=F(s,a)/F(s)$ the probability of sampling a _trajectory_ which ends in $x$, i.e. a transition such that $x\\equiv s$ is a terminal state, is proportional to the reward of $x$.\n\n> unequal pairs of states in the trajectory\n\nthis is a slightly confusing phrasing, it might be easier to understand \"such that $s_i \\neq s_j \\forall \\tau,i\\neq j$, but this is already the definition of an acyclic graph; I'm not sure this sentence is even needed.\n\n> with outputs proportional to the predefined reward function\n\nThis is also somewhat incorrect, and it's not clear what \"output\" refers to. In the general case, for a non-terminal state with multiple parents and multiple children, state flows and edge flows are not likely to be proportional to some reward. Even for a terminal state, if this state has multiple parents, then only the sum of the incoming flows is equal to $R(x)$, not any particular flow on an edge leading to $x$. Similarly for trajectories, it is the sum of flows of trajectories which is equal to the reward, i.e. $\\sum_{\\tau:x\\in\\tau} F(\\tau)=R(x)$, not the \"output\" for any one given trajectory.\n\n> $s_f$\n\nIn Bengio et al. 2021a, terminal states are denoted by $x$, in Bengio et al. 2021b, the \"GFlowNet Foundations\" paper, the authors switch to using $s_f$ (see Definition 3 of a Pointed DAG), but it denotes a _single_ final sink state to which every terminating state transitions to, i.e. $s$ is terminating if $s \\to s_f$ exists. I'd recommend keeping notation compatible with prior work.\n\n\n\n> Definition 1 (Flow Non-Stationary)\n\nShould be \"Non-Stationarity\"? Also, Definition 1 seems to define independent flows and policies, and says nothing of non-stationarity. Am I missing something?\n\n> This transition uncertainty and spurious rewards can cause the flow non-stationary property\n\nI agree that the partially observable system now appears _stochastic_ to any one agent and intuitively causes non-stationarity during learning, but, this \"flow non-stationary property\" is never defined. What is non-stationary? The reward? The transition function? The flow itself?\n\n\nIt seems counterproductive to introduce _two methods_ which are designed to fail, why not go straight to the point? It is well known in the multiagent literature that these problems exist (e.g. centralized execution leading to an exponential number of actions). I don't think this exposition was necessary.\n\n> Definition 2, $F(s,a) = \\prod_i F_i(o^i_t,a^i_t)$\n\nI wonder what the partial observability assumptions are here. What must hold so that $F(s,.)$ is computable from individual observations $o^i$ and the product of $F_i$? This is non-trivial and is left unaddressed. Consider the common multiagent scenario of a (grid) world where agents' observations are restricted to some radius $r$ around them--it may be possible there that $F(s,a)$ is uncomputable under this parameterization, but it computable and Markovian under a joint/centralized parameterization.\n\n> The uniform policy is also used as an assumption in Bengio et al. (2021a).\n\nIt is my understanding that the uniform policy is used in Proposition 4 of Bengio et al. (2021a) to show the equivalence _in a special case_ between a flow function and an action value function, and to conjecture that this equivalence _may_ be more general.\n\nI'm also not sure I understand the point of Lemma 2. It seems trivial that if $f({\\bf x}) = \\prod f_i(x_i)$ then $\\mbox{argmax}_ {\\bf{x}} f(x) = \\prod \\mbox{argmax}_ {x_i} f_i (x_i)$ when $f_i>0$. If one assumes that the flow $F(s,{\\bf a})$ is decomposable as a product, then trivially its argmax is a product of argmaxes. It follows from Proposition 4 of Bengio et al. (2021a) that there is an equivalence between $F$ and $Q$ on a tree-structured MDP. The IGM equivalence simply follows from assuming Definition 2.\n\nIt seems the attempt here is to provide an equivalence between the proposed flows and some $Q$ function as in Bengio et al. (2021a), or am I missing something deeper?\n\n> the learned policies of GFlowNets can achieve that the sampled actions are proportional to the reward\n\nThis is also technically incorrect, see above; $p(x) \\propto R(x)$, not $p($actions$)$ nor $p(\\tau)$ (in the general case).\n\n> The reward function is designed as [hypercube reward]\n\nIt seems this reward is taken from Bengio et al. (2021a), which it would be good to mention. Moreover, I'm not sure I understand what is multi-agent about this reward. All the agents get an independent reward, judging from the (incomplete) provided code, all the agents are allowed to move in all dimensions (it seems). The only explanation I can think of is that when the authors write \"where $x = [x_1, ... , x_k]$ includes all agent states\", they mean that in the $R$ computation $k$ is $k=N*m$ where $N$ is the hypercube dimension and $m$ is the number of agents.\n\nIt's also unclear in the main text nor the appendix what $m$ is exactly; I see it specified in Table 2 for MAPPO but not for the other algorithms, is it the same?\n\n> we define the empirical L1 error as\n\nThis is again following the grid example of Bengio et al. (2021a), there seems to be an $\\mbox{abs}$ or $|\\cdot|$ missing in the expectation.\n\n> Figure 3, L1 row\n\nIt makes very little sense to measure the L1 error for RL methods, that is not something they optimize for and not something we'd ever expect them to minimize even by accident.\n\nI just looked at Figure 2 of Bengio et al. (2021a) and their L1 metric goes from about $5\\times 10^{-4}$ all the way down to $10^{-5}$, so it drops about one and a half orders of magnitude, whereas the reported L1 error here seems to plateau very close to the initial values. The magnitude of the measure is also puzzling, how can an average absolute difference of things that are less than 1 (because they are probabilities) be greater than 1?\n\nI'll admit this is worrying. Is the proposed method capable of learning $p(x)\\propto R(x)$? These grid experiments should be very quick; would it be possible to show a setting where the L1 error gets low? Would it be possible to visualize the learned distribution to see if it matches what is expected? (cf. Bengio et al. (2021a) Figures 10,12,13)\n\nIt's also not mentionned how long one epoch is, how many samples does the model see?\n\n> Figure 3, modes found row\n\nWhat are FDN and JFN? Typos?\n\n> Figure 4\n\nI can imagine that two agents editing the same molecule is harder than the original setup, but the orders of magnitude are again a bit worrying. The original GFlowNet paper seems to find about 1500 modes with $R > 8$ within having seen 1M molecules. It's again not mentioned what an epoch is so it's hard to compare but even being generous about what one epoch might be, the contrast is worrying.\n\nLet's suppose that this is expected. It would be nice to have more MARL baselines, and it would be nice to test the proposed method on actual standard multiagent problems. It would also be nice to show that the trained models actually are learning a flow (see Figure 16 of Bengio et al. 2021a or Nica et al [1])\n\nAs is, the empirical evaluation of the method is quite limited.\n\n> GMFlowNets are constrained by DAGs and discrete environments, which makes GMFlowNets temporarily unavailable for multi-agent continuous control tasks\n\nThere is a simple fix to this, proposed by Bengio et al. (2021b), which is to augment the state description with the current timestep. This naturally forms a DAG because it is impossible for the agent to go back in time, therefore making it impossible to go back to a previous state.\n\nNote on formatting: it seems all the non-inline citations are missing paretheses. Are you using the `\\citep{...}` command?\n\n\n[1] Evaluating Generalization in GFlowNets for Molecule Design, Andrei Cristian Nica, Moksh Jain, Emmanuel Bengio, Cheng-Hao Liu, Maksym Korablyov, Michael M. Bronstein, Yoshua Bengio, MLDD 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "cBQIEqRBiU", "reviewer": "Reviewer_YqtU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In a multi-agent environment, the joint action space can be exponential in the number of agents. A naïve application of the GFlowNet framework requires learning a flow function over |A_i|^N actions, which can be hard to optimize. Another naïve approach which treats each agent independently also fails due to difficulties in estimating the reward w.r.t. to a single agent.\n\nThe authors propose a factorized flow function F(s_t, a_t)=\\prod_i F_i(o_t^i, a_t^i) where i runs over the agents. This allows an individual F_i to model only |A_i| actions at the expense of having to sample actions independently on every step.\n\nEmpirical results demonstrate the superiority of CFN and FCN over MCMC and RL methods in terms of number of modes found and L1 error. On harder tasks, GFlowNet-based methods also outperform RL and MCMC in terms of the best reward found.", "summary_of_the_review": "This paper applies GFlowNets to multi-agent learning scenarios, which can be of interest to many in the community. However, the paper can benefit from a clearer motivation, better experiments, and a more polished delivery. I encourage the authors to rework the manuscript and submit again.", "strength_and_weaknesses": "The GFlowNet formalism does not specify how the flow function F should be parametrized. Indeed, in a multi-agent setting where agents are expected to act independently with limited information, it can be a useful inductive bias to factorize the flow function. However, it is not clear what the contributions of this paper are besides the empirical validation that this straightforward factorization works. Note that CFN is the GFlowNet formalism applied naively to the multi-agent setup where all agents are modeled jointly, while IFN is another naïve application of the formalism where agents are modeled completely independently.\n\nA scalability issue the authors do not address is the sum over |A_i|^N actions when calculating the in- and out-flow, which is required for both CFN and FCN. A paper made public in Jan. 2022 (https://arxiv.org/abs/2201.13259) proposes a new training objective for GFlowNets which circumvents this large sum. If the authors are motivated by the scalability issue in the multi-agent setup, I encourage them to include the aforementioned paper in their investigation.\n\nLast but not the least, I have several questions regarding the experiments. The central claim is that CFN is not scalable which necessitates factorizing the flow estimator. However, FCN is not clearly better than CFN even on the hard Hyper-Grid environments (v2, v3). (Note that the legend says FDN and JFN.) It is not clear what the authors mean by “[combining] these policies for sampling” for FCN v2 whereas FCN v1 “[samples] actions independently”. I could not find further elaboration elsewhere in the paper.\n\nFurthermore, based on table 4 and 5, both FCN and CFN use the same model architecture for the flow estimator. However, CFN has a single estimator while FCN has one per agent. Could the authors clarify if these two approaches use the same number of trainable parameters? If FCN were using N times more parameters where N is the number of agents, this would further weaken the empirical result.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WmHiHoHRNn", "reviewer": "Reviewer_pmcj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Considering recent success of GFlowNets, the paper aim to extend the GFlowNet framework to the multi-agent setting, which focuses on using the flow matching constraint. The authors propose three variants of multi-agent GFlowNets, including centralized flow network, independent flow network, and flow conservation network. The authors conduct experiments based on the hypergrid and molecule generation task to validate the effectiveness of the proposed methods by comparing it with MAPPO, MASAC and MCMC.", "summary_of_the_review": "The paper extends the GFlowNets (based on flow matching) to the multi-agent setting. Although the paper is easy to follow, I think it can be improved greatly in terms of originality and quality. Although this is the first paper that extends GFlowNets to the multi-agent setting, I am concerned about its novelty. It combines existing methodology in the MARL literature and flow matching-based GFlowNets without enough in-depth analysis. The experimental benchmark is not very appropriate to fully investigate the value of multi-agent GFlowNets.", "strength_and_weaknesses": "- Clarity: The paper is easy to follow, although some of the terms (e.g., the experimental section) and the discussion of GFlowNets literature could be made clearer (discussion of the detailed balance and trajectory balance loss).\n\n- Originality: The paper extends the GFlowNets (based on flow matching) to the multi-agent setting. Authors propose three kinds of methods: centralized flow network (related to directly training joint DQN in the MARL literature that faces the challenge of the curse of dimensionality), independent flow network (related to independent Q-learning in the MARL literature that faces the challenge of non-stationarity), and flow conservation network (related to CTDE methods in MARL like QMIX that allows for centralized training and decentralized execution). Although this is the first paper that extends GFlowNets to the multi-agent setting, I am concerned about its novelty. It combines existing methodology in the MARL literature and flow matching-based GFlowNets without enough in-depth analysis.\n\n- Quality and significance: Besides the novelty concern, I am also concerned about the methodology part and the experiment part.\n    - It has been shown that trajectory balance is one of the most competitive losses of GFlowNets, and it is worth discussing the possibility and potential of the proposed method based of trajectory balance.\n    - I think the experimental benchmark is not very appropriate to evaluate multi-agent GFN (considering the single-agent feature of the experimental tasks including hypergrid and molecule generation originally used in the GFlowNets papers). I think it is more reasonable to evaluate multi-agent GFlowNets in standard multi-agent benchmarks such as StarCraft II micromanagement benchmarks.\n    - Table 1 should also include the standard variation to better investigate different methods.\n    - The details of the hypergrid task are not clear. For example, the author discuss the difficulty of changing R_0 closer to 0. It should be mentioned clearly in the paper about the size of the grid and the value of R_0, etc.\n    - The comparison in Section 5.2 is somewhat weak, and should at least compare FCN with the most competitive method in Section 5.1 like MCMC and MASAC besides MAPPO.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Iqwf4cWTFe", "reviewer": "Reviewer_4dkX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Several variants of GFlowNets are proposed in the context of multi-agent systems with centralized training and decentralized execution. The challenge considered in this paper is that we want to resulting policy to factorize across the agents, so that each agent can independently sample its next action given only a local observation.", "summary_of_the_review": "The paper suffers from too many problems in both clarity of the math and of the text to be acceptable for publication. There may also be a fundamental flaw in the formulation which would make it unscalable (scale exponentially in compute time wrt number of agents).\n\n", "strength_and_weaknesses": "The question of factorizing the policies (and the flows) of a GFlowNet is interesting and could be relevant in several domains, especially if this makes it possible to obtain efficient algorithms.\n\nUnfortunately, I found the paper very poorly written, difficult to understand (both in the text and in the math), and left me with a major concern about the actual scalability of the proposed approach. The concern arises is Eq. 9, which requires performing sums over the cross-product of all the actions of all the agents (both forward or backward). The authors give no indication that these sums can be computed tractably: it looks like they require computation scaling exponentially with the number of agents. Unless there is a fix (and I did not see one even discussed), this makes the proposal not scalable.\n\nAnother issue is that the statement in definition 2 is not really a definition but rather an assumption, and more importantly, *it is not clear at all that under this assumption there generally exists a flow function* (that satisfies the flow matching constraints). This would make the mathematical foundation for all the theoretical results somewhat shaky. On page 5, the authors present this statement as as a FACT, \"Theorem 1 states two facts. First, the joint state-action flow function F(s,a) can be decomposed into the product...\". But it is not shown in theorem 1, it is an *assumption*.\n\nEqn 13 is wrong (it should have a square at the end).\n\nThere is also an incorrect statement about Bengio et al 2021a in Remark 1: \"The uniform policy is also used as an assumption in Bengio et al 2021a\", page 6. There is no such assumption in that paper. The only mention of uniform distribution in that paper is its use for defining an exploratory policy (which samples the training trajectories of the GFlowNet): they take a mixture of the GFlowNet policy P_F and the uniform, to make sure that all actions are given a non-zero probability. But that is no assumption at all, it is a heuristic to choose the training trajectories. The paper also has a theorem showing that any full-support distribution would asymptotically be sufficient to guarantee convergence to a proper flow. This is concerning because it suggests a misunderstanding of the original GFlowNet paper. It is also concerning because this argument is used to justify a completely different assumptio used here (the one stated as definition 2 discussed above).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject