Learning to Share in Networked Multi-Agent Reinforcement Learning
Authors: Yuxuan Yi
Ge Li
Yaowei Wang
Zongqing Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cooperative Multi-Agent Reinforcement Learning
Networked Multi-Agent Reinforcement Learning

Abstract: In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where a number of agents are deployed as a partially connected network and each interacts only with nearby agents. Networked MARL requires all agents to make decisions in a decentralized manner to optimize a global objective with restricted communication between neighbors over the network. Inspired by the fact that sharing plays a key role in human's learning of cooperation, we propose LToS, a hierarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors so as to encourage agents to cooperate on the global objective through collectives. For each agent, the high-level policy learns how to share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize the local objective induced by the high-level policies in the neighborhood. The two policies form a bi-level optimization and learn alternately. We empirically demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenarios across scales.

Learning to Share in Multi-Agent Reinforcement Learning
Yuxuan Yi!
Ge Li'
Yaowei Wang Peking University
Zongqing Lu
Peng Cheng Lab
(touma.geli)ppkrt edu cn
wangywũpel.co www.elllc..... zongqing lu@pku edu
Abstract
In this paper, we study the problem of networked multi-agent reinforcement learn- ing (MARL). where number of agents are deployed as partially connected network and each interacts only with nearby agents. Networked MARL requires all agents to make decisions in decentralized manner to optimize global objective with restricted communication between neighbors over the network. Inspired by the fact that sharing plays key role in human's 's learning of cooperation, we propose LToS, a hicrarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors on the global objective through collectives. For each agent, the high-level policy learns how share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize the local objective induced by the high-level policies in the neighborhood. The two policies form bi-level optimization and learn alternately. We empirically demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenarios across scales.
encourage agents cooperate
Introduction
In fully cooperative multi-agent reinforcement learning (MARL), there multiple agents interacting with the environment via their joint action to cooperatively optimize an objective. Many methods of centralized training and decentralized execution (CTDE) have been proposed for cooperative MARL, such as COMA (Foerster et al., 2018), QMIX (Rashid et al., 2018), QPLEX (Wang et al., 2021), and FOP (Zhang et al., 2021). However, these methods suffer from the overgeneralization issue: employed value functions cannot estimate well because agents sometimes choose uncoordinated actions, and thus the optimal policy cannot be learned (Castellini et al., 2019). Moreover, they may not easily scale with the number of agents due to centralized learning (Qu et al., 2020a).
In many MARI. applications, there are a large number of agents that are deployed as partially connected network and collaboratively make decisions to optimize the globally averaged return, such communication networks (Kim al., 2019) and traffic signal control (Wei et al., 2019). To deal with such scenarios, networked MARL formulated to decompose the dependency among all agents into dependencies between only neighbors. To avoid decision-making with insufficient information, agents are permitted to exchange messages with neighbors over the network. In such settings, is feasible for agents learn to make decisions in decentralized way (Zhang et al., 2018; Qu et al., 2020b). However, there are still difficulties of dependency if anyone attempts to make decisions independently, e.g., prisoner's 's dilemma and tragedy of the commons (Pérolat al., 2017). Existing methods tackle these problems by consensus update of value function (Zhang et al., 201 8), credit assignment (Wang al., 2020), reward shaping (Chu al., 2020). However, these methods rely
Corresponning Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

either access to the global state and joint action (Zhang et al., 2018) or hand-crafi reward functions (Wang et al., 2020; Chu et al., 2020).
Inspired by the fact that sharing plays a key role in human's learning of cooperation (Fisenberg and Mussen, 1989), we propose learning to share (LToS), hierarchically decentralized learning framework for networked MARL. LToS enables agents to learn to dynamically share reward with neighbors so as collaborativlly optimize the global objective. The high-level policies decompose the global objective into local ones by determining how to share their rewards, while the low-level policies optimize local objectives induced by the high-level policies. LToS learns in a decentralized manner, and we prove that the high-level policies are mean-field approximation of the joint high-level policy, Moreover, the high-level and low-level policies form a bi-level optimization and alternately learn to optimize the global objective.
L.TS is general hierarchical framework for networked MARL and can be easily realized by diverse combinations of RL algorithms. We currently implement LToS by DDPG (Lillicrap et al., 2016) as the high-level policy and DGN (Jiang et al., 2020) as the low-level policy. We empirically demonstrate that LToS outperforms existing methods for networked MARL in both social dilemma and networked MARL scenarios.
2 Related Work
There are many recent studies for collaborative MARL. Most adopt CTDE (Rashid et al., 2018; Wang et al., 2021; Zhang et al., 2021; Su and Lu, 2022). Many of them are constructed on the basis of factorizing the joint Q-function (Rashid et al., 2018; Wang et al., 2021). However, these factorized methods suffer from the overgeneralization issue (Castellini et al., 2019). Other studies focus more on decentrulized training, to which our work is more closely related, as summarized as follows.
Zhang et al. (2018) and Qu et al. (2019) proposed consensus update of local value functions, where each agent keeps a local copy of the global value function but is assumed to have global information. Qu et al. (2020a) proposed intention propagation between agents, where each agent updates its policy based on intentions shared by other agents, but the policy may converge slowly due to propagated intentions over the network. Qu et al. (2020b) and Lin et al. (2020) investigated the exponential decay property, Le., the impact of agents on each other decays exponentially graph distance, while Chu et al. (2020) introduced a spatial discount factor to capture the influence between agents, which remains hand-tuned. However, none of these studies provide an explicit mechanism to solve social dilemmas in networked MARL.
Networked MARL.
their
Reward Design. Hostallero al. (2020) aimed maximizing social welfare, but they simply used temporal difference error for reward shaping. As temporal difference error in deep RL hardly converges to zero, it still biases the optimization objective. Mguni et al. (2019) added an extra part to the original reward as non-potential based reward shaping and used Bayesian optimization to induce the convergence to desirable equilibrium between agents. However, the extra part remains fixed during an episode, which makes it less capable of dealing with dynamic environments. Moreover, the reward shaping alters the original optimization problem. Hughes et al. (2018) proposed the inequity aversion model to balance agents' selfish desire and social fairness. Wang et al. (2020) considered learning the Shapley value as the credit assignment. However, these methods still rely on hand-crafted reward designs. Lupu and Precup (2020) added gifting as an extra action into the original MDP to modify the MDP and objective. Yang et al. (2020) proposed that each agent learns an incentive function and optimizes the policy in terms of both reward and incentives given by other agents. Obviously, both methods alter the original objective of optimization.
Unlike existing work, LToS enables agents to learn to dynamically share reward with other agents without the hias of the optimization objective such that they can collaboratively optimize the global objective in networked MARL.
3 Background
3.1 Networked Multi-Agent Reinforcement Learning
Assume N agents interact with an environment. Let V 21, .N) be the set of agents. The multi-agent system is modeled as an undirected graph G(V, E), where each agent i serves vertex
2

---

i and & V x is the set of all edges. Two agents i,j V can communicate with each other if and only if eij ji E E. We denote agent i and its all neighbors the graph together as a set Ng The state of the environment E S transitions upon joint action a E A according to transition probability Pa
S + [0, 1], where joint action set A XievA, Each agent i has a policy x; E IT, S A, > (0, 1.. and we denote the joint policy of all agents л E IT = X (Zhang et al., 2018). For networked MARL, a common and realistic assumption is that the reward of each agent i just depends on its action and the actions of its neighbors (Qu et al., 2020a), i.e., ri(s, a) = (s, ax.). Moreover, each agent may only obtain partial observation o, E O,, but can approximate the state by the observations of Ns (Jiang et al., 2020) the observation history (Chu et al., 2020), which are all denoted by o, for simplicity. The global objective is to maximize the sum of cumulative rewards of all agents, i.e., Ei=o Ei=1 7"1
A
3.2 Markov Game
In such a setting, each agent could individually maximize its own expected return, which known as the Markov game. This may lead to stable outcome or Nash equilibrium, which however is usually sub-optimal terms of the global objective. Given the joint policy æ, the value function of agent i is given by
(s) Σ
T (als) E s, a)lr + (s))l
where Pa E Pa describes the state transitions. Then, Nash equilibrium defined as (Mguni et al., 2019)
(s) 2 2u (6), Vay E
where --; = T.)
4 Method
LToS is a decentralized hierarchy. At each agent, the high-level policy determines the weights of reward sharing based low-level policies while the low-level policy directly interacts with the environment to optimize the local objective induced by the high-level policies. Therefore, they form a bi-level optimization and alternately learn towards the global objective.
4.1 Reward Sharing
The intuition of reward sharing is that if agents share their rewards with others, each agent has to consider the consequence of its actions on others, and thus it promotes cooperation. In networked MARL, as the reward of an agent is assumed to depend on the actions of neighbors, we allow reward sharing only between neighboring agents. This is because the change of actions of neighbors directly affects the reward while the agents outside the neighborhood can only affect the return of the agent indirectly by the change of state distribution. Moreover, this also fits the setting of networked MARL. with restricted communication between neighbors.
For the graph of V, we additionally define set of directed edges, D. constructed from E. Specifically, we add loop dæ E 'D for each agent i and split each undirected edge eij E & into two directed edges: dej (i and d,; (j. E D. Each agent determines a weight way E [o. 1] for each directed edge dzy. Vj E N₁, subject to the constraint ΣjEN, Waj = 1, so that Wsy proportion of agent i's environment reward ry will be shared to agent j. Let w E W = X Xd., ED be the weights of the graph. Therefore, the shaped reward after sharing for each agent is defined
Σ WHTjj. JEN,
(2)
3

---

4.2 Hierarchy
Assume there is joint high-level policy + S x W [0. determine w. Given @ and w, we can define the value function of at each agent based on (1)
" (s: Φ) Σ Φ(1l/1 ())
π (aass, w) Σ Pa 11 + +o, (d':
(3)
Σ T w) E Epols'ss,albz + (s':
2 (s: w,
It is noteworthy that w is multidimensional action for an allocation scheme rather than a probability distribution. In our derivation, we express ur as a discrete action for simplicity. also holds for continuous action as long as we change all the summations to integrals. Let V (s; = Eiev ", (s; Φ) and (s,,, uz; ς = Eiev u. (s; (. Φ).
Proposition 4.1. Given n,, V/ (s; πT) and (5 w; T) are respectively the value function and action- value function of Φ.
Proof. The proof is deferred Appendix A.
Proposition 4.1 implies that & directly optimizes the global objective by generating w, given π. Unlike existing hierarchical RL methods, can directly construct the value function and action-value function of & based on the value function of π at each agent.
As Φ optimizes the global objective given while ni optimizes the shaped reward individually at each agent given & (assuming π convergent to Nash equilibrium or stable outcome, denoted as lim), they form bi-level optimization. Let (T(T) and (Ø) denote the objectives of &b and π respectively. The bi-level optimization can be formulated as follows,
max +
J (#))
s.l. T T(()) arg lim Jn (Φ))
4.3 Decentralized Learning
We start from collective learning to achieve global optimization of average reward. So far, the joint high-level policy is still in centralized form. Note that the scenario needs a decentralized method and each agent has its own reward. Now we turn to learning the joint high-level policy in a decentralized way. Let with! = [wayl j E } and w - (wiel E Mi. The following proposition proves each agent's independence of each other on the high level.
Proposition 4.2. The joint high level policy &o can be learned in a decentralized manner, and the decentralized high-level policies of all agents form a mean-field approximation of &.
Proof. The proof is deferred to Appendix A.
Proposition 4.1 and 4.2 indicate that for each agent i, the low-level policy simply leams local [a, w to optimize the cumulative reward of r since r is fully determined by w," according (2) and denoted as r from now on. And the high-level policy de just needs to locally determine witl to optimize the cumulative reward of r
Therefore, for decentralized learning, (4) can be decomposed locally for each agent i as max di Jo, (0-,, x (
(x))
(5)
s.t. (Φ)) arg max Jz, (T (TT)
Now we use max instead of lim because local policies can be compared and improved in a de- centralized manner in a Markov game. We abuse the notation and let ø and x also denote their parameterizations respectively. To solve the optimization, we have
V Jo, (0-1, (Ø). -((())
(6)
V Vo, Jo, (0-n, aV Jn, (0), ... .......T AN + a V JaN (+)),

---

where a is the learning rate for the low-level policy, Let n. denote Tú +
we have
Vo, Jo. (8 ii()), -.,()),),))
2 Vo, ... , + + >
V2 Bu, Jn, (0VV,
,
The second-order derivative is neglected due to high computational complexity, without incurring significant performance drop such as meta-learning (Finn al., 2017) and neural architecture search (Liu et al.. 2019). Differently, our low-level policy requires more than one gradient step until convergence. Similarly, we have
V nR (T ( (T(),, (Tv(TT) V (T, (T- VV Jos (TT) ON VV Jox (T));
where 3 is the learning rate of the high-level policy. Therefore, solve the bi-level optimization (4) by the first-order approximations in a decentralized way. For each agent i, di and मं are alternately updated.
In distributed learning, as each agent i usually does not have access to state, further approxi- mate ói(w (u("""))s s) and n, (a,ass, w(")) by di (u (o)) and network
agent
neighbors
Φ (wufnt (o.)
(a, w respectively. Moreover, MARL each agent i is closely related to neigh- boring agents, (5) can be further seen as т. maxi- mizes the cumulative discounted reward of r given ΦN. where ΦN = XJeA; Φj. and o, equivalently optimizes the global objective given TM, where TA = XJE,, π. During training, TA, and d.v, are implicitly considered by interactions of withl and w." respectively. The architecture of LToS is illustrated in Figure At each timestep, the high-level policy of each agent makes decision of action wittt the weights of reward sharing based on the observation. Then, the low-level policy takes the observation and
noighbors
xy (a:la, (a, w
gradient flow flow
environment
Figure 1: L.TS
with as an input and outputs the action. Agent ob-
tains the shaped reward according to with for both the high-level and low-level policies. The gradients are backpropagated along purple dotted lines.
Further, from Proposition 4.1, we have: "s, ; v, (s; dx,), where gf is the action-value function of & given TN; u²" is the value function of m, given ΦAa; and conditioned on w.t As aforementioned, we approximately have qf" (o,,,,")) w;'") 2, (o;; w.")). We can see that the action-value function of o; equivalent to the value function of m;. That said, we can use a single network to approximate these two functions simultancously. For deterministic low-level policy, the high-level and low-level policies can share the same action-value function. the current instantiation of I.ToS, we use DGN (Jiang et al., 2020) (Q-learning) for the low-level policy and DDPG (I (Lillicrap al., 2016) for the high-level policy. Thus, the Q-network of DGN also serves as the critic DDPG, and the gradient of w."h is calculated based on the maximum Q-value of
For completeness, Algorithm 1 (see Appendix gives the training procedure of LToS based on DDPG and DGN. More discussions about training LToS are also available in Appendix C. The code of LToS available at https /ttpttggttttttt// com PKU- RL/ Roaadentts.
5 Experiments
For the experiments, we adopt three scenarios fissoner,jjugle and traffic depicted in Figure 2, where prisoner and jungle (Jiang et al., 2020) are grid games about social dilemma that easily measures agents' cooperation, while traffic is realistic scenario of networked MARL. We obey the principle of networked MARL that only allows communication in neighborhood Zhang et (2018) and Chu et al. (2020).
5

---

dolcct+
cooperate
cooperato
(a) prisoner
rever altching ugent
reward lor allacke
meadd of for saci
(b) jungle
(c) traffic
Figure 2: Three experimental scenarios: (a) prisoner, (b) jungle, and (c) traffic.
To illustrate the reward sharing scheme each agent learned, reward proportion that an agent chooses to keep for itself. For ablation, we keep the sharing weights fixed for each agent, named fixed LToS. Throughout the experiments, we additionally compare with the baselines including DQN and DGN, where DGN also serves the ablation of LToS without reward sharing DGN is the low-level policy of LToS. To maximize the global return directly by centralized learning, we use QMIX (Rashid et al., 2018) as a baseline throughout the three scenarios and two other ones in prisoner. Moreover, LToS aims to bring harmonious cooperation by reward sharing in networked MARL, we compared LToS to three methods for networked MARL, i.e., ConseNet (Zhang al., 2018), NeurComm (Chu et al., 2020) and Intention Propagation (abbreviated IP) (Qu et al., 2020a), and LIO (Yang et al., 2020) for incentivized learning, all of which use recurrent neural network (RNN) or graph neural network (GNN) for the partially observable environment. More details of hyperparameters are available in Appendix D.
simple indicator: selfishness, the
5.1 Prisoner
We use prisoner, a grid game version of the well-known matrix game prisoner's dilemma from Sodomka et al. (2013) empirically demonstrate that LToS is able learn cooperative policies to achieve the global optimum (i.e., maximize globally averaged return). As illustrated in Figure 2a, there are two agents A and that respectively start on two sides of the middle of a grid corridor full observation. At each timestep, each agent chooses an action left or right and moves to the corresponding adjacent grid, and every action incurs cost -0.01. There are three goals, two goals at both ends and one in the middle. The agent gets reward +1 for reaching the goal. The game ends once some agent reaches goal or two agents reach different goals simultaneously. This game resembles prisoner's 's dilemma: going for the middle goal ("defect") will bring more rewards than the farther one on its side ("cooperate"), but if two agents both adopt that, a collision occurs and only one of the agents wins the goal with equal probability. On the contrary, both agents obtain higher return they both "cooperate", though takes more steps. The highest possible return is 1.
Figure 3a illustrates the learning curves of all the methods in terms of average return. Note that for all three scenarios, we present the average of training runs with different random seeds by solid lines and the min/max value by shadowed areas. As result of self-interest optimization, DQN converges to the "defect/defect" Nash equilibrium where each agent receives an expected reward about 0.5. So does DGN since it only aims to take advantage of its neighbors' observations while prisoner is fully observable cnvironment already. Given maximize average return, NeurComm and fixed LToS agents are able to cooperate eventually. So are ConseNet and QMIX. However, they converge slowly. In contrast, IP agents Iearn at a slower pace and its performance is only little higher than 0.5. LIO agents cooperate soon enough at the beginning, but they cannot form steady cooperation and perhaps need longer training to get rid of such instability.
reward shaping factor to direct agents to
JointDQN, Coco-Q (Sodomka et al., 2013), and LToS perform similarly and outperform other methods. JointDQN is one centralized DQN that takes control of joint actions of both agent A and B, and thus should be able to achieve the best performance but still takes time to converge even such simple two-agent scenario. As modified tabular Q-learning method, Coco-Q introduces the coco

---

6.15
6.00
DOX
4.0
250 506
150
3050
150
1700
(a) prisoner
(b) jungle
(c) trafficc
Figure 3: Learning curves in (a) prisoner, (b) jungle, and (c) traffic. All the curves are plotted using 5 training runs with different random seeds, where the solid line is the mean and the shadowed area is enclosed by the min and max value.
value (Kalai and Kalai, 2010) as substitute for the expected return in the Bellman equation and regards the difference as transferred reward. However, it is specifically designed for some games, and it is hard to be extended beyond two-player games. LToS can learn the reward sharing scheme where one agent at first gives all the reward to the other so that both of them are prevented from "defect", and thus achieve the best average return quickly, as observed in the experiment. By prisoner, we verify that LToS can escape from local optimum by learning to share reward.
5.2 Jungle
Jungle is a scenario about moral dilemma proposed by Jiang et al. (2020) based on MAgent (Zheng al., 2017). As illustrated in Figure 2b. there are N agents and L stationary foods. At each timestep, each agent can attack or move to one adjacent grid. Eating (attacking food) brings a positive reward +1. but attacking other agents obtains a higher reward +22 The victim, however, suffers a negative reward -4, which makes each attack between agents a negative-sum action. Moreover, attacking a blank grid gets small negative reward - -0.01 (inhibiting excessive attacks). We follow the original setting of Jiang et al. (2020): map size = 30 x 30 grids, N = 20, L = 12, and the observation consists of one's coordinates and field of 11 x 11 grids nearby. Each agent has closest agents as its neighbors. Compared to prisoner, jungle has much more agents, and thus JointDQN and Coco-Q are disregarded for this scenario. Another challenge of jungle that the network topology is dynamic since each agent can always move. Fortunately, the topology change slowly and predictably, and algorithms may get the varying neighbor set Mf as part of the input. Therefore, it still likely to estimate the shaped rewards and value functions well.
Table 1: Average reward per step of all the methods in jungle.
DQN DGN fixed LToS
LToS
NeurComm
ConseNet
LIO IP
QMIX
upper bound
0.24
0.66
0.71
0.86
-0.05
-0.04 0.00
0.63
0.04
0.95
Figure 3b illustrates the learning curves of all the methods, and their performance after convergence also summarized in Table 1. NeurComm, ConseNet and QMIX do not perform well in this task. In NeurComm, each agent gets a "delayed global information". However, a stable pattern of delayed global information cannot be formed when the communication is conducted via dynamic topology. ConseNet is constructed on the basis of a premise of full observation, and it can hardly learn well when the input is not only partial but also fairly varying sequence and content. LIO agents also perform badly, since they cannot be distinguished from one another in the dynamic topology and thus fail to learn a proper incentive function. Moreover, LIO requires opponent modeling, but is hard to simultaneously model all other agents in dynamic environment. QMIX is free from these problems. While aiming at global optimization, like LIO, realizes that attacking usually means a negative-sum action, but as a result, it avoids attacking as well as eating most of the time and thus only achieves a reward slightly higher than O. Another possible reason to explain the performance QMIX is its scalability. As there are 20 agents in the scenario, it can be hard to learn the joint action-value function to directly optimize the average return (Qu al., 2020a). Also, we can see that a fixed reward sharing scheme does not bring any gain over DGN. This is because fixed reward sharing does
7

---

Caatin 2004
Nottttin whin
Co
atroriion swidios
(a) NeurComm
(b) ConseNet
(c) IP
(d) DGN
(c) LTbS
Figure 4: Representative behaviors of agents learned by (a) NeurComm, (b) ConseNet, (c) IP, (d)
DGN, and (e) LToS in jungle.
not adapt to the dynamic topology. By learning proper reward sharing and adjusting to changing circumstances, L.TS outperforms all other baselines. Note that there is an upper bound for average reward per step for jungle. By estimating the average distance between each agent and the food that the closest to it at the beginning of one episode, we can give a loose upper bound around 0.95 to reflect our improvement.
# agents
DGN
fixed LToS
LToS
upper bound
10
0.52
0.63
0.71
0.93
20
0.66
0.71
0.86
0.95
30
0.77
0.79
0.88
0.96
0.84
0.80
0.90
0.96
50
0.86
0.79
0.91
0.97
(a) statistics of performance
(b) line graph of performance
Figure 5: Average reward per step of methods in jungle with different number of agents.
Figure 4 illustrates the representative behaviors agents learned by difference methods. For NeurComm, ConseNet (and the same with LIO), most agents are not even close to the foods, so as to avoid being attacked. Even though, there are still conflict and sneak attack between agents sometimes. IP and DGN agents learn much better, but agents may still be cautious about each other, which leads hesitation when they are near the same food. LToS agents learn to properly share the food even if the foods are close (i.e., agents are casy to be attacked) as depicted in Figure 4e and demonstrate much better cooperation than the agents learned by other methods. The experimental results jungle verify that L.TSS can also adapts considerably varying topology in networked MARL..
Besides, we compared LToS and ablation baselines (i.e., DGN and fixed LToS) with different number of agents (i.e., from 10 to 50) to verify the scalability of LToS. All the setting remains the same except that the number of agents and food grows proportionally (i.e., #agents/ # foods 5/3). As depicted in Figure 5, LToS can always achieve the best performance as the agent population size increases.
neighbors
LToS
0.83
2
0.84
0.86
0.66
0.80
U00 mopoooss-1
(a) performance
(b) learning curves
Figure 6: Average reward per step of LToS in jungle with different number of neighbors.
We also conducted study on the impact of graph density. To specific, we run LToS with different number of neighbors (i.e., from I to 5). As illustrated is Figure 6, the number of neighbors indeed affects the performance. With the increase of neighbors, the performance increases first and then decreases in indefinitive way. We think the reason may be that when the graph density increases.

---

the learning difficulty also increases as the agent needs to decide how to share rewards with more neighbors. Beyond certain threshold, the learning difficulty outweighs the gain. This is similar to learning to communicate, where the graph density increases, the gain of communication decreases due to learning difficulty (Ding et al., 2020). However, as networked MARI. the graph structure typically assumed to be sparse and given, this should not be a problem.
5.3 Traffic
In traffic, as illustrated in Figure 2c, we aim to investigate the capability of LToS in dealing with highly dynamic environment through reward sharing. We adopt the same problem setting as Wei et al. (2019). In a road network, each agent serves as traffic signal control at an intersection. The observation of an agent consists of one-hot representation of its current phase (directions for red/green lights) and the number of vehicles on each incoming lane of the intersection. At each timestep, an agent chooses a phase from the pre-defined phase set for the next time interval, i.e., 10 seconds. The reward is set to be the negative of the sum of the queue lengths of all approaching lanes at current timestep. The global objective is to minimize average wait time of all vehicles in the road network, which is equivalent to minimizing the sum of queue lengths of all intersections over an episode (Zheng et al., 2019). The experiment was conducted on traffic simulator, CityFlow (Zhang et al., 2019). We use 6 x 6 grid network with 36 intersections. The traffic flows were generated to simulate dynamic traffic flows including both peak and off-peak period, and the statistics are summarized in Table 2.
Table 2: Statistics of traffic flows
Time (second)
Arrival Rate (vehicles/s)
0 600
600 - 1, 200
1/4
1, 200 - -1,800
1 1/3
1. 1,800 - 2,400
2
2. 2,400 - 3, 3,000
1/5
3,000- - 3, 600
1/2
Table 3: Average number of red lights one vehicle waits for at per intersection of methods in traffic.
Network
DQN
DGN
fixed LToS
LToS
NeurComm
ConseNet
LIO
IP
QMIX
lower bound
6 6
0.90
0.78
0.80
0.58
0.71 2.27
0.78
7.74
4.67
5.94
0.50
Shenzhen
13.99
2.02
1.94
1.71
7.63
19.15
5.52
19.74
1.50
For better demonstration, we choose to show the normalized metric of wait time: the average number of red lights one vehicle waits for at per intersection, and we can also give a loose lower bound 0.50 to reflect our improvement. Figure 3c shows the learning curves of all the methods in terms of that in logarithmic form. The performance after convergence is summarized in Table 3, where LToS outperforms all other methods. LToS outperforms DGN, which demonstrates the reward sharing scheme learned by the high-level policy indeed helps to improve the cooperation of agents. Without the high-level policy, i.e., given fixed sharing weights. fixed LToS does not perform well in dynamic environment. This indicates the necessity the high-level policy. The performance of IP agents increases so slowly that they cannot converge efficiently. Although NeurComm and ConseNet both take advantage of RNN for partially observable environments. LToS still outperforms these methods, which verifies the great improvement of LToS in networked MARL. QMIX is contined to suboptimality (Mahajan et al., 2019). As observed in the experiment, QMIX tries to release traffic flows from one direction while stopping flows from the other direction all the time, because this will only make two rows of intersections on the border blocked but keep most of the intersections from any traffic jam all the time. However, the global optimality actually does not need to be constructed on the sacrifice of anyone. Some similar thing happens to LIO. It is likely because LIO contains some sensitive parameters (Yang et al., 2020) and agents are hard to learn and coordinate their incentive functions since the original reward functions change acutely once an improper operation causes traffic congestion. An introduction of some explicit coordination mechanism may also alleviate the problem, like that of NeurComm and ConseNet.
We visualize the variation of selfishmess of all agents during an episode in trafiic in Figure 7a and 7b. Figure 7a depicts the temporal variance of selfishness for each agent. For most agents, there are two valleys occurred exactly during two peak periods (i.e., 0 O- 600s and 1, 800 - 2, 400s). This is because for heavy traffic agents need to cooperate more closely, which can be induced by being less selfish. We can see this from the fact that selfishness is even lower in the second valley where the traffic is even heavier (i.e., vs. 1 vehicles/s). Therefore, this demonstrates that the agents learn to adjust their extent of cooperation to deal with dynamic environment by controlling the sharing weights.

---

OTT 1200
406
1
1
614
2408
(a) temporal pattern
(b) spatial pattern
Figure 7: Patterns of selfishness traffic.
Figure 7b shows the spatial pattern of selfishness different timesteps, where the distribution of agents is the same as the road network in Figure 2c. The edge and inner agents tend to have very different selfishness. periods, while they diverge and present cross-lik like patterns during peak periods. This shows that handling heavier traffic requires more diverse reward sharing schemes among agents to promote more sophisticated cooperation.
addition. inner agents keep their selfishness more uniform during off-
another road network that is part of Shenzhen, China with 33 intersections as illustrated in Figure 8, and one-hour real traffic flows (Xu al., 2022). The performance is also summa- rized in Table 3. It is shown that LToS still outperforms the baselines by large margin. The experimental results in traffic verify that LToS can also handle highly dynamic environment in networked MARL.
In addition,
Figure Shenzhen network
Conclusion
In this paper, we proposed LToS, hierarchically decentralized framework for networked MARL. LToS enables agents to share reward with neighbors so as to encourage agents to cooperate on the global objective through collectives. For each agent, the high-level policy learns how to share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize the local objective induced by the high-level policies in the neighborhood. Experimentally, we demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenario across scales.
Acknowledgments and Disclosure of Funding
This work is supported in part by NSF China under grant 61872009, Shenzhen Fundamental Research Program (GXWD 20201 231 165807000-220000866 00000007772200000006666665533333 63656003), and Peng Cheng Lab. The authors would like to thank the anonymous reviewers for their valuable comments.
References
Jacopo Castellini, Frans A Olichock, Rahul Savani, and Shimon Whiteson. The representational capacity of action-value networks for multi-agent reinforcement learning. In AAMAS, 2019.
Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. In ICLR, 2020.
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. NeurIPS, 2020.
Nancy Fisenberg and Paul Henry Mussen. The roots of prosocial behavior children. Cambridge University Press, 1989.
10

---

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Mgdel-agmostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.
M. Hammersley and P. Clifford. Markov fields on finite graphs and lattices. unpublished manussir manuscript, 1971.
David Earl Hostallero, Daewoo Kim, Sangwoo Moon, Kyunghwan Son, Wan Kang, and Yung Yi. Inducing cooperation through reward reshaping based on peer evaluations in deep multi-agent reinforcement learning. In AAMAS, 2020.
Edward Hughes, Joel Z. Leibo, Matthew G. Phillips, Karl Tuyls, Edgar A. Duéñez-Guzzáne Anto- nio Garefa Castañeda, lain Dunning, Tina Zhu, Kevin R. McKee, Raphael Koster, Heather Roff, and Thore Graepel. Inequity aversion improves cooperation in intertemporal social dilemmas. In NeurlPS, 2018.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. ICLR, 2020.
Adam Tauman Kalai and Ehud Kalai. Cooperation and competition in strategic games with private information. In EC, 2010.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning schedule communication in multi-agent reinforcement learning. In ICLR, 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in time-varying networked systems, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019.
Andrei Lupu and Doina Precup. Gifting in multi-agent reinforcement learning. AAMAS, 2020.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In NeurlPS, 2019.
David Mguni, Joel Jennings, Emilio Sison, Sergio Valcarcel Macua, Sofia Ceppi, and Enrique Munoz de Cote. Coordinating the crowd: Inducing desirable equilibria in non-cooperative systems. In AAMAS, 2019.
Julien Pérolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common pool resource appropriation. NeurlPS, 2017.
Chao Qu, Shie Mannor, Huan Xu, Yuan Qi, Le Song, and Junwu Xiong. Value propagation for decentralized networked deep multi-agent reinforcement learning. In NeurIPS, 2019.
Chao Qu, Hui Li, Chang Liu, Junwu Xiong, James Zhang, Wei Chu, Yuan Qi, and Le Song. Intention propagation for multi-agent reinforcement learning. arXiv:2004.08883 2020a.
Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. In NeurIPS, 2020b.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018

---

Eric Sodomka, Elizabeth Hilliard, Michael Littman, and Amy Greenwald. Coco-q: Learning in stochastic games with side payments. In ICML, 2013.
Kefan Su and Zongqing Lu. Divergence-Regularizeed Multi-Agemt Actor-Critic. ICML, 2022.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex ducling multi-agent q-learning. In ICLR, 2021.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. In AAAI, 2020.
Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. In CIKM, 2019.
Bingyu Xu, Liwen Zhu, Yuxuan Yi, Zongging Lu, and other contributors. Roadnetsz. https: //github.
Www.RRRoRooaadtSZ 2022. Accessed: 2022-05-01.
Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning incentivize other learning agents. In NewrIPS, 2020.
Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In WWW, 2019.
Kaiqing Zhang. Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Baar. Fully decentralized multi- agent reinforcement learning with networked agents. In ICMI,, 2018.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maxinum-entropy multi-agent reinforcement learning. ICML, 2021.
Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash Gayah, Kai Xu, and Zhenhui Li. Diagnosing reinforcement learning for traffic signal control. Arx5::1905.0470 04716, 2019.
Lianmin Zheng, Jiacheng Yang. Han Cai. Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence, 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] As we just propose new reinforcement learning method, we do see any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] All our experiments do not claim for much computation.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 'rure using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "gfMyGL7fl3M", "reviewer": "Reviewer_cVVf", "summary": "This paper considers a networked multi-agent reinforcement learning (MARL) setting, where the goal is to maximize the sum of discounted cumulative rewards of agents, where at each step, each agent's reward depends on its own action, as well as that of its neighbors in the network graph. A hierarchical scheme is proposed, where the high-level policy learns what fraction of each agent's reward to be shared with its neighbors, while the low-level policy is trained to maximize the agent's local objective induced by the high-level policy. Numerical results on three different multi-agent environments demonstrate the efficacy of the proposed method when compared to multiple state-of-the-art MARL baselines in the literature.", "questions": "- I believe a line of work on using graph neural networks in MARL is missing from the Related Work Section, including, but not the following:\n\n  - Liu, Yong, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. \"Multi-agent game abstraction via graph attention neural network.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 7211-7218. 2020.\n  - Liu, Iou-Jen, Raymond A. Yeh, and Alexander G. Schwing. \"PIC: permutation invariant critic for multi-agent deep reinforcement learning.\" In Conference on Robot Learning, pp. 590-602. PMLR, 2020.\n  - Naderializadeh, Navid, Fan H. Hung, Sean Soleyman, and Deepak Khosla. \"Graph convolutional value decomposition in multi-agent reinforcement learning.\" arXiv preprint arXiv:2010.04740 (2020).\n  - Shen, Siqi, Yongquan Fu, Huayou Su, Hengyue Pan, Peng Qiao, Yong Dou, and Cheng Wang. \"Graphcomm: A graph neural network based method for multi-agent reinforcement learning.\" In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3510-3514. IEEE, 2021.\n  - Li, Sheng, Jayesh K. Gupta, Peter Morales, Ross Allen, and Mykel J. Kochenderfer. \"Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning.\" In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 764-772. 2021.\n  - Siu, Chapman, Jason Traish, and Richard Yi Da Xu. \"Dynamic Coordination Graph for Cooperative Multi-Agent Reinforcement Learning.\" In Asian Conference on Machine Learning, pp. 438-453. PMLR, 2021.\n\n- Does the last term in the second equation in (3) need to be replaced by $v_i^{\\boldsymbol{\\pi}}(s';\\boldsymbol{w},\\phi)$?\n\n- It is slightly unclear whether, at the end of the day, the average **shaped** reward is being optimized over the entire network or the average **original** reward.\n\n- Please define the *selfishness* metric precisely using the notion already introduced in the paper.\n\n- Following up on the above comment, the selfishness metric is an aggregate metric, which does not highlight more details on the learned reward sharing mechanism. Are there any deeper insights on the learned high-level policies that the authors can share in terms of what the weights $\\\\{w_{ij}\\\\}_{ij \\in \\mathcal{E}}$ depend on, for example, are they correlated with how much agents are geographically close, etc.?\n\n- Could you please add more details on how the outer bounds for the *jungle* and *traffic* environments (i.e., upper and lower bounds, respectively) were derived?", "limitations": "The authors describe some of the limitations of implementing the proposed method, but as I mentioned above, I would also love to see a discussion of how the graph construction impacts the performance of the proposed method. Also, as MARL, in general, might have potential negative societal impacts, it would be helpful to also add those as potential application domains where LToS might enable such negative impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and easy to read. The considered problem is timely and interesting, and the proposed solution shows promising performance in practice. In my opinion, the considered number of environments and baselines to compare with in the experiments is a great positive point of the paper.\n\nThat being said, the main weakness of the paper I can think of is the construction of the network graph. It seems that in the experiments, a nearest-neighbor type of graph is used, where each agent is connected to a few neighboring agents in its physical vicinity. However, such a geometric and symmetric graph construction method might be suboptimal in practice. In general, I wonder how the graph should be constructed and whether there is an automatic way to do so. Some ablation studies on the impact of graph density (for example, controlled by the number of neighbors per node) might be helpful to shed some light on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fv7bdD0W361", "reviewer": "Reviewer_gcHc", "summary": "This paper considers networked cooperative MARL settings where agents can communicate with a set of neighboring agents. The authors propose an approach for sharing rewards among neighboring agents in which a policy for sharing rewards between each neighboring node is learned along with the main policy learning task within a bilevel alternating optimization framework. A series of multi-agent experiments are conducted while learning the main policy leveraging a DQN with a reward sharing policy based on DDPG. The proposed LToS framework outperforms a set of MARL baselines on the prisoner, jungle, and traffic domains considered for evaluation. Additionally, LToS is demonstrated to learn a more effective conditional reward sharing policy than is learned by a fixed reward sharing policy baseline. ", "questions": "As I am very interested in the comparison with the fixed or average reward sharing approach, I had some clarifying questions to ask about the \"fixed LToS\" baseline:\n- For “fixed LToS” what are the sharing weights fixed at? Average over all neighbors? \n- Performance improvements on the “prisoner” domain are not terribly clear. Why does fixed LTOS perform so poorly relative to the DQN baseline? I would have expected the same or better performance because of reward sharing, why would it be so much worse? \n- Can you explain this comment about fixed LTOS? I am having a hard time understanding what is meant by this seemingly crucial point: “This is because fixed reward sharing does not adapt to the dynamic topology. By learning proper reward sharing and adjusting to changing circumstances, LToS outperforms all other baselines.” ", "limitations": "It would be nice to expand a bit more in the main text about the significance of proposition 4.2 particularly with reference to the Deep MARL based experiments conducted.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n- The overall networked communication setting of this paper is an interesting one, and learning a reward sharing policy seems like a potentially important operation in this setting. I am not aware of any other paper that explicitly learned to share rewards. \n- Proposition 4.2 seems to be a potentially interesting result. \n- Their are a nice set of experiments across a variety of multi-agent domains that illustrate the performance of the proposed framework.\n\nWeaknesses\n- The paper does not compare empirically or theoretically with other frameworks that learn to share in cooperative MARL such as action advising [1], goal-state advising [2], or sharing abstract messages [3, 4, 5]. It seems like reward sharing is only one aspect of what could potentially be useful to learn to share along with states, actions, and more abstract forms of communication. \n- The motivation for learning dynamic weights for sharing rewards over time was not made clearly and missing details about how the fixed weight sharing was implemented made it hard to know what conclusions to draw from the experiments. Given the global objective, why not just send the average local reward to each agent? Can you explain more about how dynamic weight sharing schemes work and why they would be preferable to the best fixed weight reward sharing approach?\n- Minor Comment: Figure 3 is very small. \n\nOverall, I am on the fence about this paper and look forward to receiving a response form the authors to the questions I have raised. I lean slightly towards rejection at the moment given that I am a bit unclear about the motivation for the approach and have questions about the baselines in the experiments. \n\n[1] \"Learning to Teach in Cooperative Multiagent Reinforcement Learning\" Omidshafiei et al., AAAI-2019. \n[2] \"Learning Hierarchical Teaching Policies for Cooperative Agents\" Kim et al., AAMAS-2020. \n[3] \"Emergence of Grounded Compositional Language in Multi-Agent Populations\" Mordatch and Abbeel, AAAI-2018. \n[4] \"Biases for Emergent Communication in Multi-agent Reinforcement Learning\" Eccles et al., NeurIPS-2019. \n[5] \"TarMAC: Targeted Multi-Agent Communication\" Das et al., ICML-2019. \n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FktuShs1pjf", "reviewer": "Reviewer_98u6", "summary": "This paper proposes a hierarchical method for networked multi-agent reinforcement learning, where the agents can share their local rewards with nearby agents over a partially connected network. In the proposed method, each agent optimized its value function in a decentralized manner, using a weighted sum of local rewards shared by its neighbors. To maximize the global objective, it uses a two-level structure of policies: A high-level policy learns optimal weights for shared rewards, and a low-level policy learns to optimize the local objective induced by the weighted sum of shared rewards. Propositions in this paper imply that optimizing individual objectives can achieve optimizing the global objective of the global perspective, and the experimental results demonstrate the effectiveness of the proposed method in three environments.", "questions": "1. Is it fair to compare the centralized training algorithm with the setting suitable for decentralized training? Even if it is, why does QMIX have low performance in the prisoner environment?\n\n2. The authors chose DGN to optimize the low-level policy with shaped rewards. Is there a specific reason for using DGN instead of other algorithms? It would be meaningful to make a novel framework specialized in optimizing the shaped rewards.", "limitations": "1. Networked MARL setting seems to be a subfield of MARL that has recently become popular. In this setting, rewards are given globally or often individually. But I'm not sure how useful the individual reward setting is. The global reward setting is more general and actively studied regardless of centralized or decentralized training. I think the individual reward setting presented in this paper is quite narrow.\n\n2. The proposed method is limited only to applying to environments in which each agent has a fixed number of neighbors. For example, in the jungle environment, each agent has 3 closest agents as neighbors, and the proposed method can be applied to such environments. On the other hand, we can consider a general setting of choosing neighbors, that is, defining neighbors as all agents within a d ball from the agent. Such a definition makes the number of neighbors change over time, and it would be meaningful if the high policies adapt to the time-varying cases.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The hierarchical structure for optimizing the global objective through reward sharing is well organized and may be applied in various scenarios with massive agents in the real world.\n2. Propositions show well that two-level policies trained in a decentralized way can reach global objectives.\n3. Evaluations explain and visualize the effectiveness of the proposed method well.\n\nWeaknesses:\n1. The main idea of this paper, which maximizes the weighted sum of local rewards from its neighbors to enhance cooperative behavior, seems a little lacking in novelty. \n2. All proofs assume a global perspective and the process of bringing this to the decentralized setting is explained poorly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept