Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a hierarchical method for networked multi-agent reinforcement learning, where the agents can share their local rewards with nearby agents over a partially connected network. In the proposed method, each agent optimized its value function in a decentralized manner, using a weighted sum of local rewards shared by its neighbors. To maximize the global objective, it uses a two-level structure of policies: A high-level policy learns optimal weights for shared rewards, and a low-level policy learns to optimize the local objective induced by the weighted sum of shared rewards. Propositions in this paper imply that optimizing individual objectives can achieve optimizing the global objective of the global perspective, and the experimental results demonstrate the effectiveness of the proposed method in three environments.
strengths_and_weaknesses: Strength:
1. The hierarchical structure for optimizing the global objective through reward sharing is well organized and may be applied in various scenarios with massive agents in the real world.
2. Propositions show well that two-level policies trained in a decentralized way can reach global objectives.
3. Evaluations explain and visualize the effectiveness of the proposed method well.

Weaknesses:
1. The main idea of this paper, which maximizes the weighted sum of local rewards from its neighbors to enhance cooperative behavior, seems a little lacking in novelty. 
2. All proofs assume a global perspective and the process of bringing this to the decentralized setting is explained poorly.

questions: 1. Is it fair to compare the centralized training algorithm with the setting suitable for decentralized training? Even if it is, why does QMIX have low performance in the prisoner environment?

2. The authors chose DGN to optimize the low-level policy with shaped rewards. Is there a specific reason for using DGN instead of other algorithms? It would be meaningful to make a novel framework specialized in optimizing the shaped rewards.
limitations: 1. Networked MARL setting seems to be a subfield of MARL that has recently become popular. In this setting, rewards are given globally or often individually. But I'm not sure how useful the individual reward setting is. The global reward setting is more general and actively studied regardless of centralized or decentralized training. I think the individual reward setting presented in this paper is quite narrow.

2. The proposed method is limited only to applying to environments in which each agent has a fixed number of neighbors. For example, in the jungle environment, each agent has 3 closest agents as neighbors, and the proposed method can be applied to such environments. On the other hand, we can consider a general setting of choosing neighbors, that is, defining neighbors as all agents within a d ball from the agent. Such a definition makes the number of neighbors change over time, and it would be meaningful if the high policies adapt to the time-varying cases.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers networked cooperative MARL settings where agents can communicate with a set of neighboring agents. The authors propose an approach for sharing rewards among neighboring agents in which a policy for sharing rewards between each neighboring node is learned along with the main policy learning task within a bilevel alternating optimization framework. A series of multi-agent experiments are conducted while learning the main policy leveraging a DQN with a reward sharing policy based on DDPG. The proposed LToS framework outperforms a set of MARL baselines on the prisoner, jungle, and traffic domains considered for evaluation. Additionally, LToS is demonstrated to learn a more effective conditional reward sharing policy than is learned by a fixed reward sharing policy baseline. 
strengths_and_weaknesses: 
Strengths
- The overall networked communication setting of this paper is an interesting one, and learning a reward sharing policy seems like a potentially important operation in this setting. I am not aware of any other paper that explicitly learned to share rewards. 
- Proposition 4.2 seems to be a potentially interesting result. 
- Their are a nice set of experiments across a variety of multi-agent domains that illustrate the performance of the proposed framework.

Weaknesses
- The paper does not compare empirically or theoretically with other frameworks that learn to share in cooperative MARL such as action advising [1], goal-state advising [2], or sharing abstract messages [3, 4, 5]. It seems like reward sharing is only one aspect of what could potentially be useful to learn to share along with states, actions, and more abstract forms of communication. 
- The motivation for learning dynamic weights for sharing rewards over time was not made clearly and missing details about how the fixed weight sharing was implemented made it hard to know what conclusions to draw from the experiments. Given the global objective, why not just send the average local reward to each agent? Can you explain more about how dynamic weight sharing schemes work and why they would be preferable to the best fixed weight reward sharing approach?
- Minor Comment: Figure 3 is very small. 

Overall, I am on the fence about this paper and look forward to receiving a response form the authors to the questions I have raised. I lean slightly towards rejection at the moment given that I am a bit unclear about the motivation for the approach and have questions about the baselines in the experiments. 

[1] "Learning to Teach in Cooperative Multiagent Reinforcement Learning" Omidshafiei et al., AAAI-2019. 
[2] "Learning Hierarchical Teaching Policies for Cooperative Agents" Kim et al., AAMAS-2020. 
[3] "Emergence of Grounded Compositional Language in Multi-Agent Populations" Mordatch and Abbeel, AAAI-2018. 
[4] "Biases for Emergent Communication in Multi-agent Reinforcement Learning" Eccles et al., NeurIPS-2019. 
[5] "TarMAC: Targeted Multi-Agent Communication" Das et al., ICML-2019. 
  
questions: As I am very interested in the comparison with the fixed or average reward sharing approach, I had some clarifying questions to ask about the "fixed LToS" baseline:
- For “fixed LToS” what are the sharing weights fixed at? Average over all neighbors? 
- Performance improvements on the “prisoner” domain are not terribly clear. Why does fixed LTOS perform so poorly relative to the DQN baseline? I would have expected the same or better performance because of reward sharing, why would it be so much worse? 
- Can you explain this comment about fixed LTOS? I am having a hard time understanding what is meant by this seemingly crucial point: “This is because fixed reward sharing does not adapt to the dynamic topology. By learning proper reward sharing and adjusting to changing circumstances, LToS outperforms all other baselines.” 
limitations: It would be nice to expand a bit more in the main text about the significance of proposition 4.2 particularly with reference to the Deep MARL based experiments conducted.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers a networked multi-agent reinforcement learning (MARL) setting, where the goal is to maximize the sum of discounted cumulative rewards of agents, where at each step, each agent's reward depends on its own action, as well as that of its neighbors in the network graph. A hierarchical scheme is proposed, where the high-level policy learns what fraction of each agent's reward to be shared with its neighbors, while the low-level policy is trained to maximize the agent's local objective induced by the high-level policy. Numerical results on three different multi-agent environments demonstrate the efficacy of the proposed method when compared to multiple state-of-the-art MARL baselines in the literature.
strengths_and_weaknesses: The paper is clearly written and easy to read. The considered problem is timely and interesting, and the proposed solution shows promising performance in practice. In my opinion, the considered number of environments and baselines to compare with in the experiments is a great positive point of the paper.

That being said, the main weakness of the paper I can think of is the construction of the network graph. It seems that in the experiments, a nearest-neighbor type of graph is used, where each agent is connected to a few neighboring agents in its physical vicinity. However, such a geometric and symmetric graph construction method might be suboptimal in practice. In general, I wonder how the graph should be constructed and whether there is an automatic way to do so. Some ablation studies on the impact of graph density (for example, controlled by the number of neighbors per node) might be helpful to shed some light on this.
questions: - I believe a line of work on using graph neural networks in MARL is missing from the Related Work Section, including, but not the following:

  - Liu, Yong, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. "Multi-agent game abstraction via graph attention neural network." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 7211-7218. 2020.
  - Liu, Iou-Jen, Raymond A. Yeh, and Alexander G. Schwing. "PIC: permutation invariant critic for multi-agent deep reinforcement learning." In Conference on Robot Learning, pp. 590-602. PMLR, 2020.
  - Naderializadeh, Navid, Fan H. Hung, Sean Soleyman, and Deepak Khosla. "Graph convolutional value decomposition in multi-agent reinforcement learning." arXiv preprint arXiv:2010.04740 (2020).
  - Shen, Siqi, Yongquan Fu, Huayou Su, Hengyue Pan, Peng Qiao, Yong Dou, and Cheng Wang. "Graphcomm: A graph neural network based method for multi-agent reinforcement learning." In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3510-3514. IEEE, 2021.
  - Li, Sheng, Jayesh K. Gupta, Peter Morales, Ross Allen, and Mykel J. Kochenderfer. "Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning." In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 764-772. 2021.
  - Siu, Chapman, Jason Traish, and Richard Yi Da Xu. "Dynamic Coordination Graph for Cooperative Multi-Agent Reinforcement Learning." In Asian Conference on Machine Learning, pp. 438-453. PMLR, 2021.

- Does the last term in the second equation in (3) need to be replaced by $v_i^{\boldsymbol{\pi}}(s';\boldsymbol{w},\phi)$?

- It is slightly unclear whether, at the end of the day, the average **shaped** reward is being optimized over the entire network or the average **original** reward.

- Please define the *selfishness* metric precisely using the notion already introduced in the paper.

- Following up on the above comment, the selfishness metric is an aggregate metric, which does not highlight more details on the learned reward sharing mechanism. Are there any deeper insights on the learned high-level policies that the authors can share in terms of what the weights $\\{w_{ij}\\}_{ij \in \mathcal{E}}$ depend on, for example, are they correlated with how much agents are geographically close, etc.?

- Could you please add more details on how the outer bounds for the *jungle* and *traffic* environments (i.e., upper and lower bounds, respectively) were derived?
limitations: The authors describe some of the limitations of implementing the proposed method, but as I mentioned above, I would also love to see a discussion of how the graph construction impacts the performance of the proposed method. Also, as MARL, in general, might have potential negative societal impacts, it would be helpful to also add those as potential application domains where LToS might enable such negative impacts.
review_title: null

Meta-review: metareview: While the scores are borderline, reviewers found the paper interesting and the experiments convincing, so I recommend acceptance. While there were originally concerns about the appropriateness of comparison with QMIX and the relationship to related work, I think these were adequately addressed in the rebuttal.
confidence: Less certain
Decision: Accept