Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a theoretical framework that studies Shapley value in the context of Markov Games as a useful technique for value factorization and credit assignment in agents coalitions. Leveraging this framework, the authors proposed Shapley Q-Learning (SHAQ) derived from a novel definition of a Shapley-Bellman Operator. The proposed algorithm is compared with a suite of existing algorithms (COMA, VDN, QMIX) in predator-prey and the StarCraft MA Challenge, contrasting competitive results while showing interesting properties of interpretability.
strengths_and_weaknesses: **Strengths**
1. The paper is well-written and properly motivated. The work is well-placed among the existing and vast literature in Multiagent Reinforcement Learning (MARL).

2. The combination of Shapley's theory with Q-Learning seems a novel contribution in the interesting and always challenging setting of MARL.

**Weaknesses**
1. The experimental section would benefit from a discussion on the interpretability of SHAQ in the predator-prey setting, which seems to be missing in the current manuscript.
questions: 1. Would the authors be able to reproduce the interpretability results offer for SMAC on the simple predator-pray setting?
limitations: The authors addressed the limitation of the work, including the assumptions and restrictions imposed in the scenarios considered.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a new framework and corresponding algorithm to solve value factorization in global reward games. Specifically, it derives the Shapley-Bellman optimality equation from evaluating the optimal Markov Shapley value and proposes the Shapley-Bellman operator to solve it, which is also proved in the paper. Furthermore, Shapley Q-learning is presented to implement the theoretical framework for predator-prey and SMAC environments. 

Contributions: The paper proposes a new theoretical cooperative game framework and Shapley Q-learning algorithm for solving global reward games. Moreover, the authors give proof for the theoretical framework and evaluate SHAQ on Predator-Prey and StarCraft tasks, which shows good performance and interpretability. 
strengths_and_weaknesses: Strength: 
1.	well written, easy to follow
2.	novel cooperative game framework for global reward game justified both theoretically and empirically
3.	well literature review on relevant fields
4.	proof details and codes provided

Weakness:
1.	Figures 1,2,3 are too small to read easily
2.	Improvements seem not significant compared to SOTAs

questions: Authors assume games (Predator-Prey and StarCraft) are satisfied those conditions of MCG in Eq. 1 (line 78), does it always satisfy?
limitations: The assumption in line 78 for Markov convex games looks too strong, is it possible to extend the same results to general cooperative games?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper considers multiagent reinforcement learning in a global (cooeprative) reward game. It contrasts the results value factorization frameworks, and proposes an alternative via the Shapley value from cooperative game theory. Basically, the authors consider a form of game with coalition structures, and apply the Shapley value to decompose the reward, and derive and Shapley-Bellman optimiality equation (SBOE) corresponding to the optimal joint determinisitic policy. They propose a Shapley-Bellman opeator (SBO) that solves for the SBOE. These finally give rise to a new multiagent reinforcement learning algorithm, called Shapley Q learning, SHAQ for short,  somewhat akin to existing value factorization methods. 

Empirically, on a few settings (predator-prey and starcraft)  SHAQ exhibits better performance than existing approaches, and also provides some interpertability foundation. 
strengths_and_weaknesses: The key strength of the paper is in applying cooperative game theory tools to multiagent Q learning. Recently the Shapley value has become a very popular tool in machine learning due to its ability to decompose the performance of a model to the relative influence of specific features. This has proven a very strong tool for analyzing supervised learning models. The authors propose now propose to use this theoretic foundation to multiagent reinforcement learning. 

The key weakness in my opion is not having a clear, crisp takeaway from this work. If the main claim is the superior performanc on multiagent reinforcement learning, then the empirical analysis seems to be somewhat lacking, as it covers relatively few domains (there are now enough multiagent gyms that allow a wider variability of tasks). If the main claim is a theoretical foundation, then one might expect a tighter analysis and bounds as compared to existing approaches. 

Either way, I think the writing is very formal, and could be imporved. What is the main driving intuition here? MARL is typically considered through a non-cooperative game theory prism (Markov Game). Here you are trying to use cooperative game theory, which means you consider subsets of agents, and have some function mapping each such subteam to its success in the task. Then, one might view the Shapley value as a decomposition allocating each single agent its individual reward / impact in the team's success. But why use the Shapley value rather than other solution concepts (such as the Core, which you mention, or the least-core), or the Nucleolous, or the Kernel, or other similar power indices such as the Banzhaf index? Are you using some of the axiomatic foundations to the Shapley value? If so, then where? 

All in all, I really love the topic of the paper, but the execution could be improved (more domains for empirical evaluation, tighter theoretic bounds versus baselines). And the writing should focus on the intuitions before jumping to the technical definitions 
questions: What happens in domains which are not completely cooperative (team reward), such as social dillemas or mixed motive games. Does the algorithm still runs? Does it fail? What happens when agents are trained using a mixture of algorithms (SHAQ agent with other approaches - are they compatible). Can you replace the Shapley value with other cooperative solution concepts (the Banzhaf index seems to be the closest, basically your equation just with different weights for the subteams) - does the whole method fail? 
limitations: As I wrote, the empirical analysis is somewhat limited (but certainly a decent foundation). Also the writing could be improved - at the very least I'd give the formal definitions of a transferable utility cooperative game, coalition structures, the core (as applied to a general CS or characteristic function game). The paper does a better job on the RL side (where things are fully defined). Also, you should have the discussion on what happens in non team reward (non fully cooperative) settings. 

All in all, a very interesting paper, if only for the nice connection between RL and cooperative game theory.
review_title: null

Meta-review: metareview: Reviewers appreciate that the paper is making an insightful contribution to the important field of cooperative MARL and its connection with cooperative game theory. The paper is clear and mostly well motivated, and the theoretical analysis and empirical evaluation are sufficient.
confidence: Certain
Decision: Accept