Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an important and interesting approach on fully decentralized MARL. Fully decentralized Q-learning is highly applicable to realistic and real-world applications. The method is evaluated extensively, showing a great potential.
strengths_and_weaknesses: ## Strengths
- Theoretical analysis
- Extensive evaluations
- Interesting perspective to the MARL problem with potential real-world applicability

## Weaknesses
- missing out on some related work on fully decentralized MARL
- Lack of SOTA baselines
questions: - I like the idea behind this approach and I believe that this is a strong work with potential for real-world applicability. 

- Some important related work on fully decentralized MARL are not covered [1-2]. These works and similar prior work must be discussed to cover similarities and differences so that readers can select the proper method based on their applications. 

[1] Zhang, Kaiqing, et al. "Fully decentralized multi-agent reinforcement learning with networked agents." International Conference on Machine Learning. PMLR, 2018.

[2] Konan, Sachin G., Esmaeil Seraj, and Matthew Gombolay. "Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming." International Conference on Learning Representations. 2021.

- The approach potentially needs to be compared against SOTA fully decentralized MARL work to evaluate its utility and feasibility.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new MARL algorithm under the DTDE paradigm. Specifically, the proposed algorithm (I2Q) is introduced based on the ideal transition probability (where each agent assumes that the others adopt the optimal actions for each decision) and a previous idea named QSS-Learning. Theoretical guarantee on the convergence of the proposed algorithm is provided under certain conditions. For experimental studies, the significant superiority of I2Q is demonstrated in matrix games, MPE, MA MuJoCo and SMAC.
strengths_and_weaknesses: Pros:
+ This paper is clearly written.
+ The experimental part is relatively diverse and adequate.

&nbsp;

Cons:
- The novelty of the proposed algorithm is limited. 

&nbsp;


Minor issues and typos:
- l.299 "SAMC" → "SMAC"
questions: Questions:
+ If we add a fixed randomly initialized reward function and the performance tolerance to the other algorithms, is there a performance improvement? 


I appreciate the authors’ reasonable introduction of QSS for DTDE.
However, my major concern is the lack of novelty in the proposed algorithm. The current algorithm seems to be a simple application of the QSS-Learning in the MARL scenario, the QSS and the prediction of next state in this paper are consistent with the original QSS paper. 

&nbsp;

Suggestions:

+ I recommend the authors to combine the proposed approach with more base RL algorithms, such as PPO, SAC and TD3 to better evaluate of the generality of the method.

+ The authorss should add a discussion of QSS and the differences between I2Q and QSS in the related work or background.

limitations: The rationality and limitations of the main assumptions adopted in this paper are discussed in Sec.3.4.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents I2Q, an algorithmic approach for decentralized MARL. The authors present the non-stationarity problem in this setting and propose to use "ideal transition probabilities" to solve it. Particularly, these are transition probabilities for which all agents are ensured to converge to an optimal solution when trained in a decentralized manner. The authors then propose to use the next state (in deterministic environments) as a representation of an action, and show that it induces an ideal transition probability, which ensures convergence to an optimal solution. They experiment on many baselines in various domains, showing the benefit of their approach.
strengths_and_weaknesses: The paper proposes an elegant solution to the non-stationarity problem of decentralized MARL. I'm not able to say if it is the first method to solve this problem, and I hope one of the other reviewers will address this. The paper is clearly written, and the presentation is great. Also, I found everything to be easy to read and follow. Finally, the experiments section seems to have chosen a wide variety of tasks, and I'm glad the authors also chose to show results on the high dimensional problem of SCII.

----------------------------------------------------------------------------------------------------------

The paper doesn't have strong flaws, but there are some issues that make it a borderline paper for Neurips.

First, the theory is not very deep. There are many questions that remain open that the authors don't address theoretically, and I think are important for a better understanding of the problem. One of these, is convergence proof of I2Q, which the authors don't really prove, but only discuss informally.

Second, I feel that the deterministic assumption in the paper is a strong one, unless carefully addressed. In favor of the authors, they do discuss this in the paper, showing a result of the value gap, and also experiments on a wide variety of tasks. Still, I believe this is not adequately addressed. A stronger result for stochastic environments should be provided. I assume there exist some "ideal transition probabilities" for this setting. If it is the case that such are impossible to theoretically find, then this is an important point to address in the paper. Overall, I find Theorem 3 to be a trivial result. I wish to see an approach that tackles stochasticity explicitly, and provides a tighter bound for approximation errors.

Third, the fact that I2Q must learn a forward model is troubling, as model-based methods usually fail againt state of the art model-free methods on high dimensional tasks (unless latent spaces are used, such as in MuZero). The authors don't address the problem of estimating $f$ in their work. Moreover, I feel that this is not addressed fully in the experiments either.

Finally, while the experiments show results on different types of environments, I find that I2Q was not compared against enough baselines. There are a lot of new baselines on MARL, and particularly I would expect the authors to compare I2Q to at least three more baselines which are considered SOTA, and not only IQL - even if they are not decentralized.

----------------------------------------------------------------------------
Strengths:
1. A new solution for decentralized MARL
2. Proofs to formal statements seem correct
3. Paper is clearly written and presentation is great
4. Experiments show a variety of interesting tasks

Weaknesses:
1. Theory is weak
2. Stochastic environments should be addressed
3. Forward model should be addressed theoretically and in experiments
4. Experiments are lacking comparison to other algorithms
questions: Most of my questions relate to things I already mentioned above.
1. Can stochasticity be addressed with "ideal transition probabilities" more explicitly? How can we learn such probabilities? How would this affect training? If not possible, what are the limitations?
2. How does approximation of $f$ affect convergence?
limitations: The authors discuss limitations of their work. Some of these limitations coincide with points I've already raised. As mentioned above, I believe some of these points should be addressed more thoroughly in the paper.
review_title: null

Meta-review: metareview: The paper presents a novel method for dealing with nonstationarity in decentralized multi-agent reinforcement learning (MARL). While there are some concerns about the level of novelty, the approach is interesting and presented well. There are also concerns about the discussion and comparison with the state-of-the-art in decentralized MARL methods. We suggest the authors include comparisons to other decentralized MARL methods (such as the ones below) or state why such comparisons are not reasonable. 

Omidshafiei, Shayegan, et al. "Deep decentralized multi-task multi-agent reinforcement learning under partial observability." International Conference on Machine Learning. PMLR, 2017.

Palmer, Gregory, et al. "Lenient Multi-Agent Deep Reinforcement Learning." Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems. 2018.

Lyu, Xueguang, and Christopher Amato. "Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning." Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 2020.
confidence: Less certain
Decision: Accept