Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper discusses a hierarchical, iterated reasoning scheme for learning policies conditioned on the other agents' policies at that stage,  maximizing both the mutual information between agent policies and agent rewards. The authors leverage cognitive hierarchical theory wherein each agents has multiple levels of abstraction and reasoning about the actions other agents might take, and take a game-theoretic approach to policy learning in which each subsequent policy is the best response based on other agents' previous policies. The authors proceed to show this approach yields results which outperform several baselines in complex environments such as StarCraft II and multiwalker.
main_review: While there are plenty of interesting ideas in this paper, and this kind of approach to multiagent learning that relies both on information theory and game theory seems to hold great promise, I find there are multiple issues with the paper that must be addressed for it to be of greater utility to the ML and RL communities.

- First and foremost, it seems the authors omit from their literature review (and subsequently from their consideration of eligible baselines) works relating to two heavily relevant bodies of work: work on hierarchical learning in Dec-POMDPs (for instance Amato et al. 2019), and work on learning to communicate in multiagent RL (for instance Foerster et al. 2016, or Sheikh and Boloni 2019).

- At the core, what the paper seems to propose, ultimately, is a framework for jointly learning to communicate *and* act simultaneously, which is both interesting and useful, but again, certain related work seems to be missing in that regard (consider Ghavamzadeh and Mahadevan 2004 as a relatively early - but still related - example). 

- A potentially critical consideration which the paper does not seem to explicitly address in its theoretical analysis is that it's unclear from the setting and the (rather muddled, more on that later) formulation of the algorithmic approach how the proposed framework avoids the plague of decentralized learning in multiagent RL, which is nonstationarity. In fact, it almost seems baked into the solution that if each policy learned is the best response to the policies at the lower level, then nothing is necessarily guaranteed to converge. There's also the risk that implicitly the extent of communication baked into the framework renders the approach analogous to single agent learning (theory of mind taken to its extreme), and it is unclear to me from the paper to what extent this might be the case (there is a proof in the supplementary material that eq. 2 does converge, but because the entire framework is so muddled - more on that later - it's not clear how that resolves the stability issue for the larger framework, and also, one cannot rely on supplementary material to establish key aspects of their approach).

- in practice the communication setup proposed in the paper, as much as I can understand it, seems rather contrived and unrealistic. This may not be a disqualifying aspect of the approach - we often need to make certain simplifying assumptions to establish initial results and study novel approaches, but it does raise the question of how well this approach will hold up if certain constraints on communication are imposed. 

- Unfortunately my biggest issue with the paper is lack of clarity. Terms are introduced without sufficiently elucidating their meaning in the specific context of the paper (even as simple an example as "rationalizability" right in the abstract), Section 4.2 which should be the backbone of the paper is very confusing to read, Section 4.3 was even harder for me to follow, and ultimately by the end of Section 4, before discussing the empirical evaluation, while I retained a high level mental picture of the general outline of the proposed approach, I could not technically describe how the actual learning framework was designed, how the hierarchical approach works in practice, and how the framework is trained (the supplementary material helps somewhat, but not enough, and again, authors should not rely on supplementary material to present any key aspect of their work, they are called "supplementary" for a reason).

summary_of_the_review: This paper has interesting ideas but is lacking in how it anchors its contribution to related work and compares against it. More glaringly, this papers suffers from substantial lack of clarity both in terms of its theoretical grounding, and more critically, in terms of how it presents its core framework, how it is designed, and how it is applied. I believe it should be revised substantially before it is suitable for publication at a venue such as ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces InfoPG for learning coordination in fully decentralized multi-agent games. InfoPG models agent policies following *k*-level reasoning. The authors present a theoretical analysis to show that, under InfoPG, policy gradient optimizes a form of mutual information related to coordination. Empirical demonstrations show that InfoPG enables better performance than conceptually related baselines, with some demonstration of how InfoPG better adapts to a version of a Byzantine Generals Problem.
main_review: # Strengths
- The paper is well written. For the most part, each section makes its points clearly and the paper is easy to follow.
- Insofar as it is defensible (see Weaknesses below), the policy architecture seems straightforward and enables PG to implicitly target an MI metric associated with good coordination.
- The authors provide both a theoretical analysis of InfoPG (although, again, see Weaknesses below) and empirical demonstration of its effectiveness.
- InfoPG accounts for the possibility that not all agents provide meaningful communication.

# Weaknesses
- The abstract and introduction talk about bounded rationality, but that concept doesn't seem relevant thereafter.
- Much of the paper is devoted to demonstrating the *implicit* connection between MI of agents' actions and the InfoPG policy formulation. This comes across as something of a *post hoc* motivation for the setup. It makes good enough sense why MI and coordination are related, but (as the paper points out) treating MI maximization as the objective is potentially short sighted. In sum, I question if MI is the best lens to motivate/understand this approach.
- Good performance is encouraging, but the most relevant analyses for validating the intuitions of the method are largely ignored in the main text.
- From the description in A.9, it sounds like the connection between *k*-level reasoning and the actual architecture implementation is misleading. It seems more appropriate to think of the agents' policies as one large joint policy. Please justify this connection and/or clarify the mechanisms by which agents' actions become correlated. From my understanding, InfoPG (with this architecture) would not cause the MI between the 3rd and 2nd piston to actually change over time in the BGP experiments -- that is, learning would not bring about any reduction in their MI as the main text describes.

# Questions/Comments
- It is hard to understand the aspects of InfoPG that set it apart from prior work. It may help to reconsider where MOA and PR2 are first detailed.
- In the Section 3 paragraph about MI, you write: ```In our work, X and Y are policy distributions of two interacting agents.``` This is vague. Please be more clear about what the "policy distributions" are. I'm assuming it's the action distribution of each policy, but the paragraph is not very concrete.
- Consider making it a bit more clear what "InfoPG" refers to. The k-level encoding/decoding policy definition? Something to do with the objective? Perhaps Section 4.3 is sufficiently clear, but just as a note I got a little bit confused about what "InfoPG" is meant to describe.
- For the bottom row of Figure 2, how are you estimating MI? It would be good to state that in the paper. Also, what makes this an "ablation" study?
summary_of_the_review: My starting recommendation is: weak reject.

I hope the authors can use the rebuttal period to correct any misunderstandings that may contribute to my lower score.

However, assuming I have understood things accurately, I am primarily concerned with the notion that InfoPG is a genuinely decentralized approach. It seems to me that the policy is centralized because of how the communication policy works. I wonder how much the benefit of InfoPG simply comes from the fact that each agent naturally gets more information about the state of the world. More generally, I worry that the work misrepresents itself, using phrases like "communication" and "*k*-level reasoning" where they may not be appropriate. I am interested in how the authors defend these choices and characterizations.

I also cannot escape the sense that the MI lens is somewhat *post hoc.* It may be possible to show that other MARL algorithms implicitly target MI, if it is indeed a corollary of coordination. Bottom line: a lot of the paper is spent on the theoretical connections between InfoPG and an abstract metric; I think that space would have been better spent on empirical analysis. Again, if I have overlooked the importance of (this form of) MI in these task settings, I hope the authors will correct me.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed  Info-PG, by reformulating the individual policy conditioned on state or partial observation to conditional on actions of other agents and with K-level rationalizability from cognitive hierarchy theory, the policy is conditioned recursively on lower levels of actions from all agents.
The optimization of the policy is in the same form of policy gradient algorithm.
The author showed that with reward shaping or applying relu on advantage, optimizing the reformulated policy is implicitly maximizing lower bound of mutual information across agents' policies.
In the case of Byzantine General's Problem where at least one agent is not cooperative, which could be detrimental to multi-agent learning, the author showed that by removing the relu on advantage, the policy gradient on the reformulated policy lead to tuning the upper bound of mutual information. The mutual information between the agent in question and the fraudulent agent will has its upper bound tuned down in case of negative reward, thus the fraudulent agent won't reduce the uncerntainty of policy of the agent in question, thus won't poison the multi-agent learning.
The algorithm is evaluated on a variety of multi-agent decentralized learning environments with promising results.


main_review: ## Strong points

I find it very innovative 
1. to reformulate the policy $p(.|s)$ to be $p(.|a^{j,k}, a^{i,{k-1}},  a^{i,{k-1}}, ...)$
2. to use the K-level Bayesian tree to represent the interaction between agents.

The experiments are abundant

## Issues
1. Line 12 of Algorithm 1, how the MAP sample is carried out? For discrete actions, one can choose the max posterior action by enumeration, what about continuous actions?

2. Line 19 of Algorithm 1: which equation does this correspond to?
In the explanation text, line 19 is not mentioned.

3. What is $J(\theta)$ exactly in Eqn (2) and (15)? Could you elaborate on that?

4.  In Appendix A.5 Full proof of Theorem 1, page 17, paragraph between equation (35) and eqn. (36),  for a Bayesian tree, why
  $\sum_x \sum_y p(x'|x,y)p(x,y|y')=p(x'|y')$?

summary_of_the_review: In general the paper is innovative and solid, there are several issues as in the main review section that need to be addressed.
review_title: null

Meta-review: The paper proposes a method for decentralized learning of cooperative games by maximizing the mutual information between the agents. The paper is novel and interesting and well evaluated.

Prior to the rebuttal, most of the reviewers saw presentation as the biggest weakness. Specifically, it was not clear what InfoPG refers to, and how it is related to the mutual information. During the rebuttal the authors cleaned up the misunderstandings around the presentation and provided a detailed analysis in the Appendix.

While the author responses provided helpful clarification and analysis, the authors should revise the paper holistically to remove unnecessary terminology and connections, and bring the analysis in the main text.
Decision: Accept (Poster)