Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method, Learnable Intrinsic-reward Generation Selection (LIGS) to improve coordinated exploration. LIGS incorporates an extra agent, called Generator to learn what state to give what intrinsic reward for each agent. The intrinsic reward is potential-based, so it preserves the optimality. Experimental results on several domains show its advantages over several MARL methods.

main_review: From the perspective of learning intrinsic rewards and preserving the optimality, instead of hand-craft intrinsic rewards, it is a worthwhile investigation direction.

However, there are some confusions about the method and experiments. If all these concerns are addressed well by the authors, I am willing to increase the score.

First, I think the challenges mentioned in Section 1 are redundant. The first two points can be concluded into one point.

Second, in common, each agent should have its own reward function R_i(o_i,a_i), not from the global reward function. Only if each agent receives the team reward at each step, each agent’s objective is to maximize the team reward. However, in this paper, each agent maximizes its own expected return. Especially, this paper assumes some setting that each agent’s reward is non-monotonic to the team reward. Therefore, it is not suitable to obtain an individual reward from the global reward function.

Third, about the switch controls, I am confused about equation 2, where it uses I_t, while how to calculate or define I_t is not described. Instead, I_{\tau_{k+1}} is defined as the reverse value of I_{\tau_{k}}. Does this mean at each step, if I_t is 1, the value of I_{t+1} at the next state will switch to 0, and vise versa? Why is this changing flow? How does it find important states? And as a consequence, why equation 2 can be reformulated as equation 3 is confusing.

What does ” Generator constructs intrinsic rewards that are tailored for the specific setting” mean?

Finally, about the experiments, since I have a rich understanding of starcraft II, I am concerned that the selected maps in starcraft II may not well support the method. In these maps, just simply increase the exploration rate, many MARL algorithms can achieve better results than that shown in this paper, which can also be found in [1].

There exists some typos throughout the paper. So I recommend authors repolish the paper to make it clearer.
Such as, “with bilevel approach” ->” with a bilevel approach”
“Generator’s objective is:” repeats twice.
“(potentially away from suboptimal trajectories, c.f. Experiment 2) and enable”  -> enables

[1] Revisiting the Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning. 2021

summary_of_the_review: From the perspective of learning intrinsic rewards and preserving the optimality, instead of hand-craft intrinsic rewards, it is a worthwhile investigation direction.

However, there are some concerns about the method and experiments needed to be addressed. So I give a borderline reject currently. If all these concerns are addressed well by the authors, I am willing to increase the score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a learned centralized exploration reward for multi-agent settings. The exploration reward is factorized into an on/off gate (dubbed ‘switching control’) and a scale function. Some mathematical derivations are included (sketched in the main text, with details in the appendices) to provide theoretical guarantees on how the exploration reward changes the solutions the training procedure might find. Evaluations on gridworld environments that target specific difficulties of multi-agent exploration show promising results. Some maps from the SMAC benchmark are also included, and again show good results.
main_review: The first thing that needs to be said about this paper is that it is hard to read. There are several elements that lead to this overall conclusion:
- There are many small errors, inconsistencies, and unnecessary elements in the text and equations.
- The presentation of the contribution is smeared out over several pages, instead of providing one clear overview followed by further details where necessary.
- The paper needs some language editing.

Examples of the small issues with the text (this is not meant to be an exhaustive list, I would ask the authors to critically re-read and edit their work):
- Section 3, second line: does the set $\mathcal{N}$ contains the indices $1$ through $N$, or the agents? It seems that in most places where you refer to $\mathcal{N}$, the sentence would be just as clear without it.
- Section 7.2: you promise four SMAC maps, and then list (and show results on) only three
- Algorithm 2:
  - Line 1: what is $\pi_0$? Is it the set of agent policies {$\pi^i_0$}? 
  - Line 3 is informative, please have a similar line for the second loop starting on line 16.
  - Lines 8-9, 15, 18: are $g_t$ and $q_t$ the same?
  - Input: the parameters of $h$ and $\hat{h}$ are probably $\theta_h$ and $\theta_\hat{h}$
  - Lines 18-21: no need for control flow here, that's taken care of by $r_t^i$ being zero if $q_t$ is zero itself.
  - What happens on line 27?
  - Section 4.2, which is supposed to show the learning algorithm (PPO), does not seem to be present in the paper. Section 4, for which the same claim is made, does not introduce PPO.
- Page 18 (section / Appendix 14): in the long derivation, there are inconsistencies in how $t$ is shifted to $t+1$. It is also not clear to me where the the last term on the second line comes from. In the second half of that derivation, a capital $K$ appears, without it being made clear what it means. As it stands, I'm not convinced of the correctness of that derivation. 

In spite of the errors and inconsistencies, I found Algorithm 2 to be the best presentation of the paper’s contribution. Please consider replacing sections 3-5 in the main text with a corrected version of Algorithm 2.

All that being said, the results do look promising, and the method is interesting. Some more detailed comments and requests:
- While the results look good, MAPPO isn’t quite state of the art. Please compare to something that beats MAPPO, e.g. EMC, https://openreview.net/forum?id=cLYyCXHU7g1n.
- The paper's contribution could be more significant if a bit more time/space was spent on the results (this should not be a problem if the presentation of the method is condensed). In particular investigating how the method works in practice and in that way answering the question why it actually works would be very valuable. Suggested questions to answer (this might be done with the gridworld environments):
  - What does the exploration reward generator actually learn? What does it reward?
  - What does the gate / switching control mechanism learn? When does it switch on and off?

Other comments:
- ‘Novel framework’ sounds a bit grand. The centralized exploration shaping reward that is introduced looks like a good idea, but it’s not a novel framework.
- Central coordination of exploration, like the method here is doing, reduces the independence of the agents. Centralized training in general does that too, though, and there are no gradients between the agents being introduced here, so it seems okay. But it’s good to clearly position the contribution in this respect; the method does further reduce the independence, and thereby the multi-agent aspect of MARL.
- The fact that the gate is binary seems targeted to environments like the gridworld. Have you experimented with a continuous gate, or no gate at all? What are the states in the SMAC environments where it switches on? More generally, what are the assumptions that make the switching work?

summary_of_the_review: While the results look good, the text and equations are unclear. My current recommendation is to reject the paper, because the presentation quality falls short of the expected standard. However, there are interesting aspects to the proposed method, and the results look promising. If the presentation is improved significantly, I would be open to recommending a weak accept. I would strongly recommend to the authors to condense the presentation of the method in the main text and spend more time on evaluating the method. With more evaluation and clarification of how the method functions in practice, and if the further results are also good, I would consider raising my recommendation to accept.

EDIT: the resubmission addresses a significant portion of my readability concerns, I'm raising my score to a weak accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a novel reinforcement learning algorithm for multi-agent system (MARL) that employs a generator of intrinsic reward and a switching control system that helps to regulate intrinsic control. Crucially, the intrinsic reward is learned to better fit the particular task being learned. The paper claims that the proposed algorithm helps with exploration as well as preservation of known policies. The paper has a strong theoretical background with a section that illustrates the properties of convergence and optimality. The experimental results appear to justify the approach with superior performance with respect to the baselines. The paper deals with an emerging and interesting area of RL and proposes a new mechanism for co-ordinated RL agents. 

main_review: The paper is well written and relatively easy to follow, although the notation is at time heavy. This reviewer noted the extensive support material, including relevant material for reproducing the experiments, an ablation study and further the mathematical derivations to support the work. I could not go through section 14 containing the proofs of the technical results, which are several pages of extensive mathematical notation. So I cannot comment on the correctness and I wonder whether such an extensive material is really essential, although as part of the support material does not affect the fruition of the main paper.

[General readability] In section 3, paragraph 1, "the system is in a state s and each agent takes an action a”. The concept of “system” is not defined here. At first, I would assume that it is one agent, and not a system, that can be in a state. Similarly, as I came to the concept of joint-action, it was not clear to me as I was under the impression that each agent makes one individual choice. Also unclear is how a joint action can produce a reward for one agent, and lead to one transition since a I previously assumed a transition to defined for a state-action pair, and not a state-multiple-actions pair. This lack of clarity might be due to my limited familiarity of this very particular setup, but I feel that other readers in the area of RL could benefit from a slightly more explicit explanation on this point. 


My main observation is that more explanations would be beneficial in relation to  how the properties of the algorithm described in section 6 relate to the experimental evidence in section 7. In particular, what specific aspects of the simulations complement the theoretical discussion above, and in which way? Are the experiment meant to assess specific metrics of interest, to provide a quantitative advantage over existing SOTA algorithms in the proposed benchmarks? Or else? While the narrative appears to assume that the answer to these questions are self-evident, I would argue that more justifications on the precise aims of both theoretical and experimental evidence would strengthen the paper.

I would have liked to read more about the limitations and drawback of the proposed approach in relation to existing algorithms, in particular those used for comparison. While performance is improved, what is the price to pay for that, if any? Are there particular implications from a computational perspective that need to be considered? Considerations on tuning hyper-parameters, or other elements that could help appreciate possible limitations?  



summary_of_the_review: This is a solid paper that proposed a promising approach to MARL. Many details in the paper require extensive examination, care, and previous knowledge. The paper is fairly notation-heavy, but it makes a fair effort to explain concepts also in plain English. The results are promising and I believe it can provide a valid contribution to ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on learning intrinsic rewards for multi-agent reinforcement learning, which is an important problem. Different from previous works on this topic, the authors propose to train an agent with a learnable gating function that incentives other agents. Theoretical analysis and empirical evaluation are provided to prove the effectiveness of the proposed method.
main_review: As fas as the reviewer is concerned, the proposed method is novel. However, there are some drawbacks that can be improved.

***Soundness***

The objective of Generator contains four terms: (1) environmental rewards, (2) learned intrinsic rewards, (3) rewarding costs, and (4) a RND-style exploration bonus. Learning incentivizing policies of Generator may not be easier than learning without it because the Generator faces a large space that has a similar size of joint action-observation space.

Some claims need more consideration. For example, the authors hold that "CT-DE methods are however, prone to convergence to suboptimal joint policies (Mahajan et al., 2019). ... In such situations, (random) occurrences of successful coordination are improbable, moreover value factorisations, e.g. QMIX (Rashid et al., 2018), cannot represent non-monotonic team rewards." Not all CTDE methods have this problem. QTRAN, QTRAN++, weighted QMIX, and QPLEX can represent non-monotonic Qs and are free from the issue in Mahajan's paper.  The reviewer recommend rewriting the third paragraph of the Introduction section to better motivate the proposed method

Discussion of related works should be improved. For example, there are many multi-agent exploration methods expect for MAVEN, and previous works on learning (or designing) multi-agent intrinsic rewards (e.g., using transformers or curiosity-based intrinsic rewards) should be discussed. 

***Evaluation***

The reviewer was expecting an ablation study regarding the exploration bonus. Previous work has shown that RND works well on SMAC. The reviewer would increase the score if the authors can prove that LIGS performs relatively well without the RND-like objective.


summary_of_the_review: The reviewer finds the idea novel and the method solid. However, the empirical evaluation, the discussion of related works, and the presentation of the paper can be further improved.
review_title: null

Meta-review: The paper addresses coordination improvement in the MARL setting by learning intristic rewards that motivate the exploration and coordination. The  paper is theoretically founded and the empirical evaluations back up the claims.

During the rebuttal the carried out an impressive amount of work. They provided several additional studies and substantially improved the presentation, addressing all of the reviewers' requests. Although not all the reviewers responded to the authors, the authors' response was taken into the account when recommending the decision.

Minor:
- The authors should comment on the learning intristic rewards with evolution (Faust et al, 2019): https://arxiv.org/abs/1905.07628
Decision: Accept (Poster)