Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve the exploration-exploitation tradeoff problem in the context of multi-agent reinforcement learning. While there have been a myriad of works on exploration-exploitation tradeoff on single age reinforcement learning, there are not many on the multi agent RL. This work proposes an adaptive entropy-regularization framework that learns adequate amount of exploration for each agent. To this end, this work proposes to decompose the joint soft value function into pure return and entropy sum. This disentanglements enable a more stable while updating the temperature parameters. This work focuses entropy-based MARL. 
strength_and_weaknesses: - The strength of the paper comes from the idea that, while previous works encourage same level of exploration across agents, this work proposes to differentiate the level of exploration across agents in multi-agent RL setting. 
- Another strength comes from the core idea of this work: joint soft value function decomposition / separated factorization. 
- The motivation part 3.1 sounds convincing to me; one agent’s exploration can hinder other agent’s exploitation, resulting that simultaneous exploration of multiple agents can make learning unstable. Need a framework that can adaptively learn proper levels of exploration for each agent. 
- Experiments are well done, not extensive though. 

Questions. 

Q1. Question about ADER performance shown in Figure 2a. It seems that ADER outperforms other methods like SER-DCE, SER-MARL, but there is a point where ADER’s performance suddenly jumps up in the middle. Is there any explanation on why this happens? 

Q2. In Appendix B, could you give me more justification on setting the coefficient beta_i? Especially, line B.7, beta_i are defined as softmax of expectation of \partial V^R_{JT}(s,\tau) / \partial H (pi (|))) ? Could you give us more detailed explanations on it? And can you explain why it is difficult to directly obtain the partial derivative in discrete-action case, and using chain rule is justified? 
clarity,_quality,_novelty_and_reproducibility: This paper is well-written with high clarity. Somewhat novel, but not groundbreakingly novel. I think the authors showed good amount of experiments and evaluations on various benchmarks, and ablation studies, which seem to be reproducible.  
summary_of_the_review: I would give marginally above the acceptance threshold. It would be good if the authors could answer my questions. There might be some issues that I didn’t catch, and if other reviewers have raised issues, I’m happy to discuss.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an adaptive entropy-regularization framework (ADER) to address the multi-agent exploration-exploitation trade-off problem in MARL. The key insight of ADER is that the amount of exploration every agent needs to perform is different and can change over time, so we need to adaptively control the amount of exploration each agent conducts and learn this amount across agents and over time. To achieve this, ADER adaptively learn an individual target entropy for each agent over time (to control the amount of exploration for each agent), assuming a fixed total entropy budget. And it uses the change in the joint pure-return value w.r.t. the change in agent $i$'s policy entropy to estimate the benefits of increasing the target entropy of agent $i$.  In addition, in ADER, the exploration and exploitation is disentangled by disentangling the return from the entropy (i.e., the joint soft Q-function is decomposed into one joint Q-function for reward and one joint Q-function for the entropy). 
strength_and_weaknesses: - Strengths:
	- The proposed framework is well motivated and seems to work well in a variety of complex cooperative tasks that require adaptive exploration across agents over time.
	- The main idea of adaptively learning an individual target entropy for each agent over time to better balance the exploration and exploitation in the dimension of agents is very interesting and seems novel to me. 
	- The paper is well-written and easy to follow overall. 
- Weaknesses:
	- It is not clear how the proposed method performs in cooperative tasks that do not require adaptive exploration across agents over time.
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity overall. Most of the key components of ADER are well motivated and clearly explained, although the necessity of using monotonic value function factorization is not very clear to me. 

The key idea of ADER looks novel to me. 
summary_of_the_review: I like the main idea of ADER. It seems to provide a simple yet effective way to better coordinate the level of exploration across different agents over time in cooperative tasks. I think the authors do a very good job at explaining the main intuitions of the proposed method. For instance, the motivation example is quite nice. It clearly explains how one agent's exploration can change over time and how it can hinder other agents' exploitation, thus showing the necessity of adaptively controlling the trade-off between exploitation and exploration across multiple agents. 

My only (small) concern of the paper is most of the cooperative tasks tested (e.g., predator-prey and SMAC) in the experiments were different from the original settings. They were specifically designed to be difficult for the agents to gain positive rewards under simultaneous exploration (or similar levels of exploration). While I understand this might be because exploration is not a big problem in these original tasks, it seems that the authors could have evaluated their method on MARL benchmark like Google Research Football (with the original setting), which is known to have hard exploration issues. Also, it'd be nice to know if ADER could perform well in the original SMAC tasks, such that we could know if continuously learning the target entropy for each agent could hurt performance in cooperative tasks that do not require adaptive exploration across agents. 

Some minor comments/suggestions:
- I think the introduction could be improved some. For example, the introduction says "This disentanglement alleviates instability which can occur due to the updates of the temperature parameters and enables applying value factorization to return and entropy separately." It suddenly mentions value function factorization, but does not explain the motivation/necessity for using it. 
- The related work section mentions that LICA does not maximize the cumulative sum of entropy but regularize the action entropy. Why maximizing the cumulative sum of entropy can be better than regularizing the action entropy? This could be explained some. 
- The ablation study section mentions "we compare ADER with and without the monotonic constraint to show the necessity of the monotonic constraint." For ADER without the monotonic constraint, how does it learn the joint $Q$ or $V$? Does it learn a centralized critic that directly conditions on the global state and joint action of agents? Also, I do not understand why it is necessary to use a *monotonic* value factorization. ADER learns both actors and critics and only the actors are needed during execution. So one can also use a *nonmonotonic* value function factorization.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an MARL algorithm to adpatively handle the entropy regularization in multi-agent RL. In this method, the level of exploration of each agent is controlled by its time-varying target entropy, which severs as a constraint in the optimization problem. To determine a proper target entropy for each agent, the authors propose a partial derivative-based metric to evaluate the benefit of exploration and then use this term to decide the target entropy for each agent. Empirically, in a customized SMAC environment, their method is demonstrated to be better than other value-decomposition methods.

strength_and_weaknesses: ### Strengths

1. This paper studies adaptive entropy regularization, which is important in entroy-regularized MARL and well-demonstrated in the motivation section.

2. The authors provide a comprehensive literature review of entropy-based MARL algorithms.

3. A variety of baselines is included in their SMAC experiments. 

### Weaknesses

1. **Standard SMAC maps and more baselines should be included.** In the experiment section, MAPPO should also be included as a baseline, as it is one of the SOTA MARL algorithms. Also, experiments on the *standard* (unmodified) SMAC environment should be included, as it is a standard of all MARL papers. Moreover, for the matrix game, FOP should also be included as a baselines. As far as I understand, FOP uses a weight network to determine individual temperature parameters in their tasks except SMAC. That said, adaptive entropy regularization over agents and time is indeed considered in FOP. 

2. **The writing can be largely improved.** Some paragraphs are too long, e.g., Section 1 is one paragraph, and Section 3.1 is one paragraph. Some references to equations are misused in section 3.4. The authors should refer to where those equations appear in the main text, not where they are in the appendix. For example, eq B.7 should be eq 15 in the main text.

4. **The presentation of the proposed method can be improved.** The overall workflow of this algorithm is complicated and unclear. An algorithm should be included to make it clear. Figure 6 does not provide a clear illustration. 

### Questions

1. Two mixer networks are used in this method. The update of Q-mixer and $Q_i$ is common and easy to understand. However, It seems two methods are mentioned for computing $V_{jt}$. One is using V-mixer $V_{jt} = f_{mix}(s, V_1, \cdots V_n)$ and another is using expectation $V_{jt} = E[Q_{jt}]$. Do these two methods result in the same value? Or do you use those two values in different places?

2. As stated in eq 13, the policy update is achieved in a joint form using joint value function $Q_{jt}$, so the $Q_i$ will only be used for the computation of target entropy? If so, why do we need a mixer network for $Q_{jt}$, as we can directly parameterize $Q_{jt}$ via a single neural network? The learning of the proposed method is unclear in many ways, though I may misunderstand some parts. 

3. As mentioned in Section 3.4, the partial derivative $\frac{\partial V_{jt}^{R}}{\partial H(\pi^i)}$ is used to determine the target entropy of agent $i$. For the categorical policy of discrete action space, as stated in eq 16, $\frac{\partial V_{jt}^{R}}{\partial H(\pi^i)} = \frac{\partial V_{jt}^{R}}{\partial V_i^R} \times \frac{\partial V_i^R}{\partial H(\pi^i)}$. The second term is approximated by $\frac{\Delta V_i^R(\tau^i)}{\Delta \mathcal{H}(\pi^i_t)}$. However, it is still unclear to me how it is computed numerically. 
clarity,_quality,_novelty_and_reproducibility: The Clarity of this paper can definitely be improved. Some references to equations are misused. Also, the authors should shorten the length of the motivation section and spend more paragraphs explaining the purpose of each component, as this is a relatively complex framework. Moreover, I think figure 6 can be improved by associating the loss function in it to better explain how each component is trained.

The Novelty of this paper heavily lies in the method of computing target entropy for each agent. However, as mentioned above, the computation of those partial derivative terms has to be further explained in order to judge the importance of this work.

summary_of_the_review: This paper studies adaptive entropy regularization in MARL, and provides a naive yet reasonable solution for this problem. However, the presentation of this paper can be substantially improved, and some technical details about the computation of partial derivatives have to be further explained.  In short, I will not recommend acceptance of this paper, but I will be happy to reconsider it if the authors can address all those problems.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
I thank the authors for their submission and active engagement during the discussion period.  This is a borderline paper with extensive internal discussion. On the positive side, the motivation of the method is convincing [5qy9,dyHK], the method is somewhat novel [5qy9,dyHK] and interesting [dyHK], and the paper is written clearly [5qy9] with a thorough literature review [mfnN], and overall and easy to read [dyHK]. On the negative side, reviewers noted experiments could have been more thorough [5qy9], for example demonstrating their method on more complex MARL benchmark [dyHK] as well as comparing to original tasks in the SMAC benchmark [dyHK,mfnN]. The authors have tried to address in particular the last two points in their rebuttal. However, they still have not provided a comparison against MAPPO on the original SMAC tasks. I side with reviewer mfnN on this issue and believe the paper requires a resubmission. I strongly encourage the authors to use the feedback by the reviewers to improve their paper.
justification_for_why_not_higher_score: Missing comparison to MAPPO on original SMAC tasks.
Decision: Reject