Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Based on the setting of decentralized learning, the paper proposes a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee, where each agent independently optimizes the surrogate. The experiment results show that the proposed algorithm DPO outperforms IPPO in most tasks.
strength_and_weaknesses: **Strengths**
1. The paper proposes a well-motivated technique for monotonic improvement in multi-agent optimization based on decentralized learning. The derivation is clear and easy to follow.
2. The proposed method and IPPO are compared in multiple multi-agent environments, including MPE, multi-agent MuJoCo, and SMAC.

**Weaknesses**
1. The experiment is not sufficient enough to convince the benefits of the proposed method.
   a) The details of IPPO are not mentioned. To limit the KL divergence between the last and updated policies, one can use a clip function or an adaption of beta as DPO. Which one is used in this paper?
   b) Fig.1 (b) shows that d_target dramatically influences the final performance. The performance of DPO with the worst hyper-parameter is worse than IPPO. If IPPO uses an adaption of beta as DPO, will its performance become better? Is IPPO’s best d_target the same as that in DPO?
   c) The hyper-parameter d_target is changed a lot across different domains. Is there not a single parameter configuration that could perform well in all environments? More studies are required to show the sensitivity of this hyper-parameter in different multi-agent environments. 
   d) Following c), fine-tuning d_target might lead to a dramatically different performance in various environments. The reviewer wonders whether IPPO is also fine-tuned for a fair comparison.

2. The paper proposes a new policy optimization objective for each agent as shown in Eq. 16 based on the lower bound introduced in Eq. 15. However, the sqrt of the KL divergence in Eq. 15 can be easily replaced by the KL divergence itself (for example, based on a piecewise function with a different weight) for a further lower bound. Based on this further lower bound, the policy optimization objective of each agent will be similar to that in IPPO. Thus, it is quite important for the paper to show more experiments and details, as mentioned above, to clarify the importance of explicitly optimizing the sqrt of the KL divergence between the last and updated policies.

3. The paper holds an assumption for the theoretical guarantee that each agent could receive the state. This is reasonable because the analysis in partially observable environments is much more difficult, and the problem may be undecidable in Dec-POMDP. However, for decentralized learning, one crucial factor influencing agents' cooperation is incomplete information. It will be helpful to discuss the influence of partial observation on theoretical analysis.

4. This paper does not discuss the limitations of the proposed method.



clarity,_quality,_novelty_and_reproducibility: The paper is generally easy to follow, with a clear introduction, a rounded discussion of related work, and a well-motivated method. The final policy optimization objective of each agent with two regularization terms is somewhat interesting. However, more discussion and experiments are needed to show the necessity of using two regularization terms.
summary_of_the_review: Based on the assumption that each agent could receive the state, the paper proposes a new optimization objective to guarantee the whole team's monotonic improvement with an sqrt of the KL divergence between the last and updated policies. However, further discussion and experiments are required to show the necessity of the new optimization term based on a fair comparison with IPPO. Meanwhile, further analysis of the influence of partial observation in both theoretical and experimental parts is necessary. Therefore, the paper requires a major revision.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a fully decentralized policy optimization (DPO) algorithm in cooperative multi-agent reinforcement learning (MARL). Based on the surrogate loss of the trust-region policy optimization for the joint policy, the authors derive a lower bound of the joint policy improvement under the assumption that each agent can obtain the state, and joint policy is represented as the product of individual policy. This agent-wise separable loss is optimized using two adaptive coefficients to overcome small step sizes in a naive application. Experiments in several cooperative MARL environments show that the proposed DPO can perform better than independent PPO under fully decentralized settings.

strength_and_weaknesses: Pros: The authors dealt with a theoretical derivation of the suggested surrogate loss.

Cons: Several parts of the paper should be clarified, and additional experiments should be performed. It was hard to check the reproducibility. Please refer to the detailed questions below.


clarity,_quality,_novelty_and_reproducibility: \
<Novelty and Quality>

The implementation part mainly relies on the techniques of TRPO and PPO.

- In my opinion, citing TRPO is required to “replace the maximum KL-divergence with the average KL-divergence.” In addition, using the adaptive coefficient technique requires citation of PPO, not to mention the value of $\delta$ and $\omega$.
- The PPO paper states that PPO “is not very sensitive to” the value of $\delta$ and $\omega$. DPO also requires the same sensitivity check as a multi-agent setting can be more unstable than a single-agent case due to the non-stationary issue. 


<Reproducibility>

Since the authors did not provide their source code, it was hard to check the reproducibility of the experiments and several experimental details.


<Clarity>

Many parts of the paper should be clarified. Please refer to the detailed questions below.

1. This paper states that “the network architectures and common hyperparameters of DPO and IPPO are the same for a fair comparison.” (page 6) While the original IPPO paper uses Generalized Advantage Estimation (GAE), DPO does not use GAE and uses the standard TD error for advantage estimation (stated on page 4).
- Is there any specific reason why DPO does not use GAE?
- Does the compared IPPO in the experiments use GAE? Does IPPO use value clipping and the entropy term?
- For clarity, DPO does not use value clipping and the entropy term, does it?


2. Regarding the proposed DPO loss, further ablation study is required. 
Naively thinking, the second and the third term in Eq. 17 may have a similar role. What if we do $\beta^i_1 = 0$ or $\beta^i_2 = 0$?
- If $\beta^i_1 = 0$, it produces IPPO-KL in Appendix C. I think the other environments also require IPPO-KL as a baseline for a fair comparison. If the gap between DPO and IPPO-KL is small, the effect of the proposed method may weaken. 
- If $\beta^i_2 = 0$, we may still consider trust-region due to the square root term. A related ablation study is required.


3. Can we use more tighter bound for implementation, e.g., Eq.14? Although Eq.14 is not agent-wise separable, we can easily optimize it with modern deep-learning libraries. If the gap between the true loss and the lower bound becomes larger, the learning efficiency becomes worse. Experiments using tighter bound are required to support the necessity of Eq.15, i.e., less-tight but agent-wise separable loss.

4. The legends of Figures 3 and 4 should be larger than the current ones. It is hard to read the details in each graph. In addition, several readers may overlook that authors reported average reward instead of win rate in ‘27m vs 30m’ and ‘MMM2’. This part should also be explicitly and more vividly described in the paper for the readers.

5.  ‘agent_obsk=2’ should be explicitly stated in the main paper (not only in the Appendix) for readers.

6. In the second line on page 4, it seems that $\mathbb{E}_{a_{-i}' \sim \pi^{-i}(\cdot|s’)}$ may be redundant.

7. The claim that "the better performance of DPO in the 17-agent Humanoid task could be evidence of the scalability of DPO" (page 9) can be better supported if there is any experiment showing a marginal performance gap. (The improvement ratio is the smallest among the 5 figures in Fig. 3.)

8. What is the SC2 version used?

9. For clarity, does the DPO use synchronized sampling (i.e., each agent samples the same episodes), or does each agent sample independently given an (either on-policy or off-policy) batch data?




<Discussion>

- The fine-tuned value of $d_{target}$ becomes lower as the environment becomes more complex. Could the authors provide any discussion regarding this observation?
- Could the authors provide any discussion why there is a reward drop at the 1.5M steps in MMM2?




summary_of_the_review: Although the authors provided theoretical insight into policy optimization in a fully decentralized MARL setting, the novelty and reproducibility are not well supported. In addition, several parts of the paper should be clarified by performing additional experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigated the possibility of decomposing the trust region surrogate over joint policies into the respective surrogates for each individual agent's policies for cooperative multi-agent reinforcement learning. The reported experiment results show that the newly developed DPO algorithm based on the decomposed surrogates can outperform a multi-agent system with independent PPO learners.
strength_and_weaknesses: Strength:
It is important to study new learning mechanisms to support multi-agent reinforcement learning in a decentralized manner. This paper explored an interesting direction of decomposing the trust region surrogate to facilitate independent learning among multiple agents. Experiment results show that this direction of research is promising.

Weakness:
The authors argued that independent learners in a multi-agent system can often outperform agents that adopt CTDE for policy training. However, it remains unclear why independent learners without considering inter-agent interactions can outperform a learning system that explicitly handles inter-agent interactions. Under what conditions will independent learners perform well and why? Besides the potential performance advantage, what are the key benefits of supporting independent learners? The research motivation of this paper may need to be strengthened to answer all these questions.

The paper assumes that all learning agents have direct access to the full state information. However, this assumption is often not valid in a multi-agent system. In fact, with full observability, it is not difficult for each agent to understand the impact of its decision/action on the learning environment and other agents. Therefore, achieving decentralized policy training may not be very difficult. Hence the technical contribution of building a decentralized policy training algorithm under this assumption may need to be justified more. The practical usefulness of the new algorithm also should be investigated more, given the algorithm's critical reliance on the full observability assumption.

The technique for critic training in (7) does not seem to have any difference from critic training in the single-agent setting. Existing research works showed that using a single-agent approach to train critics in a multi-agent setup can be unstable and ineffective under certain conditions, especially when all agents are constantly improving their own policies. It is not clear under what conditions the stability (or convergence) of critic learning can be guaranteed and whether those conditions are realistic for multi-agent reinforcement learning.

Regarding the new lower bound for each agent in (16), the middle term with $\sqrt{D^{max}_{KL}}$ can be simply removed by adjusting $C$ in the third term. Given that, the new surrogate in DPO is essentially the same as the TRPO surrogate for single-agent reinforcement learning. As a result, I can hardly understand how the new surrogate, as a result of including a removable middle term, can facilitate effective inter-agent coordination/cooperation. It also remains questionable how DPO is fundamentally different from a direct extension of TRPO to the multi-agent setup. What are the technical advantages of DPO, in comparison to independent TRPO agents?

The new DPO algorithm introduces several new hyper-parameters such as $d_{target}$, $\delta$, $\omega$, $\beta_1^i$ and $\beta_2^i$, that may need to be fine-tuned. The introduction of these hyper-parameters may make it difficult to apply DPO to a wide range of problems.

More technical details may need to be provided regarding some statements in the paper. For example, it is stated on page 7 that "DPO actually some approximations to the decentralized surrogate". What approximations does this statement refer to?

The authors argued that DPO can be guaranteed to converge. However, I cannot find a thorough analysis on the convergence properties of DPO. In my opinion, a high-level argument regarding the adoption of the trust-region surrogate is not sufficient to show the convergence of the DPO algorithm. Perhaps the claim on algorithm convergence should be adjusted in the paper.

While the experiment results appear promising, DPO is only compared with IPPO in the experiments. It is not clear whether DPO can outperform state-of-the-art CTDE algorithms on any of the benchmark problems. It also remains unclear whether DPO can outperform independent TRPO agents. Hence it is questionable whether the observed performance difference is due to the use of TRPO for policy training or due to the development of the new decentralized surrogate.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. However, more technical details may need to be provided regarding some statements in the paper.

The technical novelty of this paper needs to be clearly justified. Specifically, this paper assumes that all learning agents have direct access to the full state information. However, this assumption is often not valid in a multi-agent system.

It remains questionable how the new surrogate is different from the TRPO surrogate and why the difference allows agents to learn to coordinate/cooperate effectively.

The experiment results need to be significantly expanded in order to clearly understand the true advantage of the newly developed algorithm.
summary_of_the_review: It is important to study new learning mechanisms to support multi-agent reinforcement learning in a decentralized manner. However, the key assumption of this paper seems to have some major limitations. The technical novelty of the newly developed performance bound also remains questionable to a certain extent. Furthermore, experimental study is at the limited side and does not convincingly show the true advantage of the newly developed algorithm.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents early results towards a decentralized actor critic algorithm for fully observable, cooperative multi-agent tasks with theoretical guarantees. However, the reviewers raised many issues requiring significant further work which the authors acknowledged but could not complete due to the time limits of the ICLR review period. Hopefully this feedback is useful to the authors as they continue to develop this line of work for a future publication.
justification_for_why_not_higher_score: Preliminary results not yet ready for publication
Decision: Reject