Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper extends soft actor-critic (SAC) to Markov games, or in other words multi-agent reinforcement learning setting. The paper is very nicely written, derives MA-SAC in a fairly general way, and introduces a variational approximation of the distribution over optimal trajectories which enables centralized training and decentralized execution. While I like the paper, I find the novelty aspect of it quite limited, since it's quite a straightforward combination of centralized training and decentralized execution idea with an algebraic extension of SAC to Markov games. The paper would have been much stronger if it had a much more thorough evaluation of the properties and limitations of MA-SAC as well as better comparison with the related work.


Questions/comments:

1. One of the key points of the paper is equation 4 that proposes the variational approximation of the distribution over trajectories. The authors assume that agents take actions independently which enables decentralized execution. However, it looks like this construction neglects the fact that optimal policies *must* take into account the other agents. It seems that with q structured this way, dependencies between agent actions are not taken into account even when training is centralized (all equations 5-7 fully factorize, neglecting all dependencies). In other words, given the proposed q, what is the benefit of centralized training?

2. Following up on the previous question, from Levine (2018) we know that Eq. (3) results in a particular soft-Q function that can be computed using the forward-backward algorithm (assuming the knowledge of the dynamics), which would account for dependencies between agent policies/actions. On the other hand, it's unclear whether/how the Q function obtained through centralized training (Eq. 8) approximates the optimal soft-Q function. Can the authors comment on that?

3. As mentioned, the proposed MA-SAC is really a fairly straightforward extension of SAC to Markov games. What could make the paper interesting in my opinion is a much more detailed (experimental) analysis of the approximations the authors had to make in order to enable decentralized execution and the corresponding advantages and limitations. The current evaluation falls short on that front as it just shows that the proposed algorithm works better than MADDPG in a few standard multi-agent environments.

4. Although the authors position this paper as the first that introduces a probabilistic perspective on RL in multi-agent systems, there is other recent work (https://arxiv.org/abs/1901.09207) that already does that and, in fact, takes one more step and enables decentralized training with (probabilistic) reasoning about other agents. Discussion of advantages/disadvantages and comparison with the previous work I see as necessary.

----

I acknowledge that I have read the author's response. My assessment of the paper stays the same.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a new algorithm named Multi-Agent Soft Actor-Critic (MA-SAC) based on the off-policy maximum-entropy actor critic algorithm Soft Actor-Critic (SAC). Based on variational inference framework, the authors derive the objectives for multi-agent reinforcement learning. In experiments section, the authors compare the proposed algorithm with the previous algorithm called Multi-Agent Deep Deterministic Policy Gradient (MADDPG) on several multi-agent domain.

Comments:
- Based on inference, the authors derive the objectives as the equation (8) and (9). However, the proposed objectives are almost identical to SAC. First, the objectives for Q functions are just replacing \hat{Q} in the equation (7) in SAC by \bar{Q}, which has a very similar meaning. Also, the objectives for policy \pi are exactly the same as in the SAC with only the added index. Thus, the proposed algorithm seems to be a naive extension of SAC into multi-agent cases. To avoid such questions, authors need to emphasize the difference from simple extension.
- Is there a reason not to use additional neural networks to estimate value function like SAC even the proposed algorithm is based on SAC?
- Additional experimental results are needed to ensure the algorithm since there is no theoretical guarantee. 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 21st November Update: Thank you for your detailed response. I agree with the suggested future work and revisions to the paper. However, as no updates were made to the submitted paper, I will not be raising my score as the revisions represent significant changes that I cannot support acceptance of without further peer review. I encourage the authors to carefully consider all reviewers advice in updates to the paper for a future submission elsewhere, which I think will significantly improve the paper and its potential impact on the community.

--
This paper contributes a probabilistic framework for multi-agent RL and demonstrates a derivation of multi-agent SAC using it. The framework could be more broadly applicable if it included partial observability (as is often a requirement of multi-agent systems). The derivation could be improved by showing the full working for equation 5, as this would make the work self contained instead of assuming prior knowledge of ELBO by the reader.

The derived algorithm (previously published by Iqbal and Sha [ICML 2019] as noted in the paper) is then evaluated on 4 existing benchmark tasks against the baseline algorithm originally proposed with the environments - MADDPG. The environments represent a good range of multi-agent scenarios of suitable complexity to test modern deep RL algorithms. However, the empirical evaluation and methodology have issues that reduce the significance of their contribution.

On Page 8, it is noted that "the value of alpha_i is empirically tuned for each environment" but for which algorithm was it optimized? It is then noted that the values for alpha were "found using grid search" but details of the range of the search are not included nor details of how any other hyperparameters were set. Were other parameters tuned? If so please report all values searched for both algorithms and how they were set.

At the end of page 8, the caption of Figure 2 concludes "it can be seen that MA-SAC controlled agents outperform MADDPG controlled agents on majority of tasks." This statement is not supported by the graphs in this figure. I suspect Figures 2a and b show no significant difference as the confidence intervals overlap and that Figure 2d is not significantly different throughout training but may be with a small effect size at the current end of training. Figure 2d also looks like training for longer may be beneficial. Therefore, Figure 2c is the only environment that shows a significant improvement. Please provide further evidence that MA-SAC outperforms MADDPG or weaken this conclusion. It would also be interesting to investigate deeper, why MA-SAC shows such higher performance than MADDPG in the Predator-Prey domain.

On Page 9, the conclusion is reiterated and claimed to be in comparison to a state-of-the-art algorithm. However, the benefits of SAC over DDPG have been previously shown both in single agent [Haarnoja et al, ICML 2018] and multi agent domains [Iqbal and Sha, ICML 2019]. A stronger baseline to compare against would improve the significance of any resultant improvements.

The research direction is interesting but the earlier publication of the derived algorithm (Iqbal and Sha, ICML 2019) and the issues discussed above with the experimental results lead me to conclude that the contribution is not yet sufficient to warrant publication. With further work I believe this line of work could lead to a high impact publication, but feel the paper requires more changes than are feasible within the time frame of the ICLR rebuttal period. 

Minor Comments:
- Page 4, "the transition function of underlying Markov game" -> the transition function of the underlying Markov game
- Page 9, "in Figure 2c, the red curve corresponds to" -> dark blue curve
- Page 9, "MA-SAC performs at least at par with MADDPG" -> at least on par with
- Page 9, "outperforms it on majority of the tasks" -> outperforms it on the majority of tasks
review_title: Official Blind Review #3

Meta-review: The paper takes the perspective of "reinforcement learning as inference", extends it to the multi-agent setting and derives a multi-agent RL algorithm that extends Soft Actor Critic. Several reviewer questions were addressed in the rebuttal phase, including key design choices. A common concern was the limited empirical comparison, including comparisons to existing approaches. 
Decision: Reject