Probabilistic View of Multi-agent Reinforcement Learning: A Unified Approach
Authors: Shubham Gupta
Ambedkar Dukkipati
Venue:  (2020)
Keywords: multi-agent reinforcement learning
maximum entropy reinforcement learning

Abstract: Formulating the reinforcement learning (RL) problem in the framework of probabilistic inference not only offers a new perspective about RL, but also yields practical algorithms that are more robust and easier to train. While this connection between RL and probabilistic inference has been extensively studied in the single-agent setting, it has not yet been fully understood in the multi-agent setup. In this paper, we pose the problem of multi-agent reinforcement learning as the problem of performing inference in a particular graphical model. We model the environment, as seen by each of the agents, using separate but related Markov decision processes. We derive a practical off-policy maximum-entropy actor-critic algorithm that we call Multi-agent Soft Actor-Critic (MA-SAC) for performing approximate inference in the proposed model using variational inference. MA-SAC can be employed in both cooperative and competitive settings. Through experiments, we demonstrate that MA-SAC outperforms a strong baseline on several multi-agent scenarios. While MA-SAC is one resultant multi-agent RL algorithm that can be derived from the proposed probabilistic framework, our work provides a unified view of maximum-entropy algorithms in the multi-agent setting.

Under review as a conference paper at ICLR 2020
PROBABILISTIC VIEW OF MULTI-AGENT REINFORCE- MENT LEARNING: A UNIFIED APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Formulating the reinforcement learning (RL) problem in the framework of prob- abilistic inference not only offers new perspective about RL, but also yields practical algorithms that are more robust and easier train. While this connec- tion between RL and probabilistic inference has been extensively studied in the Sinle-agent setting, it has not yet been fully understood in the multi-agent setup. In this paper, we pose the problem of multi-agent reinforcement learning as the problem of performing inference in a particular graphical model. We model the environment, as seen by each of the agents, using separate but related Markov de- cision processes. We derive a practical off off-policy maximum-entropy actor-critic algorithm that call Multi-agent Soft Actor-Critic (MA-SAC) for performing ap- proximate inference in the proposed model using variational inference. MA-SAC be employed in both cooperative and competitive settings. Through exper- iments, we demonstrate that MA-SAC outperforms a strong baseline on several multi-agent scenarios. While MA-SAC one resultant multi-agent RL algorithm that can be derived from the proposed probabilistic framework, our work provides unified view of maximum-entropp algorithms in the multi-agent setting.
INTRODUCTION
The traditional reinforcement learning (RL) paradigm, that formalizes learning based on trial and error, has primarily been developed for scenarios where a single trainable agent is learning in an environment. In this setup, although the agent changes its behavior as learning progresses, the environment dynamics themselves do not change. Thus, the environment appears to be stationary from the point of view of the learning agent. However, in a setting where several agents are learning in the same environment simultancously (multi-agent setting), this is not true as a change in one agent's behavior manifests itself as change in environment dynamics from the point of view of other agents (Busoniu et al., 2008). It has been established that stability issues can arise each agent is independently trained using standard single-agent RL methods (Tan) 1993).
While, in theory, it is possible to treat a collection of multiple agents as a single centralized meta- agent be trained, in practice, this approach becomes infeasible the action space of the central- ized meta-agent grows exponentially with the number of agents. Moreover, executing the resultant centralized policy is not always possible due to various reasons like geographic separation between agents, communication overhead and so on (Foerster et al. 2018b). Even if these issues are taken care of, when the agents are competitive, designing reward function for the centralized meta-agent is very challenging and thus, in general, such a setup cannot be used with competitive agents.
There are numerous practical scenarios that require several intelligent agents to function together (either cooperatively competitively). Consider, for instance, a soccer game between two teams: agents within a team must cooperate while being competitive with the opponents. Considering that traditional single-agent RL methods cannot satisfactorily handle problems from the multi-agent domain, completely new RL algorithms that explicitly acknowledge and exploit the presence of other intelligent agents in the environment required.
In this paper, we pose the multi-agent reinforcement learning problem as the problem of performing probabilistic inference in particular graphical model. While such formulation is well known in the single-agent RL setting (Levine, 2018)) its extension to the multi-agent setup is non-trivial especially in the general case where the agents may be cooperative and/or competitive. We model

---

Under review as a conference paper at ICLR 2020
the environment seen by each of the agents using separate but related Markov Decision Processes (MDPs). Each agent then tries to maximize the expected return it gets from the environment under its own MDP (Section T
Using our framework, we derive an off-policy maximum entropy actor-critic algorithm that general- izes the Soft Actor-Critic (SAC) algorithm (Haarmoja et al., (01 to multi-agent setup. We refer to this algorithm as Multi-agent Soft Actor-Critic (MA-SAC). Like SAC, it is a maximum entropy algorithm, i.e., the learned policies try to maximize the rewards while at the same time maximizing entropy of the stochastic actor. Such algorithms are known to be more stable and easier to train (Haarooj et al. 2018a).
MA-SAC follows the centralized training. decentralized execution paradigm. As we demonstrate in Section 4 each agent learns its own policy while being actively aware of the presence of other agents. The learned policy of any given agent only utilizes its local observation at test time. Thus, MA-SAC avoids the pitfalls of both independent training of agents (being unaware of other agents leads to non-stationarity and hence instability) and training a centralized agent (centralized policies are hard to execute) as described above.
By setting a tunable temperature parameter (Section 4.3 to zero, MA-SAC yields an algorithm that is very similar to the Multi-agent Deep Deterministic Policy Gradients algorithm (MADDPG) Lowe et al. (2017) apart from minor change in updating the actor. The utility of this modification is clearly reflected in our derivation of the inference procedure. When the temperature parameter non-zero, agents trained using MA-SAC outperform agents trained using MADDPG on multiple cooperative and competitive tasks as we demonstrate in Section 5.3
Our main contributions are: (i) we present a probabilistic view of the multi-agent reinforcement learning problem where each agent models the environment using a separate but related MDP; (ii) derive an off-policy maximum entropy actor-critic algorithm (MA-SAC) by performing struc- tured variational inference in the proposed model; (iii) we empirically demonstrate that MA-SAC performs well in practice and highlight different ways in which our framework can utilize ideas from other existing approaches in multi-agent RL; and (iv) although we only present an actor-critic algorithm in this paper, our framework allows derivation of maximum-entropy variants of other reinforcement learning algorithms in the multi-agent setting.
2 RELATED WORK
In recent years, RL has made significant progress in complicated domains such as game playing (Mnih et al. 2013 Silver et al. 2016) and robotics (Levine et al. 2016) to name a few. Inspired by these successes, researchers have tried independently training all agents in multi-agent setup using single-agent RL algorithms as in the case of independent Q-learning (Tan 1993) and independent
deep Q-learning (Tampuu et al. 2017). However, discussed in Section such approach does not perform well in practice. Recently, [Forster et al. (2018a) have proposed a modification to independent Q-learning in an attempt to stabilize the training process.
To deal with non-stationarity, agents need additional information to explain the changes in their environment. Existing approaches exploit either (i) multi-agent communication or (ii) centralized training of decentralized policies to provide the required additional information to agents. In com- munication based approaches (Sukhbaatar et al. 2016 Foerster et al. 2016 Lazaridou et al. 2017,
Cao et al. 2018), agents use an emergent language to jointly execute a task. Such approaches require communication between agents even after the training is over. This needs additional computational resources and communication infrastructure which limits the applicability of these methods.
The centralized training and decentralized execution paradigm can be thought of as learning with the help of coach that coordinates activities while training. Once the training process is over, agents can independently take decisions based on their local observations (Gupta et al. 2017). These ap- proaches learn centralized critic(s) for all agents along with decentalized policies. While some ap- proaches only support cooperative agents (Sunehag et al., 2018 Foerster et al.) 2018b 8b, Rashid et al, 2018 de Witt al. 2018) others support competitive agents as well (He et al. 2016 Lowe et al.
2017 Iqbal & Sha 2019). MA-SAC follows the centralized training and decentralized execution
paradigm and it supports both cooperative and competitive agents.
2

---

Under review as a conference paper at ICLR 2020
MA-SAC is a maximum-entropp based algorithm. In the single-agent RL literature, maximum- entropy variant of many standard RL algorithms have been proposed like coft-Q-learning (Haarnoja et al. 2017) and soft-actor-critic (Haarnoja et al. 2018a). These algorithms are derived by casting the problem of learning optimal behavior as an inference problem in an appropriate graphical model (Levine 2018). This not only provides different perspective on the control problem, it leads to algorithms that are more stable and easier to train (Haarnoja et al. 2018a). A multi-agent variant of soft-Q-learning exists (Wei al., 2018), but it is only applicable in cooperative environments. Note that MA-SAC is just one of the algorithms that can be derived using our proposed framework. As in the case of single-agent reinforcement learning, in the multi-agent setup as well, one can derive maximum entropy variants of other algorithms based on our proposed probabilistic model.
Most relevant to our work is the multi-agent variant of DDPG (Liliicrap et al., 2016) algorithm (called MADDPG) that has been proposed in Lowe et al. (2017). MADDPG is based on an actor- critic framework where each agent has its own centralized critic. In Section (.3 we show that we recover algorithm that is very similar to MADDPG by setting a tunable temperature parameter used in MA-SAC to zero. In Section 5 tasks.
we show that MA-SAC outperforms MADDPG on several
If one ignores the attention mechanism proposed in Iqbal & Sha (2019), the resultant algorithm is equivalent to MA-SAC. While the main contribution in Iqbal & Sha (2019) is improvement in scalability of MADDPG using an attention based mechanism in the critic, it is mostly heuristic based and principled derivation of the algorithm is missing. On the other hand, we provide unified probabilistic framework for multi-agent RL and use it to derive MA-SAC.
3 NOTATION AND PRELIMINARIES
A Markov Decision Process (MDP), specified by the tuple (S, p,r r, 7), models an environment with a single trainable agent (Sutton& & Barto 2018). Here, is the set of states, A is the set of actions, S x A + A (S) specifies environment dynamics, r S x A + IR is the reward function and y E [o, 1] is the discount factor. We use A(S) to denote the set of all probability distributions over set S. A Markov Game (MG) generalizes MDPs to support multiple agents in the environment and is specified by the tuple (S. 1 (=:)) [Littman] 1994). Here, n the number of agents in the environment. Symbols S and y have the same meaning as before. A, and
Ti S X À X ..... X An + R respectively denote the action space and reward function of agent The transition probability function, p S x A x... x An ASS), now considers the actions taken by all agents. The same holds for the reward functions f2, T2 ........ Tn. In practical cases, it is also common to assume that each agent can only observe part of the environment. Let O denote the observation space of agent i, then, a function f. : S > O2 computes the observation of agent i from the environment state S.
Note that both MDP and MG simply specify the evolution of environment in response to the behavior of agent(s), and as such, they do not encode any notion of optimal behavior. An agent formulates its own notion of optimality which it strives to achieve by modulating its policy π S + A(A) where, as before, A(A) is the set of all probability distributions over the set of actions A. If there are multiple agents, we use π. : S + A(A,) to denote the policy followed by agent i. When the environment is partially observable, the policy of an agent can only take its local observation as input and hence π,; O, + (AA)).
In single-agent RL, the agent often solves the following optimization problem (either exactly or approximately) to learn optimal behavior:
T Σ (=0
J(T))
max EST
r(s(0) a(t)))
(1)
The agent acts in the environment only for finite number of time steps T' (a finite horizon problem).
The state and action at time are given by (t)) E S and (1)) E A respectively. The agent uses its stationary policy to choose an action (!) ~ π (()) at each time step. In MARL setting, different agents may have different reward functions. Moreover, because agents act simultancously in the same environment, the optimal behavior of an agent now depends on the behavior of other agents.
The notion of best response policy of an agent becomes important in this setup. Each agent wants
3

---

Under review as a conference paper at ICLR 2020
to maximize its own expected sum of rewards but has to do so in the context of the policies being followed by other agents.
4 THE PROPOSED MODEL
4.1 MDP FOR EACH AGENT
We use probabilistic graphical models formalize the idea of best response strategy. Note that from the perspective of each agent, all the other agents are part of the environment. Thus, we model the world, as seen by each of the agents, as a separate MDP. The MDP for ith agent is depicted in Figure | For now, let us ignore the variables of and all arrows connected to them for 1 1,2,.... T. We will these variables while deriving MA-SAC in the next two sections. For agent j A i, let a.
( where T, is the policy being followed by agent j. As these policies part of the environment from the perspective of agent i, they have been encoded into the graphical model in Figure 1 In this MDP, the probability of transitioning from state s(O) to state s g(++1) is given by:
((s(++1)
510 a (1)
/ p(s((+1)
s!0, s!0 a
II (a, ((()) da Jz1
(2)
Here, a_; represents the actions taken by all agents except agent at time t. The first term inside in- tegral in equation the transition function of underlying Markov game. The term on left hand side is the transition function of MDP for agent i. As other agents learn along with agent i, their policy Ty changes and hence, from the perspective of agent the environment dynamics are non stationary. Thus, one cannot satisfactorily use standard single-agent RL algorithms to train each agent indepen- dently. In Fig I we have enclosed the actions of all agents j # i and the environment state s(*) in a box to indicate that all these variables are part of the environment from the perspective of agent In addition to observing the current state s(t), if agent is also allowed to observe the actions taken by all other agents aj then the state tran- sition dynamics can be specified by using a stationary transition function (the first term inside the integral in equation 2). This motivates central- ized training of agents, where they are allowed to use this additional in- formation so that the environment ap- pears stationary to them, hence aid- ing in the learning process. Al- though the agents learn in central- ized fashion, we will see in the next two sections that the learned policies are completely decentralized, thereby avoiding any communication over- head during the execution of policies.
M
4.2 FINDING OPTIMAL POLICIES
Figure 1: Augmented graphical model of MDP for agent Symbol o denotes the optimality variable for agent at time t.
As discussed earlier, MDPs them- selves do not encode notion of opti- mality. Hence, in Figure ! we have augmented the graphical model of MDP for agent i with additional optimality variables 0.") 1 = 1,2...., Binary random vari-
ables of (t) indicate the optimality of action taken by agent i time t. Following (Levine, 2018), assume that (s!),, a(!)) < (this can be assumed without loss of generality for bounded reward
4

---

Under review as a conference paper at ICLR 2020
functions) and model the optimality variables as:
pl (1) = -1 | = exp (ri
(3)
Here ( = [a] 2 a. represents the joint action that has been taken by all agents. One
(4)
way to find the optimal policy is by setting π. (a(" s .PP) p(a,) s(0) a (8) (E:T) 1)
Levine 2018). Note that it is enough to condition on of for 4,1++, time independent of of for
as the action at given s(f) and a This policy is not stationary. Further, practical to
it relies on actions taken by other agents a and hence is neither decentralized, execute if all agents use a similar policy simultaneously. It is also known that such a policy risk seeking in the sense that it always assumes the best case scenario and may choose execute actions that will most likely lead to a bad outcome but may yield high reward infrequently due to unlikely state transitions (Levine 2018). In real world, one has no control over environment dynamics and hence such risk seeking policies are undesirable.
To solve these issues, use structured variational inference. We want approximate the posterior (s(*) a ) f-=o conditioned all optimality variables for agent being one. However, We want the posterior to stay faithful to the environment dynamics to avoid risk seeking behavior. As the environment dynamics are fixed, optimal policies can be easily recovered from this posterior because of the structure of the approximating distribution (given in equation 4
distribution over trajectories
We approximate the posterior over T using the following distribution:
T n
T-1
qe: aa = p()
IIII [=0
n. (a, j=1
s s()))
[I
p (++1)
1s(),
a())
(4)
We assume that the policy of agent j. i... Ty is parameterized by 0g Rd. To avoid notational clutter we suppress 0j and write Ta, s()),, instead of (a, s 0,)).
There are two important features of this approximate posterior: (i) it uses the same model of en- vironment dynamics the underlying Markov game, and (ii) it assumes that agents take actions independently of each other. Because of (i), the learned policies would not be risk seeking and, because of (ii), they can be executed
decentralized fashion. Also, note that while considering the MDP for agent i given in Figure [ 0, is the only parameter that can be adapted to bring the other policies are assumed
approximate posterior close to the true posterior. The parameters to be constant.
Following the usual variational inference procedure (Blei et al., 2017), we write the expression for ELBO using the joint distribution p(T, (o = 1) and the approximate posterior distribution given equation 4
(L, E Eog loggp(T fo!" T) loggo. 90. (T)
T
T Σ
E Eragoe
log p(a((()))) (2)
(s( a())
log ग. (a |s())]
1=
T
T log nT
= Errggo
(1(
(())
s [()) ] + const.
(5)
1=0
t=0
We assume that the prior distribution over actions is uniform, hence logp(a) (9) term has been ab- sorbed in the constant in eq Next, we derive MA-SAC that maximizes ELBO; for all agents in an actor-critic framework.
4.3 MULTT--GGENT SOFT ACTOR-CRITIC
Although one can directly optimize ELBO; over 0; for all agents using the REINFORCE gradient estimation trick (Willams) 1992), training agents in this way is hard due high variance in the
5

---

Under review as a conference paper at ICLR 2020
gradient estimates. Thus, we cast the problem in actor-critic framework. Define the function Qi(s a()) as:
Qi(s()) = =r((s( (a)) + B 1p Bauis (r((s(tt)) a
((++)))-
log (a, (++2) s((+1)) + Bg E Baltten
(6)
Here, is the transition function of Markov game and, with slight overloading of notation, we have assumed that π is a product distribution where each of the individual factors are given by T j = 1,2..... One can recursively write (i(s!t) (())) as:
Qi(s!)) aaa) rs a())) Betett EACDD (i(s((-1), ((ss(-"),attt a(tt)) log (a) ((+1) s s(tt))
If the log n. term were not there in equation ( a()) could have been interpreted as mea- suring the expected reward-to-go for agent when all agents start in state s(), take action (!)) and thereafter follow the joint policy л. This aligns with the commonly used notion of Q function in RL. Note that we have a separate Q function for each agent i. Q, uses actions of all agents at time I,
i.., (+)) as input, as opposed to just using the action of agent because the actions of all agents are needed to sample the next state (t-1) from p.
The log गT, term can be seen as an augmentation of the agent's reward function that increases the value of received rewards when the agent follows a policy with high entropy. We parameterize the function Q, using Φ E Rdd, Using equation 7 Φi be optimized by minimizing the following error:
Eq, = Es00 (0)) (Qi(s(0), - (ss(0,,
(8)
where, is the replay buffer. We use equation to compute (s) (() using one-step looka- head. (s(t)), (1)) refers to the value of Q2 under the current parameters Φ.. The lookahead term (i(((t+)), a (t+1)) appearing in equation is also computed using the current value of parameters Φi. Both (s(())) (1)) and Qi(s(t+1), a ((F1)) are treated as constants while optimizing equation R over oi. In practice, all expectations are approximately evaluated by taking samples from the cor- responding distributions. While we sample fixed size batch of examples from D, only one sample from y and π is used to approximate equation Also, note that we need centralized training setup because computation of εq, requires the ability to sample from policies of all agents.
This Q; estimate can now be used to find 0; that maximizes ELBO in equation 5 To optimize the policy of agent for choosing the correct action at time t, we have to maximize the term in (t) is sampled. Thus, the objective (to be minimized)
nested expectation in equation where action a." becomes:
Ex, (0.) = Bg E (s(( a()))))))))) - log प. (a. (1) ((1))]
We approximate the above expectation using samples. Gradients can either be computed using RE- INFORCE (Williams) 1992) or, if the policy permits, reparameterizztion can be used to get lower variance gradient estimates. Algorithm [ summarizees MA-SAC. In MADDPG, during optimization of equation g only actions of agent are sampled rather than sampling actions of all agents.
In practice. a temperature parameter a; E R+ is multiplied to the log π. term in equation 7 and equa- tion 9 to assign relative importance to entropy and reward maximization (Haarnoja et al. 201 8a) Although there are methods for automatically selecting an approriate value of a; (Haarnoja et al., 2018b), in all our experiments we treat it as a user specified hyperparameter.
Another important quantity that must be incorporated in equation is the discount factor 7. One way to think about discounting in the probabilistic framework is by considering the possibility that an agent can die with probability 1 - y at each time step and enter an absorbing state in which all subsequent rewards will be O. The transition probability of going non-absorbing state now gets multiplied by y and hence y should be multiplied to the expectation term in equation 7 to incorporate discounted rewards. We do this in all our experiments.

---

Under review as a conference paper at ICLR 2020
We parameterize agent policies and Q functions using neural networks. To improve learning sta- bility, we also use target policy and Q function networks. Additionally, each of our Q function network consists of pair of twin sub-networks and the minimum of the value produced by these sub-networks in a pair is taken as the output Q value (Fujimoto et al. 2018).
EXPERIMENTS
Algorithm Multi-agent Soft-Actor-
Initialize parameters (0., 0., Φi, 6. for each iteration do for each environment step do
5.1 ENVIRONMENTS
We use the multi-agent environments proposed in|Lowe et al. 2017)( to per- form experiments. These envi- ronments are built on the grounded communication framework that was proposed in Mordatch & Abbeel (211).. Each environment has a cer- tain number of agents and landmarks. At each step, each agent observes a part of the global environment state and chooses an action from its dis- crete action space. In certain environ- ments, the agents also have the abil- ity to communicate with each other.
(a, s0), for = ..,,.., .1.,
s ((+ ((s(t+1) ( (1)
- DU {(s(0) {a. (s(()))) (0) ) (((1)))
end for
for each gradient step do
0, 0: Ax Vo, 0. Ex i i= 1,2, 2,..... ................................
Φ + Ø XQ V s. &.E. i = 1,2 ...
0 + 10; - (1 -1--7
(1-r)0, 0
..,,....
6 +-++ r (1 T) i ...,
end for
end for
In such cases, in addition to choosing a physical action, the agents also choose a communication action. The resultant communication message generated by agent (represented as a one-hot en- coded vector) is then broadcasted to all other agents where it becomes part of their observation in the next time step. Upon taking an action, each agent receives a reward from the environment which may potentially be distinct for different agents. Thus, based on the reward structure, the agents can be cooperative or competitive. We briefly describe the environments that we have used in our experiments here. For more details see Lowl et al. (2017).
Cooperative navigation: There are agents and n landmarks in this environment. All agents get the same reward. This reward is calculated based on the distance of closest agent from each landmark. Additionally, agents occupy physical space and are penalized for colliding with each other. The optimal solution for agents is spread out and cover all the landmarks simultancously such that each landmark has an agent ideally overlapping with its position. Since everyone gets the same reward, this task requires cooperation among all agents. We use 3 in experiments with this environment.
Cooperative communication: There are 3 landmarks, each colored differently. An agent (called listener) is rewarded based on its distance from the target landmark of a chosen color, however, it is unaware of this choice of color. A different immobile agent (called speaker) knows the color of target landmark. The speaker can only take communication actions and the listener can only take physical actions. The speaker receives the same reward as the listener and hence it must coopera- tively communicate with the listener to transmit information about the color of target landmark.
This environment mimics the classic predator-prey game where k slower cooper- ating agents chase a faster adversary. All cooperating agents get positive reward each time any of them collides with an adversary while the adversary gets a negative reward. This mixed cooperative-competiive environment where the & cooperating agents compete with the adversary. We use 3.
Predator-prey:
There is one adversary and two cooperating agents in the environment. The environment also has two landmarks one of which is the target landmark. While the cooperating agents know which one is the target landmark, the adversary does not have this information. All agents want to reach the target landmark. The cooperating agents are rewarded based on minimum distance between target landmark and any of the cooperative agents. They penalized based on distance between adversary and target landmark. Adversary gets rewarded for being close to the target landmark. The cooperative agents must go both the landmarks so that the adversary is
Physical deception:
7

---

Under review as a conference paper at ICLR 2020
unable to identify the target landmark based on the behavior of cooperative agents. This is again mixed cooperative-competiiic setting.
5.2 HYPERPARAMETERS AND OTHER IMPLEMENTATION DETAILS
We parameterize the Q functions Q1 Q2...... On and agent policies WL. T2, Tu using fully connected neural networks. All networks have hidden layers having 128 units each. The hid- den layers use ReLU activation function. We use Adam optimizer (Kingma & Ba 2015) with learning rate 0.01. Following Lowe et al. (2017), we use Gumbel-Softman (Jang et al., (017] to approximate discrete sampling of actions in the policy networks (inverse temperature parameter of Gumbel-Softman distribution was fixed 1). We also use target policy and Q networks. Target network parameters are exponentially moving averages of the corresponding policy or Q network parameters. We update the target networks with each gradient step. For all experiments, we set 0.01 (in Algorithm]] [ to compute the exponential average. For all environments, we train for 15000 episodes, each of length 25. One gradient step is taken after every 100 environment steps. We replay buffer of size 10th which is populated in round-robin fashion. A randomly sampled batch of 1024 examples from replay buffer used for each gradient step. The value of a; is empir- ically tuned separately for each environment. We use the following values: cooperative navigation: 0.02, cooperative communication: 0.005, predator-prey: 0.26 and physical deception: 0.01. These values were found using grid search. We use the same value of a; for all cooperating agents in an environment. The value of discount y was set to 0.95 in all experiments.
5.3
COMPARISON WITH MADDPG
-140
-10-
-160
-15
Mean Agent neward
Mean Reward
-180
-20
-200
-25
-220
-30-
MADDPG
MADDPG
-240
MSAC
MSAC
-35
2000
4030
6000 8300 10020 12000 14000
2000
4000
6000
10300 Number Episodes
8000
12000
14000
Number Episodes
(a) Cooperative navigation
(b) Cooperative communication
MSAC chesing MADDPG MADDPG chasing MSAC
50
Arrth M Myrana
40
Agent Reward Reward
Reward
6-
30
20
10.
MSAC deceiving MADDPG MADDPG deceiving MSAC
2000
4300
6000
RODO 10300 12000 14000
2000 4000
6000
8000
10000
12000 14000
Number
(c) Predator-prr
(d) Physical deception
Figure 2: Comparison of MA-SAC (our approach) with MADDPG Lowe al. 2017). The tem- perature values used by MA-SAC controlled agents for each environment given in Section 5.2 can be seen that MA-SAC controlled agents outperform MADDPG controlled agents on majority of the tasks.
8

---

Under review as a conference paper at ICLR 2020
We chose to compare with the MADDPG algorithm (Lowe et al., 2017) for two reasons: (i) MA- SAC is closely related to MADDPG and a comparison with MADDPG allows us to demonstrate that the changes incorporated in MA-SAC do lead to better performance; and (ii) it was demonstrated in Lowe et al. (2017) that MADDPG outperforms state-of-the-ane single agent reinforcement learning
algorithms like DDPG (Lillicrap et al. 2016). TRPO (Schulman al. 2013), DQN Mnih et al.
(2015) and so on and thus it is a strong baseline.
Figure shows the comparison between rewards obtained by MADDPG controlled agents and MA- SAC controlled agents on different environments. To obtain these plots, we executed the most recent learned policy at the time after every 100 episodes. The rewards were averaged over 100 independent episodes. Before testing, the policies were first converted into deterministic policies that select the most probable action. The entire process was independently repeated 5 times with different random seeds. The mean and standard deviation have been plotted in Figure 2
For cooperative navigation and communication tasks, both agents were controlled by the same al- gorithm. For predator-pres and physical deception tasks, agents that compete with each other were controlled by different algorithms. For example, in Figure [c] the red curve corresponds to a setting where the three cooperative agents are controlled by MA-SAC whereas the adversary is controlled by MADDPG. It can be seen that MA-SAC performs at least at par with MADDPG on all tasks and outperforms it on majority of the tasks.
5.4 OTHER REMARKS
The temperature parameter o; plays an important role in determining the performance of MA-SAC. The log rT, term in equation 9 augments the Q-function based on uncertainty of policy π,, in state s!t) A very high value of ag leads to policies that maximize entropy (i.e. behave randomly) and ignore the rewards obtained from the environment. At very low value of a;, the learned policies do not exhibit sufficient entropy and hence are susceptible to getting trapped in local optima. Based on the scale of rewards obtained from the environment, the value of a must be tuned for achieving optimal performance. One can alternatively try to automatically tune a; using techniques presented in (Haarnoja et al. 2018b), however, we leave these experiments for future work.
MA-SAC produces stochastic policies. Such policies are preferred during training as they are easier to train (Haarnoja et al. (0018)). Moreover, we believe that training with/against an agent that uses a stochastic policy would result in a more robust policy as it would have the same effect as training with ensemble of deterministic policies as done in (Lowe et al. 2017). Once trained, it is often better to convert these stochastic policies to their deterministic counterparts to improve performance. This is in the same spirit as turning off e-greedy exploration during test time while performing Q- learning. As noted in Section [.3] we follow this practice in our experiments.
Scaling efficiently the number of agents increases is common challenge associated with cen- tralized training. In particular, the complexity of networks modeling Q; increases with the number of agents. A number of approaches have been proposed to parameterize the Q network in various ways to improve scalability in certain settings (Yang et al. [2018] Rashid et al. 2018 Iqbal & Sha 2019). For example, Yang et (2018) train each agent to play against the average opponent which allows training with hundreds of agents. These alternative parameterizations of Q, can naturally be integrated with MA-SAC. Finally, centralized training can still be done without access to policies of other agents as long as their actions are observable. To do so, Lowe et al.] (207)) propose that each agent can train a local approximation to all opponents' policies supervised manner using their observed actions. This again can be used with MA-SAC. We leave detailed experiments in this direction for future work.
CONCLUSION
In this paper we posed the multi-agent RL problem as the problem of performing probabilistic infer- ence in a graphical model where each agent views the environment as separate MDP. We derived an off policy maximum entropy actor-critic algorithm based on the centralized training, decentralized exccution paradigm using proposed model. Our experimental results show that the proposed al- gorithm outperforms a strong baseline (MADDPG) on several cooperative and competitive tasks. As noted in Section 5.4 various existing ideas for parameterizing Q-funciooss (Yang et al. 2018 Rashid

---

Under review as a conference paper at ICLR 2020
al. 2018 Iqbal & Sha 2019) can be naturally integrated with MA-SAC to improve its scalability as the number of agents increases. Our framework can also be used for deriving mexximum-entroppe variants of other RL algorithms in the multi-agent setting. We leave these ideas for future work.
REFERENCES
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisti- cians. Journal of the American Statistical Association, 112(518):859 877, 2017.
L. Busoniu, Babuska, and B. De Schutter. comprehensive survey of multiagent reinforce- ment learning. JEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172, 2008.
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z. Leibo, Karl Tuyls, and Stephen Clark. Emer- gent communication through negotiation. In 6th International Conference Learning Represen- tations (ICLR), 2018.
Christian A. Schroeder de Witt, Jakob N. Foerster, Gregory Farquhar, Philip H. Torr, Wendelin Boehmer, and Shimon Whiteson. Multi-agent common knowledge reinforcement learning. CoRR, abs/1810. 11702, 2018.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 29, pp. 2137-2145. Curran Associates, Inc., 2016.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MulliAgent Systems (AAMAS), pp. 122-130, 2018a.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. AAAI Conference on Artificial Intelligence, 2018b.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Proceedings of the 35th International Conference on Machine Learning (ICML), 80:1587-1596, 2018.
Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. Autonomous Agents and Multiagent Systems (AAMAS), pp. 66-83, 2017.
Tuomas Haarnoj,, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning (ICML), 10:1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. Proceedings of the 35th International Conference on Machine Learning (PMLR), 90:1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/18 1812009015 2018b.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daum III. Opponent modeling in deep rein- forcement learning. In Proceedings of The 33rd International Conference on Machine Learning (ICML). 48: 1800-181181 2016.
Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. Proceedings of the 36th International Conference on Machine Learning (ICML). 97:2961-2970, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. Pro- ceddings of the 5th International Conference on Learning Representations (ICLR), 2017.
Diederik Kingma and Jimmy Lei Ba. Adam: method for stochastic optimization. Proceddings of the 3rd International Conference Learning Representations (ICLR), 2015.
10

---

Under review as a conference paper at ICLR 2020
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agen agent cooperation and the emergence of (natural) language. 5th International Conference on Learning Representations (ICLR), 2017.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, abs/1805.00999, 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research (JMLR), 17(39):1-400 2016.
Timothy Lillicrap, Jonathon J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yubal Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Proced- dings of the 4th International Conference on Learning Representations (ICLR), 2016.
Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. Pro- ceedings of the Eleventh International Conference on Machine Learning (ICML). pp. 157-163, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compeiixxs environments. Neural Information Processing Systems (NIPS), 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 51 5129, 2015.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. AAAI Conference on Artificial Intelligence, 2018.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. Proceedings of the 35th International Conference on Machine Learning (ICML), 80:4295-4304, 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. Proceedings of the 32nd International Conference on Machine Learning (ICML), 37:1889-1897, 2015.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(484), 2016.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems 29, pp. 2244-2252. Curran Associates, Inc., 2016.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Pro- ceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2018.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE, 12(4), 2017.
11

---

Under review as a conference paper at ICLR 2020
M. Tan. Multi-agees agent reinforcement learning independent cooperative agents. Proceedings of the tenth International Conference on Machine Learning (ICML). pp. 330-337, 1993.
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning, AAAI Spring Symposium Series, 2018.
Ronald Williams. Simple statistical gradient-following algorithms for connectionissreiiforeeenttt learning. Machine Learning. 8(3):229-25) 256, 1992.
Yaodong Yang, Rui Luo, Minne Li. Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. Proceedings of the 35th International Conference on Machine Learning (ICML), 80:5571-5580. 2018.
12


Reviews:
[{"review_id": "r1etri0TtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new algorithm named Multi-Agent Soft Actor-Critic (MA-SAC) based on the off-policy maximum-entropy actor critic algorithm Soft Actor-Critic (SAC). Based on variational inference framework, the authors derive the objectives for multi-agent reinforcement learning. In experiments section, the authors compare the proposed algorithm with the previous algorithm called Multi-Agent Deep Deterministic Policy Gradient (MADDPG) on several multi-agent domain.\n\nComments:\n- Based on inference, the authors derive the objectives as the equation (8) and (9). However, the proposed objectives are almost identical to SAC. First, the objectives for Q functions are just replacing \\hat{Q} in the equation (7) in SAC by \\bar{Q}, which has a very similar meaning. Also, the objectives for policy \\pi are exactly the same as in the SAC with only the added index. Thus, the proposed algorithm seems to be a naive extension of SAC into multi-agent cases. To avoid such questions, authors need to emphasize the difference from simple extension.\n- Is there a reason not to use additional neural networks to estimate value function like SAC even the proposed algorithm is based on SAC?\n- Additional experimental results are needed to ensure the algorithm since there is no theoretical guarantee. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lHqUjRKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extends soft actor-critic (SAC) to Markov games, or in other words multi-agent reinforcement learning setting. The paper is very nicely written, derives MA-SAC in a fairly general way, and introduces a variational approximation of the distribution over optimal trajectories which enables centralized training and decentralized execution. While I like the paper, I find the novelty aspect of it quite limited, since it's quite a straightforward combination of centralized training and decentralized execution idea with an algebraic extension of SAC to Markov games. The paper would have been much stronger if it had a much more thorough evaluation of the properties and limitations of MA-SAC as well as better comparison with the related work.\n\n\nQuestions/comments:\n\n1. One of the key points of the paper is equation 4 that proposes the variational approximation of the distribution over trajectories. The authors assume that agents take actions independently which enables decentralized execution. However, it looks like this construction neglects the fact that optimal policies *must* take into account the other agents. It seems that with q structured this way, dependencies between agent actions are not taken into account even when training is centralized (all equations 5-7 fully factorize, neglecting all dependencies). In other words, given the proposed q, what is the benefit of centralized training?\n\n2. Following up on the previous question, from Levine (2018) we know that Eq. (3) results in a particular soft-Q function that can be computed using the forward-backward algorithm (assuming the knowledge of the dynamics), which would account for dependencies between agent policies/actions. On the other hand, it's unclear whether/how the Q function obtained through centralized training (Eq. 8) approximates the optimal soft-Q function. Can the authors comment on that?\n\n3. As mentioned, the proposed MA-SAC is really a fairly straightforward extension of SAC to Markov games. What could make the paper interesting in my opinion is a much more detailed (experimental) analysis of the approximations the authors had to make in order to enable decentralized execution and the corresponding advantages and limitations. The current evaluation falls short on that front as it just shows that the proposed algorithm works better than MADDPG in a few standard multi-agent environments.\n\n4. Although the authors position this paper as the first that introduces a probabilistic perspective on RL in multi-agent systems, there is other recent work (https://arxiv.org/abs/1901.09207) that already does that and, in fact, takes one more step and enables decentralized training with (probabilistic) reasoning about other agents. Discussion of advantages/disadvantages and comparison with the previous work I see as necessary.\n\n----\n\nI acknowledge that I have read the author's response. My assessment of the paper stays the same.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgSsm3TKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "21st November Update: Thank you for your detailed response. I agree with the suggested future work and revisions to the paper. However, as no updates were made to the submitted paper, I will not be raising my score as the revisions represent significant changes that I cannot support acceptance of without further peer review. I encourage the authors to carefully consider all reviewers advice in updates to the paper for a future submission elsewhere, which I think will significantly improve the paper and its potential impact on the community.\n\n--\nThis paper contributes a probabilistic framework for multi-agent RL and demonstrates a derivation of multi-agent SAC using it. The framework could be more broadly applicable if it included partial observability (as is often a requirement of multi-agent systems). The derivation could be improved by showing the full working for equation 5, as this would make the work self contained instead of assuming prior knowledge of ELBO by the reader.\n\nThe derived algorithm (previously published by Iqbal and Sha [ICML 2019] as noted in the paper) is then evaluated on 4 existing benchmark tasks against the baseline algorithm originally proposed with the environments - MADDPG. The environments represent a good range of multi-agent scenarios of suitable complexity to test modern deep RL algorithms. However, the empirical evaluation and methodology have issues that reduce the significance of their contribution.\n\nOn Page 8, it is noted that \"the value of alpha_i is empirically tuned for each environment\" but for which algorithm was it optimized? It is then noted that the values for alpha were \"found using grid search\" but details of the range of the search are not included nor details of how any other hyperparameters were set. Were other parameters tuned? If so please report all values searched for both algorithms and how they were set.\n\nAt the end of page 8, the caption of Figure 2 concludes \"it can be seen that MA-SAC controlled agents outperform MADDPG controlled agents on majority of tasks.\" This statement is not supported by the graphs in this figure. I suspect Figures 2a and b show no significant difference as the confidence intervals overlap and that Figure 2d is not significantly different throughout training but may be with a small effect size at the current end of training. Figure 2d also looks like training for longer may be beneficial. Therefore, Figure 2c is the only environment that shows a significant improvement. Please provide further evidence that MA-SAC outperforms MADDPG or weaken this conclusion. It would also be interesting to investigate deeper, why MA-SAC shows such higher performance than MADDPG in the Predator-Prey domain.\n\nOn Page 9, the conclusion is reiterated and claimed to be in comparison to a state-of-the-art algorithm. However, the benefits of SAC over DDPG have been previously shown both in single agent [Haarnoja et al, ICML 2018] and multi agent domains [Iqbal and Sha, ICML 2019]. A stronger baseline to compare against would improve the significance of any resultant improvements.\n\nThe research direction is interesting but the earlier publication of the derived algorithm (Iqbal and Sha, ICML 2019) and the issues discussed above with the experimental results lead me to conclude that the contribution is not yet sufficient to warrant publication. With further work I believe this line of work could lead to a high impact publication, but feel the paper requires more changes than are feasible within the time frame of the ICLR rebuttal period. \n\nMinor Comments:\n- Page 4, \"the transition function of underlying Markov game\" -> the transition function of the underlying Markov game\n- Page 9, \"in Figure 2c, the red curve corresponds to\" -> dark blue curve\n- Page 9, \"MA-SAC performs at least at par with MADDPG\" -> at least on par with\n- Page 9, \"outperforms it on majority of the tasks\" -> outperforms it on the majority of tasks", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper takes the perspective of "reinforcement learning as inference", extends it to the multi-agent setting and derives a multi-agent RL algorithm that extends Soft Actor Critic. Several reviewer questions were addressed in the rebuttal phase, including key design choices. A common concern was the limited empirical comparison, including comparisons to existing approaches. 
Decision: Reject