Decentralized Policy Optimization
Authors: Kefan Su
Zongqing Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning

Abstract: The study of decentralized learning or independent learning in cooperative multi-agent reinforcement learning has a history of decades. Recently empirical studies show that independent PPO (IPPO) can obtain good performance, close to or even better than the methods of centralized training with decentralized execution, in several benchmarks. However, decentralized actor-critic with convergence guarantee is still open. In this paper, we propose decentralized policy optimization (DPO), a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee. We derive a novel decentralized surrogate for policy optimization such that the monotonic improvement of joint policy can be guaranteed by each agent independently optimizing the surrogate. In practice, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. Empirically, we compare DPO with IPPO in a variety of cooperative multi-agent tasks, covering discrete and continuous action spaces, and fully and partially observable environments. The results show DPO outperforms IPPO in most tasks, which can be the evidence for our theoretical results.

Under review as a conference paper at ICLR 2023
DECENTRALIZED POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The study of decentralized learning or independent learning in cooperative multi- agent reinforcement learning has a history of decades. Recently empirical studies show that independent PPO (IPPO) can obtain good performance, close to or even better than the methods of centralized training with decentralized execution, in several benchmarks. However, decentralized actor-critic with convergence guarantee is still open. In this paper, we propose decentralized policy optimization (DPO), a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee. We derive a novel decentralized surrogate for policy optimization such that the monotonic improvement of joint policy can be guaranteed by each agent independently optimizing the surrogate. practice, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. Empirically, we compare DPO with IPPO in a variety of cooperative multi-agent tasks, covering discrete and continuous action spaces, and fully and partially observable environments. The results show DPO outperforms IPPO most tasks, which can be the evidence for our theoretical results.
INTRODUCTION
In cooperative multi-agent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) has been the primary framework (Lowe et al., 2017: Foerster et al., 2018; Sunchag al.. 2018; Rashid al., 2018; Wang al.. 2021a; Zhang et al., 2021; Yu al., 2021). Such a framework can settle the non-stationarity problem with the centralized value function, which takes as input the global information and is beneficial to the training process. Conversely, decentralized learning has been paid much less attention. The main reason may be that there are few theoretical guarantees for decentralized learning and the interpretability insufficient even if the simplest form of decentralized learning, i.e., independent learning, can obtain good empirical performance in several benchmarks (Papoudakis et al., 2021). However, decentralized learning itself still deserves attention as there are still many settings in which the global information cannot be accessed by each agent, and also for better robustness and scalability (Zhang et al., 2019). Moreover, the idea of decentralized learning is direct, comprehensible, and easy to realize in practice.
Independent Q-learning (IQL) (Tampuu et al., 2015) and independent PPO (IPPO) (de Witt et al., 2020) are the straightforward decentralized learning methods for cooperative MARL, where each agent learns the policy by DQN (Mnih et al., 2015) and PPO (Schulman et al.. 2017) respectively. Empirical studies (de Witt et al., 2020; Yu al., 2021: Papoudakis et al., 2021) demonstrate that these two methods can obtain good performance, close CTDE methods. Especially, IPPO can outperform several CTDE methods in a few benchmarks, including MPE (Lowe et al., 2017) and SMAC (Samvelyan et al., 2019), which shows great promise for decentralized learning. Unfortunately. to the best of our knowledge, there is still no theoretical guarantee or rigorous explanation for IPPO, though there has been some study (Sun et al., 2022).
In this paper, we make a further step and propose decentralized policy optimization (DPO), a decentralized actor-critic method with monotonic improvement and convergence guarantee for cooperative multi-agent reinforcement learning. Similar to IPPO, DPO is actually independent learning, because in DPO each agent optimizes its own objective individually and independently. However, unlike IPPO, such an independent policy optimization of DPO can guarantee the monotonic improvement of the joint policy.
From the essence of fully decentralized learning, we first analyze Q-function in the decentralized setting and further show that the optimization objective of IPPO may not induce the joint policy

---

Under review as a conference paper at ICLR 2023
improvement. Then, starting from the surrogate of TRPO (Schulman et al., 2015) and together considering the characteristics of fully decentralized learning, we introduce a novel lower bound of joint policy improvement the surrogate for decentralized policy optimization. This surrogate can be naturally decomposed for each agent, which means each agent can optimize its individual objective make sure that the joint policy improves monotonically. In practice, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. The idea of DPO is simple yet effective, and suitable for fully decentralized learning.
Empirically, we compare DPO and IPPO in a variety of cooperative multi-agent tasks including a cooperative stochastic game, MPE (Lowe et al., 2017), multi-agen MuJoCo (Peng et al., 2021). and SMAC (Samvelyan et al., 2019), covering discrete and continuous action spaces, and fully and partially observable environments. The empirical results show that DPO performs better than IPPO in most tasks, which can be evidence for our theoretical results.
2 RELATED WORK
CTDE. In cooperative MARI, centralized training with decentralized execution (CTDE) is the most popular framework (Lowe al., 2017; Iqbal & Sha, 2019; Foerster al., 2018: Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2021a; Zhang et al., 2021; Peng et al., 2021). CTDE algorithms can handle the non-stationarity problem in the multi-agent environment by the centralized value function. One line of research in CTDE is value decomposition (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Wang et al., 2021a), where a joint Q-function is learned and factorized into local Q-functions by the relationship between optimal joint action and optimal local actions. Another line of research in CTDE is multi-agent actor-critic (Foerster et al., 2018; Iqbal & Sha, 2019; Wang et al., 20:1b; Zhang et al., 2021; Su & Lu, 2022), where the centralized value function is learned to provide policy gradients for agents to learn stochastic policies. More recently, policy optimization has attracted much attention for cooperative MARL. PPO (Schulman al., 2017) and TRPO (Schulman et al., 2015) have been extended to multi-agent settings by MAPPO (Yu et al., 2021), CoPPO (Wu al., 2021), and HАPPO (Kuba al., 2021) respectively via learning a centralized state value function. However, these methods are CTDE and thus not appropriate for decentralized learning.
Fully decentralized learning. Independent learning (OroojlooyJadid & Hajinezhad, 2019) is the most straightforward approach for fully decentralized learning and has actually been studied in cooperative MARL since decades ago. The representatives are independent Q-lcarning (IQL) (Tan, 1993; Tampuu et al., 2015) and independent actor-critic (IAC) as Foerster et al. (2018) empirically studied. These methods make agents directly execute the single-agent Q-learning or actor-critic algorithm individually. The drawback of such independent learning methods is obvious. As other agents are also learning, each agent interacts with a non-stationary environment, which violates the stationary condition of MDP. Thus, these methods are not with any convergence guarantee theoretically, though IQL could obtain good performance in several benchmarks (Papoudakis et al., 2021). More recently, decentralized learning has also been specifically studied with communication (Zhang al., 2018; Li et al., 2020) parameter sharing (Terry al., 2020). However, in this paper, we consider fully decentralized learning as each agent independently learning its policy while being not allowed to communicate or share parameters as in Tampuu et al. (2015); de Witt et al. (2020). We will propose an algorithm with convergence guarantees in such a fully decentralized learning setting.
IPPO. TRPO (Schulman et al., 2015) is important single-agent actor-critic algorithm that limits the policy update in a trust region and has a monotonic improvement guarantee by optimizing a surrogate objective. PPO (Schulman et al., 2017) is a practical but effective algorithm derived from TRPO, which replaces the trust region constraint with a simpler clip trick. IPPO (de Witt et al., 2020) is recent cooperative MARL algorithm in which each agent just learns with independent PPO. Though IPPO is still with no convergence guarantee, it obtains surprisingly good performance in SMAC (Samvelyan et al., 2019). IPPO is further empirically studied by Yu et al. (2021); Papoudakis et al. (2021). Their results show IPPO can outperform a few CTDE methods in several benchmark tasks. These studies demonstrate the potential of policy optimization in fully decentralized learning. which will focus in this paper. Although there are some calue-based algorithms for fully decentralized learning (Tan, 1993; Matignon et al.. 2007; Palmer et al., 2017), most of them are heuristic and follow the principle different from polecy-based algorithms, so will not focus on these algorithms.
2

---

Under review as a conference paper at ICLR 2023
3 METHOD
From the perspective of policy optimization, in fully decentralized learning, we need to find an objective for each agent such that the joint policy improvement can be guaranteed by each agent independently and individually optimizing its own objective. Thus, we propose a novel lower bound of the joint policy improvement to enable decentralized policy optimization (DPO). In the following, we first discuss some preliminaries; then we analyze the critic of agent in fully decentralized learning; next we derive the lower bound and the proof for convergence; finally we introduce the practical algorithm of DPO.
3.1 PRELIMINARIES
Dec-POMDP. Decentralized partially observable Markov decision process is general model for cooperative MARL. A Dec-POMDP is tuple G = (S, A, P. O, N,,,N. S is the state space, N the number of agents, 7 E [0, 1) is the discount factor, and / = (1,2.... N) is the set of all agents. A A A₁ x A2 x ... x AN represents the joint action space, where A, is the individual action space for agent i. P(s'|s, a) S x A
[0, 1] is the transition function, and r(s, a) SS A - R is the reward function of state E S and joint action
A. Y is the observation space, and (s,,t) : S X I - Y is a mapping from state to observation for each agent i. The objective Dec- POMDP is to maximize J(Tr)
E, ( a,)] thus need to find the optimal joint policy π = arg max (TT). ). To settle the partial observable problem, history E T; = (Y x A:)" is often used to replace observation 0g E YY In fully decentralized learning, each agent i independently learns an individual policy n' (a; [2) and their joint policy π can be represented as the product of each n². Though each agent learns individual policy as T' T', in practice, in our analysis, we will assume that each agent could receive the state 8, because the analysis in partially observable environments much more difficult and the problem may be undecidable in Dec-POM (Madani et al., 1999). Moreover, the V-function and Q-function of the joint policy π are as follows,
V (s) Eant 1Q (s, a)]
(5, a) r(s, a) + P( [V (s')).
Joint TRPO Objective. In Dec-POMDR we can still obtain TRPO objective for the joint policy π from the theoretical results in single-agent RL (Schulman et al., 2015), which is referred to as the joint TRPO objective,
](neww - -[(ool)) 2
(TTew)
DRT. (Told [uew)
(3)
Rold
where Cloint (Tnew) = > Pold Σ Tnew(a (als) ( a),
where DIAXX (Told [new) max, DKL. Pxcl(Total (Tola (-1s)l [ (-|s()))), (s) Er=o Pr(ss s/Taaa) the discounted stationary distribution of the state given Toldd Aold the advantage function under Told, and C' is constant.
The joint TRPO objective, i.e., RHS of (3), is a lower bound for the difference between the new joint policy Tnew and the old joint policy Told in term of expected return. Therefore, we can use this objective as a surrogate, and maximizing this surrogate can guarantee that the policy is improving monotonically. However, the joint TRPO objective cannot be directly optimized in fully decentralized learning as this objective is involved in the joint policy, which cannot be accessed in fully decentralized learning.
We will propose a new lower bound (surrogate) for ((Tnew) ((moaa: which can be optimized in fully decentralized learning. Before introducing our surrogate, need to first analyze the critic of agent in fully decentralized learning, which is referred to as decentralized critic.
3.2 DECENTRALIZED CRITIC
In fully decentralized learning, each agent learns independently from its own interactions with the environment. Therefore, the Q-function of each agent i is actually the following formula:
(s, a:) Fr i(s,a, a;) + YLE 5, (,a'
P(-ep e. aas of 12 a()),
(5)
3

---

Under review as a conference paper at ICLR 2023
where (5-1(s, 0.) E.--iife (r(s, ds, a-1)], and x- and a-; respectively denote the joint policy and joint action all agents expect agent i. If we take the expectation Eg' Sa',~==i( ((1s), (-1s) over both sides of the Q-function of joint policy (2), then we have
Bx- 1QT ai a.i)] (s, a:) + y ( a(w.p(aaaaaaaaaa
1E.-.107 d d'll
We can see that 0 (s, a;) and Bx :(Q (s, ag a-))] satisty the same iteration. Moreover, we will show in the following that Q (8, 0z) and Ex (8, ai a-s) are just the same.
We first define an operator --4 as follows,
m Q(s, (4) - az) + (o (Q(s,,
Then we will prove that the operator r is a contraction. Considering any two individual Q-functions Q1 and Q2, we have:
mmxxx 7 I io pw aaftor 1Q 0.) 0z(s',a')) | Q1 r Q21loo
8,0,
S BB Be www.:rtt....PPl...............
[max IQ: (6', (6',')) Q2(6',a()1) s'
max 111('' (s's,aa) at) Q2(s',a)) d'
=> - Q211
So the operator TT: a-t has one and only one fixed point, which means
Q= a,) = 1Q= az a-:)], (s= = EV- (s)..
With this well-defined decentralized critic, et al., 2020). In IPPO, the policy objective of each agent can be essentially formulated as follows:
further analyze the objective of IPPO (de Witt
C (Tew)) = ''eer (s) Σ Tnew(ayls) (oa
where Asaa a:) QTole (s, as) Exjnn 1 (s, a:)] E (Aoia dis am:)]: aold
However, (6) is different from (4) in the joint TRPO objective. Thus, directly optimizing (6) may not improve the joint policy, and thus cannot provide any guarantee for convergence, the best of our knowledge. Nevertheless, it seems that (sia o;) is the only advantage formulation that can be accessed by each agent in fully decentralized learning. So, the policy objective of DPO will be derived on (6) but with modifications to guarantee convergence, and we will introduce the detail in the next section. In the following, we discuss how to compute this advantage in practice in fully decentralized learning.
As we need to calculate (s, = B (s, a, a) + Vi (s') Molaa (s) for the policy aii
update, we can approximate (sia az) with A'(s, ((s)a:) = r >VT (s),, which is an unbiased estimate of A (s, a.). though it may be with a large variance. In practice, we can follow the traditional idea in fully decentralized learning, and let each agent independently learn an individual value function ('(s). Then, we further have Á'(s,a) a;) r (s' (s)... The loss for the decentralized critic is as follows:
Centie E [(V'(s) }= where y: VV or Monte Carlo return.
(7)
There may be some ways to improve the learning of this critic, which however beyond the scope of our discussion.
3.3 DECENTRALIZED SURROGATE
We are ready to introduce the decentralized surrogate. First, we derive our novel lower bound of the joint policy improvement by the following theorem.
4

---

Under review as a conference paper at ICLR 2023
Theorem 1. Suppose wold and neew are two joint policies. Then, the following bound holds:
N
N
(Tnew)-JJC (TTuia) 2 N
Cam (Toww)- M -
Σ
Dmax (Fold (ew) C.
DKL. (Told I new)
where M = m (A and C'
44 maxi,a (
1-y
(1-7)²
are two constants.
We first consider Clolut (Teew) Lirara (Thww) According to (4) and (6), we have the following equation:
Proof.
Cloint (Tnew)
Lir ( (Teww)
Σ Pold(s)
new (asls) Σ
Σ a_d
Nnww (aoils)Aollss, a.)) - Abials, ai)
8
aj
-Baaae Brrine
Σ
(Taee (a_sls) Tod (a=rls) Aota As,
Then, have the following inequalities:
mold (Tnew)
Linesaa (Nrew)) )
< Epora Bxiiio
Σ T Told (a=:18) J Nnew (a=:))) Anta(s ci اm)ه
< Epola Exi
M "oldd (a_ Tnew (a=ss))
(M max (Ataa a))
a.a
2M poid Drv aT old (-|s)] | (-1s))]
< 2M
7 max Drv (1ss)[ (-1s)
2M
M Dme (Told (
(M 1 1
My DRa. (Fridll
(8)
S M.
Dmax (T llaalitte Thow),
where (8) is from the relationship between the total variance and KL-divergence that (rv < DKL(pi (plg)) (Schulman et al., 2015), and (9) a property of the KL-divergence, which can be obtained follows,
mmax (Total [nww) max DKL. (Tatal (-|s)] new (-1s))
a
max DELL (F (-|s)] F (-1s))
Σ
8
max DELL ( (- (-1s)
Dinax (m (ow)) Σ
(10)
From (9), further obtain the following inequality,
Cloint (Tnew)
Ci ( (Ti
2 -M
Diaax (Tr [
(11)
5

---

Under review as a conference paper at ICLR 2023
Next will prove this theorem, starting from (3),
new
J (Todd) 2 Cjoint (Tnww) C. DKT.
knass (Told Tnow)
N
Cioint (Teww) -
C. DRL (Told know)
M
(T
DL (TotallTiew))
C. D nnww (old
(12)
N
N
N
N
Wold (Tnww)
M
Σ
DKL D (Totall - C DKL (Tnew Told)
(13)
N
N
N
N
N
N
Wold (
(new)
M
DLL (T FotallThew
C
Dmax (Tolel (eew) (14)
N
N
N
N
N
(neww
M
DRE ( (Toial -
max (T (how)
N
Wold
L Rold
(neww)
M
Σ
V KIL (T (hww) C.
Dmax (T || k(ew).
(15)
The inequality (12) the direct application the inequality (11), The inequality (13) is from the Cauthy- Sehhh-- inequality,
Duax KL (Taaa
T knew)
Σ jti
DRL ( (Tola wdew)
jtt
N(N 1)
Σ
Die ( (l
The inequality (14) is from (10), while the inequality (15) is from the simple inequality VE, a E. Va (a, 2 0, Vi).
The lower bound Theorem 1 is dedicated to decentralized policy optimization, because it can be directly decomposed individually for each agent decentralized surrogate. From Theorem 1, if set the policy optimization objective of each agent as
+ ( (T
.M / DRI. (Taal ( C DRL ( ITT)):
A new
arg Max
(16)
then we have neww) 2 J (TToaa from TRPO (Schulman al., 2015). Moreover, the objective J(T) is bounded, the convergence of (J(x')) } is guaranteed, where n' is the joint policy after t iterations according to (16). Therefore, the joint policy of agents improves monotonically and converges to sub-optimum by fully decentralized policy optimization, i.e., independent learning. Note that this result is under the assumption that each agent can obtain the state, and practice each agent will take the individual trajectory T; the approximation to the state.
3.4 ALGORITHM
DPO is with simple idea that each agent optimizes the decentralized surrogate (16). However, face the same trouble as TRPO that the constant M and C are large and if we directly optimize this objective, then the step size of the policy update will be small.
To settle this problem, absorb the idea of the adaptive coefficient in PPO (Schulman et al., 2017).
We adaptive coefficients 8 and B3 to replace the constant M and C and additionally replace

---

Under review as a conference paper at ICLR 2023
Algorithm 1. DPO
1:
for episode = 1 to M do eepssode_enngeee do
2:
for = to
3:
select action
(-(ss
4: execute action a, and observe reward r and next state 8
5:
collect ",,
6:
end for
7: Update decentralized critic according to (7)
8: Update policy according to the surrogate (17)
9: Update øi and 83 according (18).
10:
end for
the maximum KL-divergence with the average KL-divergence (Schulman et al., 2015). In practice, we will actually optimize the following objective
nnw = arg max ( fmmia (T') (
Bi avz (Taall==)) - 122Dii avy ( (molll
(17)
where DaVE (Ta ( Eswnan
As for the adaption of 88 and $2, we need to define a hyperparameter dtargets which can be seen as a ruler for the average KL-divergence DRT, (T (aalll for each agent. D (T [ Thew) is close to dearget, then we believe current B1 and B3 are appropriate. If DKL. om ll (eew exceeds dearget too much, we believe Bi and 82 are small and need to increase and vice versa. practice, we will use the following rule to update 34 and f3z
If D Di KL (Toid ((ewww dearget 8, then Bj + 8) #
Vj {1,2) If D Davg (# (eoall/few) < dearget 18, then + B [
(18)
We choose the constants 8 1.5 and critic, we just follow the standard method in PPO. Then, we can have the fully decentralized learning procedure DPO for each agent in Algorithm
the choice PPO (Schulman et al., 2017). As for the
The practical algorithm of DPO actually uses some approximations to the decentralized surrogate. Most of these approximations are traditional practice in RL or with no alternative in fully decentralized learning yet. We admit that the practical algorithm may not maintain the theoretical guarantee. However, we need to argue that we go one step further to give decentralized surrogate in fully decentralized learning with convergence guarantee. We believe and expect that a better practical method can be found based on this objective in future work.
EXPERIMENTS
In this section, we compare the practical algorithm of DPO with IPPO (de Witt et al., 2020) in variety of cooperative multi-agent environments, including cooperative stochastic game, MPE (Lowe et al., 2017), multi-agent MuloCo (Peng et al., 2021), and SMAC (Samvelyan et al., 2019), covering both discrete and continuous action spaces, and fully and partially observable environments. As we consider fully decentralized learning, in the experiments agents do not use parameter-sharing sharing parameters should be considered as centralized learning (Terry al., 2020). all experiments, the network architectures and common hyperparameters of DPO and IPPO are the same for fair comparison. More details about experimental settings and hyperparameters are available in Appendix. Moreover, all the learning curves are from 5 random seeds and the shaded area corresponds the 95% confidence interval.
4.1 DIDACTIC EXAMPLE
First, cooperative stochastic game didactic example. The cooperative stochastic game is with 100 states, agents and each agent has 5 actions. All the agents share a joint reward function. The reward function and the transition probability are both generated randomly. This stochastic
7

---

Under review as a conference paper at ICLR 2023
Sramare Same
Storbadd Game
100
402
200
305
200
200
102
2P1
39TMAA
mua
100
+0.01
100200
200003
300000
400000
500030
100002
200000
200000
400000
500000
steps
Stees
(a) DPO compared with TPPO
(h) the influence of drarget
Figure 1: Empirical studies of DPO on the didactic example: (a) learning curve of DPO compared with IPPO and the global optimum; (b) the influence of different values of dearget on DPO, x-axis is environment steps.
umpossppad
Iea control
orde contro
-10
muan 4260000 PorurrS
-20
-30
-30
-10
-00
-40
DPO
-50
DPC
DPC
-20
I1F0
P7C
IPP
0.0
0.2
0.4
9.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
stess
105
steps
165
steps
100
Figure 2: Learning curve of DPO compared with IPPO in 5-agent simple spread, 5-agent line control,
and 5-agent circle control MPE, where x-axis is environment steps.
game has a certain degree of complexity which is helpful to distinguish the performance of DPO and IPPO. On the other hand, this environment tabular which means training in this environment is fast and we can do ablation studies efficiently. Moreover, we can find the global optimum by dynamic programming to compare with in this game.
The learning curves in Figure la show that DPO performs better than IPPO and learns a better solution in this environment. The fact that DPO learns sub-optimal solution agrees with our theoretical result. However, the sub-optimal solution found by DPO is still away from the global optimum. This means that there still improvement space.
On the other hand, we study the influence of the hyperparameter dearger on DPO. We choose deorget = 0.001, 0.1, The empirical results are shown in Figure 1b. We find that when deorget small, the coefficient 31 and 32 are more likely to be increased and the step size of the policy update limited. So for the case that diarget = 0.001, 0.01, the performance of DPO is relatively low. And when dearget is large, the policy update may be out of the trust region. This can be witnessed by the fluctuating learning curve of the case dzarget = 1. So we need to choose an appropriate value for dearget and in this environment we choose dearget the learning curve of DPO in Figure 1a. We found that the appropriate value for dearget changes in different environments. In the following, we keep dearger to be the same for tasks of the same environment. There may be some better choices for drarget, but it is bit time-consuming and out of the range of our discussion.
0.1, which is also
4.2 MPE
MPE is a popular environment in cooperative MARL. MPE is 2D environment and the objects in MPE environment are either agents or landmarks. Landmark is part of the environment, while agents can move in any direction. With the relation between agents and landmarks, we can design different tasks. We use the discrete action space version of MPE and the agents can accelerate or decelerate in the direction of x-axis or y-axis. We choose MPE for its partial observability. We take dearget = 0.01 for all MPE tasks.
The MPE tasks we used for the experiments are simple spread, line control, and circle control which are originally used in Agarwal et al. (2020). In our experiments, we set the number of agents N = in all these three tasks. The empirical results are illustrated in Figure 2. We can find that although DPO may fall behind IPPO at the beginning of the training in some tasks, DPO learns a better policy in the end for all three tasks.
8

---

Under review as a conference paper at ICLR 2023
201
D70
D70
303
DPC
DPO
3P0
P10
P70
P70
PP0
1P0
Figure 3: Learning curve of DPO compared with IPPO in 3-agent Hopper, 3-agent HalfCheetah. 3-agent Walker2d, 4-agent Ant, and 17-agent Humanoid in multi-agent MuloCo, where x-axis is environment steps.
DF0
310
D10
DPO
DPO
PH0
1P0
IPP
IPPO
Figure 4: Learning curve of DPO compared with IPPO in 2s3z, 8m, 3s5z, 27m_ in SMAC, where x-axis is environment steps.
/_____________________________ vs_ 30m, and MMM2
4.3 MULTI-AGENT MuJoCo
Multi-agent MuJoCo is a robotic locomotion control environment for multi- mullti-aeent settings, which is built upon single-agent MuJoCo (Todorov et al., 2012). In multi-agent MuJoCo, each agent controls one part of a robot to carry out different tasks. We choose this environment for the reason of continuous state and action spaces. We select 5 tasks for our experiments: 3-agent Hopper, 3-agent HalfCheetah, 3-agent Walker2d, 4-agent Ant and 17-agent Humanoid. In all these tasks, we set agent_obsk=2. We take diarget 0.001 for all multi-agent MuJoCo tasks.
The empirical results are illustrated in Figure 3. We can find that in all five tasks, DPO outperforms IPPO, though in 3-agent HalfCheetah DPO learns slower than IPPO at the beginning. The results on multi-agent MuJoCo verify that DPO is also effective in facing continuous state and action spaces. Moreover, the better performance of DPO in the 17-agent Humanoid task could be evidence of the scalability of DPO.
4.4 SMAC
SMAC is a partially observable and high-dimensional environment that has been used in many cooperative MARL studies. We select five maps in SMAC, 2s3z, 8m, 3s5z, [rm__v_30mm and MMM2 for our experiments. We take dearget 0.02 for all SMAC tasks.
The empirical results are illustrated in Figure 4. The two super hard SMAC tasks (27m_vs. 1Fm__ss_3mm_ and MMM2) are too difficult for both DPO and IPPO to win, so we use episode reward as the metric to show their difference. DPO performs better than IPPO in all five maps. We need to argue that though we have controlled the network architectures of DPO and IPPO to be the same, in experiments each agent has its individual parameters which increases the difficulty of training. So our results in SMAC may be different from other works. Although IPPO has been shown to perform well in SMAC (de Witt et al., 2020; Yu et al., 2021; Papoudakis et al., 2021), DPO can still outperform IPPO, which verifies the effectiveness of the practical algorithm of DPO in high-dimensional complex tasks and can also be evidence of our theoretical result. Again, the better performance of DPO in 27m_ vs 30m shows its good scalability in the task with many agents.
5 CONCLUSION
In this paper, we investigate fully decentralized learning in cooperative multi-agent reinforcement learning. We derive a novel decentralized lower bound for the joint policy improvement and we propose DPO, fully decentralized actor-critic algorithm with convergence guarantee and monotonic improvement. Empirically, we test DPO compared with IPPO in a variety of environments including cooperative stochastic game, MPE, multi-agent MuJoCo, and SMAC. covering both discrete and continuous action spaces, and fully and partially observable environments. The empirical results show the advantage of DPO over IPPO, which can be evidence for our theoretical results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent team. In International Conference on Autonomouus Agents and Multiagent Systems (AAMAS), 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.05333 2020.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Shariq Iqbal and Fei Sha. ector-attention-criiee for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2019.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109. 11511, 2021.
Wenhao Li, Bo Jin, Xiangfeng Wang, Junchi Yan, and Hongyuan Zha. F2a2: Flexible fully- decentralized approximate actor-critic for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2004.1111 11145, 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed
environments. In Advances in Neural Information
Processing Systems (NeurIPS), 2017.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and partially observable markov decision problems. In AAAI/IAA,, pp. 541-548, 1999.
Læëtitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning algo- rithm for decentralized reinforcement learning in cooperative multi-agent teams. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2007.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529-53) 2015.
Afshin OroojlooyJadid and Davood Hajimezhad. A review of cooperative multi- deep reinforce- ment learning. arXiv preprint arXiv:1908.033966 2019.
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. arXiv preprint 1707:11777.04402 2017.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. Advances Neural Information Processing Systems (NeurIPS), 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902.04043 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICMI., 2015.
10

---

Under review as a conference paper at ICLR 2023
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Kefan Su and Zongqing Lu. Divergence-reguulriize multi-agent actor-critic. In International Conference on Machine Learning (ICML), 2022.
Mingfei Sun, Sam Devlin, Katja Hofmann, and Shimon Whiteson. Monotonic improvement guaran- tees under non-stationarity for decentralized ppo. arXiv preprint arXiv:2202.00082 2022.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi. Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Interna- tional Conference on Autonomous Agents and MultiAgent Systems (AAMAS), 2018.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. arXiv preprint arXiv:1511.08779 2015.
Ming Tan. Multi-agent agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning (ICML), 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005. 1..5..1366 13625, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. In International Conference on Learning Representations (ICLR), 2021a.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjic Zhang. Dop: Off-policy multi- agent decomposed policy gradients. In International Conference on Learning Representations (ICLR), 2021b.
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo, et al. Coordinated proximal policy optimization. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.03393 03939, 2020.
Hsuan- Yu Yao, Ping-Chun Chun Hsieh, Kuo-Hao Ho, Kai-Chun Hu, Liang-Chur Ouyang, Wu, al. Hinge policy optimization: Rethinking policy improvement and reinterpreting ppo. arXiv preprint arXiv:2 10. 13799, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Baar. Fully decentralized multi- agent reinforcement learning with networked agents. In International Conference on Machine Learning (ICML), 2018.
Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint 1911v:9911.10635, 2019.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropp multi-agent reinforcement learning. In International Conference Machine Learning (ICML), 2021.

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL SETTINGS
A.1
MPE
The three tasks built the origin MPE (Lowe al., 2017) (MIT license) and are originally used in Agarwal et al. (2020) (MIT license). The objective in these three tasks are listed as follows:
Simple Spread: There are N agents who need occupy the locations of N landmarks.
Line Control: There are N agents who need to line up between 2 landmarks.
Circle Control: There are N agents who need to form a circle around a landmark.
The reward in these tasks is the distance between all the agents and their target locations. We set the number of agents N = for these three tasks in our experiment.
A.2 MULTI-AGENT MuJoCo
Multi-agent MuJoCo (Peng et al., 2021) (Apache-2.0 license) is a robotic locomotion task with continuous action space for multi-agent settings. The robot could be divided into several parts and each part contains several joints. Agents this environment control part of the robot which could be different varieties. So the type of the robot and the assignment of the joints decide a task. For example, the task "HalfCheetah-3xx x2 means dividing the robot "HalfCheetah" into three parts for three agents and each part contains 2 joints.
The details about our experiment settings in multi-agent Mujoco are listed in Table 1. The configura- tion defines the number of agents and the joints of each agent. The 'agent obsk' defines the number of nearest agents an agent can observe.
Table 1: The task settings of multi-agent MuloCo
task
configuration
agent obsk
HalfCheetah
3x2
2
Hopper
3xl
2
Walker2d
3x2
2
Ant
4x2
2
B TRAINING DETAILS
Our code is based on the open-source code' of MAPPO (Yu et al., 2021) (MIT license). We modify the code for individual parameters and ban the tricks used by MAPPO for SMAC. The network architectures and base hyperparameters of DPO and IPPO are the same for all the tasks in all the environments. We use 3-layer MIPs for the actor and the critic and ReLU non-linearities. The number of the hidden units of the MLP is 128. We train all the networks with an Adam optimizer. The learning rates of the actor and critic samples is 15 which is the recommended value in Yu et al. (2021). For IPPO, the clip parameter is 0.2 which is the same as Schulman et al. (2017). For DPO, the initial values of the coefficient Bf and 3 are 0.01. The value of drarget is 0.1 for the cooperative stochastic game, 0.01 for MPE, 0.001 for multi-agent MuJoCo, and 0.02 for SMAC.
both 5e-4. The number of epochs for every batch of
The version of the game StarCraft2 in SMAC is 4.10 for our experiments in all the SMAC tasks. We set the episode length of all the multi-agent MuloCo tasks as 1000 in all of our multi-an multi-agent MuJoCo experiments. We perform the whole experiment with a total of four NVIDIA A100 GPUs. We have summarized the hyperparameters in Table 2.
12

---

Under review as a conference paper at ICLR 2023
Table 2: Hyperparameters for all the experiments
hyperparameter
value
MLP layers
hidden size
128
non-linear
ReLU
optimizer
Adam
actor |r critic_In
5e-4
Se-4
numbers of epochs
15
initial 18]
0.01
initial 83
0.01
1.5
dearget
different for environments as aforementioned
clip parameter for IPPO
0.2
C ADDITIONAL RESULTS
Schulman al. (2017) actually proposed two versions of PPO. The first version, which is also the most popular version, is with the clip trick. The second version is directly optimizing the penalty formula with adaptive coefficients and we refer this algorithm as PPO-KL. IPPO (de Witt et al., 2020) is actually extended from the first version, while the practical algorithm of DPO is similar to the second version. The main difference between DPO and PPO-KL is the term of the square root of the KL-divergence in the policy loss. We modify IPPO by making each agent learn with PPO-KL to obtain IPPO-KL. On the other hand, independent Q-learning (IQL) (Tan, 1993) is a classic independent learning algorithm. IQL could obtain good performance in some tasks but is not with any theoretical guarantee, to the best of our knowledge.
Stochastic
500
meen epivnde rewernts
400
300
200
100
OPTIMAL
DPO
IQL
1000000
200000
300000
400000
500000
steps
Figure 5: Learning curve of DPO compared with IPPO, IPPO-KL, IQL, and the global optimum in the cooperative stochastic game, where x-axis environment steps.
For the completeness of our experiments, we test the performance of IPPO-KL and IQL in the cooperative stochastic game and the empirical result is illustrated in Figure 5. We find that the performance of IPPO-KL is close to DPO but little bit lower and more unstable. This can be explained as the policy loss of -PLO-KLIP is actually biased approximation of DPO, which omits the square root term. The theoretical guarantee of DPO may also help IPPO-KL perform better than IPPO in this game, while the bias in the policy loss of IPPO-KI. makes it perform worse than DPO. As for IQI., performance lower than IPPO though converges faster. Since IQL. performs worse than IPPO in such a didactic environment and IQL is value-based algorithm and is out of the scope of our discussion, we do not take IQL as a baseline in other environments. Moreover, we think the essential relations among IPPO, IPPO-KL, and DPO can be clearly witnessed from the empirical results in the cooperative stochastic game. For other tasks, we focus on the mainstream version of IPPO in de Witt et al. (2020); Yu et al. (2021); Papoudakis et al. (2021).
Besides our empirical results, we would like to share our views on the difference between DPO and IPPO and give some intuitive ideas. KI. regularization and ratio clipping similar in the
13

---

Under review as a conference paper at ICLR 2023
single-agent setting, but they not supposed to be similar in multi-agent settings. The 'correct' ratio clipping in multi-agent setting according to the theory of PPO should clip over the joint policy ratio [olw(a]s]
Told (a)s) ()) IPPO just clips individual policy ratio [au=(asls) for each agent i which may be
[aig()
enough to realize the "correct' ratio clipping. We could find more discussion about this in the CoPPO
(Wu et al., 2021) paper. So IPPO is not supposed enjoy the theoretical results of DPO.
We could rewrite the objective of IPPO for agent with similar formulation in HPO (Yao et al., 2021) as follows:
CLIPPO (T (Tnww) =
new (a, Abia az)l (sign(Asia a:)), us (s, a,) l,e), Σ (ota(s) Σ
Bold
where ((y, 2y,,,cc) = max c - x x is the hinge loss and u a.)
is the ratio.
If we follow the same idea PPO, then IPPO is the 'correct' ratio clipping version for the surrogate of DPO. However, the effectiveness of this ratio clipping formulation in theory is still open in decentralized learning since there is not any convergence guarantee for IPPO, the best of our knowledge.
Though the effectiveness of IPPO in theory is beyond the scope of our paper, we could provide intuitive explanation for the fact that the performance of DPO can surpass IPPO from this formulation and the analysis in HPO. In the proof of HPO, there critical assumption that the sign of the estimated advantage is the same as that of the true advantage (Assumption 4 in Section 2.3 in Yao al. (2021)). And HPO also shows that the sign of the advantage more important than the value for this formulation of PPO-clip. In decentralized learning, both DPO and IPPO are facing the difficulty of learning the individual advantage function as there may be noise in the individual value function. However, the objective of DPO is continuous and the objective of IPPO is discrete for Aiga(Arla a))). So the impact of the noise in the value function may be larger on IPPO than DPO.
D DISCUSSION
In the paper, we derive a novel lower bound that can be naturally divided into independent surrogate (16) for each agent. By each agent optimizing this surrogate, the monotonic improvement of the joint policy can be guaranteed in fully decentralized settings. However, the practical algorithm of DPO takes the formula of (17) with several approximations. How to solve the optimization of (16) more precisely is left as future work.
Moreover, we expect our work could provide some insights for future studies on fully decentralized multi-agent reinforcement learning, since current methods still have a gap from the optimum as shown in Figure 5,
E ADDITIONAL RESULTS FOR REBUTTAL
We first add some experiments results of the ablation study about the objective in (17). We consider two ablation methods: in the first one, we will keep / = to eliminate the influence of the term
HalfCheeth--22
Whi 2
Antr2
4000
1600
1330
epteate
2000
1400
w/t
500
DPO win
1000
000 w/n
D70 W/s spr
DPQ ano sert
800
DPO w/t wor:
5.0
0.5
1.0
1.5
2.0
0.00
0.50
0:75
1.00
1.25
1.75
0.0
0.5
1.5
2.0
265
24b
Figure 6: Learning curves of DPO compared with DPO _/o_ Wor_srtt and DPO_w/o kl in 3-agent HalfCheetah, 3-agent Walker2d, and 4-agent Ant multi-agent MuJoCo, where x-axis is environment steps.
14

---

Under review as a conference paper at ICLR 2023
253z
0.-
0.
0.4
02
0.9
SPO_whh_amm
0.0
0.2
0.4
0.5
50000
100000 150000 200000 250000 300000 350000 400000
stegs
led
Figure 7: Learning curves of DPO compared with DPO__Wo_sqT and DPO_w/d w/o_//ookkk kl on 2s3z and 8m in SMAC, where x-axis is environment steps.
V (T ". which is actually the same as IPPO-KI; in the second one, we will keep 83 = 0 to eliminate the influence of the term DiL (T l²). The other parameters are controlled to be the same as DPO. We will call these two methods DPO w/o_ sqrt and DPO_ _/wo kl respectively. We select three multi-agent MuJoCo tasks (3-agent HalfCheetah, 3-agent Walker2d and 4-agent Ant) and two SMAC tasks (2s3z and 8m) for this ablation study. The empirical results are included in Figure
and Figure We find that eliminating either the term v DT.. (Tala [[')) or the term DRT, aaa ( will lower the performance of DPO in all these tasks. This could be evidence for the significance of our novel lower bound (16) and the objective (17).
Moreover, further study the influence the values of 5 and for the adaptive adjustments of the coefficient 31 and 83. We test seven different choices of á and w as (6, = (1.5, 2), (1.5, 4), 6). (1.1, 2), (3, (), (3, 6). (1.1, 6) in the cooperative stochastic game. The pirical results are included in the Figure 8. We find that the influence different 5 and w is relatively limited and the adaptive adjustment is not very sensitive to them. This conclusion is similar the PPO paper (Schulman et al., 2017).
Stochastic Game
400
mean episode rewards
300
200
6=1.5 w=2
6=1.1,w==
100
IPPO
6=3_w=2
6=1.5 w=4
6m3,w=6 6=1.5_ 6
0
6=1.1_w=6
0
100000
200000
300000
400000
500000
steps
Figure 8: Learning curves of DPO with different values of 6 and w in the cooperative stochastic game, where x-axis is environment steps.
15


Reviews:
[{"review_id": "t8SVEM9N6K", "reviewer": "Reviewer_KSta", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated the possibility of decomposing the trust region surrogate over joint policies into the respective surrogates for each individual agent's policies for cooperative multi-agent reinforcement learning. The reported experiment results show that the newly developed DPO algorithm based on the decomposed surrogates can outperform a multi-agent system with independent PPO learners.", "summary_of_the_review": "It is important to study new learning mechanisms to support multi-agent reinforcement learning in a decentralized manner. However, the key assumption of this paper seems to have some major limitations. The technical novelty of the newly developed performance bound also remains questionable to a certain extent. Furthermore, experimental study is at the limited side and does not convincingly show the true advantage of the newly developed algorithm.", "strength_and_weaknesses": "Strength:\nIt is important to study new learning mechanisms to support multi-agent reinforcement learning in a decentralized manner. This paper explored an interesting direction of decomposing the trust region surrogate to facilitate independent learning among multiple agents. Experiment results show that this direction of research is promising.\n\nWeakness:\nThe authors argued that independent learners in a multi-agent system can often outperform agents that adopt CTDE for policy training. However, it remains unclear why independent learners without considering inter-agent interactions can outperform a learning system that explicitly handles inter-agent interactions. Under what conditions will independent learners perform well and why? Besides the potential performance advantage, what are the key benefits of supporting independent learners? The research motivation of this paper may need to be strengthened to answer all these questions.\n\nThe paper assumes that all learning agents have direct access to the full state information. However, this assumption is often not valid in a multi-agent system. In fact, with full observability, it is not difficult for each agent to understand the impact of its decision/action on the learning environment and other agents. Therefore, achieving decentralized policy training may not be very difficult. Hence the technical contribution of building a decentralized policy training algorithm under this assumption may need to be justified more. The practical usefulness of the new algorithm also should be investigated more, given the algorithm's critical reliance on the full observability assumption.\n\nThe technique for critic training in (7) does not seem to have any difference from critic training in the single-agent setting. Existing research works showed that using a single-agent approach to train critics in a multi-agent setup can be unstable and ineffective under certain conditions, especially when all agents are constantly improving their own policies. It is not clear under what conditions the stability (or convergence) of critic learning can be guaranteed and whether those conditions are realistic for multi-agent reinforcement learning.\n\nRegarding the new lower bound for each agent in (16), the middle term with $\\sqrt{D^{max}_{KL}}$ can be simply removed by adjusting $C$ in the third term. Given that, the new surrogate in DPO is essentially the same as the TRPO surrogate for single-agent reinforcement learning. As a result, I can hardly understand how the new surrogate, as a result of including a removable middle term, can facilitate effective inter-agent coordination/cooperation. It also remains questionable how DPO is fundamentally different from a direct extension of TRPO to the multi-agent setup. What are the technical advantages of DPO, in comparison to independent TRPO agents?\n\nThe new DPO algorithm introduces several new hyper-parameters such as $d_{target}$, $\\delta$, $\\omega$, $\\beta_1^i$ and $\\beta_2^i$, that may need to be fine-tuned. The introduction of these hyper-parameters may make it difficult to apply DPO to a wide range of problems.\n\nMore technical details may need to be provided regarding some statements in the paper. For example, it is stated on page 7 that \"DPO actually some approximations to the decentralized surrogate\". What approximations does this statement refer to?\n\nThe authors argued that DPO can be guaranteed to converge. However, I cannot find a thorough analysis on the convergence properties of DPO. In my opinion, a high-level argument regarding the adoption of the trust-region surrogate is not sufficient to show the convergence of the DPO algorithm. Perhaps the claim on algorithm convergence should be adjusted in the paper.\n\nWhile the experiment results appear promising, DPO is only compared with IPPO in the experiments. It is not clear whether DPO can outperform state-of-the-art CTDE algorithms on any of the benchmark problems. It also remains unclear whether DPO can outperform independent TRPO agents. Hence it is questionable whether the observed performance difference is due to the use of TRPO for policy training or due to the development of the new decentralized surrogate.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qtF0OPp9djo", "reviewer": "Reviewer_9jhx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on the setting of decentralized learning, the paper proposes a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee, where each agent independently optimizes the surrogate. The experiment results show that the proposed algorithm DPO outperforms IPPO in most tasks.", "summary_of_the_review": "Based on the assumption that each agent could receive the state, the paper proposes a new optimization objective to guarantee the whole team's monotonic improvement with an sqrt of the KL divergence between the last and updated policies. However, further discussion and experiments are required to show the necessity of the new optimization term based on a fair comparison with IPPO. Meanwhile, further analysis of the influence of partial observation in both theoretical and experimental parts is necessary. Therefore, the paper requires a major revision.", "strength_and_weaknesses": "**Strengths**\n1. The paper proposes a well-motivated technique for monotonic improvement in multi-agent optimization based on decentralized learning. The derivation is clear and easy to follow.\n2. The proposed method and IPPO are compared in multiple multi-agent environments, including MPE, multi-agent MuJoCo, and SMAC.\n\n**Weaknesses**\n1. The experiment is not sufficient enough to convince the benefits of the proposed method.\n   a) The details of IPPO are not mentioned. To limit the KL divergence between the last and updated policies, one can use a clip function or an adaption of beta as DPO. Which one is used in this paper?\n   b) Fig.1 (b) shows that d_target dramatically influences the final performance. The performance of DPO with the worst hyper-parameter is worse than IPPO. If IPPO uses an adaption of beta as DPO, will its performance become better? Is IPPO’s best d_target the same as that in DPO?\n   c) The hyper-parameter d_target is changed a lot across different domains. Is there not a single parameter configuration that could perform well in all environments? More studies are required to show the sensitivity of this hyper-parameter in different multi-agent environments. \n   d) Following c), fine-tuning d_target might lead to a dramatically different performance in various environments. The reviewer wonders whether IPPO is also fine-tuned for a fair comparison.\n\n2. The paper proposes a new policy optimization objective for each agent as shown in Eq. 16 based on the lower bound introduced in Eq. 15. However, the sqrt of the KL divergence in Eq. 15 can be easily replaced by the KL divergence itself (for example, based on a piecewise function with a different weight) for a further lower bound. Based on this further lower bound, the policy optimization objective of each agent will be similar to that in IPPO. Thus, it is quite important for the paper to show more experiments and details, as mentioned above, to clarify the importance of explicitly optimizing the sqrt of the KL divergence between the last and updated policies.\n\n3. The paper holds an assumption for the theoretical guarantee that each agent could receive the state. This is reasonable because the analysis in partially observable environments is much more difficult, and the problem may be undecidable in Dec-POMDP. However, for decentralized learning, one crucial factor influencing agents' cooperation is incomplete information. It will be helpful to discuss the influence of partial observation on theoretical analysis.\n\n4. This paper does not discuss the limitations of the proposed method.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "45MxourXhw", "reviewer": "Reviewer_byTG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a fully decentralized policy optimization (DPO) algorithm in cooperative multi-agent reinforcement learning (MARL). Based on the surrogate loss of the trust-region policy optimization for the joint policy, the authors derive a lower bound of the joint policy improvement under the assumption that each agent can obtain the state, and joint policy is represented as the product of individual policy. This agent-wise separable loss is optimized using two adaptive coefficients to overcome small step sizes in a naive application. Experiments in several cooperative MARL environments show that the proposed DPO can perform better than independent PPO under fully decentralized settings.\n", "summary_of_the_review": "Although the authors provided theoretical insight into policy optimization in a fully decentralized MARL setting, the novelty and reproducibility are not well supported. In addition, several parts of the paper should be clarified by performing additional experiments. \n", "strength_and_weaknesses": "Pros: The authors dealt with a theoretical derivation of the suggested surrogate loss.\n\nCons: Several parts of the paper should be clarified, and additional experiments should be performed. It was hard to check the reproducibility. Please refer to the detailed questions below.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject