A Fully Decentralized Surrogate for Multi-Agent Policy Optimization
Authors: Kefan Su
Zongqing Lu
Venue: Accepted by TMLR ()

Abstract: The study of fully decentralized learning or independent learning in cooperative multi-agent reinforcement learning has a history of decades. Recent empirical studies have shown that independent PPO (IPPO) can achieve good performance, comparable to or even better than the methods of centralized training with decentralized execution, in several benchmarks. However, a decentralized actor-critic algorithm with a convergence guarantee is still an open problem. In this paper, we propose decentralized policy optimization (DPO), a decentralized actor-critic algorithm with monotonic improvement and convergence guarantee. We derive a novel decentralized surrogate for policy optimization such that the monotonic improvement of joint policy can be guaranteed by each agent independently optimizing the surrogate. For practical implementation, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. Empirically, we evaluate DPO, IPPO, and independent Q-learning (IQL) in a variety of cooperative multi-agent tasks, covering discrete and continuous action spaces, as well as fully and partially observable environments. The results show DPO outperforms both IPPO and IQL in most tasks, which serves as evidence for our theoretical results. The code is available at https://github.com/PKU-RL/DPO.

Published in Transactions on Machine Learning Research (01/2024)
A Fully Decentralized Surrogate for Multi-Agent Policy Opti- mization
Kefan Su School of Computer Science Peking University
eukefon@pku.edu..om
Zongqing Lut
zongging. edu.ccm
School of Computer Science Peking University
Reviewed on OpenReview: https: // openreview. net/ per/ff Ed=NpppU90wlD
Abstract
The study of fully decentralized learning or independent learning in cooperative multi-agent reinforcement lcarning has a history of decades. Recent empirical studies have shown that independent PPO (IPPO) can achieve good performance, comparable to or even better than the methods of centralized training with decentralized execution, several benchmarks. However, docentralized actor-critic algorithm with a convergence guarantee is still an open problem. In this paper, we propose decentralized policy oplimization (DPO), decentralized actor-critic algorithm with monotonic improvement, and convergence guarantee. We derive novel decentralized surrogate for policy optimization such that the monotonic improvement of joint policy can be guaranteed by each agent independently optimizing the surrogate. For practical implementation, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. Empirically, we evaluate DPO, IPPO, and independent Q-learning (IQL) in a variety of cooperative multi-agent tasks, covering discrete and continuous action spaces, as well as fully and partially observable environments. The results show DPO outperforms both IPPO and TQL in most tasks, which serves evidence for our theoretical results. The code is available at https: attpp://gittes //gtth//....tt/// COR/O.URLL/DOO.
1 Introduction
In cooperative multi-agent reinforcement learning (MART.), centralized training with decentralized execution (CTDE) has been the dominant framework (Lowe et al., 2017; Foerster et al., 2018; Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2021a; Zhang et al., 2021; Yu et al., 2021). Such framework resolves the non-stationarity problem with the centralized value function that takes the global information input, making it beneficial to the training process. Conversely, decentralized learning has received less attention. The main reason is likely due to the fact that few theoretical guarantees exist for decentralized learning and the interpretability is insufficient even though the simplest form of decentralized learning, í.e., independent learning, may achieve good empirical performance several benchmarks (Papoudakis et al., 2021). However, decentralized learning itsclf still should be considered as there are still many scttings in which the global information is unavailable, and also for better robustness and scalability (Zhang al., 2019). Moreover, decentralized learning is straightforward, comprehensible, and easy to implement in practice.
Independent Q-learning (IQL) (Tampuu et al., 2015) and independent PPO (IPPO) (de Witt et al., 2020) are the straight.forwal decentralized learning methods for cooperative MARL, where each agent learns the policy by DQN (Muih et al., 2015) and PPO (Schulman et al., 2017) respectively. Empirical studies (de Witt et al., 2020; Yu et al., 2021; Papoudakis al., 2021) demonstrate that these two methods can obtain good
*Corresponding Author

---

Published in Transactions on Machine Learning Research (01/2024)
performance, close to CTDE methods. Especially, IPPO can outperform several CTDE methods in few benchmarks, including MPF (Towe et al., 2017) and SMAC (Samvelyan et al., 2019), which shows great promise for decentralized learning. Unfortumately, to the best of our knowledge, there is still no theoretical guarantec or rigorous explanation for IPPO, though there has been some study (Sun et al., 2022).
In this paper, we take a step further and propose decendralized policy oplimization (DPO), a fully decentralized actor-critic method with monotonic improvement and convergence guarantee for cooperative MARL. Similar to IPPO, DPO is actually independent learning as each agent optimizes its own objective individually and independently in DPO. However, unlike IPPO, such an independent policy optimization of DPO can guarantee the monotonic improvement of the joint policy.
From the essence of fully decentralized learning, we first analyze the Q-function in the decentralized set setting and further show that the optimization objective of IPPO may not induce joint policy improvement. Then, starting from the joint monotonic objective from the thcorctical results in singlo-agent RL (Grudzien et al., 2022; Lan, 2023) and together considering the characteristics of fully decentralized learning, we introduce a novel lower bound of joint policy improvement as the surrogate for decentralized policy optimization. This surrogate can be naturally decomposed for each agent, which means each agent can optimize its individual objective to make sure that the joint policy improves monotonically. Practically, this decentralized surrogate can be realized by two adaptive coefficients for policy optimization at each agent. The idea of DPQ is simple yet effective and well-suited for fully decentralized learning.
Empirically, we evaluate the performance of DPO, IPPO, and IQL in a variety of cooperative multi-agent tasks, including a cooperative stochastic game, MPF (Towe et al., 2017), multi-agent MuJoCo (Peng et al., 2021), and SMAC (Samvelyan et al., 2019). Our evaluation covers discrete and continuous action spaces, well as fully and partially observable environments. The results indicated that DPO outperforms both IPPO and IQL in most tasks, which serves as evidence for our theoretical results.
2 Related Work
CTDE. In cooperative MART, centralized training with decentralized execution (CTDE) is the most popular framework (Lowe et al., 2017; Iqbal & Sha, 2019; Foerster et al., 2018; Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2021a; Zhang et al., 2021; Peng et al., 2021). CTDE algorithms address the non-stationarity problem in the mulli-agent environment by the centralized value function. One line of research in CTDE is value decomposition (Sunchag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang al., 2020; Wang et al., 2021a), where joint Q-function is learned and factorized into local Q-functions by the relationship between optimal joint action and optimal local actions. Another line of research in CTDE is multi-agent actor-criti (Foerster et al., 2018; Iqbal & Sha, 2019; Wang et al., 2021b; Zhang et al., 2021; Su & Lu, 2022; Wang et al., 2023a), where the contralized value function is Iearned to provide policy gradients for agents to learn stochastic policies. More recently, policy optimization has attracted much attention for cooperative MARL. PPO (Schulman et al., 2017) and TRPO (Schulman al., 2015a) have been extended to multi-agent scttings by MAPPO (Yu et al., 2021), CoPPO (Wu al., 2021), HAPPO (Kuba et al., 2021) and A2PO (Wang et al., 2023b) respectively via learning centralized state value function. However, these methods are CTDE and thus not appropriate for decentralized learning.
Fully decentralized learning. Independent learning (OroojlooyJadii & Hajinezhad, 2019) is the most straightforward approach for fully decentralized learning and has been a subject of study in cooperative MARL for decades. The representatives are independent Q-learning (TQL.) (Tan, 1993; Tampuu et al., 2015) and independent actor-critic (IAC) as Foerster et al. (2018) empirically studied. These methods enable agents actor-critic algorithm individually. The drawback of such
to directly execute the single-agent Q-learning independent learning methods is obvious. As other agents are also learning, each agent interacts with a non-stationary environment, which violates the stationary condition of Markov decision processes (MDPs). Thus, these methods are not with any convergence guarantee theoretically, though IQL could obtain good performance in several benchmarks (Papoudakis et al., 2021). More recently, decentralized learning has also been specifically studied with communicat communication (Zhang et al., 2018; Li et al., 2020) or parameter sharing (Terry et al., 2020). However, in this paper, we consider fully docentralized learning in the strictest sense with each agent independently learning its policy while being not allowed to communicate or share parameters
2

---

Published in Transactions on Machine Learning Research (01/2024)
in Tampuu et al. (2015); de Witt et al. (2020). We will propose an algorithm with convergence guarantees in such a fully decentralized learning setting.
IPPO. TRPO (Schulman et al., 2015a) is an important single-agent actor-critic algorithm that limits the policy update in a trust region and ensures monotonic improvement by optimizing a surrogate objective. PPO (Schulman et al., 2017) is a practical but effective algorithm derived from TRPO. PPO replaces the trust region constraint with simpler clip trick. IPPO (de Witt al., 2020) is recently emerged cooperative MARL algorithm in which each agent just learns with independent PPO. Though IPPO is still with no convergence guarantee, it obtains surprisingly good performance in SMAC (Samvelyan et al., 2019). IPPO is further empirically studied by Yu et al. (2021); Papoudakis et al. (2021). Their results show IPPO can outperform a few CTDE methods in several benchmark tasks. These studies highlight the potential of policy optimization in fully decentralized learning, a topic on which this paper focuses.
3 Decentralized Policy Optimization
From the perspective of policy optimization, in fully decentralized learning, we need to find an objective for each agent such that joint policy improvement can be guarantood by each agent independently and individually optimizing its own objective. Therefore, we propose novel lower bound of the joint poliey improvement to enable decentralized policy optimization (DPO). In the following, first discuss some preliminaries; then we analyze the critic in fully decentralized learning; next, derive the decentralized surrogate and prove the convergence; finally, we introduce the practical algorithm of DPO.
3.1 Preliminaries
Dec-POMDP. Decentralized partially observable Markov decision process is general model for cooperative MARL. A Dec-POMDP is a tuple g = P. 1J.. S is the state space, N is the number of agents, y E [0, 1) is the discount factor, and T= (1,2... NY is the set of all agents. A= A x Ag x
... x AN represents the joint action space, where A, is the individual action space for agent i. P(s'ls, a) : S A + [0, 1] is the transition function, and r(s, a): :
A - [-Tmax:Tmas is the reward function of state s S and joint action a E A, where Tmax is bound of the reward function. Y the observation space, and is) 8 x + Y is mapping from state to observation for each agent í. The objective of Dec-POMDP is to maximize J(TT) Ex 7 a:)], thus we need to find the optimal joint policy πt = arg max J(T). To settle the partial observable problem, history т, E T, (Y x A.)* is often used to replace observation O E YY In fully decentralized learning, cach agent é independently learns an individual policy x <<a, (r,) and their joint policy æ can be represented as the product of each æ'. Though each agent learns individual policy as n (a((T))) in practice, in our analysis, we will assume that each agent could receive the state s because the analysis in partially observable environments is much more difficult and the problem may be undecidable in DeMPPOMDP (Madani et al., 1999). Moreover, the V-function and Q-function of the joint policy are follows,
V (T(()) = Eana 1 (s, a]] (x) r(s, a) + P( (V*('))
(1)
(2)
In our discussion, assume the initial state 80 is sampled from fixed distribution μ. The objective ((T) can be rewritten as J(r) [V"~ (so)]. We will use the value function without the agent index to represent. the joint value function related to the joint action such as (T a) and the value function with the agent
index i to represent the individual value function of the agent i such O as) which will be discussed later.
Joint Objective for Monotonic Improvement. In Dec-POMDP, we can still obtain an objective for the joint policy π from the theoretical results in single-agent RL (Grudzien et al., 2022; Lan, 2023) to ensure the joint policy can improve monotonically.
Lemma 3.1. Suppose nnew and wold are two joint policies. If Tnew and oold satisfy the condition
Cloiut (neww, s) C. DKL (Tota (-oa(-1s) (-Is) 2 0, Vs E S,
(3)
3

---

Published in Transactions on Machine Learning Research (01/2024)
where Cioint ( () Ea [[a]s) a), Aold is the advantage function under wold; and C' is constant, then Viow 2 (8) (8).
This Lemma the corollary of Lemma 3.3 in Grudzien et al. (2022).
From Lemma 3.1, we can define an objective as C (TT, s) C DRL (TToLa-)s))]] which referred as joint monotonic objective. Maximizing this objective can guarantee that the joint policy is improving monotonically. However, the joint monotonic objective cannot be directly optimized in fully decentralized learning as this objective is involved in the joint policy, which cannot be accessed in fully decentralized settings.
We will propose a new lower bound (surrogate) for the joint monotonic objective, which can be optimized in fully decentralized learning. Before introducing our new surrogate, we need to first analyze the critic of the agent in fully decentralized learning, which referred to as decentralized critic.
3.2 Decentralized Critic
In fully ralined learning, each agent learns independently from own interactions with the environment. Therefore, the Q-function of each agent i is the following formula:
(7-((8, In (8, BBa P Pll.e....a.)aaapppp (Q""- a()),
(4)
where Ta (s, a,) = E (r(s, a, aaa4.) a)), and π -i and a-, respectively denote the joint policy and joint action of all agents expect agent If we take the expectation (10)) over both sides of the Q-function of joint policy (2), then we have
1==:(Qm a.)] Tx -8, + B (, ((s.a((()) [E~-i (4',
We can see that -= (s, a.) and E, - 12 az a))] satisfy the same iteration. Moreover, we will show in the following that O (s, as) and -,,
(0.0"(((s.)))))) (s, as a))] are just the same.
We first define an operator r n- follows,
r Q az) = (x, az) + + - (Q(s', a()]-
(5)
Then we will prove that the operator rg'.. is a contraction. Considering any two individual Q-functions Q: and Q2, we have:
| Q1 - 121 max //E,, nn , 10 at) Q2(s2, a()1] 8,89 [max 1Qi(s', a ) Q2(s', ab)l max
$ SB, i, d'
(1's at) Q2(s',a))
d'
= = Q2 |xa-
(6)
So the operator I has one and only one fixed point, which mcans
O (s, = - (Q ( (s. 04, a-1),, V (s) = R.-i (x)) VT))).
This conclusion is from the perspective of definition and policy evaluation. Further discussion about the learning or iteration of 0% (s, aq) is included in Appendix F.
With this well-defined decentralized critic, we can further analyze the objective of IPPO (de Witt et al., 2020).
In IPPO, the policy objective (without clipping) of each agent i in state s can be essentially formulated follows:
L C = Σ n' (ayss as))
(7)
where Absa(s, a;) = nnda (s, az) - E 12 Tald
1Q (s, a:)l (Aot(Aaasss, a;, am:)].
4

---

Published in Transactions on Machine Learning Research (01/2024)
However, (7) is different from (3) in the joint monotonic objective. Thus, directly optimizing (7) may not improve the joint policy, and thus cannot provide any guarantee for convergence, to the best of our knowledge. Nevertheless, it seems that Abia (s, a;) is the only advantage formulation that can be accessed by each agent in fully decentralized learning. So, the policy objective of DPO will be derived on (7) but with modifications to guarantee convergence, and we will introduce the details the next section. In the following, we discuss how to compute this advantage practice in fully decentralized learning.
As we need to calculate (sia a.) B
aii [r(s, a, a_,) + >VV
(s)] for the policy update,
approximate (s,a a.) with A'(s, a.) = =r+ -V) which is an unbiased estimate of (s, az), though it may be with a large variance. In practice, we can follow the traditional idea in fully decentralizod learning, and let each agent i independently learn an individual value function Vi()). Then, we further have A'ss, a.) + (s') (i().. The loss for the decentralized critic is as follows:
Ceiiii E (s) y;) wherey; + +V'(s') other target values.
(8)
Here we could take the target value y/ according to different methods like Monte Carlo returns or GAE (Schulman et al., 2015b). There may be some ways to improve the learning of this critic, which however is beyond the scope of this paper.
3.3 Decentralized Surrogate
We are ready to introduce the decentralized surrogale. First we will discuss the relationship between the joint policy objective Cisint (TT s) and the individual policy objective (T),), We have the following lemma. Lemma 3.2. Suppose Vold and π are two joint policies. Then, the sollowing bound holds for any agent i:
2Tmax
(TI, Cloin (TI, 8) Cinaa (T' M
DKL (at)a((sss]1 (11))) where M
1 1-
Proof. We first consider Cjoine(n) (TT, 8) - -i (T²,, 8). ( Mold n 8) - Cinaaaa (T))s) = Eyp
According to (3) and (7), we have the following equation:
Cloint
Σ
x (1==)))) F (1)) Aota (0, ai a..)
Then, we have the following inequalities:
Cloinl (TT, s) - Ct (T', s)) S E
b (a_1s)) x =a as, as,
< F
M Σ
a oool (a_; x (a>
Tmax 2 max |ala a)l) (M
1 -
s.a
2M DTv
DDv (1)
= (M=2MM
S M DRI (191
(10)
= MM Σ DKL (15) j
(11)
where (10) is from the relationship between the total variation distance and KL-divergence that Drv(plg)) < (KI,(Pll) (Schulman et al., 2015a), and (11) is property of the KI-divergence that DiI. (Toaaa-)s)) (T) E, DKL (aa (+s) From (11), we can further obtain the following inequality, which completes the proof,
Cloint (TT, 5) Li (T' -M Σ Dia (-1))[[=(((11))
5

---

Published in Transactions on Machine Learning Research (01/2024)
We need to emphasize that the inequality (9) connects the joint policy objective with the individual policy objective, which is essential for our purpose of fully decentralized learning.
Next, we will derive our novel lower bound of joint policy improvement by the following theorem.
Theorem 3.3. Suppose vold and are two joint policies. We have
N
N
Cinint
(TT, s) C- -KL Wold
2
c
Wold (T', M1
Σ
N
N
C
DKL ( (T (-15) (| (-ss) 2 0, as S,
(12)
where M
N- 2rmax and C >> 0 is any constand.
V
Proof. We will start prove this theorem from the objective Lioint (TT, 8) C (Ta
Cloint (TT 5) C D (To)(((ss)](
Cloint (T. 5) C. (Ds
N
M
(TT"
DKL (F (115) - -C (To
(13)
N
Rold
N
jatis
N
Rold (IT (T's))-
M.
DKL. ( (1l((()))))) - C DkL. (Taa(
(14)
N
N
N
Mar (1) M,
DKL (Tin
C
DKL (m (1))
(15)
N
N
N N
N
N
masa (T')))
M
DKL (T (+)))))))))))))) - C
DeL
(16)
The inequality (13) is the direct application of the inequality (9) in Lemma 3.2. The inequality (14) is from the Schwar-Schwarz inequality.
N
DKL (T (15))[ <
N
DKL (To (-1s)]]((((11s)
==1 sFC
N(N 1)
( (1
The inequality (15) is from a property of the KL-divergence (see the proof of Lemina 3.2), while the inequality (16) is from the simple inequality / V., a < E, Va, (a 2 0, Vi).
The lower bound in Theorem 3.3 is dedicated to decentralized policy optimization because it can be directly decomposed individually for each agent as a decentralized surrogate. From Theorem 3.3, if we set the policy optimization objective of each agent as
Cn (F', MR (F (-Is)] (-as(()))) (+|s) C DRI. (Foia ([T))))
new (-|s) arg IIXX x²
(17)
then we have [(Tnew)) 2 J(Fota) from Lemma 3.1. Finally, obtain the following theorem.

---

Published in Transactions on Machine Learning Research (01/2024)
Theorem 3.4. If we define joint policy sequence {गeh) as follows:
[ ( My (KK C (rL Vi E I, E
1++1 arg max x²
(18)
then the sequence (J(m)) will improve monotonically and converye to sub-oplimum.
Proof. At each itcration, as the policy of cach agent is obtainod by (18), all agents jointly maximize the RHS of (12). Thus, from Theorem 3.3, we have Vs
Cloint (T) 8) - C. (T) ((1((s))) T,
N DKL (T( c
C m (1(+1:8) -M
Σ
DKL (
N
N
N
M Σ v ( - c cE ( (1)) = 0, Li m (T/ 8)
(19)
where (19) is from the definition of 1/41 (-1s in (18). From Lemma 3.1, we know V (s) 2 (s), Vs E S. From the definition J(T)) Es (so)), we know J(T(+1) 2 JTT)), which means that the sequence {J(T)) improves monotonically. Moreover, as {J(++)] is bounded (the reward function is bounded), (J(T+))
will converge to sub-optimum, which completes the proof.
As (17) can be independently optimized al each agent, the monotonic improvement and convergence of joint policy can be achieved by fully decentralized policy optimization. It is worth noting that the result above is under the assumption that each agent can obtain the state, and in practice, each agent can take the individual trajectory T, as the approximation to the state.
3.4 Remarks About Our Theoretical Results
In this section, we would like to summarize our theoretical results and discuss an important problem: how could DPO overcome the non-stationarity problem in decentralized learning and obtain the convergence guarantee?
Our general idea is that finding surrogate for the joint monotonic improvement condition ( s) -C- (=s)aa-(s)[[Tmeer
and this surrogate can be optimized independently for each agent. Then optimizing this surrogate will make sure that the joint policy will improve monotonically.
DPO realizes this idea in two stages. In the first stage, according to the inequality (9) in Lemma 3.2, we have a bound between Cleint (πT, 8) and Ci (T), This bound actually help us to replace E= [Aoia a, a=)) in Lioint (TT, s) with (sta(s, a;) = E (s, a,, a)]], where ग. Tee is not accessible in training. But the extra terms M, EEHz (Tei (+Is)| (-Is)) in (9) is still from other agents and not accessible for agent i. So, in the second stage, we apply (9) for all agents and rearrange the terms in M VVV,,, Ejyr (1 by incqualities in Theorem 3.3 to make sure that the term ( (-Is)]] is optimized by agent j instcad of agent i. Finally, we have a new surrogate (12) which could be divided into N parts, and each part is only related to one agent. Our general idea mentioned above is eventually realized after these two stages.
We also would like to discuss the relation between DKL and VDLL in the surrogate (12). It seems that DKL and VDki. are similar and could be merged into one term. However, in fact, we are not able to merge these two terms. From the perspective of theory, VDkL is from the bound (9). The relation between DKL. and VDRi. is different around DKL. = 1 and we cannot find an incquality to merge these two terms. From the perspective of experiments, we could find in the empirical results in Section 4.5 that eliminating either the term VDKL or the term DKL. lowers the performance of DPO.
3.5 The Practical Algorithm
7

---

Published in Transactions on Machine Learning Research (01/2024)
DPO is proposed with the simple idea that each agent optimizes the decentralized surro- gate (17). However, we face the same trouble as TRPO in that the constant M is large and if we directly optimize this objective, then the step size of the policy update will be too small.
Algorithm The practical algorithm of DPO
1: for episode to M do
for t= to max episode length do
select action a, (+|ss)
execute a; and observe reward r and next state s
5: collect (s,0,.r,s') end for
To settle this problem, we absorb the idea of the adaptive coefficient in PPO (Schulman et al., 2017). We use two adaptive coefficients Øf and 33 to replace the constant M and C' (Schul- man et al.; 2015a). Since any constant > 0 is available theory, we can also obtaín an appropriate value of in an adaptive way. In practice, we will optimize the following objective:
6:
7:
Update decentralized critic according to (8)
8: Update policy according to the surrogate (20)
Update 3i and 32 according (21).
10: end for
( ( (T' BiV DRI.(T (1s)) B3 DKr. (T (++))l
Fhew(-108 arg max 3
(20)
As for the adaption of 31 and $3, we need to define a hyperparameter dearget: which can be seen as ruler for the average KL-divergence DKL (Taall (eww) for cach agent, where DKL (Fatallr))) Banmoss [DKI, (T)d
If DavE (Trial (ew) is close to deorget, then we believe current 3i and 88 are appropriate. If DRT, (Talll fiew) exceeds dearget too much, we believe Sj and 82 are small and need increase and vice versa. In practice, we will use the following rule to update 81 and $3:
If DEL (Taal (oow) > dearget * 5, then 83 B3
33*# *
Vj {1,2}
(21)
If D (T ( < ddarget/00 0 then 8) + B3/w Vj (1,2}.
We choose the constants 8 = 1.5 and w = as the choice in PPO (Schulman et al., 2017). As for the critic, we just follow the standard method in PPO. Then, we can have the fully decentralized learning procedure of DPO for each agent in Algorithm 1.
The practical algorithm of DPO uses some approximations of the decentralized surrogate. Most of these approximations are traditional practices in RL or with no alternative in fully decentralized learning yet. We admit that the practical algorithm may not maintain the theoretical guarantee. However, we need to argue that we go one step further to give a decentralized surrogate in fully decentralized policy optimization with a convergence guarantce. We believe and expect that better practical method can be found basod on this objective in future work.
4 Experiments
In this section, we compare the practical algorithm of DPO with IPPO (de Witt et al., 2020) and IQL (Tan, 1993) in a variety of cooperative multi-agent environments, including a cooperative stochastic game, MPE (Lowe et al., 2017), multi-agent MuJoCo (Peng et al., 2021), and SMAC (Samvelyan et al., 2019), covering both discrete and contimuous action spaces, as well as fully and partially observable environments. In our experiments, we focus on the comparison between DPO and IPPO, and just add IQL as a baseline for reference. Q-learnéng value-based methods out of our scope of discussion. As we consider fully decentralized learning, in the experiments agents do not use porometer-sharing as sharing parameters should be considered as centralized lcarning (Terry et al., 2020). In all experiments, the network architectures and common hyperparameters of DPO and IPPO are the same for a fair comparison. Note that IPPO (de Witt et al., 2020) the widely used clip version of PPO, while the KL version of PPO serves as ablation of DPO, which is also included in the ablation study. More details about experimental sottings and hyperparaneters are available in Appendix A and B. Moreover, all the learning curves are from 5 random seeds and the shaded area corresponds to the 95% confidence interval.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Stochastic Game
300
400
320
200
carost - 0:001
Crarger - 0.01
(132)
200
140-
OPTIMAL
IPPO
dtarget=1
DPO
IQL
J00020
202000
530000
400200
520005
100000 200000 30C0CO
400000
500000
steps
(a) DPO vS. IPPO and TQT.
(b) the influence of dearget
Figure 1: Empirical studies of DPO on the didactic example: (a) learning curve of DPQ compared with TPPO, IQL, and the global optimum; (b) the influence of different values of drarget on DPO, x-axis environment steps.
4.1 A Didactic Example
cooperative stochastic game didactic example. The cooperative stochastic game has 100 First, states and 6 agents. Each agent has 5 actions. All the agents share joint reward function. The reward function and the transition probability are both generated randomly. This stochastic game has a certain degree of complexity which is helpful to distinguish the performance of DPO, IPPO, and IQL. On the other hand, this environment is tabular which means training in this environment fast and we can do ablation studies efficiently. Moreover, we can find the global optimum by dynamic programming compare with in this game.
The learning curves in Figure 1(a) show that DPO performs better than two baselines and learns better solution in this environment. IQL is the most unstable among the three algorithms according to its variance. The fact that DPO learns a sub-optimal solution agrees with our theoretical result. However, the sub-optimal solution found by DPO is still far from the global optimum. This means that there is still improvement space.
On the other hand, we study the influence of the hyperparameter diarget on DPO. We choose drarget = 0.001, 0.01, 1. The empirical results are shown in Figure 1(b). We find that when diarget is small, the coefficients 31 and $2 are more likely to be increased and the step size of the policy update is limited. So for the case that drarget- 0.001, 0.01, the performance of DPO is relatively low. And when dearget is large, the policy update may be out of the trust region. This can be witnessed by the fluctuating learning curve of the case drarget = 1. So we need to choose an appropriate value for drarget and in this environment we choose dearget = 0.1, which also the learning curve of DPO in Figure 1(a). We found that the appropriate value for dearget changes different environments. In the following, we keep dearget to be the same for tasks of the same environment. There may be some better choices for diorgets but it is a bit time-consuming and out of the focus of our discussion.
simple spread
line control
dirck control
-10
mean opisade rewards
-20
-20
-30
-30
DPO
DPO
DPO
-40
IPPO
IPPO
IPPO
IQL
-50
IQL
IQL
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
166
le5
1e6
Figure 2: Learning curve of DPO compared with IPPO and IQL in the 5-agent simple spread, 5-agent line control, and 5-agent circle control in MPE, where the x-axis is environment steps.
9

---

Published in Transactions on Machine Learning Research (01/2024)
Hopaeno
2712
2.00
1200
DP0 399
DP0
IPPO
IPPO
IPPO
153PG
190P
IDDM
IDPG
0A
355
3.4
Figure 3: Learning curves of DPO compared with IPPO and IDPG in 3 x 1 Hopper, 3 x 2 HalfCheetah, 3 x 2 Walker2d, x Ant, and 17 Humanoid in multi-agent MuJoCo, where x-axis is environment steps.
1.-
633
DPO IPPO
D150
DPO
IPPO
IPPO
IPPO
IPPO
10L
IOL
IQL
05
15
00
0.
45
04
95
20
Figure 4: Learning curves of DPO compared with IPPO and IQL in 2s3z, 8m, 355z, 27m VS 30m, and MMM2 in SMAC, where x-axis environment steps.
4.2 MPE
MPE popular environment in cooperative MARL. MPE is a 2D environment and the objects in MPE environment. are either agents or landmarks. Landmark is part of the environment, while agents can move in any direction. With the relation between agents and landmarks, discrete action space version of MPE and the agents can accelerate or decclerate in the direction of the x-axis or y-axis. We choose MPE for its partial observability. We take deørget = 0.01 for all MPE tasks.
can design different tasks. We use the
The MPE tasks we used for the experiments are simple spread, line control, and circle control which were originally used in Agarwal et al. (2020). In our experiments, we set the number of agents N = 5 in all three tasks. The empirical results are illustrated in Figure? 2. We can find that although DPO may fall behind IPPO and IQL at the beginning of the training in some tasks, DPO learns better policy in the end for all three tasks. As for the drop in IQL's performance, we checked the learning curves of different random seeds and found that this phenomenon is actually caused by the instability of TQL. IQL, learns very fast but converges to different sub-optima in different random seeds.
4.3 Multi-Agent MuJoCo
Multi-agent MuJoCo robotic locomotion control environment for multi-agent settings, which is built upon single-agent MuJoCo (Todorov et al., 2012). In multi-agent MuJoCo, cach agent controls one part of robot to carry out different tasks. We choose this environment for the reason of continuous state and action spaces. We use independent DDPG (Lillicrap et al., 2016) (IDDPG) to replace IQL, for continuous action spaces. We select 5 tasks for our experiments: 3-agent Hopper, 3-agent HalfCheetah, 3-agent Walker2d, 4-agent Ant and 17-agent Humanoid. In all these tasks, we set poont__bsk-2, We take dearget = 0.001 for all multi-agent MuJoCo tasks.
The empirical results are illustrated in Figure 3. We can find that in all five tasks, DPO outperforms IPPO and IDDPG, though in 3-agent. HallCheetah DPO learns slower than IPPO at the beginning. The results on multi-agent MuJoCo verify that DPO is also effective in facing continuous state and action spaces. Moreover, the better performance of DPO in the 17-agent Humanoid task could be evidence of the scalability of DPO.
4.4 SMAC
SMAC a partially observable and high-dimensional environment that has been used in many cooperative MARL studies. We select five maps in SMAC, 2s3z, 8m, 3s5z, 27m vs 30m, and MMM2 for our experiments. We take dtarget 0.02 for all SMAC tasks.
10

---

Published in Transactions on Machine Learning Research (01/2024)
simple spread
line control
circle contral
mesin epicode rewards
-20
-20
-20
-30
DPO
-30
DPO
DPO
-30
DPO w/o kl
DPO w/o kl
DPO w/o_kk kl
-40
-40
DPO sqrt
DPO w/o sqrt
40
DPO w/o sqrt
-50
IPPO
-50
IPPO
IPPO
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.
stops
165
steps
stops
Figure 5: Learning curves of DPO compared with DPO _PO_w/o__ kl and DPO DPO_w/o w/o_strt in three MPE tasks, where x-axis environment sleps.
The empirical results are illustrated in Figure 4. The two super hard SMAC tasks (27m vs 30m and MMM2) are loo difficult for all DPO, IPPO, and IQL to win, so we use episode reward as the metric to show their difference. DPO performs better than IPPO and IQL in all five maps. We need to argue that though we have controlled the network architectures of DPO and IPPO be the same, in our experiments each agent has its individual parameters which increases the difficulty of training. So our results in SMAC may be different from other works. Although IPPO has been shown to perform well in SMAC (de Witt et al., 2020; Yu et al., 2021; Papoudakis et al., 2021), DPO can still outperform IPPO, which verifies the effectiveness of the practical algorithm of DPO in high-dimensional complex tasks and can also be evidence of our theoretical result. A Again, the better performance of DPO in 27m vs 30m shows its good scalability in the task with many agents. As for the performance of IPPO, we have fine-tuned the clip parameters and found that the impact of different clip parameters is relatively small. The empirical results and more discussions are includod in Appendix C.
4.5 Ablation Study
We carry out the ablation study about the objective in (20). We consider two ablation methods: in the first one, we keep 8 = 0 to eliminate the influence of the term / (T otal (m)l) which the same as IPPO-KT; in the second one, we keep B3 to eliminate the influence of the term DavE ( (F (Ta) The other parameters are controlled to be the same as DPO. We call these two methods as DPO w/o sqrt and DPO w/o kl, respectively.
We compare DPO with DPO DPO_w/o kl and DPO_ S_l/____saa sqrt for ablation study in MPE, multi-agent MuJoCo, and SMAC. We add the performance of IPPO in the empirical results for reference. The tasks selected for each environment are the same as previous experiments in Section 4.2, 4.3 and 4.4. The empirical results for MPE, multi-agent MuJoCo, and SMAC are illustrated in Figure 5, Figure 6, and Figure 7, respectively. In general, we can find that climinating either the term (F or the term DRI ||=') lowers the performance of DPO in all these lasks. Though the influence of eliminating these two terms is relatively small in several tasks such as Walker2d-V2 3X2 in multi-agent MuJoCo and 2s3z in SMAC, the absence of these two terms will obviously lower the performance of DPO in most tasks. This could be evidence for the significance of our novel lower bound (17) and the objective (20). The performance drop also shows empirically that we are not able to merge the two KT-divergence terms into one.
5 Conclusion
In this paper, we investigate fully decentralized learning in cooperative multi-agent reinforcement learning. We derive novel decentralized lower bound for the joint policy improvement and we propose DPO, a fully decentralized actor-critic algorithm with convergence guarantee and monolonic improvement. Empirically, we test DPO compared with IPPO and IQL in a variety of environments including a cooperative stochastic game, MPE, multi-agent MuJoCo, and SMAC, covering both discrete and continuous action spaces, as well as fully and partially observable environments. The results show the advantage of DPO over IPPO and IQL, which can be evidence for theoretical results.
11

---

Published in Transactions on Machine Learning Research (01/2024)
2010
2.00
1930
1200
2330
D00
010
070
710
070 WC
070 WL 11
370 m0
0.4
0.5
0.5
3.
155
29
Figure 6: Learning curves of DPO with compared with DPO _///_///____ kl and DPO_ W// ww/p///ooooooo_oo sqrt in 3-agent Hopper, 3-agent HalfCheetah, 3-agent Walker2d, 1-agent Ant, and 17-agent Humanoid in multi-agent MuJoCo, where x-axis is environment steps.
RM
The M
32-
82
L65
140
Figure Learning curves DPO with compared with DPO _PO_w/o kl and DPO __/////oo___ w//__qri on 2s32, 3s5, MMM2, and 27m vs 30m in SMAC, where x-axis is environment steps.
We have to admit that there are still some limitations to DPO. To optimize the decentralized surrogate objective, the practical algorithm of DPO needs to take some approximations, which however may not preserve the theoretical results. In future work, we hope to find better practical method for our theoretical results.
Acknowledgements
This work was supported by NSF China under grant 62250068. The authors would like to thank the anonymous reviewers for their valuable comments.
References
Akshat Agarwal, Sumit. Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent team. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whitcson. independent learning all you need in the starcraft multi-agent challengo? arxiv preprinl (0::::0011005333 2020.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counter- factual multi-agent policy gradients. In AAAT Conference on Artificial Intelligence (AAAT), 2018
Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, pp. 7825-7844. PMLR, 2022.
Shariq Iqbal and Fei Sha. Actor-allention-critie for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.119: 11251, 2021.
Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming. 198(1):1099-1106 2023.
Wenhao Li, Bo Jin, Xiangfong Wang, Junchi Yan, and Hongyuan Zha. F2a2: Flexible fully-decentralized ap- proximate actor-critic for cooperative multi-agent reinforcement learning. orXie preprint arXiv:200,,11445, 2020.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International conference learning representations (ICLR), 2016.
Ryan Lowe, Yi T Wu, Aviv Tamar, Jean Harb, OpenAT Pieter Abbeel, and Igor Mordatch. Multi-agent. actor-critic for mixed cooperative-compeittie environments. In Advances in Neural Information Processing Systems (NewrIPS), 2017.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistie planning and infinite-horizon partially observable markov decision problems. In AAAI/IAAI, pp. 541-548 1999.
Francisco S Melo. Convergence of q-learning: simple proof. Institute Of Systems and Robotics, Tech. Rep, pp. 1-4, 2001.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through doep reinforcement lcarning. Nature, (18(7540):529--55) 2015.
Afshin JaddjlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement lcarning. arviv preprint arXiv: 1908.03333, 2019.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforeement learning algorithms in cooperative tasks. In Advances in Newral Information Processing Systems (NeurIPS), 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Plerre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. In Advonces in Neural Information Processing Systems (NeurIPS), 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2018
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arvivv preprint arXiv:1902.0/ 04033, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensiona continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02/4: 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint arXiv:1107.06344 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Kefan Su and Zongqing Lu. Divergence-reguulrtiee multi-agent actor-critic. In International Conference Machine Learning (ICML), 2022.
Mingfei Sun, Sam Devlin, Katja Hofmann, and Shimon Whiteson. Monotonic improvement guarantees under non-stationarity for decentralized ppo. arXiv preprint arXiv:2202.0008) 2022.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Jocl Z Leibo, Karl Tuyls, et al. Value-dcoompsition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). 2018.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. orxiv preprinl WwXi::151.08879, 2015.
Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In International Conference Machine Learning (ICML), 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter
sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005.1900 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robols and Systems (IROS), 2012.
Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized execution:
Multi-agent conditional policy factorization. In International conference on learning represenlations (ICLR), 2023a.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang, Qplex: Duplex dueling multi-agent q-learning In International Conference Learning Representations (ICLR), 2021a.
Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang, Order matters: Agent-by- agent policy optimization. arXiv preprint or wwxiv:2302066200 2023b.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjic Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations (ICLR), 2021b.
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo, et al. Coordinated proximal policy optimization. Advances in Newral Information Processing Systems (NeurIPS), 34, 2021.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.0390 03939, 2020.
Hsuan-Yu Yu Yao, Ping-Chun Hsieh, Kuo-Hao Ho, Kai-Chun Hu, Liang-Chun Ouyang, Wu, et al. Hinge policy optimization: Rethinking policy improvement and reinterpreting ppo. arXiv preprint ar Xiv:21 13799999 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. Xiviv preprint arXin:2103. 01955, 2021.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning (ICML), 2018.
Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint arXiv:1911.10633 2019.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximun-entrrop multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
14

---

Published in Transactions on Machine Learning Research (01/2024)
A Experimental Settings
A.1
MPE
The three tasks are built on the origin MPE (Lowe et al., 2017) (MIT license) and are originally used in Agarwal et al. (2020) (MIT license). The objectives in these three tasks are listed as follows:
Simple Spread: There are N agents who need occupy the locations of N landmarks.
Line Control: There are N agents who need to line up between 2 landmarks.
Circle Control: There are N agents who need form a circle around landmark.
The reward in these tasks is the distance between all the agents and their target locations. We set the number of agents N for these three tasks in our experiment.
A.2 Multi-Agent MuJoCo
Multi-agent MuJoCo (Peng et al., 2021) (Apache-2.0 license) is robotic locomotion task with continuous action space for multi-agent settings. MuJoCo's reward function is about the distance the robot has moved from the original position. In Multi-Agent MuJoCo, the robot could be divided into several parts and each part contains several joints. Agents in this environment control a part of the robot which could be different varietics. So the type of the robot and the assignment of the joints decide a task. For example, the task HalfChhetahh-3x2 means dividing the robot 'HalfCheetah' into three parts for three agents and each part contains 2 joints.
The details about our experiment settings in multi-agent Mujoco are listed in Table 1. The configuration defines the number of agents and the joints of each agent. The 'agent obsk' defines the number of nearest agents an agent can observe.
Table 1: The task settings of multi-agent MuJoCo
task
configuration
agent obsk
HalfCheetah
3x2
2
Hopper
3x1
2
Walker2d
3x2
2
Ant
4x2
2
Humanoid
17x1
2
A.3 SMAC
SMAC (Samvelyan et al., 2019) popular environment for MARL. The agents in SMAC are rewarded soon they attack kill an enemy unit. The rewards for an episode in SMAC affected by the mumber of agents, the environment has normalized the maximun episode rewards for all tasks to 20. If the agents in SMAC kill all enemy units in an episode, then they have 'won' that episode. The observation space of the agents in SMAC is related to the number of units in the task. In general, the observation is a vector with 100+ dimensions over the information of all units in difficult tasks, and the information of the units that are outside the agent's field view is denoted by zero in the observation vector. More details on SMAC can found in the original paper (Samvelyan al., 2019).
B Training Details
Our code of IPPO is based on the open-source code of MAPPO (Yu et al., 2021) (MIT license). We modify the code for individual parameters and ban the tricks used by MAPPO for SMAC. The network architectures
15

---

Published in Transactions on Machine Learning Research (01/2024)
and base hyperparametters of DPO and IPPO are the same for all the tasks in all the environments. We use 3-yyyer MLPs for the actor and the critic and use ReLU as non-linearities. The number of the hidden units of the MLP is 128. We train all the networks with an Adam optimizer. The learning rates of the actor and critic are both 5c-4. The number of epochs for every batch of samples is 15 which is the recommended value in Yu et al. (2021). For IPPO, the clip parameter 0.2 which the same as Schulman et al. (2017). For DPO, the initial values of the coefficient, /i and B3 are 0.01. The value of dearget is 0.1 for the cooperative stochastic game, 0.01 for MPE, 0.001 for multi-agent MuJoCo, and 0.02 for SMAC. Our code of IQL is based on the open-source code² PyMARL (Apache-2.) 2.0 license) and modify the code for individual parameters. The default architecture in PyMARL is RNN so we just follow it and the number of the hidden units is 128. The learning rate of IQL is also 5e-4. The architectures of the actor and critic of IDPGG are 3-layer MLPs. The learning rates of the actor and critic are both 5e-4.
Table 2: Hyperparameters for all the experiments
hyperparameter
value
MLP layers hidden size
3
128
non-linear
ReLU
optimizer
Adam
actor Ir
5e-4
critic_lh Ir
5e-1
numbers of epochs
15
initial Bf
0.01
initial $
0.01
1.5
w drarget
2
different for environments as aforementioned
clip parameter for IPPO
0.2
The version of the game StarCraf12 in SMAC is 4.10 for our experiments in all the SMAC tasks. We set the episode length of all the multi-agent MuJoCo tasks as 1000 in all of our multi-agent MuJoCo experiments. We performed the whole experiment with a total of four NVIDIA A100 GPUs. We have summarizod the hyperparameters in Table 2.
C
Additional Results
Schulman et al. (2017) actually proposed two versions of PPO. The first version, which is also the most popular version, is with the clip trick. The sccond version is directly optimizing the penalty formula with adaptive coefficients and we refer to this algorithm as PPO-KL. IPPO (de Witt et al., 2020) is actually extended from the first version, while the practical algorithm of DPO is similar to the second version. The main difference between DPO and PPO-KL is the term of the square root of the KL-divergence in the policy loss. We modify IPPO by making each agent learn with PPO-KL to obtain IPPO-KL.
For the completeness of our experiments, we test the performance of IPPO-KI, IQI. and DPO DPO_w/o w/// kl that eliminates the term (TT (Tsall)"))) for ablation study, in the cooperative stochastic game. The empirical result is illustrated in Figure 8. We find that the performances of IPPO-KL, IPPO, and DPO DPO_w/o kl are close and are all lower than DPO. This could be evidence of the effectiveness of DPO. As for IQL., its performance is lower than IPPO though it converges faster.
Moreover, we further study the influence of the values of 8 and w for the adaptive adjust- ments of the coefficient 88 and B3. We test seven different choices of 8 and w as (8, w) (1.5,2), (1.5, 4), (1.5,6), (1.1, (3. 2), (3. 6), (1.1,6) in the cooperative stochastic game. The empirical results are included in Figure 9. We find that the influence of different å and w is relatively limited and the adaptive adjustment is not very sensitive to them. This conclusion is similar to the PPO paper (Schulman et al., 2017).
16

---

Published in Transactions on Machine Learning Research (01/2024)
Stochastic Game
Stochastic Game
500
400
mean opitodle rewards
400
300
200
200
6m1.5_ wm2
6m1.1_ u
100
100
OPTIMAL
IPPD-KL
IPPO
4=2
DPO
IPPD
6m1.5 5_mm4
6m3,um
0-
DPO w/o
IQL
5-1.5 u--6
6-1.1 w-6
100000
200000
300000
400000
500000
0
100000
200000
300000
400000
500000
Steps
steps
Figure 8: Learning curve of DPO compared with DPO_ DPO_w/o kl, IPPO, IPPO-KL, IQL, and the global optimum in the cooperative stochastic game, where the x-axis is environment steps.
Figure 9: Learning curves of DPO with different values of and w in the cooperative stochastic game, where the x-axis is environment steps.
27m vs 30m
MMM2
10
mean episade rewards
6-
10
DPO
DPO
IPPO_ dipp_param___pp_ perapaaamm__ 0.1
IPPO ctcp_plpppp__e clip_param,___ param_aaaaa 001
IPPO_ dipp sip_param 0.2
IPO chtpppppppppcee param 0.2
2-
IPPO alp_param D.3
IPPO llt___ppppppaam._____
0.0
0.5
1.0
1.5
2.0
0.0
1.0
1.5
2.0
steps
1e6
steps
1e6
Figure 10: Learning curves of DPO with compared with IPPO with different clip parameters in 27m vs 30m and MMM2 in SMAC, where x-axis is environment steps.
IPPO performs poorly in super-hard SMAC tasks such as 27m vs 30m and MMM2 where the number of agents relatively more and IPPO can hardly win. So we finetune the clip parameter of IPPO in these tasks to find the impact of hyperparameters on IPPO. The empirical results are illustrated in Figure 10. We could find that different clip parameters only affect the performance of IPPO during the training process, but at the end of the training, these algorithms perform similarly. We need to argue again here that in settings parameter sharing is not permitted so all the methods will have difficulties in training as the number of agents increases. To determine whether the bound in (9) is trivial in theory is actually not simple. But we need to argue that the bound will be tighter if the KL-divergence between the new policies and the old policies is smaller. So we empirically evaluate the mean KL-divergence of all agents in the training process in two SMAC tasks, 27m vs 30m and MMM2. The empirical results are illustrated in Figure 12. We could find that the KT-divergence is actually small and stable during the whole learning process, which could be evidence for the effectiveness of our bound.
Table 3: The two-player matrix game where each agent has two actions. p and g represent the poli- cies of two agents respectively. o, b, c and d are the rewards for the corresponding joint actions respectively.
1
G
1 p
To verify the monotonic improvement property of (17), we use a simple two-player matrix game Table 3. In this matrix game, each agent has two actions and the reward for four joint actions are a, b, c and d respectively. To obtain more general result, take four different sets of the payoff matrix: Matrix Game (a, b, (5, 7, 6, 4); Matrix Game (a, b, d) (1, 3,5, 4); Matrix Game 3 (a, c, (7,1, Matrix Game 4 (a, b, c, d) = (20, 0, 0, 10). Since there is no closed-form solution to (17), we use a numerical method to find the solution for policy updates. The constant M is defined by the payolf matrix and the constant C' can be chosen arbitrarily. We choose C = 10 in all the experiments. The empirical results are illustrated in Figure 11. We use the payoff matrix and the joint policy म, (:) to calculate the value of J(T,). We can find the phenomenon of monotonic improvement all four matrix games, Morcover, we can find that the joint policy म. (Pt,9) converges in all four matrix games. From the results in Matrix Game and Matrix
17

---

Published in Transactions on Machine Learning Research (01/2024)
250
3."
15.
5.43
36-
12.5
160
5.00
Uveery (wam
10
20
50
10
70
10
0.3
0.6
0.4
0.0
0.5
0.4
02-
5.7
30
Figure 11: The learning curves of the iteration (17) four different matrix game. Each column corresponds to one matrix game. The first line is the learning curves of J ( TThe second line is the learning curves of the policies and q.
MMM2
0.025
0.03
0.015
0.02
0.010
0.01
0.005
0.000
DPD
0.0
0.5
1.0
2.0
0.0
0.5
1.0 steps
1.5
2.0
steps
1e6
1e6
Figure 12: The mean KT--divergecce curves of all agents for DPO in 27m vs 30m and MMM2 in SMAC, where the x-axis environment steps.
Game 2 we know that the iteration (17) may be trapped in the sub-optimum. These results can empirically verify our theoretical results in Theorem 3.4.
D Discussion
Besides our empirical results, we would like to share our views on the difference between DPO and IPPO and give some intuitive idcas. KL regularization and ratio clipping are similar in the single-agent setting, but they are not supposed to be similar in multi-agent settings. The 'correct' ratio clipping in the multi-agent setting according to the theory of PPO should clip over the joint policy ratio Rnow (a|s) IPPO just clips individual (aja)
policy ratio (o.18) for cach agent i which may not be enough to realize the 'correct' ratio clipping, We To [a (a:]s)
could find more discussion about this in the CoPPO (Wu et al., 2021) paper. So IPPO is not supposed to enjoy the theoretical results of DPO.
We could rewrite the objective TPPO for each agent with a similar formulation in HPO (Yao et al., 2021) as follows:
CIIPO (TP) (-²)= Epoa(s) Σ n' (asls))Abb a:ll (Ai a:)). uu(s, a.) 1,e),
Hold
where 1(y, max(0, x xt is the hinge loss and u.(s,a.) a.)
x (arls) is the ratio.
(aa()))))
If we follow the same idea as PPO, then IPPO is the 'correct' ratio clipping version for the surrogate of DPO. However, the effectiveness of this ratio clipping formulation in theory is still open in decentralized learning since there is not any convergence guarantee for IPPO, to the best of our knowledge.
Though the effectiveness of IPPO, in theory, is beyond the scope of our paper, we could provide an intuitive explanation for the fact that the performance of DPO surpass IPPO from this formulation and the
18

---

Published in Transactions on Machine Learning Research (01/2024)
analysis in HPO. In the proof of HPO, there is critical assumption that the sign of the estimated advantage is the same as that of the true advantage (Assumption 4 in Section 2.3 in Yao et al. (2021)). And HPO also shows that the sign of the advantage is more important than the value for this formulation of PPO-clip. In decentralized learning, both DPO and IPPO are facing the difficulty of learning the individual advantage function as there may be noise the individual value function. However, the objective of DPO is continuous and the objective of IPPO discrete for sign( (Agiaa (s, az)). So the impact of the noise in the value function may be larger on IPPO than DPO.
Recently, there have been several works with theoretical analysis of monotonic improvement in MARL including HAPPÓ (Kuba et al., 2021), CoPPO (Wu et al., 2021) and A2PO (Wang et al., 2023b). Unfortunately, these algorithms are not appropriate for fully decentralized learning. НАРPO needs agents to maintain joint advantage function over the joint action and the agents need the information of the critical function M from other agents. These conditions cannot be satisfied in fully decentralized learning. CoPPO needs the ratios of other agents' policies for policy updates but this information cannot be obtained in fully decentralizod learning. A2PO uses the joint advantage function and agents need the joint policy in the off-policy correction for the calculation of A Moreover, A2PO uses the double-clip trick similar CoPPO which requires the ratios of other agents' policics. Thus, A2PO is also not appropriate for fully decentralized learning.
E Additional Proof for Decentralized Critic
In this section, we will discuss the convergence of the iteration of decentralized critic following the idea of Melo (2001). We have a lemma from Melo (2001). Lemma E.1. The random process {Ai} taking values in R" and defined
(++1(x) (1 ( ac())))
converges to zero w.p.1 under the following assumplions:
0 S a4 $ 1, E, (x) = xo and Σ < c0;
[[[[)( S 7| A.l with < 1;
(ar(F(()) & C (1+ [|.))()) for C C>0 0.
Next, we define an update rule given any initial Q-function Qi for any agent i and the joint policy π as follows:
1i af) = (st, at) + af at) (ri(se, at) + ylB (+1e4s))) [2 63)] - Qi(s,, ail).
(22)
where 0 af(s), is the step-size for agent and the reward as) (s, Er-i(r(s, e.))
Then we have the following result.
Proposition E.2. Given any initial Q-function Qa for any agent i, the update rule (22) converges to Q- (s, w.p.1 1 long
Eai(s,ai) = 0o and Eai < x
(23)
for all (s) ESX A,.
Proof. We can rewrite (22) as follows
Qiei at) = (1 a (f a))) Qi(sn at) + as(s af) (re(se, at) + T [Q((+++,,6)))
(24)
We define of as) Qi(s az) o (s),;), a;), then we have
(s,, at) (1 -a((s,;)))) (s at) aj(st at) (r:(st,a)) + 21 [Q((s 68]] Q-- at))
(25)
19

---

Published in Transactions on Machine Learning Research (01/2024)
If we define F/ a,) rs(ss,,) + (B Qiis' (6)( - (5, then we have (s at) (1 (5), a (5 af) + ai of) (sc, at)
(26)
To apply Lemma E.I, consider E [f)((, and Var [F((s.
For E (s. we have
(F -E (i(8, a:) + (a, [Q (o((s',b)]] b.]] (s,
=r:(s,a;) -BBgs 17 (6)),,,)]] Q=
-I= (8,
(27)
1/-,Q((s, - 25
(28)
where Pi(s'|s, 0:) (() -(a0ll1 of agent i given fixed m-i. The step (27) is from the definition of r in (5). The step (28) is from the property of the fixed point in (4). From the contraction property of r in (6), we know
as. a-;) is the individual transition probability from the perspective
| [F/(s, a)) = |
- Qi r r=-::== | s 0 ||= => >1
(29)
For Var [F/ (s. [F]] we have
E (i(ssa)) a.) + 71Bo) [Qi(s', Q (s) - 00100-111100 (f
-z=
Var
E (ri(s, a.) + ( [Q) (00((ss',))) b.)] - - Q(s,,a))
Var (r:(s,a;) + 71B (s)(s', [Fi]
Given the fact that ry is bounded, we know that Var (s) a. [FF}] S (1 (1 || for some constant. C. Finally from the Lemma E.1, we know {Ai} converges to zero w.p.1, i.e., {Qi) converges O w.p.1. 0
F Future Work
In the paper, we derive a novel lower bound that can be naturally divided into independent surrogate (17) for each agent. By each agent optimizing this surrogate, the monotonic improvement of the joint policy can be guaranteed in fully docentralized scttings. However, the practical algorithm of DPO takes the formula of (20) with several approximations. How to solve the optimization of (17) more precisely is left as future work. Moreover, expect our work could provide some insights for future studies on fully decentralized multi-agent reinforcement learning since current methods still have gap from the optimum as shown in Figure 8.
20


Reviews:
[{"review_id": "PfkSTHec2z", "reviewer": "Reviewer_Cmkb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis paper analyzes the potential of using trust-region based in decentralized multi-agent reinforcement learning and propose practical algorithms, with abundant ablation studies.\n\n## Weaknesses\n\nTheoretically, there exists **an error** in proving the monotonic improvement bound, which is the **main theoretical contribution** of this paper.\nIn the proof of the following equation, the authors only prove that the operator $\\Gamma_{\\pi^{-i}}^{\\pi^i}$ is a contraction mapping.\n$$\nQ_{\\pi^{-i}}^{\\pi^i}(s, a_i)=E_{\\pi^{-i}}[Q^\\pi(s, a_i, a_{-i})]\n$$\n\nThere are two main mistakes:\n\n1. One can not prove that a variable under an operator such as the well-known Bellman operator, converges to a fixed point by only showing the operator is a contraction mapping, since in the algorithm the variable is updated using gradients. Therefore, the authors should consider model learning $Q_{\\pi^{-i}}^{\\pi^i}(s, a_i)$ as a random process (or random approximation) [1].\n\n2. Even if \n$Q_{\\pi^{-i}}^{\\pi^i}(s, a_i)$ and $E_{\\pi^{-i}}[Q^\\pi(s, a_i, a_{-i})]$\nare accurately proved to converge to the same fixed point, the authors can not directly derive that $E_{\\pi^{-i}}[Q^\\pi(s, a_i, a_{-i})]$. The two variables are not proved to be equal at all steps. **So the mismatch between the two variables should also be considered when proving the monotonic improvement.**\n\n   [1] Melo F S. Convergence of Q-learning: A simple proof.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel algorithm for cooperative multi-agent reinforcement learning (MARL) in fully decentralized settings, where each agent learns its own policy without communication or parameter sharing. The main contributions and new knowledge are:\n\n1. The authors propose decentralized policy optimization (DPO), a decentralized actor-critic method that guarantees monotonic improvement and convergence of the joint policy by optimizing a novel surrogate objective.\n2. The authors derive a decentralized surrogate that can be decomposed into individual objectives for each agent, and show that it is a lower bound of the joint policy improvement.\n3. The authors evaluate DPO on various cooperative MARL tasks, covering discrete and continuous action spaces, as well as fully and partially observable environments, and demonstrate that DPO outperforms existing decentralized methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical to securing my recommendation\n\n1. Correct the proof as mentioned in the `Weaknesses` section. \n2. Add discussions about recent works that theoretically analyze the monotonic improvement in MAS, such as CoPPO, HAPPO and A2PO.\n\n###Simply strengthen the work\n\n1. Add the performance of IPPO into the ablation for a clear comparison.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u5FK8OKrLL", "reviewer": "Reviewer_PNJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The authors develop a new surrogate loss for MARL that guarantees monotonic improvement when the parameters are properly chosen. \n\nWeakness: My major concerns are (i) the theoretical results seem some direct extension from single-agent TRPO, with the help of strong and even unrealistic assumptions, (ii) the algorithm with a theoretical guarantee of monotonic improvement might be hard to implement and the algorithm implemented is different from the theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a fully decentralized policy optimization (DPO) algorithm in cooperative multi-agent reinforcement learning (MARL). \nThe assumption of the decentralized MARL setting is that each agent observes the state and the joint policy is factorized into a product of local policies of each agent. Motivated by the surrogate loss of the trust-region policy optimization in the single-agent case, the authors devise a surrogate loss of the joint policy that factorizes into a sum of the individual loss of each agent. This surrogate loss involves the square-root of the max-KL divergence and the max-KL-divergence itself. In practice, the agent-wise separable loss is optimized using two adaptive coefficients and the max-KL-divergence is replaced by an average of KL-divergence over roll-out data. Experiments in several cooperative MARL environments show that the proposed DPO can perform better than independent PPO under fully decentralized settings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Strong assumptions: Using the max-KL divergence seems very stringent. By taking the maximum of $s$, this terms can be very large. This essentially means that, when updating policy, the new policy has to be close to the old policy in each state, which is hard to satisfy when you have a neural network policy. In contrast, in single-agent TRPO/PPO literature, there are already existing works showing that variants of TRPO/PPO archives **global optimality** even when the regularization term is an averaged KL-divergence. E.g., see \"Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes\". I would suggest the authors consider the weaker KL divergence. \n\n2. Unrealistic algorithm: It seems that the algorithm is hard to implement in a decentralized fashion. The reason is that the algorithm needs to solve a policy evaluation subproblem when updating the policies. When implementing the algorithm, essentially we need to let the agents hold their policies fixed until the policy evaluation problem is solved. \n\n3. How do you solve (19) practically?\n\n4. It seems unclear to me why you need both the square-root KL and the KL itself. When the update is small, then squareroot KL is always larger right?\n\n5. Is it possible to use a smaller-scale environment to implement the theoretically verified algorithm in a fully decentralized fashion, and empirically show monotonic improvement?", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gwo0hW81E7", "reviewer": "Reviewer_ToSz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this is a good paper that provides a detailed theoretical study of decentralized policy optimization. Further, it contains a practical algorithm with detailed experimental results in different multi-agent settings. Despite this, I have some important comments regarding the weaknesses of the paper. \n\n1) In the introduction the motivation says that \"many settings in which global information is unavailable\", which led me to believe that the authors are studying partially observable settings. However all the theory in the paper uses the global state and not the observation of individual agents. The experiments seems to use partially observable settings (from the configurations in the appendix). So, I am confused what the authors are trying to accomplish and what setting they are using the paper. \n\n2) There are some important details missing in the paper to ensure reproducibility. First, the reward functions of some environments (Mujoco and SMAC) are not mentioned anywhere in the paper (main paper or appendix), so I am not able to fully interpret the results with the mean episodic rewards. Second, while SMAC is mentioned to be partially observable, how big is the state space and the observation space of agents in this environment, as used in the experiments of the paper? Also, what conditions are needed for \"winning\" as plotted in figure 7 of the paper. \n\n3) Several methods in the experiments show unlearning (ex: IDDPG in Ant, DPO w/o Sqrt in circle control etc.). This makes me feel that the DPO algorithm could show unlearning if run for more steps in the future. Do the authors have any intuitions for why this may not be the case? If not, I would recommend experiments to be conducted for longer periods. Especially several methods in the paper in all experiments still seem to be improving/changing and have not reached a stable point in training.\n\n4) In the experiments different performances of algorithms in different experiments do not have clear explanations. For instance, why does IQL show instability in MPE, but performance close to the best performing algorithm in SMAC? This may show that the hyper-parameters have not been carefully tuned for these baselines in all experiments. \n\n5) In general, I am confused as to what the experiments are trying to accomplish. The paper says that \"focus on comparison between DPO and IPPO and just add IQL as baseline\". Why are such comparisons helpful? Also, I strongly disagree with the conclusion in the paper \"the results show the advantage of DPO over IPPO and IQL, which can be evidence for our theoretical results\" -- the algorithm has important differences with respect to the theory and I am not convinced it is acting as an evidence for any theoretical findings. \n\n6) There seems to be some abuse of notations in the theory which makes it hard to follow everything. The objective $J(\\pi)$ is not conditioned on the initial state (though it should be conditioned on some initial state). The policy (and joint policy) was never defined. In some cases the Q function and value function are not parameterized by the agent index (Eq for $A^i_{old}$ following Eq.6), and in other cases they are parameterized by the agent index (Eq 7), which makes me more confused.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a decentralized policy optimization algorithm (DPO) with convergence and monotonic improvement guarantees. Extensive experiments in different types of cooperative MARL testbeds show good performances of DPO.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please address my points on weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of reviewers were in favor of publication, and the remaining concerns weren't sufficient to override that (e.g. whether the phrasing overly suggested a focus on partially observed domains).
Decision: Accept (poster)