Model-Based Decentralized Policy Optimization 
Authors: Hao Luo
Jiechuan Jiang
Zongqing Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Agent Reinforcement Learning

Abstract: Decentralized policy optimization has been commonly used in cooperative multi-agent tasks. However, since all agents are updating their policies simultaneously, from the perspective of individual agents, the environment is non-stationary, resulting in it being hard to guarantee monotonic policy improvement. To help the policy improvement be stable and monotonic, we propose model-based decentralized policy optimization (MDPO), which incorporates a latent variable function to help construct the transition and reward function from an individual perspective. We theoretically analyze that the policy optimization of MDPO is more stable than model-free decentralized policy optimization. Moreover, due to non-stationarity, the latent variable function is varying and hard to be modeled. We further propose a latent variable prediction method to reduce the error of latent variable function, which theoretically contributes to the monotonic policy improvement. Empirically, MDPO can indeed obtain superior performance than model-free decentralized policy optimization in a variety of cooperative multi-agent tasks.

Under review as a conference paper at ICLR 2023
MODEL-BASED DECENTRALIZED D POLICY OPTIMIZA- TION
Anonymous authors Paper under double-blind review
ABSTRACT
Decentralized policy optimization has been commonly used in cooperative multi- agent tasks. However, since all agents are updating their policies simultaneously, from the perspective of individual agents, the environment is non-stationary, result- ing in it being hard to guarantee monotonic policy improvement. To help the policy improvement be stable and monotonic, we propose model-based decentralized policy optimization (MDPO), which incorporates a latent variable function to help construct the transition and reward function from an individual perspective. We theoretically analyze that the policy optimization of MDPO is more stable than model-free decentralized policy optimization. Moreover, due to non-stationarity, the latent variable function is varying and hard to be modeled. We further propose a latent variable prediction method to reduce the error of latent variable function, which theoretically contributes to the monotonic policy improvement. Empirically, MDPO can indeed obtain superior performance than model-free decentralized policy optimization in a variety of cooperative multi-agent tasks.
INTRODUCTION
Decentralized multi-agent reinforcement learning (MARL) has been commonly used in practice for cooperative multi-agent tasks, e.g., traffic signal control (Wei et al., 2018), unmanned aerial vehicles (Qie et al., 2019), and IoT (Cao et al., 2020), where global information is inaccessible. Independently performing policy optimization using local information, e.g., independent PPO (Schulman al., 2017) (IPPO). is one of the most straightforwat studies (de Witt et al., 2020; Yu al., 2021 la Papoudakis al., 2021) demonstrate that IPPO performs surprisingly well in several cooperative multi-agent benchmarks, which shows great promise for fully decentralized policy optimization.
methods for decentralized MARL. Recent empirical
However, since all agents updating policies, from the perspective of an individual agent, the environment is non-stationary (Zhang et al., 2019). Thus, the monotonic policy improvement, which can be achieved by policy optimization in single-agent settings (Schulman et al., 2015; 2017), may not be guaranteed in decentralized MARL. Concretely, in policy optimization, the state visitation frequency is assumed be stationary since the agent policy is limited slight updates, which is necessary to guarantee monotonic policy improvement (Schulman et al., 2015). However, in decentralized multi-agent settings, as all agents are updating policies simultaneously, the state visitation frequency will change greatly, which contradicts the fundamental assumption of policy optimization, thus the monotonic improvement of policy optimization may not be preserved.
To address this problem, we resort to exploiting the environment model to stabilize the state visitation frequency and help monotonic policy improvement. However, learning an environment model in decentralized settings is non-trivial, since the information of other agents, e.g., other agents' policies, not observable and changing. Therefore, we introduce latent variable to help distinguish different transitions resulting from the unobservable information. And then we build an environment model for each agent, which contains a transition function, a reward function, and latent variable function that learns the latent variable given observation. The agents optimization methods, e.g., TRPO (Schulman et al., 2015) or PPO (de Witt et al., 2020), on both the experiences generated by the environment model and collected in the environment.
trained using independent policy
Since the environment is non-stationary, the latent variable function is also varying during learning. We theoretically show that independently performing policy optimization experiences generated

---

Under review as a conference paper at ICLR 2023
by the environment model with the varying latent variable function can obtain more stationary observation visitation frequency than on the experiences collected in the non-stationary environment. Thus, independent policy optimization goes more stable on the environment model.
Moreover, to obtain monotonic improvement, the gap between the return of interacting with the environment and the return predicted by the environment model should be small. We theoretically analyze that the return gap is bounded by the prediction error the latent variable function. As the latent variable function varying due to non-stationarity, minimize the prediction error, we propose a latent variable prediction method that uses the historical variables to predict the future variable. Thus, the latent variable prediction can reduce the return gap and help the monotonic policy improvement.
The proposed algorithm, model-based decentralized policy optimization (MDPO), is theoretically grounded and empirically effective for fully decentralized learning. We evaluate MDPO on variety of cooperative multi-agent tasks, i.e., a stochastic game, multi-agent particle environment (MPE) (Lowe et al., 2017), and multi-agent MuJoCo (Peng et al., 2021a). MDPO outperforms the model-free independent policy optimization baseline, and the proposed latent variable prediction additionally obtains performance gain, verifying that MDPO can help stable and monotonic policy improvement in fully decentralized learning.
PRELIMINARIES
Dec-POMDP. A cooperative multi-agent task is generally modeled as a decentralized partially observable Markov decision process (Dec-POMDP) (Oliehoek & Amato, 2016). Specifically, P, R, 7). S is the state space, I is the set
Dec-POMDP is defined as a tuple = {S, Aμr is the joint action space, where A, is the action space for
of agents, and A A each agent i. At each state s, each agent E I merely gets access to the observation O, E O, which is drawn from observation function [2ss i), and selects an action a; E A,, and all the actions form a joint action a E A. The state transitions to next s' according to the transition function P(s'|s, a): : S A x S - [0, I], and all agents receive shared reward The objective is to maximize the expected return 77(T) = EE> >rra|po, π under the joint policy of all agents æ and distribution of initial state Po. where 7 E (0, 1) is the discounted factor. The joint policy π can be represented as the product of each agent's 's policy πy. Also we denote - as the joint policy of all agents except i.
R(s, a) S x A IR.
Fully decentralized learning, We consider the fully decentralized way to solve the Dec-POMDP (Tan, 1993; de Witt et al., 2020), where each agent independently learns a policy and executes actions without communication or parameter sharing in both training and execution phases. Since all agents are updating policies, from the perspective of individual agents, the environment is non- stationary, which fundamentally challenges decentralized learning (Zhang et al., 2019). The existing decentralized MARL methods are limited. Independent Q-learning (IQL) (Tan, 1993) and independent policy optimization, e.g., IPPO (de Witt et al., 2020), are the most straightforward fully decentralized algorithms. Despite good empirical performance (Papoudakis et al., 2021), due to non-stationarity. these methods lack theoretical guarantees. IQL has no convergence guarantee, to the best of our knowledge. Although there has been some study (Sun et al., 2022), IPPO may not guarantee policy improvement by independent policy optimization, since the assumption of stationary state visitation frequency for policy optimization may not hold in fully decentralized settings, which we will discuss in the following.
policy improvement. In Dec-POMDP, from centralized perspective, we can obtain a TRPO objective (Schulman et al., 2015) of the joint policy π for the monotonic improvement,
Monotonic
"<w)
now - 17 old
π
new (als)
T old (s, a) -C- -CDRLL (T old [ || """)
(I)
new (als)ATTH
(s, a) C --DAx (T old [ new,,
π
where pT pmol((s) (s) = -n Pr(st = s old) is the discounted state visitation frequency given ald
oold, similarly for p= (s), A
the advantage function under gold, DRL [ max, DKL. (7ll (-1))\ (-1s), and C is constant. From (1) to (2) is approximation or
2

---

Under review as a conference paper at ICLR 2023
assumption (Schulman al., 2015). As p=" is unknown and the policy is limited to slight updates, is approximated by p=' However, in fully decentralized MARL, this assumption may not hold, as all agents are updating their policies simultaneously and their joint policy may change significantly especially when the number of agents large. This will severely affect the performance of independent policy optimization. Although we can constrain the policy update of each agent to slight like TRPO, this leads to much slower convergence, especially in fully decentralized the joint policy has much larger search space and is merely optimized by independent learning of individual agents.
o new
MARL, where
3 MODEL-BASED DECENTRALIZED POLICY OPTIMIZATION
In this paper, we provide a novel perspective and resort to the environment model to bridge the gap between "='"" and pa* for each agent such that the monotonic joint policy improvement can be potentially achieved by fully decentralized policy optimization.
As illustrated in the following, we turn the learning process into a Dyna-style (Sutton, 1990) decen- tralized model-based method with the green path Each agent i additionally learns decentralized model using local information from policy rollout and can optionally perform policy optimization on the experiences from model rollout. When optimizing policy with model rollout, we essentially have ! which means the state visitation frequency in model rollout (Pmodel) is more stable. Thus, the approximation from (1) to (2) becomes acceptable under looser constraints of policy update. Further, we can bound the gap between the returns of policy rollout (7) and model rollout (oyd)de))), ), formally in 2. Once the bound (B) is controllable throughout the learning process, it can potentially guarantee the monotonic improvement of the joint policy in the real environment.
environment policy rollout -
policy optimization 7
Paodal
-moodet
model
model rollout
-
policy optimization 2
moddel
mymoode (n < B
Thus, ! and ② together highlight the potential benefits of incorporating environment model for decentralized policy optimization. In the following, we discuss how to learn such a decentralized model, theoretically investigate its benefits for decentralized policy optimization, and analyze the return bound for monotonic policy improvement.
3.1 LATENT VARIABLE MODEL
In decentralized learning, from the perspective of each agent i, the transition function and reward function are respectively,
Ps(''ls, a.) = Fa, ((s'|s,a,,... and Rg a:) Ba-,,t, R(s, a, a-:;
where a_; denotes the joint action of all agents except i. As other agents are also updating their policies, P. and R, are varying throughout the learning process, which is the well-known non- stationarity problem. Moreover, decentralized learning, the model can only be learned on (o,, a, of, r). Thus, challenging to construct environment model from the perspective of an individual agent.
each agent i usually obtains observation instead of state in
To build a decentralized environment model, we introduce a latent variable 2,, which helps distinguish different transitions resulting from varying unobservable information of the full state and other agents' policies. Then the transition function and the reward function can be redefined as:
Po(ofoo ais 24) and [2 di, 21).
As we discuss fully decentralized learning, we drop the subscript of for simplicity in the following.
To model the transition function and the reward function with the latent variable, we define the latent variable function from the perspective of an individual agent, (zzo)), which indicates the
Related work model-based MARL can be found in Appendix E. However, none of the existing work considers exploiting the environment model to help fully decentralized policy optimization.
3

---

Under review as a conference paper at ICLR 2023
probability of latent variable given observation As is related to the policies of other agents, (0(200 o) also varies during policy updates. A latent variable model consists of three modules: transition function Po. reward function Rg, and latent variable function Vu.. to predict the next observation and reward. As the impact of unobservable information is designed to merely reflect on the latent variable, although other agents update their policies, the transition function and reward function stay constant and only the latent variable function varies. We learn such model by maximizing the likelihood of experiences of policy rollout D,
max Ro (ooa.d/))/Dii an (oolo (Ro (0, - -) 8,0040
We examine the correlation between latent variable learned end-to-ent and inaccessible information in a simple setting, and the learned latent variable is indeed correlated with the inaccessible information. More details can be found in Appedix
Moreover, when using the learned latent variable model to train an agent, we adopt k-step branched model rollout in MBPO (Janner et al., 2019) to avoid compounding model error due to long- horizon rollout. Concretely, at each policy update of an agent, we sample htstep length experiences {(01, 41, ri). ... (o), an. 0y, rT)} from policy rollout D and perform k-step model rollout start- ing from the last observation o under current policy The policy is updated on the merged (Oh+k, ah+k, h Th++)] by policy optimization, (h + k)sttep experiences {(0), 01, o1. ri), ""
e.g., PPO (Schulman et al., 2015).
3.2 STABLE POLICY OPTIMIZATION ON MODEI.
Now, we turn to analyze the benefits of such model- method over model-free independent policy optimization. We first theoretically analyze that independently performing policy optimization egg., TRPO (Schulman et al., 2015) or PPO (Schulman et al., 2017), on the latent variable model can make the learning process more stable.
In decentralized learning, from the perspective of an agent, given the true latent variable function v/, the discounted observation visitation frequeney of D obtained by policy rollout is defined
- (0) Po n. (o) + ypT A (0) + 7 J. (o) + P2
where p 7. (o) A Pr(o: = 0) and or is the observation at timestep / of experience from D. Note that n0 (o) unbiased estimate of discounted observation visitation frequency when interacting in the environment. Similarly, = denotes the discounted observation visitation frequency for experiences obtained by model rollout. During the learning process, T" and s" respectively denote the policy and latent variable function after the rith policy update. Then, we have the following theorem. All proofs are available in Appendix A.
Theorem 1. Define (o) & pT .²" (o)
(0)) and denote ||pp||| A max(p"
qin-1 (0)), similarly for || || and ||A It holds that,
| S C (Ex + Ex),
where En A max |aax|||ATi || Egp A max ||A and C is a constant. Assume +2 (1-a)) -
72
and 1/ = 53 2, It holds that Enp > Exg., and the bound above is lower when substituting with s/w-
According to Theorem 1, the divergence of discounted observation visitation frequency is bounded by the divergence of policy and latent variable function. Again, the policy divergence can be constrained via policy optimization, like TRPO. Thus, the main difference lies in the divergence of latent variable function. As indicated by Theorem 1, the learned latent variable function vi. has smaller divergence between consecutive policy rollouts than the true latent variable function tb. Therefore, independent policy optimization on experiences generated by the latent variable model can obtain more stationary observation visitation frequency than on the experiences collected in the varying environment, so the learning process of independent policy optimization becomes stable on the model.
Since i is varying and the is continuously updated using the experiences from several recent policy rollouts, we the form of soft-update for the relation hetween 1 and vh..
4

---

Under review as a conference paper at ICLR 2023
3.3 RETURN BOUNDS
We then analyze the bound of return gap between interacting in the environment and interacting with the model. If the return improvement of interacting with the model higher than the bound, the agent can obtain the monotonic policy improvement when interacting in the environment.
However, the return of interacting in the environment is hard to analyze in decentralized learning since the policies of other agents are inaccessible, we turn to analyze the return in policy rollout, which is unbiased estimate of expected return in the environment.
Several bounds have been introduced in MBPO (Janner et al., 2019) for the return bound analysis, which however are sufficient in decentralized learning. Thus, we need introduce two new bounds that indicate the divergence of latent variable function between consecutive policy rollouts and the error of learned latent variable function. Formally, with v/", V.w and x" respectively referring to the true latent variable function, the learned latent variable function, and the policy of the nth policy rollout, we denote the bounds as follows:
reward bound T'mx A max max (R(o, a, 2), Ro a, =);
0,0,2
policy divergence E, maxDry (T||="), where Drv is total variation distance;
transition function error eg max Ba (1, Drv (P (044100 (1) PP (0041 Oz 2)));
latent variable function divergence Go max (16 (1)m)))
learned latent variable functionn error
maxDrv (W"))
Additionally, we use several notations to represent different returns. The return in rith policy rollout with the true latent variable function v is denoted as Tι(T, vb), the return in model rollout with the wih learned latent variable function dew is denoted as 17 ( (T, vi)), and the return in k-step branched model rollout with /tsttep experiences of nith policy rollout is denoted as Now we analyze the return bound of model rollout and branched model rollout with the newly introduced E and 6 in the following two theorems.
T), (v, ())).
Theorem The return gap between n 1th policy rollout and model rollout with nihh learned model bounded as:
2r 2rmax
In (T,, gn+t)) -17 model (T,, +8) S
(760 + 2ez + 2e + Ew).
(1 7)
Clcastritu
Theorem 3. The return gap between n + 1th policy rollout and branched model rollout with nith learned model is bounded
In (IT, + - branch ((T", (v.", +()) C' (Ees Exs Ewi Eur Ev).
According to Theorem 2 and 3, we can guarantee the monotonic improvement in the environment via improving the return in model rollout or branched model rollout beyond a bound linear to (eT, E ). ). In these bounds, Eo and ew are limited via supervised learning and e, is constrained by policy optimization. However, Go is left unrestricted. In the following, we try to find better bound in which all elements are controllable.
3.4 LATENT VARIABLE PREDICTION
In order to restrict the impact of divergence of the latent variable function, we introduce one new error bound, which measures the divergence between the learned latent variable function and the true latent variable function in incoming policy rollout. Formally, such an error bound in rith policy rollout is defined as:
E A maxDry (1+++1))))
Now use & in place of e to analyze the return bound of model rollout and branched model rollout again in the following two theorems.
Theorem 4. The return gap of 1th policy rollout and model rollout with nah learned model is bounded as:
|n (IT, t/ n) model (TT, (T.4)))
C (Eo; km;
5

---

Under review as a conference paper at ICLR 2023
Plu
f
D
Og dey
of
P
R
Figure 1:
The environment model includes four modules: transition function Po. reward function Rg-
latent variable prediction function fc, and latent variable functions ? tha;) ) over consecutive policy rollouts. For learning, each agent maintains the experiences of I consecutive policy rollouts, Po and Rig learn on l consecutive policy rollouts, elw,, learns on the experiences of (th policy rollout, and f learns to predict ith latent variable given I - latent variable functions.
Theorem 5. The return gap of n + 1th policy rollout and branched model rollout with nihh learned model is bounded as:
In (T,, (ptttt)) branch ((T", TT, (v",,
(V",,"))) +00) S C' (co, Exy Cun, Rw)
Now all elements of the bounds controllable, once To achieve this, we introduce a latent variable prediction function, which predicts the latent variable distribution given observation in incoming policy rollout via latent variable distributions of in the latest - policy rollouts. However, as the true latent variable function cannot be obtained directly for agent, the latent variable prediction function fc can instead minimize:
constrain & in the learning process.
maxDry (1h,, (o)llf (o), (o),
(o)))
With such a latent variable prediction function, & is controllable.
3.5 ALGORITHM
With all the theoretical analysis and discussions above, we are ready present the learning of model-base decentralized policy optimization (MDPO).
As illustrated in Figure 1, the environment model consists transition function Pe, reward func- tion R3, latent variable prediction function fc, and latent variable functions { ... Mhn over recent consecutive policy rollouts. The experiences of the consecutive policy rollouts Deme [D1 ..... D'} are also stored.
After the latest policy rollout 1, we update the transition function and reward function, and learn the latent variable function tiw, of policy rollout / by optimizing the objective:
max B de
(E rr) won (a) Po (o'lo, a, (Ro (0, =))
(4)
where uha, is obtained by updating Viyy, using D', while Pe and Rs are updated using Deny to make sure they are stable across policy rollouts. Then, the latent variable prediction function fc is updated using D' by optimizing the objective:
max Bo-D! BoNDD_=>/aa (='/e (o). s (o) [fe (=11)),....,_))))]
(5)
For model rollout, the model predicts the transition in incoming policy rollout given observation and action via - 1 latest learned latent variable functions (#22, wh) as: fc (:1²² thw, (0), .......................
than (o)) 0 Po (o, a, z) i RS (o,a, Finally, the policy is updated using the branched model rollout by policy optimization, such as PPO or TRPO. We summarize the full learning procedure of MDPO in Algorithm 1.
(6)

---

Under review as a conference paper at ICLR 2023
Algorithm 1. MDPO
Initiate Dene = (D), ,.. D'), n, Pa. Rs. (V/wss" :. Nhwnd. Sc.
2:
repeat
3:
policy rollout in environment and obtain D
4:
optimize Pe.R. and Win on Deno with (4)
5:
optimize prediction function fc on D with (5)
6:
obtain branched model rollout Dratoot based on D' using Po. RS. V, and fc with (6) optimize policy т using Droliout by PPO or TRPO ...... do 8: for Di+1 Mwe, f- twst1
7:
9:
<
10: end for
11:
until terminate
EXPERIMENTS
For evaluation, compare MDPO, MDPO without latent variable prediction (denoted by MDPO w/o prediction), and independent PPO (IPPO) (Schulman et al., 2017) on a set of cooperative multi- agent tasks including stochastic game, multi-agent particle environment (MPE) (Lowe et al., 2017), and multi-agent MuJoCo (Peng et al., 2021b). We do not consider StarCraft multi-agent challenge (SMAC) (Samvelyan al., 2019), because IPPO has been shown to perform very well in SMAC (de Witt et al., 2020; Papoudakis et al., 2021), close enough to centralized training with decentralized execution methods like QMIX (Rashid al., 2018) and MAPPO (Yu et al., 2021a). Thus, the gain of MDPO may not be clearly evidenced there.
By experiments, we try to answer following three questions:
1. Does the latent variable model help generate experiences with more stationary observation visitation frequency experimentally?
2. Does latent variable prediction help to control ?
3. Does MDPO help improve performance in decentralized learning?
For fair comparison, the network architecture and hyperparameters are the same for IPPO and MDPO. The number of environment steps taken in each round (policy rollout, network update) is consistent and thus we compare the performance of methods under the same number of environment steps and policy updates. Note that since we consider fully decentralized learning, for methods, agents do not tralized learning (Terry et al., 2020). More details on experiment settings, implementation, and hyperparameters available in Appendix C.
shrameter-ssaring. Indeed, parameter-sharing should not be allowed in decen-
4.1 STOCHASTIC GAME
The stochastic game is a cooperative game with 30 observations (states), agents, and 5 actions for each agent, and every episode consists 40 steps. The transition function and the shared reward function are randomly generated. The game is chosen to verify our theoretical results.
Figure (left) shows the learning curves of MDPO, MDPO w/o prediction, and IPPO, among which MDPO performs better throughout the learning process. With finite observation space in this game, we calculate the divergence of observation visitation frequencies (||Apl) in Section 3.2) in consecutive rollouts. Concretely, we calculate the L1 distance of observation visitation frequency over observations in consecutive rollouts (policy rollouts for IPPO and branched model rollouts for MDPO), and their curves are shown in Figure 2 (mid). We can that the latent variable model generates experiences with more stationary observation visitation frequency than IPPO. which is consistent with Theorem 1. This may account for the superior performance of MDPO w/o prediction over IPPO.
We also examine how well the latent variable prediction helps to control the prediction enror (ê.). As the real latent variable function is inaccessible, we examine é by comparing how well the learned environment model predicts with and without latent variable prediction. Specifically, measure the

---

Under review as a conference paper at ICLR 2023
storage 2oum
obscrvtion virtatte frequezcy
prodit otaralia Cnalaroyy
290
1.90
338
5355
peddec
35
MDPD wis prodetion
MDPO
10-
1.5
01
Figure 2: Learning curves of MDPO compared with MDPO w/o prediction and IPPO on the stochastic game: average return (left), observation visitation frequency divergence (mid), and model prediction errors (right). Each round is 1600 environment steps.
Cooperative Navigation
Regular Polyyon Control
14000
-200
12000
-300
10000
400
8000
6000
-600
-700
UPS
NDM wil pradecton
2000
-800
300
600
1200
1500
900
1200
1500
round
round
Figure 3: Learning curves MDPO, MDPO w/o prediction, and IPPO in Cooperative Navigation (left) and Regular Polygon Control (right). Each round is 1280 environment steps.
mean cross-entropy of the next observation distribution predicted by the nith learned model and the ground truth in the + 1th round, and the mean L1 distance of predicted reward and ground truth reward. The curves are shown Figure 2 (right). The lower prediction error of MDPO indicates that latent variable prediction error (E..) is controlled at a lower level than without latent variable prediction. Moreover, as shown in Figure 2 (mid), the divergence of observation visitation frequency of MDPO and MDPO w/o prediction are similar but much lower than IPPO, which indicates e is still under control in MDPO. This indicates that MDPO can well control both e, and éw-
As MDPO helps to handle non-stationarity in multi-agent settings from the perspective of individual agent, it will be natural to also apply MDPO to non-stationary single-agent settings. So, we modify this stochastic game into non-stationary single-agent game and show that MDPO also outperforms the baselines. More details are available in Appendix D.
4.2 MPE
MPE is multi-agent environment with continuous observation. In MPE tasks, agents observe their own positions, velocity, and others' relative positions. And agents are expected to fulfill certain goal via controlling their accelerations in every direction which is continuous in our experiments. Two tasks of MPE, 4-agent Cooperative Navigation and 5-agent Regular Polygon Control, are chosen for performance comparison. In 4-agent Cooperative Navigation, agents learn to cooperate to reach 4 landmarks respectively. In 5-agent Regular Polygon Control, agents learn to cooperate with another agent, which controlled by a fixed policy, aiming to form regular pentagon, and the reward is given according to the similarity to regular pentagon.
Figure shows the learning curves of all methods. Generally, MDPO w/o prediction performs better than IPPO, which verifies that the latent variable model can help decentralized policy improvement by making the observation visitation frequency more stationary. And MDPO outperforms MDPO w/o prediction, which verifies latent variable prediction reduce the gap between the return of interaction and the return predicted by the environment model.
It is worth noting that the unobservable information required fulfill the goal is at completely different levels in the two tasks. Concretely, acknowledging the general direction others is enough to decide which landmark approach in Cooperative Navigation. Yet the precise positions of others
8

---

Under review as a conference paper at ICLR 2023
HaliChoetah 3x2
Hopper
Ani4x2
1400
600
600
ivorage retum.
1200
300
600
400
1000
300
200
01
600
200
round
round
round
HalfCheetah 6x1
HalfChtetta S.(.1.1.1.2)
HalfCheetah 5x2
800
500-
avanug Fcn121
600
600
410-
600
200
200
IPPO
MDT wis pridicso
0-
MDPO
100
200
300
400
100
200
300
400
0
100
200
300
400
round
round
round
Figure 4: Learning curves of MDPO, MDPO w/o prediction, and IPPO in six multi-agent MuJoCo tasks. Each round 4000 environment steps for 4-agent Ant and 2000 for other tasks.
matter to form regular polygon in Regular Polygon Control and are hard learn accurately. Thus, MDPO performs well in Cooperative Navigation since the very early learning stage, while does not perform well in Regular Polygon Control before 600 rounds. Although more accurate model is required in Regular Polygon Control, MDPO still converges to better performance. And this indicates progressive pattern in prediction also works when prediction hard to be fairly accurate.
4.3 MULTI-AGENT MUJOCo
Multi-agen aggett MuJoCo is continuous multi robotic control environment, based on OpenAl's Mujoco Gym environments. In multi agett MuloCo task, each agent controls several joints of the robotic to move forward, where both the observation space and action space are continuous. We choose 3-agent Hopper, 4-agent Ant, and versions of HalfCheetah with different agent numbers or joint allocation for performance comparison. Details of joint allocation given in Appendix C.
As illustrated in Figure 4, MDPO consistently performs better in these tasks with different difficulties and various agent numbers. Compared with MPE, agents in multi-agent MuJoCo have deeper impacts on each other due to the interaction between adjacent joints. Consequently. the transitions of each agent closely related to the policies of other agents. Thus. non-stationarity caused by policy updates other agents is severer in these tasks, resulting in IPPO struggling and converging to low performance. Moreover, note that MDPO w/o prediction performs almost the same as IPPO worse in some tasks. The poor performance of MDPO w/o prediction is a consequence of larger e caused by strongly associated agents in these tasks. Thus, latent variable prediction necessary in these tasks with closely associated agents.
CONCLUSION
In this paper, we propose model-based decentralized policy optimization (MDPO). By introducing latent variable into the environment model, we theoretically show the model helps to generate experiences with more stationary observation visitation frequency and benefits decentralized policy optimization. Furthermore, We theoretically analyze that the return bound for monotonic policy improvement is controllable by the prediction error of the latent variable function. Consequently, we propose a latent variable prediction method to constrain the prediction error. We examine all the theories and designs via experiments verify our theoretical results and show MDPO indeed obtains superior performance over model-free decentralized policy optimization.
set of cooperative multi-agent tasks. Results

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zilong Cao, Pan Zhou, Ruixuan Li, Siqi Huang, and Dapeng Wu. Multiagent deep reinforcement learning for joint multichannel access and task offloading of mobile-edge computing in industry 4.0. IEEE Internet of Things Journal, 7(7):6201-622 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint 1:Xiv:2011.095333 2020.
Yali Du, Chengdong Ma, Yuchen Liu, Runji Lin, Hao Dong. Jun Wang, and Yaodong Yang. Fully decentralized model-based policy optimization for networked systems. arXiv preprint arXiv:2207.06559, 2022.
Vladimir Feinberg. Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, arXiv preprint 1803.00000000000 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 2019.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agen reinforcement learning: Intention sharing. In International Conference on Learning Representations, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettivy environments. In Advances in Neural Information Processing Systems, 2017.
Frans A Olichock and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht, Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems, 2021.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by sulf-supervised prediction. In International Conference on Machine Learning, 2017.
Bei Peng. Tabish Rashid, Christian Schroeder de Witt, Alexae-Aeex Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems, 2021a.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmae: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems, 2021b.
Han Qie, Dianxi Shi, Tianlong Shen, Xinhai Xu, Yuan Li, and Liujing Wang. Joint optimization of multi-uav target assignment and path plamning based on multi-agent reinforcement learning. IEEE access, 7: 7.146424-44627. 2019.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Conference on Autonomous Agents and MultiAgent Systems, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning (ICML), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
10

---

Under review as a conference paper at ICLR 2023
Mingfei Sun, Sam Devlin, Katja Hofmann, and Shimon Whiteson. Monotonic improvement guaran- tees under non-stationarity for decentralized ppo. arXiv preprint arXiv:2202.00082, 2022.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In International Conference on Machine Learning, 1990.
Ming Tan. Mgeti- reinforcement learning: independent versus cooperative agents. In Interna- tional Conference on Machine Learning, 1993.
Justin K Temry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint 13X2, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-basee reinforcement learning, arXiv preprint arXiv: 000000000077... 2019.
Hua Wei, Guanjje Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: reinforcement learning approach for intelligent traffic light control. In International Conference on Knowledge Discovery & Data Mining, 2018
Daniël Willemsen, Mario Coppola, and Guido CHE de Croon. Mambpo: Sample-efficient multi-robot reinforcement learning using learned world models. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021a.
Xiaopeng Yu, Jiechuan Jiang, Haobin Jiang, and Zongqing Lu. Model-based opponent modeling. arXiv preprint arXiv:2108.01843, 2021b.
Kaiqing Zhang, Zhuoran Yang, and Tamer Basar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint arXiv:/911.10663 2019.
Kaiqing Zhang, Sham M. Kakade, Tamer Basar, and Lin F. Yang. Model-based multi-agent RL in zero-sum markov games with near-optimal sample complexity. arXiv preprint arXiv:2007.07461, 2020.
Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou. Model-based multi-agent policy optimiza- tion with adaptive opponent-wise rollouts. arXiv preprint arXiv:2105.032: 2021.

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 OBSERVATION VISITATION FREQUENCY DIVERGENCE
In this section, we provide proofs for the upper bound of observation visitation frequency divergence. Lemma 1. Given two pairs of policy and latent variable function, (T1, 111) and (T2, 22). Vo E O, it
holds that
Σ I (alo) 11 (z/o) т (alo) 2/ (z|o)] < [A| ||| - T2ll + |2] - 1/411 -221.
0,2
where
[ -
- (1 (a(lo)))) 1141 b2ll 1/ - -b>(20))
0,0
Proof.
m (alo) thi (z)o) To (alo) 2/2 (2|0)]
m (a w/1 (a|o) m (alo) di (=|o)
0,z
+
m (alo)
82 (z|0) 2 (alo) 1/2 (z|o)
m (alo)
| (z|o) 2/2 (=|o)
+ Σ 1/2 (210)
Σ | (alo) m (alo)
$ ||| - ||| T2ll + +1 11 - 221l
Lemma 2 (Timestep observation visitation frequency recursion). Given two pairs of policy and latent variable function (TI, 11) and (T2, v2), we define:
App (mnd)))))
(0) - Pt (0) Pt 22. (0),
App (
max A ppt p (1 (0)
It holds that
Appti (in (o') S |A|. ||] T2ll + |2| 11v1 - 1/2 + 10| || (
Proof. For observation visitation frequency timestep t 1, there is recurrence relation:
PE (o))
Σ
Pt
Σ P P(d(a,, (alo) (2|o)
Thus, the divergence of observation visitation frequency at timestep t + 1 can be processed corre- spondingly:
App A (1020)))))))) (o') = P
T1
(o')
22.2 (o')
Pif
11.01 (o) Σ
(al)) (als) (elo))
a.2
Σ
pr
(P (c
piine (o) Σ
(P
di (z|0) ma (alo) de (xlo)))
8,5
Σ
Apl (1) (o) Σ (P 2)) 22 (=|0))
12

---

Under review as a conference paper at ICLR 2023
Using Lemma 1,
bound the divergence of observation state frequency timestep 1:
Apt41 Ap14tar)iltaae)))
(o))
+ Σ
Pi m (0)
Σ
(P(o'la,,,2) m (alo) en (=|o) - ma (alo) wa (zlo)))
+ Σ
(0)
Σ (P (2'1aa, 0, m2 (2((o))
0,2
(p (0) (1A|] - Æ2ll
2 |1 4221))
+ A
Σ (P 0, (alo) (2(00))
wall +10 Apt ( $ || || F2ll + +1 11651
Lemma 3 (discounted observation visitation frequency divergence bound). Given two pairs of policy and latent variable function (I1, 1/2 and (I2, 4/2), with the same distribution of initial observation po (o), it holds that
S <C([ - T2ll + +111/1 1/2ll).
where C' is a certain constant.
Proof. We transform it to the cumulative form of the timestep, and scale it using Lemma
Ap ( (0) =
7 A Ap (0)
71 Apt (( (0)
7 Σ 22-
e-T
Apt
(FI. (o)
t=T
(AA) || و2/l
+ |1
1/2
+7
T-2
Σ
x' Ap! (wt
+
27
1 --7
(111)) (1 ||
27
T2ll + 12 v22l)) +
1
Thus, get bound discounted observation visitation frequency divergence:
A
Si (1|=) - Tzll + |/v/ 1/2l1),
(plnnriioo < 110 m -
C² (| Æ2ll + 11 2/2 Il)
A.2 LATENT VARIABLE FUNCTION DIVERGENCE
In this section, we provide proof for divergence comparison between latent variable function of policy rollout and learned latent variable function in the model.
13

---

Under review as a conference paper at ICLR 2023
Lemma 4 (Latent variable function divergence comparison). Assume 4 (1 - + an and initially th = vb', where n is the nth policy rollout. Then,
Ev > Edis
where E A max ||06 -
and En... A max || 1/1-111-
Proof. Firstly, construct such recursive inequality: 13 (1 a:) wi ax/-1111
|1 2
11 I 11550 - -000111 + (1 - (1-a)]|man-1 -3-
Thus, we can expand it recursively:
v/3
< 1/" =/
+ (1 a) || | w-
13"" 1/2m
(1 a) | 80 -2
+... (1 a) a)==1 |/ 20.0
1/ 1/-1
+ (1 a)) |0001 2
+ (1 -)=-1 19 1//
E (1 + (1 a) + (1 (1-a)"
E
Using this inequality, we can zoom
|/ // w/8-111
1| 2i || (| // wh-1 + oz/"
-b0-11 11 a =allv"" ----11 2 < Exp.
Thus, Epp. = max الا 1タ -タ < Ev.
Now combine Lemma 3 and 4 to prove Theorem 1.
Theorem 1 (Latent variable model benefits). Define (0) & p==."""
.²" (0)()) p
(0), and
denote || & max)p" (o)). similarly for ||T' and ||A It holds that,
1Ap"l1 || S +
where & A max max/|/1r || Eg A max || and C a constant. Assume v/ n
a))
o and initially 42 = v0. It holds that ENs > Exg.. and the bound above is lower when substituting with vh..
Proof. Using Lemma 3, we can scale ||Ap"||.
||pp||| S C (lAT"l +
+ Exe
Using Lemma 4, Eg > E Thus, the bound above is lower for Exr..
A.3 LEMMAS FOR RETURN BOUND ANALYSIS
In this section, we prove several lemmas as preparations for return bound analysis.
Lemma 5 (TVD bound of joint distribution). Consider two joint distributions of n + variables like this:
Pi (1,91. ..)) = (z)
II P (u.(z)
1=1
Pa (2, vn) = Pa (x)
II Ps (3/(z)
SSince ii is varying and the is continuously updated using the experiences from several recent policy rollouts, the form of soft-update for the relation hetween 1 and viw..
14

---

Under review as a conference paper at ICLR 2023
We can bound the total variation distance of the joint distributions
.9.) Drv (Pi (a, "i... .yn) P/ (I. ".... .wo) - ...))))))))))))).............)..
Drv (Pi (x) Pa (z))
maxxry (Pi (vila)) (v.l)))
Proof. We start the proof from basis case when n 1:
Drv (PillP2)
Σ
P1 (z.
Pa
P1 (z) PP (glz) (z) (11z)
2 2.
Σ N.
P1 (z) P2 Pz (01z) + (2 (1(2)) (y|=))
Σ P1 (z) (z) Σ (x) (z) Drv (P1 (y/z) PP (-|=))
2
<Drv (P1 (z) P2 (z)) max (Pr (y|z) 2) P (y|=))
Similarly, we can prove the case of multi-variables:
Drv (P] S Drv (Pi (z) || (x)) -axDrr (Pi (on..,,0 (z) P2 (VI, ".. , -9nlx))
$ Drv (Pr (x) P2 (x)) +max Drv (Pr (yn (x:) P2 (v1lz))
maxDrv (Pi (12, ... yn P2 (y2,... "",,".",,""""" -,,lx=x)))
Drv (P) ( maxDry (Pi (y1 (x) P2 (y1(z))
x
max (P1 (y2, Un (x) P2 (y2, ,ynlx))
< Drv (x) 112 (z))
maxDrv (PI (v/lz) 1|3 (wlz))
Before proving following lemmas, we clarify the premise discuss in this section is based In Dec-POMDE denote the co-occurrence probability of tuple (o, a, 2) at timestep t as pt (o, a, e P(o: = 0, 0g = a, 21 = 2).
Consider two Dec-POMDPs different merely in transition function and reward function, G1, Gz. P represents the probability in G1 while P2 for Gg. Different policies and latent variable functions, (TI, 11) and (T22 1/2), are used rollout respectively in G1 and G2, We denote several bound
between them:
reward bound: Tmax max max R1 Ri(o, 2), R2(o, a,2);:
policy bound: ex - maxDry ( T2);
latent variable function bound:
ex max MaxxDrv (11111)2);
transition function bound: cm A maxE p pt- Drv (Pi (o:lo, a, 2) [|2 (0xlo, a, z)).
Additionally, consider the branched model rollout mentioned in Section 3. Policy, latent variable function, transition function, and reward function vary before and after the model rollout branch. We
15

---

Under review as a conference paper at ICLR 2023
denote these functions via superseripts 'Pre' for function before branch and 'Post' for functions after branch. Correspondingly, when discussing branched model rollout, extend the bounds above:
reward bound: Tmax A max max (Rr"""" a, 2), Ross (o, 2), R (o,a, 2), R R (o.a,
0,0,2
Pre e maxDrv (mf (| Pre) Post & maxDrv (xforlll (n
policy bound: 6
ct
latent variable function bound: cp
Pre maxDrv (vS (12" :c0 Post A maxDrv (
transition function bound: 6m
Pre - maxE
an pf P. Drv (Pf (odoo, a, 2) 1IP! a;))
Post A maxE
0.0,2-12 Pont Drv (pfost (oelo, IPPo (oolo, a,
cm
Lemma (Observation distributions TVD bound). The total variation distance of observation distributions at timestep t, (oz) and P2(0e), can be bounded as below:
Drv (Pi (0x) PI (o.)) S (ex c + tmm
Proof.
1
Drv (Pi (o.) |2 (oi) =
Σ
P1 (o:) Ps (o.)
1
Σ
P (0,a, Pa (olo,,,,) P
(0, Ps (o)lo,
2
Σ
Σ
pt (0,0,2) - P (0, (orlo,
02
0,0,2
Σ
Ps- (0,0, 2) PI (o)lo,a, Ps (oclo,,,,
0,,,2
Pl- (0,0,2) P (0,a, Σ
Pi a.)lo,,,2)
2
+ E
DrV (Pr (0ol0, 2) [1aboo_,;)(((((o)oo..:))))
=Drv (pt-) (0,a,2) 1| (0,a, tmm
According to Lemma 5,
Drv a0 ((.a)))) <Drv (Pi (01-1) P2 (o--1)
+ +maxDr (मı)( + maxDrv (W1l))))
=Drv (PI (o.-)) P2 (o.-1)) + ex tep
Thus,
Drv (o)) SDrv (Pi (0:-1) Pa (0)-1) + + to mm
<Drv (Pi (on) Pa (oo) + +l(ex + cp + 6m)
[l + +ew + tm)
Lemma (Rollout return bound). The gap between rollout returns in G1 with (I H2) and G2 with (T2, 122) is bounded as:
mm (m,, 1h) 722 (T22, 112) S 2r max (Y + Ev + em)
+ Ex + 6v)
1-
16

---

Under review as a conference paper at ICLR 2023
Proof.
In (1) 72 (T2, 22)2
(o,a,, [ (o, P/2
( - (o,
0,0,3
2 max
Using Lemma and
Drv (Pf (,) |/3 (o, DDvv (Pi (s (o:)) + maxDry (Tıl(TT) + maxDrv (1b2)
<t <t( t + cm) 6
Thus,
2rmax
(7(ex + cv Em) ter tcp
Im (TI, (T2, 122)
1 1-7
1-7
Lemma 8 (Branched rollout return bound). In k-step branched rollout with h-step length before the branch taking into consideration, denote the gap between branched rollout returns in G1 with
((Tire Pre Ti Post)
(v. vbPost)) and G2 with ((T Pre TT2 Post).
(v: 2//2 (.sttt)
as:
Im
m2
branch ((T , m1 Post). (vrre, (Post)) 1/2
branch
((T (mpre, mT2 Post), (v6 (o))))
which is bounded as:
bm 72 h+ /hthhml) (1 (1-
mam
(ePre + pre + pre) + (erre + Efer)
+ +++ 1 (K: (EPost Ex
Post + cm
Post) + 6x 6x Post)
Post
Proof. According to Lemma 6,
Drv (ot) PP (ox) S Drv (Pi (ot-1) [P (ot-1) 4 + c + Ems
which stays in branched rollout case. We can discuss 82 Drv (Pi (o,a, PS (o, 2)) with different t value: when h.
8, S (c Pre + e + + Pro) + e
Pre + e
when l < <t k.
8, Sh (ePre Pre + Pre + t Pre) + (t - 11) (ePost + 4 e + Poot) + Ex
Post
Post
+ 66
Post
when h+ k.
8, <h Pre
Pre Pre) (et
Post
Post + em e
Post) Post
Post Exp
Exp
cm
Ex
+ (t k) (cPre
Pre + Pre + & + c Pre + E
Pre
=(t- ( (Pree + +
Pre + cm
Pre) + fre + erre + k (cotst 4 c
Post + p
Post)
+ 6
Post
4 Co
Post
17

---

Under review as a conference paper at ICLR 2023
Using the inequalities above, write:
8 Σ ''Drv (Pi 112 (P,,,,,)))
-
Σ
1dt (t ((
Pre + +
pre + tm Pro) + Ex
pre + Ex
1=
h
Σ
y h1 pre
(c + E + +mm e + Post + Post + (t 1 hi) (Ex
Post
+ Epp Post
+ e
++1
Σ b+
Pre + (n((tt (t-k) (eree + +
erre) + Pre + cΦ
Pre
+ k (&+ Post + cv
Post Post) + + + cv
Post
Post
7dt (Ex
Pre + t
Pre + tm Pre) + Ex
Pre + Ex
Pro)
Σ
gtt 1o (cx
Pre + c + mmm
Pre) + Pre + c
Pre
b++
k (c
Post + cv post Post) + + cv
Post
Post
Σ
71((t= (t- 1c) (free + Pre
teere + err)) + Pre + c
Pre
h+++11
+ k (Ep
Post + E
post + post)
+ +,6
Post
+ cv
Post
00 r'1 Pre + to h bi*k
Pro Pre) +
(co pre + Ex
Pre)
cm
Σ 7* (k (cx
Post E
Post + cm
Post)
cx
Post + E
Post)
1=+++1
1 y + ghthit) (cPre + Pre + pre) + -
(cPre
Pre + c
Pre)
(1 71-
g
+ y (Ke (cPost + Post + &ost) + c
Post
Post
+ Eo
Thus,
1m 722 22r m
max ( Ih+
(1 (1-7)) (efre + Pre + efre) + (re + efr)
7
+ h+1 (K (c Post + p + Poet) + c
Post
Post
+ Poot)
A.4 PROOF OF RETURN BOUND
In this section, we provide proofs of return bound in different cases.
18

---

Under review as a conference paper at ICLR 2023
Theorem 2 (Rollout return bound for decentralized model). The gap between return in + 1th policy rollout and return in model rollout with nth learned model 1) moddel (τz, 2))) is bounded as:
nn (T, x 11 model (T, (,, -
2r max
(7Ea + DE 2E EV).
7)
Proof.
In (TT, (1,t)) - modee (2) < 11 (I,, w0+tt) - 1/ (TD: 2b)) + (TD. 1²) - -1 mpmode (T,, w/
(")))))
L1
L2
Apply Lemma
L1 and L2:
2r max
7 7( + Gp)
+ En +
L1 5 7-7
1-7
L2 max
7 (c + c0 + (w))
+ Ex +
1-7
Thus,
In (T. vntt))
17 model
(TT, w-) 5 max (2Ez + tGp + Eu + 766)
(1 (1-
Theorem 3 (Branched rollout return bound for decentralized model). The gap between return in n + 1th policy rollout T7(T,, 1b=+1) and return in k-step branched rollout with h-step experience with nth learned model 1f branch ((T T), (et", v2)) is bounded as:
In (Tr s/tt)) 17 branch ((T), T), (v)" w (2)) S C (EB, Ewy
Proof.
8 17 (T,, 1) branch ((TB: T). (1 x()
17 (T, 1)++1)
17 branch
((T)) TB) (vb" ))
L1
+ 1) branch ((TB, aB) (vt (²)) branch ((TB, x), (v"" (2)
ba
Apply Lemma T, and Tzz:
25 max
L
+
b+1 (cr + tp) + (cr + 6x) + (I (fz + 6x + co) + t +(p)
7
1-9 7
2r max
h+k+1
+1 (K 11) (cr co) by + c0
h
1-
2r max
n+++1
Ta2 <
h 1 1-
(Ew) (Ew) h+1 (k (cr +w
?
2r max
h+++1
e. + (k: 4+3 (Ea (ww) +1
1 -
Thus.
8 < 1- 7
2r max
( [
(1-
(k+ (1+1))111 (cz + Go (w) + t h+1 (co + hez)
19

---

Under review as a conference paper at ICLR 2023
Theorem 4 (Rollout return bound for decentralized model with prediction error) The gap between return in n + 1th policy rollout 77 + and return in model rollout with nth learned model ndmoded (x, 2 is bounded as:
2r max
In (IT, xb++t)=+++=) 17 model (TT, 16 5
(2xx + Ew + 7e8).
(1 7)
C(co,ex.(u)
Proof.
In (T, (T,t) - 17 modee (xt, v.51 < 17 (Ir, sbxtt) - 17 (TB. 2b + 177 (TB. w.) -7 modee (0r, 2/5
Apply Lemma 7 L1 and L2:
2r max
? 7(T + (w)
L1 -
+ ct + Ews
-
2r max
7 (ET + co)
L2 5
+ EG
7
-
Thus,
25 max
1 (Tr, v(tl)) model
(TT, 1/(0)
(2ex + ew + 760)
7)²
Theorem 5 (Branched rollout return bound for decentralized model with prediction error). The gap between return in n + 1th policy rollout Tι(T,, 1)*++) and return in k-step branched rollout with h-step experience with nth learned model 1f pranch ((T T), (vb", vb)) is bounded as:
In (T,, (t,
branch ((TB. T). (+b +))) I C' (EO, Ewr
Proof.
8 |n (T,, 1)xtt)
17 branch ((T)) T) (vb" (2
17 ( 1b++1)
11 branch
(TT =B) ( x
+ 17 branch ((TB, TB) (vi v.)) - n
branch ((TB, a), (.*",)) (2)
1.2
Apply Lemma to L1 and Lz:
L1 2amax
+ 1-
(Ex + Ew) + (Ex + (w) + h+l +++ (k (Ex ew + + + en + Ew)
7
2r 2maax
h
h+h+1
+1+ (k- 1 +1)x (cr + (w) + bo
k h+
1 1-y
2r max
h+++1
L2 <
(Ew) + (EW) 1- (k (Ex) + tx)
7
1 1-7
2r 2maax
h+++1
h
1 1- +1 eu + (I + 1)2 h+l Ex
Thus,
2r max
[
(
8 1 &
+ h -hh+1 1h+1 (cr + eo (w) + 1) n+ (2cz (w) + ko b
20

---

Under review as a conference paper at ICLR 2023
(0,0) (0,1) (1,0) (1,1)
(0,0)
(0,1)
(1,0)
(1,1)
Hlgh
p(a.)/=))
P(so/a..)
Figure 5: Conditional probability matrices: ~oa (left) and P(zo a (right). Deeper color
means higher probability. The order of has been adjusted for clarification.
B VERIFICATION ON LEARNED LATENT VARIABLE
To examine how related the learned latent variable and the inaccessible information are, designed a simple tabular case, where policies, transition matrix, and reward matrix are preset. There are states and agents with actions for each and the space of latent variable we collect experiences (s, ao, s', r) and train a latent variable model end-to-end. For visualization, we design 4 as an explicit network to fetch learned Eg and preset the forward pass for the state in Po to avoid the correspondence being conditioned on the state. Then, we sample 2g from learned latent variable function network th for each experience in the buffer, and then calculate the conditional probabilities, P() and P(a-ol 20). As shown in Figure 5, there is one-to-one correspondence between the latent variable and other agents' joint action. This demonstrates the latent variable model implicitly capture inaccessible information relevant to transition and reward via end-to-end learning.
to be 4. For agent
C EXPERIMENT DETAILS
C.I ENVIRONMENT SETTING
In this section, we introduce the environment settings used in the experiments.
Stochastic Game. In our stochastic game, there are 30 ohservations, 3 agents and actions for each agent, and episode length is limited to 40 steps. We generate transition matrix T and reward matrix Ht in advance as transition function and reward function. Concretely, T is a matrix in shape of [30, 5, 5, 5, 30] and Rt is matrix in shape of [30, 5, 5,5, 1]. At each timestep given observation or and agent joint actions (a), af af). the transition is:
o++1 T(o al a, az], rt Río,, a oz a.).
MPE. In our MPE tasks, agents observe their own positions, velocity, and others' relative positions. And actions of agents control their accelerations every direction which continuous our experiments. In MPE tasks, the episode length is limited to 40 steps.
4-Agent Cooperative Navigation. In 4-agent Cooperative Navigation, as shown in Figure (left), agents learn cooperate reach 4 landmarks respectively. Concretely, denote the radius of agent i as d', position of agent as (x", v() and position of landmark i as (æf, vf), and the reward is:
miny (af xz)² + (ei - ydd)²
Pe
where Pe is a collision penalty:
(V(rd æ2)² + (w1 vá)²
Pe
US
Thus, the reward upper bound each step is 4.
21

---

Under review as a conference paper at ICLR 2023
Coopenaive Naviasiion
Reaulia Pelygon Conarl
X
leaminy agent
Agonii with Eine policy
landraark
Figure 6: Illustration of MPE tasks: Cooperative Navigation (left) and Regular Polygon Control (right).
5-Agent Regular Polygon Control. In S-agent Regular Polygon Control, as shown in Figure (right), agents learn cooperate with another agent, which is controlled by a fixed policy. aiming to form regular pentagon. The fixed policy is that the acceleration of the agent is always the direction of the relative position between the center of the other 4 agents and itself. And the reward is given according to the area S of current pentagon scaled by its perimeter C,, which formally is:
10 10 S.
(
agents form a convex pentagon
Sscaled
otherwise,
and represents the area of its similar pentagon with perimeter of 10. So when the pentagon is regular pentagon, Secaled comes to maximum, cot E Additionally, two penalty items are given. Bound penalty, Ps. is used to restrict agents to stay in bounds
0,
|x|| < 0.9
bound(x)
10 * (|xl - 0.9),
| < 1.0
2/11-2
otherwise
3
Σ [bound(x") + bound(y')]
Pa =
1=0
where (x', y²) is the position of agent i. Collision penalty, Pe, is as same as that in 4-agent Cooperative Navigation. Finally, design the reward as:
r r= min max Sscated: cot - Sseated
1000
-App Phs
where max operator helps to distinguish when pentagon is relatively large and min operator handles the situation being divided by zero. This task more difficult than Cooperative Navigation.
Multi-Agen MuJoCo. In our multi-agent MuJoCo experiments, the state in MuJoCo environment, which describes the position, velocity, angular velocity of each joint, etc, is used the observation distributed to each agent. Specifically, Ant task, we only use dimensions from to the state. We limit the episode length of Halfcheetah to 250 steps, and 500 steps for Ant and Hopper. We provide the joint allocation each task Table
C.2 IMPLEMENTATION & HYPERPARAMETERS
In this section, we provide details for implementation and hyperparameters.
For the experiment environment, we adopt MPE (MIT license) and MuloCo Gym (MIT license). For PPO, we follow the version in OpenAl's Spinning Up (MIT license).
22

---

Under review as a conference paper at ICLR 2023
Table I:
Joint allocation in multi-agent MuJoCo tasks. The relation column indicates how agents control the joints of robotics.
Task
MuloCo action
Multi-agent MuloCo actions
Relation
Hopper 3x 1
(do;al,a)) ai, az)
[(a)): (aa) (a5)
0, ag
Ant 4 xx2
(0o, ,07)
00: 00.00 0i), ) ... (an. ai)
02i Oj
HalfCheetah 3x2
(ao, :.s) ass)
(ag. 44) ) ... (uō. af)
02i+ dj
HalfCheetah 6x
(00.".. .0s)
(0o), (aa)
0, 0g
HalfCheetah 5:11,1,1,1,2]
(ao, '.. (s)
[aa), ... , (ax). (ao. at)
0;44 = dg
HalfCheetah 5x2
(ap, ass)
[(a af)..... (ad. a])]
0g. 0g 2, m]
Table 2: Structure of the neural networks we used in experiments.
Stochastic Game
MPE
Multi-agent MuJoCo
Network
hidden
activation tanh
hidden
activation
hidden
activation
PPO Actor
(128, 128)
(128,128)
tanh
(128,125
tanh
PPO Critic
(128, 128)
tanh
(128,128)
tanh
(128,128)
tanh
(64,64)
ReLU
(64.64)
ReLU
(32,32)
ReLU
Ic
(128)
ReLU
(64.64)
ReLU
(256)
ReLU
Po
(128,64)
ReLU
(64)
ReLU
(128,64)
ReLU
Rs
(128,64)
ReLU
(64,64)
ReLU
(128,64)
ReLU
Table 3: Hyperparameters
Stochastic Game
MPE
Multi-agent MuloCo
latent variable dimension
3
4
0.94
0.98
C
0.2
Cyalue
10
Centropy
3e-4
Cpalue
0.5
max gradient norm
0.6
PPO batch size
32
actor learning rate
3e-4
critic learning rate
1e-3
k
h
8
6
I
8
Cor
10
1,100
5
latent variable model batch size
64
prediction batch size
32
128
J learning rate
lc-5
Js learning rate
le-4
3e-5
5e-5
Pg learning rate
le-4
3e-5
Rg learning rate
lc-4
3e-5
All neural networks used in our implementation are in the form of Multi-Layer Perception (MLP). Particularly, the transition function and reward function are respectively learned using an ensemble formed by 3 individual versions of the last layer. The likelihood of next observation is multiplied by a coefficient car to balance the scale of the elements (3) and (4). The hidden size and activation function used in the networks are provided in Table 2. And the parameters used in training are provided in Table 3.
Cor in 4-Agent Cooperative Navigation task and cor 100 in 5-Agent Regular Polygon Control
23

---

Under review as a conference paper at ICLR 2023
observation virtaaiion Freaumyy
predat ebservaiion Crasiitrary
140
137
pro rowennl
0.30
IPPO
MIPO wel prodiction
0.00
500
100
2000
Figure 7: Learning curves of MDPO compared with MDPO w/o prediction and IPPO on the single-agent stationni stochastic game: average return (left), observation visitation frequency divergence (mid), and model prediction errors (right). Each round is 1600 environment steps.
In the implementation of latent variable function, both deterministic and stochastic latent variable satisfy our analysis, and we choose between them according to environment properties and experi- mental performance. In MPE and stochastic game except for Appendix B, use deterministic latent variable with L2 regularization. In stochastic game in Appendix B, we use Category distribution. Gaussian distribution. As for the implementation
And in multi-agent MuJoCo environment, of transition function and reward function, we use Category distribution for transition function in stochastic game and deterministic output for others.
The experiments are carried out on Intel 19-10900K CPU and NVIDIA GTX 3080Ti GPU. The training of stochastic game task costs hours, while takes 14 hours for each MPE task, and 25 hours for each multi-agent MuJoCo task.
D ADDITIONAL RESULTS
Since MDPO helps to handle non-stationarity in decentralized MARL from the perspective of an individual agent, will be natural and easy to also apply MDPO to single-agent RL in non-stationary environments. In this section, we investigate how MDPO performs in such a non-stationary single- agent environment.
We adopt the cooperative stochastic game into a single-agent non-stationary version. Concretely, we fix the policies of two agents and leave only one agent to update its policy. And we generate 5 noise N4) randomly, which are in the same shape as the transition matrix (T) and will influence the transition probability in a rotating manner. Formally, in nth policy rollout, the transition matrix is T + N, mod 5, and we guarantee such transition matrix is reasonable when generating noise matrices.
matrices (No.
We compare the performance of MDPO, MDPO w/o prediction, and IPPO on the single-agent non- stationary stochastic game, and the learning curves are shown in Figure 7. As illustrated in Figure (left), MDPO still performs the best in the single-agent non-stationary environment. As shown in Figure (right), latent variable prediction helps to predict the non-stationary transition, and as no noise is applied the reward matrix, there is merely slight difference in reward prediction. Since the latent variable in this environment (noise matrices) is in a regular rotation, the prediction function easier learn than in decentralized MARL settings. However, unlike in decentralized MARL, non-stationarity in this setting will not fade away in pace with policy convergence, thus MDPO w/o prediction may keep oscillating and generate experiences with larger observation visitation frequency divergence than MDPO, which is shown in Figure (mid).
Generally, MDPO also works in single-agent non- sta-sttatooayy environments, especially when there is regular pattern of non-stationarity, More thorough studies are left as future work.
E ADDITIONAL RELATED WORK
By utilizing an environment model, model--ased RL has shown many advantages, such as sample efficiency (Wang et al., 2019) and exploration (Pathak et al., 2017). Many paradigms have been
24

---

Under review as a conference paper at ICLR 2023
Pass and Shoot
vs with Keeper
avecago rale
0,5-
0.3
0.2
0.1
0.0
800
400
500
round
round
Figure 8: Learning curves of MDPO, MDPO w/o prediction, and ITRPO in Run and Pass (left) and VS I with Keeper (right). Each round is 3200 environment steps.
proposed on how exploit the environment model. Model-based planning methods, such model predictive control, select actions through model rollouts. Dyna-style methods (Sutton, 1990; Feinberg al., 2018; Janner et al., 2019) by the learned model to update the policy. Recent studies have extended model-based methods to multi-agent settings for sample efficiency zero-sum game (Zhang et al., 2020), in stochastic game (Zhang et al., 2021) and in networked system (Du et al.. 2022), centralized training (Willemsen et al., 2021), opponent modeling (Yu al., 2021b), and communication (Kim al., 2021). However, none of them strictly tackle fully decentralized setting of our paper.
both data collected in the real environment and data generated
Specially, the DMPO (decentralized model-based policy optimization) algorithm in prior work (Du et al., 2022) is designed for a networked system, where agents are able to communicate along the edges with their neighbors. The naming of DMPO and MDPO may lead to misunderstanding of similar settings, but the two algorithms are actually concerned with different settings. In fully decentralized setting of our paper, no information sharing is allowed between agents. And when the number of neighbors is set zero in DMPO, it will degenerate into the version of MDPO w/o prediction in our algorithm.
F ADDITIONAL EXPERIMENTS
In order to enrich our baseline and test MDPO on more complex environment, we supplement experiments Google Research Football environment (GRF) with ITRPO the baseline. ITRPO the decentralized version of TRPO (Schulman et al., 2015), and we use TRPO to optimize the policy MDPO and MDPO w/o prediction. Specifically, We choose 'simplel 15v2' 15v22 as observation representation which encodes the state with 115 floats and 'scorin++checkpoin" as reward which encodes the domain knowledge that scoring aided by advancing across the pitch. We examine MDPO, MDPO w/o prediction and ITRPO in two scenarios, Run and Pass and vs with Keeper, in both of which MDPO improves the average goal rate. The learning curves shown in Figure
In our implementation, we use Category distribution for one-hot dimensions in observation and Gaussion distribution for others model the transition. We use MLP for reward function and Gaussion distribution for latent variable function. The structures are listed in Table 4 and the hyperparameters are listed Table
25

---

Under review as a conference paper at ICLR 2023
Table Structure of the neural networks we used GRF experiments.
Google Research Football
Network
hidden
activation tanh
TRPO Actor
(128,128) 128)
TRPO Critic
(128,128)
tanh
(128, 128)
ReLUU
(256)
ReLU
Po
(128,128)
ReLU
R5b
(128,128)
ReLU
Table 5: Hyperparameters in GRF experiments
TRPO hyperparameters
MDPO hyperparameters
1.0
latent variable dimension
6
0.99
k
4
KL limitation
0.06
h
16
damping coefficient
0.2
8
conjugate gradient iteration
5
Cot
backtrack iteration
5
latent variable model batch size
128
backtrack coefficient
0.8
prediction batch size
256
max gradient norm
10
learning rate
le-5
TRPO batch size
32
/c learning rate
5e-5
Coalue
10
Po learning rate
3e-5
critic learning rate
5e-5
Rg learning rate
le-5
26


Reviews:
[{"review_id": "kI20s2iXtxx", "reviewer": "Reviewer_DZj6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main contribution of this paper is to propose MDPO, a model-based decentralized policy optimization (MDPO), which incorporates a latent variable function to help construct the transition and reward function from an individual perspective.  The theoretical monotonic improvement guarantee is given and experimental results show the superiority of the proposed method.\n", "summary_of_the_review": "The problem considered in this paper is clearly motivated. I find the methodology lacks motivation and the relation between the theoretical results and algorithm design (sec 3.5) is not clearly addressed. \n\nSome detailed comments are listed below.\n\n– Can authors explain why the first two inequalities in sec 3 hold?\n\n– In sec 3.1, the authors propose to maximize likelihood of the dynamic and reward prediction model. How is $P_\\theta\\left(o^{\\prime} \\mid o, a, z\\right)$ implemented? Do you use MLP or gaussian distribution? Similarly, how do you implement $\\left(R_\\phi\\left(o, a, o^{\\prime}, z\\right)\\right.$?\n\n– There is another work on decentralized model-based policy optimization (DMPO). The setting of DMPO can be equal to this work by setting the number of neighbors as zero. \nCan the authors include this method as a baseline? \t\t\n\t\t\t\t\t\nYali Du, Chengdong Ma, Yuchen Liu, Runji Lin, Hao Dong, Jun Wang, and Yaodong Yang. Fully decentralized model-based policy optimization for networked systems. arXiv preprint arXiv:2207.06559, 2022. \n\n– Line 9 of algorithm 1, how is $\\psi_{w_{j+1}}$ updated?\n\t\t\n– The authors mentioned that MDPO is a decentralization method, but in the process of learning hidden variables, it is actually necessary to obtain the information of other agents. How could the algorithm be decentralized in this sense?\n\n", "strength_and_weaknesses": "Strength：\n– The paper is clearly motivated and well structured.\n– Theories and proofs are provided to support the algorithm.\n\nWeakness: \n– The authors made good theory contributions. But how the theory would guide the design of the algorithm and how the experiments show this relation are not clear to me. \t\t\n– the experiments can be enhanced by comparing to recent model-based RL methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SQ2m5diPvd", "reviewer": "Reviewer_p6F5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The study proposes a model-based decentralized policy optimization for MARL to overcome the non-stationary issues arising from the concurrent policy learning of multiple agents. Specifically, the current study proposes modeling individual agents’ state transition model and reward function while taking into account other agents’ policies into a latent variable. The modeled individual dynamic and reward functions along with the latent factor encoder are used to train an individual decision-making policy. Furthermore, to take into account the nonstationary aspect of other agents’ policies, the current study uses a latent variable prediction function. ", "summary_of_the_review": "The mathematical analysis is rigorous and detailed, but the description of the structure of the model used and the learning method is relatively week.", "strength_and_weaknesses": "1. This study explains in detail why decentralized MARL learning is difficult and explains in detail and theoretically the tasks that must be preceded in order to overcome the difficulties.\n\n2. The performance bound of decentralized MARL learning was analyzed very strictly by correlating the performance of the prediction model.\n\n3. The mathematical analysis is rigorous and detailed, but the description of the structure of the model used and the learning method is relatively week. In particular, reasons and justifications for using a specific model structure and choosing a learning method are not sufficiently provided.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ZQQImJ81Cm", "reviewer": "Reviewer_WQD2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new approach (MDPO) to model-based decentralized reinforcement learning by leveraging a latent variable function $\\psi(z|o)$ to help learn the observation transition $P_i(o_i'|o_i,a_i,z_i)$ and reward $R_i(o_i,a_i,z_i)$ models. \n\nThe main problem this paper tries to address is that the assumptions of TRPO may not hold in a decentralized MARL because multiple agents update their policies simultaneously, making a significant difference between the new and old joint policies. MDPO uses the environment model to bridge the gap between $\\rho^{\\pi^\\text{new}}$ and $\\rho^{\\pi^\\text{old}}$.\n\nThis paper also provides the proposed algorithm's theoretical performance bounds (Theorem 2, Theorem 3), which theoretically contribute to the monotonic policy improvement.", "summary_of_the_review": "This paper introduces a latent variable to help learn the observation transition $P_i(o_i'|o_i,a_i,z_i)$ and reward $R_i(o_i,a_i,z_i)$ models. This paper proposes a new decentralized MARL algorithm, MDRL, to solve the nonstationary problem in a model-based way. However, I am negative about this paper based on the weak points above.\n", "strength_and_weaknesses": "Strength:\n\n1. MDPO learns a latent variable to help distinguish different transitions resulting from varying unobservable information of the full state and other agents' policies. This novel approach makes the observation transition $P_i(o_i'|o_i,a_i,z_i)$ and reward $R_i(o_i,a_i,z_i)$ models more stable during training.\n2. This paper also provides theoretical performance bounds (Theorem 2, Theorem 3) for the proposed algorithm, which theoretically contributes to the monotonic policy improvement.\n\nWeaknesses:\n\n1. It needs to be clarified how MDPO solves the main problem of decentralized MARL. It seems that the main reason that makes $\\|\\rho^{\\pi^\\text{new}}-\\rho^{\\pi^\\text{old}}\\|>\\|\\rho^{\\pi^\\text{model}}-\\rho^{\\pi^\\text{old}}\\|$ hold is that the $\\psi^n_w=(1-\\alpha)\\psi^{n-1}_w+\\alpha\\psi^n$ in Theorem 1, where $\\psi^n$ is the true latent variable function. MDBP does not update $\\psi$ directly to the true value, but uses a $\\alpha$-tradeoff between the true value and the old value. Then does it mean that MDBP solves this problem by slightly updating each agent's policy like TRPO? As this paper says, such a slight update will lead to much slower convergence, especially in fully decentralized MARL. This is a key point about whether MDPO truly solves the nonstationary problem caused by the simultaneous update of policies (which also makes the assumption in TRPO not hold).\n2. The experimental part of this paper needs to be more convincing. The baseline algorithm is only IPPO, and complex multi-agent environments such as The StarCraft Multi-Agent Challenge or Google Research Football are missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject