$f$-Divergence Policy Optimization in Fully Decentralized Cooperative MARL
Authors: Kefan Su
Zongqing Lu
Venue: Accepted by TMLR ()

Abstract: Independent learning is a straightforward solution for fully decentralized learning in cooperative multi-agent reinforcement learning (MARL). The study of independent learning has a history of decades, and the representatives, such as independent Q-learning and independent PPO, can achieve good performances on several benchmarks. However, most independent learning algorithms lack convergence guarantees or theoretical support. In this paper, we propose a general formulation of independent policy optimization, $f$-divergence policy optimization. We hope that a more general policy optimization formulation will provide deeper insights into fully decentralized learning. We demonstrate the generality of this formulation and analyze its limitations. Based on this formulation, we further propose a novel independent learning algorithm, TVPO, which theoretically guarantees convergence. Empirically, we demonstrate that TVPO outperforms state-of-the-art fully decentralized learning methods on three popular cooperative MARL benchmarks, thereby verifying the efficacy of TVPO.

Published in Transactions on Machine Learning Research (03/2025)
f --Divergence Policy Optimization in Fully Decentralized Co- operative MARL
Kefan Su School of Computer Science Peking University
sukefon@pkuu.du..om
Zongqing Lu
congring.uuüpkur edu.cr
School of Computer Science Peking University
Reviewed on OpenReview: https: // openreview. net/ per/ffrmm? id- Wj8yFjIpon
Abstract
Independent learning is straightforward solution for fully decentralized learning in co- operative multi-agent reinforcement learning (MARL). The study of independent learning has a history of decades, and the representatives, such as independent Q-learning and independent PPO, can achieve good performances on several benchmarks. However, most independent. learning algorithms lack convergence guarantees or theoretical support. Tn this paper, we propose a general formulation of independent policy optimization, f-divergence policy optimization. We hope that provide deeper insights into fully decentralized learning. We demonstrate the generality of this formulation and analyze its limitations. Based on this formulation, we further propose novel independent learning algorithm, TVPO, which theoretically guarantees convergence. Empirically, we demonstrate that TVPO outperforms state-of-the- fully decentralized learning methods on three popular cooperative MARL, benchmarks, thereby verifying the efficacy of TVPO.
general policy optimization formulation will
1 Introduction
Cooperative multi-agent reinforcement learning (MARL) has shown great potential in many areas, including power control (Zhang & Liang, 2020), autonomous vehicles (Han et al., 2022), and robot control (Sartoretti et al., 2019). The main framework for cooperative MARL is centralized training with decentralized execution (CTDE) (Kraemer & Banerjee, 2016), while the MARL community pays less attention to fully decentralized learning, also known decentralized training with decentralized exccution (DTDE). Fully decentralized learning remains significant. in cooperative MARL due to its simplicily. From applications perspective, fully decentralized learning is useful in various industrial applications where agents may belong to different. parties, such as autonomous vehicles or robots. From a theoretical perspective, fully decentralized algorithms rely on less information during training and are therefore more general and worthy of further study.
For DTDE or fully docentralized settings, independent learning a straighttfrward yet effective approach that enables agents directly execute the same single-agent RL algorithm. Representative methods include independent Q-learning (IQL) (Tan, 1993) and independent actor-critic (IAC) (Foerster al., 2018; Papoudakis et al., 2021). Recently, independent PPO (IPPO) (de Witt et al., 2020), which extends PPO (Schulman et al., 2017) to MARL, has shown good performance on several benchmarks. However, these independent learning algorithms are still troubled by the non-slationarity problem and lack convergence guarantees or theoretical support.
* *Corresponding Author

---

Published in Transactions on Machine Learning Research (03/2025)
Additionally, introducing constraints of the policy update or trust region into the policy optimization is a classic and effective paradigm for ensuring policy convergence guarantees in single-agent RT. (Schulman et al., 2015; 2017; Nachum et al., 2017). Recently, DPO (Su & Lu, 2022b) has proposed a fully decentralized algorithm with a convergence guarantce by following this approach. However, DPO focuses on the KL- divergence and can be troubled by trivial policy updates in some cases due to its approximation bias in the optimization objective. Therefore, we hope that a more general policy optimization formulation can provide deeper insights into fully decentralized learning.
In this paper, we propose general formulation of independent policy optimization, f-divergence policy optimization. We demonstrate the generality of this formulation for independent learning in cooperative MARL. We also analyze the policy iteration of this formulation and discuss its limitations using a two-player matrix game. Based on this formulation, we further propose a novel independent learning algorithm, total variation policy optimization (TVPO). To thcoretically study the properties of TVPO and prove its convergence, we introduce new set of value functions and policy iteration specifically designed for fully decentralized learning and prove the monolonicity of this policy iteration. The practical implementation of TVPO can be effectively realized through an adaptive coefficient, similar to PPO (Schulman et al., 2017).
Empirically, we verify our discussion regarding the limitations of f-divergence policy optimization in the two-player matrix game and demonstrate the joint policy may converge to a sub-optimal solution with different f-divergences. Moreover, we evaluate the performance of TVPO across three popular benchmarks of cooperative MARL including SMAC (Samvelyan et al., 2019), multi-agent MuJoCo (Peng et al., 2021) and SMACv2 (Ellis et al., 2023). We compare TVPO with four representative fully decentralized learning methods: IQL (Tan, 1993), IPPO (de Witt et al., 2020), T2Q (Jiang & Lu, 2022), and DPO (Su & Lu, 2022b). The empirical results show that TVPO outperforms these baselines in all evaluated tasks, thereby verifying the effectiveness of TVPO in fully decentralized cooperative MARL.
2 Related Work
CTDE. The popular framework to address cooperative multi-agent reinforcement learning (MARL) problems is centralized training with decentralized execution (CTDE) (Lowe et al., 2017; Foerster et al., 2018; Sunehag et al., 2018; Rashid et al., 2018; Iqbal & Sha, 2019; Wang et al., 2021a; Zhang et al., 2021; Su & Lu, 2022a; Wang al,, 2023a). CTDE successfully mitigates the challenge of non-stationarity through centralized training. This line of rescarch can be categorized into two types: value decomposition algorithms (Sunchag et al., 2018; Rashid et al., 2018; Son et al., 2019; Yang et al., 2020; Wang et al., 2021a), where the optimum of the centralized Q-function aligns with the optima of the decentralized Q-functions, allowing the learning of the centralized Q-function to be factorized into the learning process of the decentralized Q-functions and multi-agent actor-critic algorithms (Focrster et al., 2018; Iqbal & Sha, 2019; Wang et al., 2021b; Zhang et al., 2021; Su& & Lu, 2022a; Wang et al., 2023a; Wen et al., 2022; Liu el al., 2023), which leverage a centralized Q-function to facilitate the learning of decentralized stochastic policies. HAPPO (Kuba et al., 2021) and MAPPO (Yu et al., 2021) extend the applicability of TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017), respectively, to the MARL setting through a centralized state value function. HASAC (Liu et al., 2023) combines the heterogeneous-agent decomposition with the entropy regularization in SAC. MAT (Wen et al., 2022) introduces Transformer and sequential modeling into the heterogeneous-agent decomposition. Nevertheless, it is important to note that these approaches remain constrained by the CTDE paradigm and are therefore unsuitable for fully decentralized learning.
Fully Decentralized Learning. There have recently been several different views on fully decentralized learning decentralized learning. Some works study decentralized learning specifically with communication (Zhang et al., 2018b; Li et al., 2020) parameter sharing (Terry et al., 2020). Both communication and parameter sharing involve exchanging information among agents (Terry et al., 2020). However, in this paper, we consider fully decentralized learning in the strictest sense - with each agent independently learning its policy without being allowed to communicate or share parameters as in Tampuu et al. (2015); Mao et al. (2022b); Wang et al. (2023c). Additionally, there are several studics (Zhan et al., 2023: Wang et al., 2023b; Mao & Başar, 2023) considering general-sum games in decentralized MARL, these studies focus on the episodic Markov game(Jin et. al., 2021), which is non-cooperative and
2

---

Published in Transactions on Machine Learning Research (03/2025)
assumes the reward function, transition probability, and policy are related to the time step. The objective of finding equilibrium in this setting differs from the fully recentralized learning concerned in this paper. Independent learning (OroojlooyJaddid & Hajinezhad, 2019) has been extensively studied in the field of cooperative multi-agent reinforcement learning (MARL) as a straightforward approach to fully decentralized learning. Represenlatives of this approach include independent Q-learning (TQL) (Tan, 1993; Tampuu el al., 2015), independent actor-critic (IAC) (Foerster al., 2018; Papoudakis et al., 2021), and independent proximal policy optimization (IPPO) (de Witt al., 2020). It should be noted that all these independent learning algorithms deviate from the stationary condition of the Markov decision process (MDP) and lack convergence guarantees, even though IQL, and IPPO perform well in various benchmarks (Papoudakis et al., 2021). Recent studies have emerged with convergence guarantees in fully decentralized MARL, namely 12Q (Jiang & Lu, 2022) and DPO (Su & Lu, 2022b). I2Q introduces the concept of QSS-value (Edwards et al., 2020) into independent Q-learning, achieving convergence guarantees. However, its applicability is restricted to deterministic environments. On the other hand, a novel docentralizod surrogate of the joint TRPO objective is proposed by DPO to ensure convergence. In terms of empérical performance, I2Q demonstrates superior performance compared to IQL, while DPO outperforms IPPO. Therefore, in our empirical studies, we comprehensively compare our TVPO with these two caaeeoof-theon methods.
Mirror Descent in RL. Recently, mirror descent (Blair, 1985) and similar ideas have been applied in single-agent RL (Wang et al., 2019; Lan, 2023; Tomar et al., 2020; Yang et al., 2022; Vaswani et al., 2021) as well as in CTDE algorithms in MARL (Su Lu, 2022a; Kuba et al., 2022; Liu et al., 2023) for theoretical guarantees. Mirror descent is method associated with the Bregman divergence (Bregman, 1967). Although Bregman divergence is general divergence class, KI-divergence has been most frequently used in previous mirror descent studies (Wang et al., 2019; Lan, 2023; Tomar et al., 2020; Yang et al., 2022; Vaswani et al., 2021). On the other hand, KL-divergence lies at the intersection of Bregman divergence and f-divergence and our analysis of KI-divergence indicates that it can become trapped in the sub-optimum even in a simple matrix game. Furthermore, Bregman divergence in mirror descent and f-divergence represent two distinct classes of divergences, and the latter can provide us with more useful properties for theoretical guarantees in fully decentralized learning. Extending mirror descent in fully decentralized learning with theoretical guarantecs remains open challenge and is beyond the scope of this paper.
3 Preliminaries
Cooperative Markov Game The cooperative Markov Game serves general model for cooperative multi-agent reinforcement learning (MARL) (Chen et al., 2022; Zhang et al., 2018a; Matignon et al., 2012). It is a special case of the Markov Game (Littman, 1994) where the reward functions of all agents are identical. It is represented by the tuple g = {S, A, P, N,I,N,T, 7} where N denotes the number of agents, and I {1, 2, ......... .N3 refers to the set of all agents. The state space is denoted as S, and the joint action space is denoted as A = A₁ A2 x. ...................... x AN, where a' represents the individual action space for agent i. The transition function P(s'|s, a): 5 x A S > [0, 1] defines the probability of transitioning from state s to s' given a joint action The discount factor is denoted y E [0, 1), and the reward function r(s, a) S A + [-rmax, Tmax] assigns rewards to state and joint action a, with Tmax serving as the upper bound of the reward function. The objective of cooperative Markov Game is to maximize J(T) = Ex 1 "rss,,, a.)]- Thus, the optimal joint policy π' = arg max, J(T) needs to be determined. In fully decentralized learning, each agent independently learns an individual policy denoted as (a))s)). The joint policy π of all agents can be represented as the product of each individual policy n².
Additionally, the V-function and Q-function of the joint policy can be defined as follows:
(T)() = Eanz (s. Qa (5T a) - r(s a) + ~( [s'(s')
(1)
(2)
Fully Decentralized Critic. The concept of the critic in fully decentralized learning has been explored in previous studies (Peshkin et al., 2000; Lyu & Xiao, 2021; Su & Lu, 2022b). To facilitate further discussion, we provide some formulations and deductions regarding the fully decentralized critic.
3

---

Published in Transactions on Machine Learning Research (03/2025)
In fully decentralized learning, each agent learns independently through its own interactions with the environment. Consequently, the Q-function for each agent can be described by the following formula:
(s- a²) = =r (s, a²) + ylE iar "(1r) (s',
(3)
where rx rx= a') = E.g Ba-ii on (ollrsss a', a ">),,, and - and a respectively denote the joint policy and joint action of all agents expect agent i. It can be shown that (=, Eg i~r 161=) 1Q (s, a' a=") For simplicity, in the following, we use QT to denote 07 given joint policy æ, if there is no confusion.
Independent Learning. Independent learning straightforward method to solve cooperative MARL problems, which makes each agent learn through the same single-agent RI. algorithm, such as TQL (Tan, 1993), IAC (Foerster et al., 2018), and IPPO (de Witt al., 2020). Though independent learning faces the non-stationarity problem, it still has the advantage of absorbing the benefit of single-agent RL. The policy iteration nnww arg max Σ. (raaas)(=all improves monotonically over Told and guarantees the convergence. We draw inspiration from policy iteration in single-agent RL, introduce a general formulation of independent policy optimization, and try to find an independent learning algorithm that can guarantee convergence in cooperative MARL.
is fundamental in single-agent RI, which ensures that Nnew
4 A General Formulation for Independent Policy Optimization
Given the condition of fully decentralized learning in coop- erative MARL, we first propose a general formulation of independent policy optimization, f-divergence policy opti- mization, and discuss its generality and limitation. Then, based this formulation, propose total variation policy optimization (TVPO), prove the convergence of TVPO in fully decentralized learning, and provide a practical algo- rithm.
U
Bob
qi
qt
Alice
Pt
b
- Pt
c d
Table The two-player matrix game for Alice and Bob with policies after the nunber of policy iterations. Alice will take action ua with probability pt and take action n' with probability 1 - Pe; Bob will take action ug with probability and take action wh with probability 1 1-q--
Before diving inlo the discussion, we need to introduce a simple two-player matrix game for later use. In this matrix game, the two agents, Alice and Bob, each have two actions, denoted as {uh, w for Alice and {ul, u for Bob. Each episode of this matrix game consists of only one step. The rewards for the joint actions (uΩ, u2), (u ub), (u), ub)) and (w',, ub)) are a, b, c, and respectively. The policics Alice and Bob can be described with pt and qe as that Alice will lake action un with probability pt and Bob will take action up with probability dt, where
represents the number of policy iterations. The full information of this matrix game is illustrated in Table
4.1 f Divergence Policy Optimization
The f-divergence policy optimization is formulated as follows: Definition 4.1. Given any fixed and aid
Thw arg max
Σ (a (s,a²) wD, (1= TT
(4)
where (p(l) & Σ, q. P1 f-divergence, f: [0, x0) + (-00, is convex and f(1) = O.
is f-divergence (Ali & Silvey, 1966) and according to the definition of
9:
This formulation contains an additional term Dy (-1s)) which describes the distance between n and T
There are several studies considering the distance between Vold and nnww The trust region in TRPO (Schulman et al., 2015) and PPO (Schulman et al., 2017) is actually KT-divergence between oald and nnew;
4

---

Published in Transactions on Machine Learning Research (03/2025)
while Nachum et al. (2017) extend entropy regularization to more general formulation with KL-divergence. Unlike these studies that just use KT-divergence as the distance measure, we would like to discuss a more general formulation. So we use f-divergence, which is widely used for describing the distance between two distributions. Also, KL-divergence is a special case of f-divergence with f(x) = x log z and we have many other choices for f-divergence, such as f(r) 2 corresponding to total variation distance Dy(pllg) Σ, lp--9.
[x=1|
and f() (1 / corresponding to Hellinger distance (p(p v El(vii vq.)².
To further discuss f-divergence policy optimization, we need to find the solution to the optimization objective (4) and we have the following lemma.
Lemma 4.2. Given a fired function and the corresponding f-dieergence Dj, let g(x) (z), then the solution to Equation (4) is
nnw (a'ls) max max(Toiala"Is))) (A. + Of (x+00 )))))) .0),
(5)
w
where X satisfies
Σ max (aiaaalaaaa ( A + Q.Tchl (s,a") ,0} =1 1.
This proof is included in Appendix A.1 and follows Yang et al. (2019).
We use the two-player matrix game between Alice and Bob (i.e., Table 1) to discuss the limitation of f-divergence policy optimization. As for the policy iteration in the matrix game, we have the following proposition.
Proposition 4.3. Suppose g(x) 2 0 and lel M bb d, p Mo and a = M If the payoff matris of the two-player matriz game satisfies M > O, and Alice and Bob update their policies with
Σ
n' (d'ls) (s, a²) - wD, ( (+s)
T+1 = arg max
(6)
a¹
then we have P1) 2 D + a+1 S 9: (2) S p + 9+1 2 q:: (3) 9 2 q = Pi+1 S Pti (4) P+1 2 Pt-
q0 S 9
The proof is included in Appendix A.2. With Proposition 4.3, we can build a case where the joint policy sequence can only converge to the sub-optimum. We assume the matrix game satisfics the condition > e > max d), then the optimal joint policy is (pt, g:) = (1,0) corresponding to the joint action (u), u/)) and reward b. Moreover, the condition b > c> maxfa, d3 also means p E (0, 1) and q (0, If at iteration know that
t, the condition 9: > 4, Pt < f is satisfied, then 90+1 > 9: > 4, Pt+1 Pt p. By induction, Vt' 2 t, qu++1 > qe q. +1 < Pr < p. As the sequence {pc} and {qi} are both bounded in the interval [O, 1], we know the sequence {pr} and {2:} will converge to p² and g², As for p" and q", we have the following corollary.
Corollary 4.4. If at iteration t, the condition qi > G, Ps P is satisfied, then the sequence {p.} and {aih will converge to p² = 0 and d" = respectively.
The proof is included in Appendix A.3. Corollary 4.4 tells us if once q > q. Pt < P, then the joint policy converges to the sub-optimal solution (p".g") = (0, 1) corresponding to the joint action (u), ug) and reward c. So if the initial policy Po and qu satisfies the condition go > q, Po < p. then the joint policy converges to the sub-optimal policy. We further illustrate this in the experiment.
4.2 Total Variation Policy Optimization
The f-divergence formulation (4) can be trapped in the sub-optimal joint policy even in a simple two- player matrix game. This shows the upper bound of f-divergence policy optimization, so we should not expect such a policy iteration could obtain the optimal joint policy fully decentralized learning in all MDPs. Fortunately, we have found an algorithm that accords with the f-divergence formulation and has
5

---

Published in Transactions on Machine Learning Research (03/2025)
the convergence guarantee. This algorithm uses the total variation distance for f-divergence, call it total variation policy optimization (TVPO). The convergence guarantee of TVPO shows the potential of the f-divergence formulation.
Before we introduce TVPO and prove its convergence, we need some definitions and lemmas. We use (pll)) = 2 E, Ip - qil to represent the total variation distance. We define a new V-function (s) and a new Q-function 07 (s, a', i) given joint polices π and p as follows:
Definition 4.5.
1 1 ><<((()ss 1 (a (1077 WD, wDy (F((1)))))()))))
(e)=
(a'ls)
(7)
(p ;a r
r(s, a + +E (g)]
(8)
As the definition (7) is a fixed-point equation, we need to prove that this definition is well-defined. So we define an operator 1 as follows:
1 N (s)
ΣΣ
<aaass)
wD, (F)
(9)
a²
Then for any value function V1 and V2, we have
- V2) - = - ΣΣ x (a'ls) (as [ (V2(s))))
a
(s) ll
So the operator rø is y-contraction, which means v is the unique fixod-point of (7) and the definition
(7) is well-defined.
To apply total variation distance to independent policy optimization, we have the following lemma.
Lemma 4.6. Suppose nnew, Told, and π are three joint policies. Let 20max then for any state s, we have
Σ Tnew Tnew(alss)1 (als)
1
T (a'1s)
N
i=1
(N
1)L
Σ DrV (1))
(10)
N
The proof is included Appendix A.4. Lemma 4.6 critical bridge between normal value function V and our new value function V and we can witness its effect in our later discussion. Moreover, we also know that V = V and Q# Q
We can also realize the monotonic improvement with fully decentralized optimization objective via the following proposition.
Proposition 4.7. Given a fized joint policy p and an old joint policy wold, if all the agents update their policies according to
mt new arg max Σ (a'ls) Ep Ea (T) Told (s. as (T((1))l)
(11)
then we have VTtu(( (s) < VT (s), ( a) s 2 a)) YsE S, a A.
The proof included in Appendix A.5. According (11), by taking Told p n. and Tnew Te+,, we can design policy iteration follows:
A+1 arg max
Σ (a(1ss) Σ गi m (s)) (s.,, WD, (1(()))
(12)
a²

---

Published in Transactions on Machine Learning Research (03/2025)
This policy iteration resolves the f-divergence formulation (4). According to Proposition 4.7, know the joint, policy sequence { has the property V (8) 2 V (s) = (s). By taking Dj Drv and
w (N-1)L we can combine these results with Lemma 4.6 to obtain the convergence guarantee.
N
Theorem 4.8. Let w = N
(N-1)L, If all agents update their policies according to
Σ क (a'ls)
Σ क.. (a (1))0= (a=))))))))) (s,))" wDTV (1111)))
TH+ = arg max
R²
= arg max
Σ
n Q, (s. - wDrv ( (1)]11))))))))))
(13)
then we have Vir ((8) 2 (s) 2 ( 2 (8). Moreover, the sequence {V*I} and ( converge to V and Th respectively, which satisfy the print equation,
नंं arg max n
Σ n (a'ls)
Σ प. (I=))ss at + (V
DrV (T (1)))
The proof is included in Appendix A.6.
We further discuss the coefficient w. Intuitively, if w is too large, then the policy will not be updated by (13), iee,, (13) only has a trivial solution कं. = A similar conclusion has been mentioned in Schulman et al. (2015). For the total variation distance case, the threshold value of w is M = 2rmax = 2||Q1I For any
I-y
@ > M, we can show that (13) only has a trivial solution ri41 =
o.. =m From the property of (13), we have
(mi QT))
2 m+ zill, 2 Q²t) = (ग - ní. Qmi) = 1 - xill, 20 0
W || Till1 20 O.
(14)
2
The step (14) is from the inequality (# - Tे. Q=i) S 1 Thus, the condition @ M indicates || rill, 0 which results in the trivial solution. Our choice of (N-1)L has two critical properties. On the one hand, if N 1, then On the other hand, < M indicates the possibility of the non-trivial update .ं+ * T.. We can show the non-trivial update of (13) in a two-player matrix game in Table 1 with both theoretical and empirical results. More details about the non-trivial update are included in Appendix F.4 and Section 5.2.
0 and (13) degenerates to a single-agent policy update.
The policy optimization objective of TVPO is (13). An important property of (13) is that it can be optimized individually and independently by each agent and the joint policy converges according to Theorem 4.8. Although (13) is similar to the surrogate of DPO (Su & Lu, 2022b), there are two main differences between TVPO and DPO. The first difference is that from the property (pllg) S (rr.(ll)), the bound Drv of TVPO is tighter than vDKL in DPO. The second difference is that TVPO obtains the convergence guarantec through policy iteration while DPO obtains the convergence guarantee through the surrogate of joint TRPO objective. A tighter bound means the iteration is less likely to be influenced by the trivial update. We also investigated their empirical performance the experiments. More details of the discussion about the difference between TVPO and DPO are included in Section 5.2.
Remark.
4.3 The Practical Algorithm of TVPO
Practically, we use the objective (13) directly, then the large coefficient w will greatly limit the step size of the policy update, and the algorithm will not work (Schulman et al., 2015). So follow previous studies such PPO (Schulman et al., 2017) to use an adaptive coefficient 3i to replace w, then the policy optimization objective can be rewritten as
n i++1 = arg max Σ (a'sss) (s, 3 Drv (())))))())))))
(15)
7

---

Published in Transactions on Machine Learning Research (03/2025)
7.0
6.--
6.-
0.-
6.0-
2.5
55
20
30
20
10
10
40
terozers
iteratonss
Aor MEA
Figure 1: Learning curves of KT-iteration, TV-iteration, X -iiteration, and H-iteration over four different sets of initialization in the matrix game (Table 1).
where (s, a²) = QT (s, at) - En; [2T (s, = Q," (*)(()) Here we use the baseline V* (s) to reduce the variance in training.
The update rule of 3 follows the practice of PPO. We can choose a hyperparameter d, which means we expect the total variation distance should be around d. Then we can update 3 according to the value of Drv (()+(()s)) in training as follows:
if Drv (+4+1(1)))) > d 6, then 8 t B a if Drv (+)))(+))]] (+1s)) d/6, then 3 /a,
(16)
where ō and a are two constants and we choose 8 1.5 and a like the choice of PPO.
For the critic, since the policy update needs to calculate AT (s, a²) ( a², a-
+ y (8)) V (s));
we take an individual state value function (s() as the critic for each agent and approximate A7 (s, a²) with A. + (s)).. The critic is updated as follows:
fortie E (s) y1)
(17)
where y >s'((' or other target values.
When facing continuous action space, we usually use Gaussian distribution as the policy. However, there is closed-form solution for total variation distance between two Gaussian distributions, to the best of our knowledge. To avoid optimization difficulties, we replace total variation distance with Hellinger distance Du(pllo) = VV,(pi V in the environment with continuous action space, since there is a closed-form solution for Hellinger distance between two Gaussian distributions. Moreover, Hellinger distance has a critical property related to total variation distance that Drv(vlla) S Du(pllq) and the proof is included in Appendix A.7.
With this property, we can replace Dry with DH in Lemma 4.6 and Theorem 4.8, while still obtain the same convergence guarantee. Thus, for the continuous action space, we use the following policy optimization objective:
{dd(())))))) D' ( (11))))))))))))
TH+1 = arg max
(18)
The practical algorithm of TVPO summarized in Algorithm in Appendix
5
Experiments
The experiments contain four main parts. The first part is to verify the limitation of f f-divergence policy optimization as have discussed in Section 4.1 through the matrix game. The second part is to compare TVPO with DPO in matrix game. The third part is to evaluate the performance of TVPO in three popular cooperative MARL benchmarks including SMAC (Samvelyan et al., 2019), multi-agent MuJoCo (Peng et al., 2021) and SMACv2 (Ellis et al., 2023), compared with stale-of-the-arl fully decentralized algorithms. The last part is the ablation study about the hyperparameters d, a and 3. All learning curves correspond to five different random soeds and the shaded area corresponds to the 95% confidence interval. To ensure reproducibility, our codes are included in the supplementary material and will be open source upon acceptance. Due to the space limit, additional experiments included in Appendix E.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Notrix Some Casg
Matik Game Case
0.5
protatiitty
0.4
TV. teration
TV. Beration
terwikn_
D52 Iacadion
taratkon
10
15
10
15
Iterations
Itorations
Figure 2: Learning curves of the iteration (13) and the DPQ iteration in the matrix game (a, b, d) = (-4, 7, 6, 4), where x-axis iteration steps. The first and second figures show the expectation J(T,) and the policies p and q of two iterations in the matrix game case 4 respectively, where J(T:) is calculated by the joint policy ग. qc) and the payoff matrix.
Matrix Game Case
Matrix Game Case
Matrix Game Case
0.8
0.8
probability
0.6
probabillty
0.6
DPO Init.
DPO Init 1.p
DPO init, 6
DPO Init,
DPO init 6.9
DPO_ini_2
DPO_init_
DPO_init_2.
DPO_iniL 7_P
@PO_inil_2,p
DPO_nii_7
1-
init
Init
8
Init,
DFO Init44
DPO iii_iitt_9
DPO Init 4_P
DPO int__int___pp
DPO Init 4_9
DPO_ 900__i99___9
DPO Ini__5
DPO Init
DP0 Init
Rerations
Beraaions
Iterations
Figure 3: Learning curves of the DPO iteration with different initial policies in the matrix game (a, b, c, d) = (-4, 7, 6, 4), where x-axis iteration steps. The three figures show the expectation J(Te), the policies p and q of nine different initial policies in the matrix game case 4 respectively, where JT,,) is calculated by the joint policy म. (pe. q:) and the payoff matrix.
5.1 Verification in Matrix Game
= 4 for the matrix game, which satisfics the condition > e max(u, d) as mentioned in Section 4.1. We use four different specific f-divergences: KT-divergence, total vari- ation distance, ddddittanc,, and Hellinger distance to build four different iterations of (4). We call these four itera- tions KT-iteration, TV-iteration, - and H- iteration respectively. We test these iterations over four sets of initialization: init 1 (Po, qo) = (0.4, 0.8); init (Po. 40) = (0.6, 0.6); init_ 3 (Po.qo) = (0.49,0.7 0.76); init (Po, 40) (0.51, 0.74). For the matrix game, we can calcu- late that (f. (0.5,0.75) as defined in Proposition 4.3. From the discussion in Section 4.1 we know that init 1 and init 3 satisfy the condition Po < 9, 4o 4 which means the converged policy should be the sub-optimal policy (p*,9") (0 with reward 6, and init_ 2 and init 4 satisly the condition Po > 9, go < ④, which means the converged policy should be the optimal policy (p".g") = (), with reward = 7. The empirical results are illustrated in Figure We can find that the empirical results agroe with our thcorctical derivation for all four itcrations over the four sets of initialization. The learning curves of the policy p and q are included in Figure 9 in Appendix E. These empirical results corroborate discussion about the limitation of f-divergence formulation.
In this section, we choose = 5,
Table The policy update types of DPO itera- tion with different initial policies in the matrix game (a, b, c, d) (-4, 7, 6, 4). T represents the trivial policy update and NT represents the non- trivial policy update.
4o
0.2
0.55
0.8
Po
0.2
T
T
T
0.55
T'
T
0.8
T T
T
9

---

Published in Transactions on Machine Learning Research (03/2025)
84
TyTO
120
02-
04
3.5
13
50
68
19
83
15
16
Figure 4: Learning curves of TVPO compared with IQI., IPPO, T2Q, and DPO on the maps 2s3z, 3s5z, 8m, MMM2 and 27m vs 30m in SMAC.
Popparcc2 2xd
da rChustan v2
Antvv
5020
DPO
6000
5920
IDOPG
2020
TVRO
2020
1030
1010
1000
Anpo
staps
Figure 5: Learning curves of TVPO compared with IDDPG, IPPO 12Q, and DPO in 3-agent Hopper, 3-agent Walker2d, 3-agent HalfCheetah and 4-agent Ant in multi-agent MuJoCo.
5.2 Comparing TVPO with DPO
From the discussion in Section 4.2, we have an intuitive idea about the difference between DPO and TVPO that the bound Drv of TVPO is tighter than vDRL DPO. A tighter bound means the iteration will be less influenced by the trivial update. We would like to build a matrix game show this phenomenon. Fortunately, the matrix game (a, b, c, = (-4, 6, 4) satisfies our requirement. The DPO iteration has no closed-form solution and haven't found any useful propertics like Appendix F.4. Thus, we use numerical method to solve the DPO iteration. First, The empirical results are included in Figure 2. We can find that the TVPO iteration has a non-trivial update but the DPO iteration only has trivial updates. This result can be evidence for our conclusion about the difference between TVPO and DPO.
keep the initial policy (Po, (0.55, 0.8) for two iteralions
Moreover, we study the influence of the initial policies on the DPO iteration. We select three candidate values C {0.2, 0.55, 0.8} for the initial policies. We traverse all the values in C for (po. qo) and conclude the performances of all 9 combinations in Figure 3 and Table 2. We can find all 9 initial policies fall into the trap of the trivial update due to the regularization term vDKL in DPO. These empirical results can partially exclude the impact of initial policies on the performances of the DPO iteration in this matrix game.
5.3 Evaluation of TVPO
We compare TVPO with four baselines: IQL (Tan, 1993), IPPO (de Witt et al., 2020), 12Q (Jiang Lu, 2022), and DPO (Su & Lu, 2022b). A brief introduction of these baseline algorithms is included in Appendix F.1. In our experiments, all the algorithms use the independent parameter to agree with the fully recentralized setting, and parameter sharing is banned. More details about the experiment settings and hyperparameters are available in Appendix B and D.
SMAC a popular benchmark in cooperative MARL with high-dimenniooall features and partial observability property. We select five maps in SMAC, 2s3z, Sm, 3s5z, MMM2 and 27m_ vss ______0_mmm for our experiments. These maps cover all three diffieulty levels in SMAC: 2s3z and 8m are easy maps; 3s5z is a hard map; MMM2 and 27m_ vs v0m are super-hard maps.
We show the empirical results of these algorithms in Figure 4. In the super-hard maps MMM2 and 27m vs 30m, all the algorithmss can hardly win, so we use episode rewards as the evaluation metric to show the difference more clearly. As illustrated in Figure 4, TVPO has the best performance in all five maps. The performance of DPO and TVPO is similar in the map 8m, and the rcason may be that 8m is very easy and both of them can obtain nearly 100% win rates within one million steps. In the other four maps, the differences bet TVPO and DPO are more clear.
10

---

Published in Transactions on Machine Learning Research (03/2025)
19900
100
141
22
04
24
12
04
04
15
24
15
25
42
Figure 6: Learning curves of TVPO compared with IQL, IPPO, 12Q, and DPO v8 5 terran,5 vs protoss,5 5 vs zerg,10 vs 10 terran, 10 vs. protoss and 10 vs. 10 zerg SMACv2.
Multi-Agent MuJoCo is robotic locomotion control environment designed for multi-agent scenarios with continuous state and action spaces, based on the single-agent MuJoCo framework (Todorov et al., 2012). In this environment, each agent controls different part of a robot to perform various tasks. We use independent DDPG (Lillicrap et al., 2016) (LDPP) to replace IQL for continuous action spaces. As discussed in Section 4.3, we use Hellinger distance to replace total variation distance for continuous action space TVPO. We select tasks for our experiments: 3-agent Hopper, 3-agent HalfCheetah, 3-agent Walker2d, and 4-agent Ant. In all these tasks, we set agent_oobsk-1
The learning curves of the multi-agent MuJoCo tasks illustrated in Figure 5. We can find that TVPO substantially outperforms the baselines except in 3-agent HalfCheetah, where DPO obtains similar performance to TVPO. The difference between the performance of the value-based algorithms and the policy-based algorithrms is larger in multi-agent MuJoCo compared with SMAC. The reason may be that the continuous action space in fully decentralized learning brings more difficulty in training for the value- based algorithms.
SMACv2 (Ellis et al., 2023) a more stochastic and difficult environment based on SMAC, where each agent controls different units and the initial position will also be randomly determined. We select two settings, __vs__ vs 5 and 10_vs 10_vs_10 10, among three races, terran, protoss and zerg, a total of six tasks from SMACv2 in our experiments. The empirical results are illustrated Figure These tasks are difficult for fully decentralized learning, so we also the cumulative reward the metric. We find that TVPO performs better than the four baselines, similar to the results in SMAC.
mean opicade rowards
14
12
10
d0000001
d-0.001
d=0.01
d=0.1
d-1.0
1.00
steps
106
Figure 7: Learning curves of the TVPO with different hyperparameter d in _0__s 10 protoss in SMAC-v2.
In all three environments, TVPO obtains the best performance in all the evaluated tasks compared with the four baselines, and the differences bet TVPO and the other baselines are obvious in most tasks. The performance of TVPO empirically verifies our discussion about the convergence guarantee of TVPO and the effectiveness of TVPO.
5.4 Ablation Study
We select the 10 vs 10 protoss task SMAC-v2 for the ablation study of the hyperparameters d, and B. All the learning curves correspond three random scods and the shaded area corresponds 95% confidence interval.
For the ablation study of d, compare the performance of TVPO with d E {0.0001, 0.001, 0.01, 0. 1.0]. The empirical results are illustrated in Figure 7. Intuitively, d represents the expected distance of Drv between the old policy and the new policy. If d is small, corresponding to the learning curves d = 0.0001 and d = 0.001, the step size of the policy update limited, which may result in relatively low performance. If d is large, corresponding to the learning curves d==1 1.0, the policy update may excood the trust region, which is away from the convergence condition and results in ascillating curves. There is a trade-off for d. Therefore, the appropriate choices 0.01 and 0.1 have the best performance in this task.
11

---

Published in Transactions on Machine Learning Research (03/2025)
2-11,3-6
p=]_6-
0.00
0:25
850
1.90
1.33
1.50
175
0.0
0:23
0.3
015
3.0
3.25
1.0
175
200
0.00
6:55
0:50
075
1.83
333
350
3.75
233
10-
10-
0.00
0.25
650
0.25
600
0.54
0.15
2.50
1.20
2.00
0.06
5.5
600
4.00
150
1.20
200
Figure 8: Learning curves of the TVPO with different combinations of hyperparameter a and ò in
10 vs 10 protoss in SMAC-v2.
For the ablation study of the hyperparameter a and ò, we choose a E {1.1, 1,5,3) and (2, 4, 6}. The empirical results are shown in Figure 8. In the first line, we control to be the same in each plot. In the second line, we control d to be the same in each plot. Intuitively, a represents the adjustment strength of the coefficient 3 and represents the tolerance of the expected distance. A smaller d means more frequent adjustments. The empirical results show that should match 8, i.e., a smaller adjustment strength (a smaller a) should correspond to more frequent adjustments (a smaller 5) and vice versa. A good combination of () means a good ability to keep the coefficient B close to the expected distance d. Specifically, among the values of a and d we chosen, from the perspective of a, a = 1.1 and = 1.5 are small values corresponding are large values corresponding to the best.
to the best value 8 = 2; from the perspective of 8, 8 and
value a
6 Conclusion and Limitations
In this paper, we propose f-divergence policy optimization, general formulation of independent policy optimization in cooperative multi-agent reinforcement learning, and analyze the policy iteration of such formulation. We discuss the limitation of this formulation, iee, convergence to only sub-optimal policy, and verify it by the empirical results in two-player matrix game. Based on f-divergence policy optimization, we propose a novel independent learning algorithm, TVPO, and prove its convergence in fully decentralized learning. Empirically, we evaluate TVPO against four basclines in three environments. The empirical results show that TVPO outperforms all the baselines, which verifies the elfectiveness of TVPO.
The main limitation of our work is the approximations in the practical algorithms which may not preserve the theoretical properties including the convergence. Additionally, though the learning of decentralized critic is unbiased, it may be troubled with the variance especially in multi-agent settings. Moreover, TVPO still requires on-policy updates which is inconvenient especially in multi-agent settings.
References
Akshat Agarwal, Sumit Kumar, Katia Sycara, and Michael Lewis. Learning transferable cooperative behavior in multi-agent team. In International Conference on Autonomous Agents and Multingent Systems (AAMAS), 2020.
Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131 142, 1966.
Gürdal Arslan and Serdar Yüksel.. Decentralized q-learning for stochastic teams and games. IEEE Transactions on Automatic Control, 62(4):1545-11555 2016.
Charles Blair. Problem complexity and method efficiency in optimization (as nemirovsky and db yudin). Siam Review, 27(2):264, 1985.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR compulational malhemalics and mathematical physics, 7(3):200-2111 1967.
Dingyang Chen, Yile Li, and Qi Zhang. Communicatior-effcient actor-critic methods for homogeneous markov games. arXiv preprint orXiv:2202.0943 2022.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXtv:2011.09533 2020.
Ashley Edwards, Himanshu Sahmi, Rosanne Liu, Jane Hung, Aukit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q (s, s') with deep deterministic dynamics gradients. In International Conference on Machine Learning (ICML), 2020.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Nico- laus Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforcement learning. In Thirly-seventh Conference on Newral Information Processing Systems Dalasels and Benchmarks Track, 2023. URL https://oppnnrevier http:://pppaaeeeeee..eee/r/tae// wtt.//torrrr/iii-51111.....
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whileson. Counter- factual multi-agent policy gradients. In AAAT Conference on Artificial Intelligence (AAAT), 2018.
Roy Fox, Stephen M Mcaleer, Will Overman, and Toannis Panageas. Independent, natural policy gradient always converges in markov potential games, In International Conference on Artificial Intelligence and Statistics, pp. 4414 4425. PMLR, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018
Songyang Han, He Wang, Sanbao Su, Yuanyuan Shi, and Fei Miao. Stable and elficient shapley value- based reward reallocation for multi-agent reinforcement learning of autonomous vehicles. In International Conference on Robotícs and Automation (ICRA), 2022.
Shariq Iqbal and Fei Sha. Actor-attention-critte for multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2019.
Jiechuan Jiang and 'Zongqing Lu. 12q: A fully decentralized q-learning algorithm. Advances in Neural Information Processing Systems (NewrIPS), 35:20469-20485 2022.
Chi Jin, Qinghua Liu, Yuanhao Wang, and Tiancheng Yu. V-learning a simple, efficient, decentralized algorithm for multiagent rl arXiv preprint 2rX 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized
planning. Neurocomputing. 190:82-94 94, 2016.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11255) 2021.
Jakub Grudzien Kuba, Xidong Feng, Shiyao Ding, Hao Dong, Jun Wang, and Yaodong Yang. Heterogeneous- agent mirror learning: A continuum of solutions to cooperative marl. arXiv preprint AXXi:2208.00688 2022.
Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathemalical programming, 198(1):1059 1106, 2023.
Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent, policy gradient in markov potential games. arXiv preprint arXiv:2106.01969, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Wenhao Li, Bo Jin, Xiangfeng Wang, Junchi Yan, and Hongyuan Zha. F2a2: proximate actor-critic for cooperative multi-agent reinforcement learning. orxiv preprint ar xrXi:22004.111 111 11145, 2020.
Flexible fully-decentrralized ap-
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, In International conference learning representations (ICLR), 2016.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, Qiang Fu, Xiaojun Chang, and Yaodong Yang. Maximum entropy beterogeneous-ageen mirror learning. arXiv preprint orXiv:2306. 10715, 2023.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-ommeetiiee environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Xueguang Lyu and Yuchen Xiao. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2021.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and infinite- horizon partially observable markov decision problems. In AAAI Conference on Artificial Intelligence (AAAT), 1999.
Weichao Mao and Tamer Başar. Provably efficient reinforcement learning in decentralized general-sum markov games. Dynamic Games and Applications, 13(1):166-118 186, 2023.
Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning, pp- 15007 15049. PMLR, 2022a.
Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2022b.
Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners cooperative markov games: survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1 31, 2012.
Martin Mundhenk, Judy Goldsmith, Christopher Lusena, and Eric Allender. Complexity of markov decision process problems. Journal of the ACM (JACM), 47(4):681 720, 2000.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advonces in Neural Information Processing Systems (NeurIPS), 2017.
Afshin OroojlooyJadii and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. ar preprint arXiv:1908. 005.00933,,,333,,,, 2019.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research, 12(3):441 440-50,4-450,,, 1987.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht, Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierro-lexxadree Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralisod policy gradients. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Leonid Peshkin, Kee-Eung Kim, Nicolas Meuleau, and Leslie Pack Kaclbling. Learning to cooperate via policy search. In Conference on Uncertainly in Artificial Intelligence. (UAT), 2000.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (TCML), 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Focrster, and Shimon Whiteson. The starcraft multi-agent challenge. arxiv preprint arXix:1902.004
2019.
Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Aulomation Letters, 4(3):2378 2385, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint or arXiv:
1oXi::177.00000 707.0037777 2017.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. Tn International Conference on Machine Learning (ICML), 2019.
Kefan Su and Zongqing Lu. Divergence- multi-agent actor-critic. In International Conference Machine Learning (ICML), 2022a.
Kefan Su and Zongqing Lu. Decentralized policy optimization. arXio preprint arXi::2111.03033 2022b.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Somnerat, Joel 7 Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based team reward. In International Conference on Autonomous Agents and MulliAgent Systems (AAMAS), 2018.
Ardi Tampuu, Tambet Matüisnn, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. orXiv preprint arXiv: 151.108117779, 2015.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In International Conference on Machine Learning (ICMI.), 1993.
Justin K Terry, Nathaniel Grammel, Ananth Hari, Luis Santos, and Benjamin Black. Revisiting parameter sharing in multi-agent deep reinforcement learning. ar preprint "XXx::005513622 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In TEFE/RSJ International Conference Intelligent Robots and Systems (IROS), 2012.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimiza- tion. arXie preprint arXiv:2005.0981,, 2020.
Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. general class of surrogate functions for stable and efficient reinforcement learning. arXiv preprint arxi0:1108. 05828, 2021.
Nikos Vlassis, Michael L Littman, and David Barber. On the computational complexity of stochastic controller optimization in pomdps. ACM Transactions on Computation Theory (TOCT), 4(4):1 8, 2012.
Jiangxing Wang, Deheng Yc, and Zongqing Lu. More centralized training, still decentralized exccution: Multi-agent conditional policy factorization. In International conference on learning representations (ICLR), 2023a.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjic Zhang. Qplex: Duplex ducling multi-agent q-learning. In International Conference on Learning Represenlations (ICLR), 2021a.
Qing Wang, Yingru Li, Jiechao Xiong, and Tong Zhang, Divergence-augmented policy optimization. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International Conference on Learning Representations (ICLR), 2021b.
Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient. decentralized multi-agent rl with function approximation. In Conference on Learning Theory (COLT), 2023b.
Yuanhao Wang, Qinghua Liu, Yu Bai, and Chi Jin. Breaking the curse of multiagency: Provably efficient decentralized multi-agent rl with function approximation. orXiv preprint arXiv:2302006606 2023c.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509-16521, 2022.
Long Yang, Yu Zhang, Gang Zheng, Qian Zheng, Pengfei Li, Jianhang Huang, and Gang Pan. Policy optimization with stochastic mirror descent. In AAAI Conference on Artificial Intelligence (AAAI), 2022.
Wenhao Yang, Xiang Li, and Zhihua Zhang. A regularized approach to sparse optimal policy in reinforcement learning. In Advances in Neural Information Processing Systems (NewrIPS). 2019.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:000.003939, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint FFF::2103.0195 2021.
Wenhao Zhan, Jason D. Lee, and Zhuoran Yang- Decentralized optimistic hyperpolicy mirror descent: Provably no-regret learning in markov games. In International Conference on Learning Representations (ICLR), 2023.
Chengwei Zhang, Xiaohong Li, Jianye Hao, Siqi Chen, Karl Tuyls, Zhiyong Feng, Wanli Xue, and Rong Chen. Sco-rfmq learning in cooperative markov games with continuous actions. arXio preprint or Xiv: 1rx:::8809.000 1809.0662,, 2018a.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International conference on machine learning, pp. 5872 5881. PMLR, 2018b.
Lin Zhang and Ying- Ohn-Chang Liang. Deep reinforcement learning for multi-agent power control in heterogeneous networks. IEEE Transactions on Wireless Communicaatios,, 20(4):2551-25) -2264, 2020.
Runyu Zhang, Zhaolin Ren, and Na Li. Gradient play in stochastic games: stationary points, convergence, and sample complexity. IEEE Transactions on Automatic Control, 2024.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of iaciimmu--nttrop multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Appendices
A Proofs
A.1 Proof of Lemma 4.2
Proof. The Lagrangian function of (4) is follows:
Σ n (a:1))) a:) w Σ Tota (ails)) ( n' ra(asss)
(asss))s))
[ - Σ (a)ls))(((als))
+ A
where A and ((a; are the Lagrangian multiplier.
Then by the KKT condition have
0L im'(a, (a,ls)
Qu az) wH (a(lss)
+ 02 + (ayls) 0,
T Taaa(a)ss))
can resolve n² (a;ls) as
(ails) x (ails)) 9
+ + (a:ls) Q (s
(19)
w
From the complementary slackness know that ((a,ls) = 0, rewrite (19) as
('alss) max {o (s, .0.. (a:ss)
(20)
W
(s,a:) 2o .0).
n' <aalss = max Taaa (ailsg
(21)
A.2 Proof of Proposition 4.3
Proof. To discuss the monotonicity of the policies and at, let (A( and (A(1) represent the expected reward Alice will obtain by taking action un and u respectively. Simlilarly, we can also define Q (0) and (B(1) for Bob.
From the definition, we have (,) q: a (1(1) d (c - d)q;, (0) e + (a c)p, and Q1( d + (b - dlpp.
..) b + (a b)g. Similarly can obtain that
Combining (21) with the condition g(z)= 2 0, then we have
(a - b)g + b AA
(c (c- d)q: d XA
Pc+1 = P:g
Pe+1 (1 pt)g
w
W
(0- (o=d)
=
1)
w
(22)
Pt+1
Pt
(a blg +0+
17

---

Published in Transactions on Machine Learning Research (03/2025)
From (22) we can find that
g (c-d) (c-d)artdt>?
P++1 S Pt.
(a la-bla- 4
21
4 (c d)qt + d 2 (a b)gt +
(23)
9 (b + a d)q: 2 b - d
qt 2
The critical step (23) is from the combination the condition g(x) 2 0 and the property g(x) non-decreasing
Similarly we can obtain that p. 2 = g++1 S qr: Pt S q++1 qei qe q = P++1 Ps and qe $
Pi+1 2 Pc.
]
A.3 Proof of Corollary 4.4
Proof. From the iteration of {Pt} we have
[a=b)e++btdp
P1+1
(24)
1 P++1
Pt
(c- (oddg+d++0 2,1
Let - in both side of (24), we know that
(a (a=b)g"+bb+))
pº
9
-1 =
(25)
1 - p"
(e- (e-dir+dd22
g
(a-b)9" +h+ A
As q² ④, we know that
So we can rewrite (25) as - and resolve p² = 0.
+d+A4
As for q", we can follow a similar idea. From the iteration of {9o} we have
(b-d)p+ddt?""
1)
w
(26)
9++1
Øt
la-clpetota$
Let / ++ in both side of (26), we know that
(b (b-d)p"+ +d+22"
9
-1 -0.
(27)
9"
(a-c)p'+otay
(b-d)p" (b-d)p"+d+12 +
As n² <
we know that
1. Then can rewrite (27) 1-9"
and obtain q² 1.
(a-c)p" +c
4"
18

---

Published in Transactions on Machine Learning Research (03/2025)
A.4 Proof of Lemma 4.6
Proof. For any fixed i, consider the following difference
Σ New (als))²
Enhew new (ails) Σ (a (1s
Σ Fnew(ay (asls)
nnew (a-ils) - Told (a-is)) (s, a-s)
(28)
Σ nnw Tnow(a:lss (asls)
Nnw (a_:/s) (a (Q as, Σ
(29)
a-s
T new (a:ls)
Σ
r new (a_ils) (8) - (amiss)
(30)
2
a
M
b (a_ils)
Told (a==ls)))
(31)
2
N
M
1:k-1 (a1:8-1 TT K:N (aw:vls) - nnww Tnew((a. (s) old (ax+1:N|s)
K+InN
(32)
2
new
N
$
1:k (a1)8-115)mmt) (awxrls) - new (a 18) ald K+INN (ax+1.11b)|
(33)
2
R-4 k=1,
N
M
Σ
Tne (apls) - Tata (axls)]
k
(34)
2
1,
ok
M
Σ
DTV (+
(35)
1= R,ggtt
where nnw meanings as a1:k-1 = a1 x a2 X ..... -g-1.. 1. In (29) and (33), we use the triangle inequality of the absolute value. Tiax M
1:k-1 denotes Tnew X nnew X ... Tnow and Thew will be skipped if involved, and disk-1 has similar
In (30), we use the property Q (s, a) S I-y 2 from the definition of Q-function. In (32), insert N --1 terms between (a-ils) and Tedd (l=ils) to make sure the adjacent two terms are only different in one individual policy.
By rewriting the conclusion above, for any agent é, we have
Σ π new (als)
Σ Thew (atls)
Σ (a_ as am1)
M Σ Drv (T ( ( (-1s))
(36)
k=1,4:
Then, by applying (36) to i i= 1,2, ..,.,..,, ,N and add all these N inequalities together, we have
Σ (als)) 2 Σ Nnew (arls)
1100 a, a.,
(N
1)M
Drv
N
19

---

Published in Transactions on Machine Learning Research (03/2025)
A.5 Proof of Proposition 4.7
Proof. By the definition of Vi we have
Viold (s)
ΣΣ (ailss N
1
(s, a-i) (a-1ssss)
DD D, (1s)))
ΣΣ T (ails
(a-ls))) (s, -w w E Ds (T (-s))lpp
(37)
N
R-1
1
ΣΣ nnew (a:1s) (a-iss)s)) (r(s, + V (s')l)
N
~ Dj ne (-|s)]lp(((sss)
(38)
(expand Vi (s') and repeat replacing vold with Thow
(39)
V (s).
(40)
In (37), we use the definition of new in (11). (38) is from the definition of Qp a; a-;) (39). we
repeatedly expand Vo according to its definition and replace Told with Thew by the optimality of keew like what we have done in (37). After we replace all Tald with Thew then we obtain VTTe((>) (s) according to the definition of V (s) in (40).
With the result Virala (s) I ViTaew (s), we know T (s, = r(s, a(,, ()s,a)++EEV EV (8') r(s, a) EV now (5') QT (s, a).
A.6 Proof of Theorem 4.8
Proof. From the Proposition 4.7, we know Val (s) 2 V
Thus, we just need to prove (*) V/ (8).
From the definition of V (s) we have
(* (s) (s,
T((as(s)
1=1
a
3
Drv (1((11))1(+11(111))
(41)
T((as(s)
N
a
w
Σ Drv (
(42)
1=1
(expand (s' and repeat replacing प, with mp
(43)
V (s).
(44)
(41) is from Lemmn 4.6, and (42) is from the definition of QT (s, ai am:.. In (43), we repeatedly expand V and replace the प, with -;-1 by Lemma 4.6 like what we have done in (41). After we replace all ,.," with क, then we obtain VAL (s) in (44) according to the definition of V (s).
From the incqualities VAA,++ (s) 2 VTh (s) 2 V (s) 2 V*)= we know that the sequence (V** improves monotonically. Combining with the condition that the sequence (V*)) bounded, we know that {V**} will converge to V According to the definition, the sequence ( and (n.) will also converge to Q* and n.
20

---

Published in Transactions on Machine Learning Research (03/2025)
respectively, where n, satisfies the following fixed-point equation:
म arg max Σn (ayls) Σ (a o, a-1) wDrV (#((+ss)]=::(11s))
A.7 Proof of Drv(plla) & Du(ela)
Proof.
(v =
Σ pp 9:l
Σ + IVDD Veillve + val
=
4
) + v
Σ IVPs -vai²)
$ 4
Saauchy-Scwaare inequality)
4 pilllla)
(2+2 vpih
(allla))
B
Experimental Settings
B.1
MPE
The Uhree lasks are based on the original Multi-Agent Particle Enviromment (MPE) (Lowe et al., 2017) (MIT license) and were initially used in Agarwal et al. (2020) (MIT license). The objectives of these tasks are:
Simple Spread: N agents must occupy the locations of N landmarks.
Line Control: N agents must line up between two landmarks.
Circle Control: N agents must form circle around landmark.
The reward in these (asks the distance between all the agents and their larget locations. We select these tasks to maintain consistency with DPO (Su & Lu, 2022b) but set the number of agents N = 10 for these three tasks in our experiment.
B.2 Multi-Agent MuJoCo
Multi-agent MuJoCo (Peng et al., 2021) (Apacho-2.0 license) is robotic locomotion task featuring continuous action space for multi-agent settings. The robot is divided into several parts, each containing multiple joints. Agents in this environment control different parts of the robot. The type of robot and the assignment of joints determine the task. For example, the task "HalfCheetah-3x2" means dividing the robot 'HalfCheetah" into three parts, with each part containing two joints. Details of our experiment settings in multi-agent MuJoCo are listed Table 3. The configuration specifies the number of agents and the joints assigned each agent. "Agent obsk" defines the number of nearest agents an agent can observe.
B.3 StarCraft2
SMAC (Samvelyan et al., 2019) (MIT license) is a widely used environment for multi-agent reinforcement learning (MARL). In SMAC, agents reccive rewards when they attack or kill an enemy unit. The rewards for episode normalized to maximum of 20, regardless of the number of agents, to ensure consistency
21

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: The task settings multi-agent MuJoCo
task
configuration
agent obsk
HalfCheetah
3x2
2
Hopper
3x1
2
Walker2d
3x2
2
Aut
4x2
2
across tasks. An episode is considered won if the agents kill all enemy units. The observation space for agents depends the number of units involved in the task. Typically, the obscrvation is vector with over 100 dimensions, containing information about all units. Information about units outside agent's field of view is represented as zero in the observation vector. More details on SMAC can be found in the original paper (Samvelyan et al., 2019). SMACv2 (Ellis et al., 2023) (MIT license) is an advanced version of SMAC. Unlike SMAC, SMACv2 allows agents to control different types of units in different opisodes, where the unit types are determined by a distribution and a type list. Moreover, the initial positions of agents are randomly selected in different episodes. With these properties, SMACv2 is more stochastic and difficult than SMAC. We keep the configuration the same the original paper (Ellis et al., 2023) among the selected tasks.
C Algorithm
Algorithm 1. The practical algorithm of TVPO
1: for episode to M do
2: for t== to max @pisode__engee do
3: select action a; ~ (-1ss
4: execute a, and observe reward and next state s
collect rs,) end for
7: Update the critic according to (17)
8: Update the policy according to (15) or (18)
9: Update He according to (16).
10: end for
Training Details
Our code of IPPO is based on the open-source code! of MAPPO (Yu et al., 2021) (MIT license). The original IPPO and MAPPO is actually implemented as a CTDE method with parameter sharing and centralized critics. We modify the code for individual parameters and ban the tricks used by MAPPO for SMAC. The network architectures and base hyperparameters of TVPO, DPO and IPPO are the same for all the tasks in all the enviromments. We use 3-layer MLPs for the actor and the critic and use RcLU as non-lincarities. The number of the hidden units of the MLP is 128. We train all the networks with Adam optimizer. The learning rates of the actor and critic are both 5e-4. The number of epochs for every batch of samples is 15 which is the recommended value in Yu et al. (2021). For IPPO, the clip paramcter is 0.2 which is the same as Schulman et al. (2017). For DPO, the hyperparameter is set as the original paper (Su & Lu, 2022b) recommends. Our code of TQL is based on the open-source code2 PyMART (A (Apache-2.0 license) and we modify the code for individual parameters. The default architecture in PyMARL RNN so we just follow it and the number of the hidden units is 128. The learning rate of IQL is also 5c-4. The architectures of the actor and critic of IDDPG are 3-layer MLPs. The learning rates of the actor and critic are both
httpe///uhhhu..cmmmmmmmmmmmmmaap httpe///ghhaabccmmm///wwnettp//p
22

---

Published in Transactions on Machine Learning Research (03/2025)
5c-4. Our code of 12Q is from the open source code of the original paper (Jiang & Lu, 2022). We keep the hyperparameter of 12Q the same as the default value of the open-source code in our experiments.
Table 4: Hyperparameters for all the experiments
hyperparameter
value
MLP layers
3
hidden size
128
non-linear
ReLU
optimizer
Adam
actor Ir
5e-4
critic_ Ir
5e-4
numbers of epochs
15
initial &
0.01
1.5
2
0.001
clip parameter for IPPO
0.2
The version of the game StarCraft2 in SMAC is 4.10 for our experiments in all the SMAC tasks. We set the episode length of all the multi-agent MuJoCo tasks as 1000 in all of our multi-agent MuJoCo experiments. We perform the whole experiment with a total of four NVIDIA A100 GPUs. We have summarized the hyperparameters in Table 4.
E
Additional Empirical Results
Figure illustrates the learning curve of the policy p and q in the matrix game of KL-iteration, TV-iteration, -iitteaaiion, and H-iteration over four different sets of initialization. We can observe the policies of all four kinds of iterations converge.
MPE is popular environment in cooperative MARL. MPE is 2D environment and the objects are cither agents or landmarks. Landmark is a part of the environment, while agents can move in any direction. With the relation between agents and landmarks, we can design different tasks. We use the discrete action space version of MPE and the agents can accelerate or decelerate in the direction of the x-axis or y-axis. We choose MPE for its partial observability.
The empirical results MPE are illustrated in Figure 10. We find that TVPO obtains the best performance in all three lasks. In this environment, the poliey-based algorithms, TVPO, DPO, and IPPO, outperform the value-based algorithms, IQL and 12Q) I2Q has a better performance than IQL in all three tasks.
For the comparison with the bascline IPG (Leonardos et al., 2021) and INPG (Fox et al., 2022), we select three 10 vs 10 SMAC-v2 lasks. The empirical results are illustrated Figure 11. We can find that IPG's performance is not stationary and may drop with the progress of training compared with other policy based algorithms. We think the main reason that IPG lack the constraints about the stepsize of policy iteration. We use the adaptive coefficient for INPG, and its performance is similar to DPO, which is reasonable as their policy objectives similar except for a square root term.
We also compare the influence of the hyperparameters with values 0.1, 0.2, 0.3 for ablation study and select the 10__s_ _0_ protoss task for experiments. The empirical results are ilustrated in Figure 12. We can see that the impact of this hyperparameter is not significant.
IPPO's performance. We choose clip parameters
23

---

Published in Transactions on Machine Learning Research (03/2025)
0.73
0.75
0.75
0.73
0.50
0.50
9.50
9.50
m1.1
TV.pint:
Ch p.
1 Wt.
0.02
0. ht1
2.50
Cn int
0.00
10
20
30
10
20
10
Nozte
40
10
Nabia Carre
40
20
1.00
1,00
1.00
0.50
0.50
2.50
0.5
p hnt
0.25
TV inte
0.25
9.2
2
Mi.r Gere
Muy Gammi
Molse Gare
Nabrit Gamte
1.00
1.50
1.50
1.00
0.75
0.75
3.25
0.2
0.00
0.00
TV ir
3.50
0.00
30
10
40
10
20
40
10
40
0.75
0.75
9.75
0.75
0.50
0.00
9.30
9.50
TV.
Chips thip.ink (i
C mL4
TV. m
3.00
Ch.4 irk
0.00
m
10
20
20
10
10
20
30
40
10
20
30
40
50
12
30
40
20
Iwndone
Paratios
Paratios
Enestion
Figure 9: Learning curves of the policy p and in the matrix game of KL-iteration, TV-iteration, x² -i-iteaatinn, and H-iteration over four different sets of initialization. Each row corresponds to one set of initialization and each column corresponds to one type iteration.
simple spread
cantral
ciicle control
-15
-10
-10
-20
DPO
-20
mman epiiodde
-25
iPP0
-30
-30
-10
IQL
120
40
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
staps
166
stops
1e6
St8p0
1e6
Figure 10: Learning curves of TVPO compared with IQL, IPPO, 12Q, and DPO in 10-agent simple spread, 10-agent line control, and 10-agent circle control in MPE.
10 vs 10 terran
10 10 protoss
10 vs 10 zerg
10
mean apisods rewards
15.0
10
12.5
TVPO
120
TVPO
120
6-
TVPo
22
DPO
IGG
7.5
DPO
IPG
DPO
IPG
2-
IQL
IPP0
IPG
5.0
IPP
IQL
INPG
4-
IQL.
IPPO
INPG
2
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
steps
steps
1e6
steps
Figure 11: Learning curves of the TVPO and other baselines including IPG and INPG in the three 10 SMAC-v2 tasks.
10
24

---

Published in Transactions on Machine Learning Research (03/2025)
protoss
mean episode rewards
15.0
12.5
10.0
TVPO
7.5
OPPO_liip. @all__pram_____ 0.1
__o__lip_pa__ 0.2
5.0
IPPO _alp__aaa_______________________ 0.3
0.00
0.25
0.50
0.75
1.00
1.25 1.50 1.75
2.00
steps
1e6
Figure 12: Learning curves of the TVPO and IPPO with different clip parameters in the 10 vs. 10 protoss.
F Discussion
F.1 A Brief introduction of baseline algorithms
We select these four baseline algorithms as representatives of fully decentralized algorithms. IQL (Tan, 1993) a basic value-based algorithm for decentralized learning. TPPO basic policy-based algorithm for decentralized learning. Both IQL and IPPO (de Witt et al., 2020) do not have convergence guarantees, to the best of our knowledge. DPO (Su & Lu, 2022b) and I2Q (Jiang Lu, 2022) are the recent. policy-based algorithm and value-based algorithm respectively, and both of them have been proved have convergence guarantec.
TQL, IDDGG, and TPPO are relatively simple to understand, where each agent updates its policy through an independent Q-learning, DDPG, or PPO. These algorithms simply extend the single-agent RL algorithms into the MARL setting. They are heuristic algorithums without convergence guarantees in fully decentralized MARL.
The idea of DPO to find lower bound of the joint policy improvement objective as surrogate which can also be optimized in a decentralized way for each agent. The formulation of DPO is follows:
arg Σ (7 (s) M. V DKL (1)()))))))(()))) C DKL. (1(())))))))))))
1+1
DPO has been proven to improve monotonically and converge in fully decentralized MARL. 12Q uses Q-learning from the perspective QSS-value Qi(s, 8'). The QSS-value is updated with the following operator:
(Q, =r max,"eN'()) @i(s',s");
where (ss') is the neighbor set of state s'. In the deterministic environment and with some assumption about the transition probability, Qi(s, s') will converge to the same Q-function for each agent i, so the joint policy of agents will also converge in fully decentralized MARL.
F.2 Unary Formulation
Before proposing the f-divergence formulation, we have studied another formulation. This formulation follows the idea of entropy regularization and the extra is only related the policy œ² instoad of the divergence between r and Thi We refer this approach as the unary formulation. Though we discovered that the unary formulation has more significant. drawbacks, the properties of the unary formulation inspire in
25

---

Published in Transactions on Machine Learning Research (03/2025)
the proof of TVPO. So we would like to provide the properties and some empirical results of the unary formulation here for discussion.
The unary formulation is
new arg max Σ (a:ls))) (s, + Σ (adss) (*
(45)
This formulation (45) follows the idea of Yang et al. (2019) which discusses the regularization algorithm in single-agent RL. From the perspective of regularization, the update rule (45) can be seen as optimizing the regularized objective J²(T)) = E [ 7 (ri(s, as) + we (a:()))))) where ri(s, a:) = Er [r(s, az, a-i)). The choice of Φ is flexible, e.g., ((x) = log x corresponds to entropy regularization and independent SAC (Haarnoja ct al., 2018); ((x)) = mcans (45) degenerates to independent Q-learning (Tan, 1993); Morcover, there are many other options for ái corresponding to different regularization (Yang et al., 2019). So we take (45) the general unary formulation of independent learning, where the 'unary' means the additional terms Eng x² (ails)ō (x" (a:ls) is only about one policy n².
For further discussion of (45), we can utilize the conclusion in Yang et al. (2019) as the following lemma.
Lemma F.1. If ((x) in [0, and satisfies the following conditions: (1) Φ(x) is non-increasing: (2) ((1) 0; (3) σ(x) is differentiable; (4) fo(x) = zι(x) is strictly concave, then we have that gø(x) = (f)) (I) exists and (x(x) is decreasing. Moreaver, the solution to the optimizalion objective (45) con be described with
(x()) as follows:
new (a:ls) max ( Tx - mold (s,ai) ,0.
(46)
w
where X₂ satisfies E., max(gs ( (No-Q.oollssa) .0 =0
Though seems that ((x) needs to satisfy four conditions, actually d(x) log for Shannon entropy and d(x) - x for Tsallis entropy are still qualified.
However, unlike the single-agent setting, the update rule in Lemma F.1 may result in the convergence to sub-optimal policy or even oscillations in policy in fully decentralized MARL.
We further discuss (45) in the two- wwo-player matrix game and have the following proposition.
(x(x) Proposition F.2. Suppose that go (x) 2 and gx(x) is continuously differentiable. If the payoff matria of
the two-player matria game sulisfies + e < d, and two agents Alice and Bob update their policies with policy iteration
#4+1 = arg max
+ (F Σ xt (o.l))))
(47)
4;
a,
then we have (1) Pt > Pt-1 7 a++1 46 (2) Pt < Pt-1 = 941 < < (3) q: > 41-1 = PL+1 > P6 (4) qo <
G--1 3 P+11 < Pt-
Proof. To discuss the monotonicity of the policies Ps and qt we need the solution in Lemma F.1. Before applying the update rule (46), we need to calculate the decentralized critic given and d.- Let Q; and (1) represent the expected reward Alice will obtain by taking action ua and w respectively. We can also define @ and Q (1) for Bob.
From the definition, we have Q (0) qi a (1 qi) b b b+ (a b)q. Similarly we could obtain that (1(1)= d (c d)q, Of (0) + (a - c)ps and Of d (b - d)pp.
26

---

Published in Transactions on Machine Learning Research (03/2025)
With (46) and the condition go(x)= 0, we have
A1 (0)
(b a qt + X, -b)
(d -)-c (ld-c) 11 -d
Pt+1 = 96
= 9
1 pc+1 = 90
w
w
w
(b - a
a)
+ Xp1 -
(d c) + Xp d
96
+ 96
=1
w
(c appt + Xp -c c
(d- b)pt + 1/
4t+1 = 96
1 qi+1 -0
(c (c- ulpt
X + gd
(d (d- b)pt + X - d
96
=1
We can rewrite these equations with some simplifications as follows,
mA(3) A (b a)x + )a(x) b nA(x)= & (d- c)x + AA(x) -d ha(x))) = 9o (ma(z))
where \x) satisfies 96 (m^(x)) 9o (na(x)) =1 mx(x)= A (c - a)pt + {B(2) -C (T( (d- (d-b)pt
(48)
b)pp + AB(L)= h1(x)) g6 (mb(#))
w
where (z(x) satisfies gs (mB(2)) + go (nø(x)) =
With these definitions, we know that p1+1 ha())) q++1 = (p())) and the monotonicity of pt and qi is determined by the property of function ha(r) and hn(r). By applying the chain rule to (48), have:
5o (ma(x)) (6 1 (1
Na(x) (b - (4-
(49)
+
Then we have:
1
(x(x)- =
Ls (ma(x)) a (Apply chain rule)
(50)
w
1
a d) Le (((
(Substitute (49) for Xx ).
(51)
(m)()(=)) L (z)(
Let M -b= and M' = -, then Na(x) M d'ann (m)))
o (((m (na())) From the condition and Lemma
F.1 know that M' 0 and (z(x) is decrcasing which means L' (x) < 0. Combining these conditions logether, we know 1,,(2) (z) > 0 and h is increasing which means that P++1 ha(w)) is increasing over a. which means that d 9--1 + p++1 pt and a 9-1 Pt+1 > Pt-
Similarly, we can obtain that N3 (x) = M (m
> 0 which could lead to the result that
(((m)())++
Pr > Pt-1 + a++ > qe and Pt < Pt-1 + qc+1 < qt.
Proposition F.2 actually tells us P++1 = (a())) is increasing over and = = hp(p:) is increasing over when M = b a- d < O. Intuitively, we can find two typical cases for policy iterations with Proposition F.2. In the first case, if in a certain iteration the conditions Pt > Pt-1 and qe > q--1 are satisfied, then we know that pe+1 > Pe +1+1 > qe V2 L. As the sequences (pt) and {4) are both bounded the interval [o, 1J, we know that {pr} and {gr} will converge to p² and q- The property of p* and g" determined by la(z) & hz(ha(z)) and (x(x) & ha (h(hh(=) respectively as PLt2 hp(h)(p)) and q42 ha(hp(())) and we have the following corollary.
Corollary F.3. [x)x U2, (1(()) & M2U where Up is constont determined by ((x)-
27

---

Published in Transactions on Machine Learning Research (03/2025)
0.4
5.00
6.4
6.7
2.0
achoco FEARnds
0.0
0.0
20
20
10
30
40
steratioss
Figure 13: Learning curves of the unary formulation in two matrix game cases, where x-axis is iteration steps. The first and second figures show the performance and the policies p and q in the matrix game case 2 respectively. The third and fourth figures show the performance and the policies and in the matrix game case 3 respectively.
Proof. As (2) is continuous, let U1 & max l6' (z))), U3 - max 10's (1 UB - [ l6 (m)(())) and U3 - maxse(o.1) (((n((z))) Moreover, let Us = max U3 U1. U then apply the chain rule to (x) and we have
[((x)]] = (h'g
= =M² (H))
(52)
(c(m (=)
= M (1((m(())))l((((mm()) (=((m(()))lo((((m(()))
(Tet y = ha(x) E [0, 1))
(((mm())))+0 (s (m)(((())] (g
S M (o(((mm()))))) + lo (m)((()))) +
lol (xx(
(53)
2
2
S U2
(54)
where (52) is from Proposition F.2, (53) is from the AM-GM inequality ab S (e+b)² definition of Ug. Similarly, we can obtain (x)( < M U3.
2 and (54) is from the
Combining Corollary F.3 and Banach fixed-point theorem, we can find that Ug is a constant, if [M'| the then we can find a constant L such that [x)( s U & S L < 1, which means that the iteration Pi+1 = (p()) is a contraction and p* is the unique fixed-point of LA- This conclusion can be seen as that smaller [M'| corresponds to larger probability of convergence. In this convergence case, the converged policies p" and q" are usually not the optimal policy as the optimal policy is deterministic, which can be seen in our empirical results.
In the second case, which may be more general, in iteration f, (p. 1 Pr-1)( - 9r-1) < O, which means pt > Pt-1 and q < q--1 or pt < Pt-1 and qi > 4t-1- Without loss of generality, we assume Pe > Pt-1 and ge < q:-1, then we know P++1 <pt and g++1 < ge from Proposition F.2. By induction we can find that for any t' " t, the sequence } and } will increase and decrease alternatively, which means that the policies may not converge but oscillate. We will show this in our experiments. As the unary formulation may result in policy oscillation, we would like to find other formulations for fully decentralized MARL.
F.3 Verification for Unary Formulation
In this section, we choose d(x) log corresponding to the entropy regularization the representation for the unary formulation. We build two cases to show the convergence to the sub-optimal policy and the policy oscillation. We choose a = 5, b = 6, c = 3, d = 5 as case 2 and a = 7, b = 5, c = 4, d = 6 as case 3. Both two cases satisly the condition b+ c < a + d as discussed above. We keep = 0.1 for all the experiments on these two matrix games. The empirical results are illustrated in Figure 13. We can find the policies p and q improve monotonically to the convergence (p",4' R (0.773, 0.227) in case 2, which policy. However, in case 3, the policies p and q oscillate between and and do not converge. These results verify our discussion about the limitation of the unary formulation.
sub-optimal joint
28

---

Published in Transactions on Machine Learning Research (03/2025)
Notrix Gome Casg
Matik Game Case 4
0.5
protabiiity
0.4
theory, return
10
10
15
Itorations
Itorations
Figure 14: Learning curves of the iteration (13) in the matrix game (a, c, d) = (-4,7, 6, 4), where x-axis is iteration steps. The first and second figures show the expectation ((T)) and the policies and q in the matrix game case 4 respectively, where J(Tz) is calculated by the joint policy л. (pt, (P) and the payoff matrix.
6,6,4,4
F.4 Non-Trivia Solution Iteration (13)
In this section, we will build a two-player matrix game like Table 1 to show the non-trivial solution to iteration (13). In general, there is no closed-form solution to iteration (13). However, for the matrix game case, can show some properties of iteration (13). With the same definitions as previous discussions, can rewrite (13) in the matrix game as follows:
P++1 arg max PQA + (1- (1) wlp - p=[0.1]
(55)
Let f(p) -1- (1-))))))))))
- pel, then Pt+1 = arg maxpe((.1] f()).
We know that f(p) is a linear funct of in both intervals [0, and [pt, 1] and the maximums of linear function are always achieved in the endpoints of one interval. Thus, we have P+-1 = arg maxpe f(),, which mcans we only neod consider
f(0) = Q1) - wpt
f(1) QA(0) - w(1 pt)
f(p.) = (1) p.(Q (0) (1))
Next, we can build matrix game with the property b max(a, d} > d > a. In this case, = = and w (N-DM
N = b. Then we consider the condition f(0) > f(r)). We have
f(0) f(p)) Pc (Q) (1) = (26 d - + d)q.) -b-d S(0) f(x) < a aa
We need ensure a feasible a can be found, which means b <
b b, > d 0 a, we find non-trivial solution to (13). To empirically verify this conclusion, we choose a matrix game with (a, d. = (-4, 7,6, 6, 4) where q = 13 ~ 0.769.... For simplicity, we call this matrix game as matrix game case 4. We also choose (po.go) = (0.55, 0.8) to ensure the condition de > q. The empirical results are illustrated in Figure 14. We can find the non-trivial update for the joint policy which verifies our conclusion discussed before.
Thus, for matrix game satisfying the condition
10
F.5 Discussions about using global state in theoretical results.
Using the global state s for theoretical analysis has been a common practice in the study of multi-agent reinforcement learning, especially in the setting of decentralized learning. There are many previous works containing theoretical results in decentralized learning. which include both cooperative settings (Jiang & Lu, 2022) and non-cooperative settings (Arslan & Yüksel, 2016; Mao et al., 2022a; Zhang et. al., 2024). The
29

---

Published in Transactions on Machine Learning Research (03/2025)
main reason for this common practice is the difficulty in solving POMDP, which has been studied for decades in Papadimitriou & Tsitsiklis (1987); Mundhenk et al. (2000); Vlassis et al. (2012). Additionally, the theoretical analysis of Dec-POMDP will be even more difficult in the multi-agent setting. If we include partial observability in the analysis, we may not obtain anything since the problem may be undecidable Dec-POMDI (Madani et al., 1999).
30


Reviews:
[{"review_id": "KXbfwj1jeQ", "reviewer": "Reviewer_G6n7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have some confusion about the contributions of this paper. I'll state them in a series of questions, hoping that the authors will help me to clarify the matter.\n\n1. Is the main contribution of the paper theoretical or empirical?\n2. Is the f-divergence formula (4) something new? In particular, why not express this in terms of a single policy outputting all actions?\n3. Following, the derivations in Sec 4.1 feel standard RL calculations. Is it new, or at which point?\n4. How does the fixed point defined in Thm 4.6 depend on $\\omega$? Could you give some examples where dependence is not trivial?\n5. Do the results of Sec 4.1 hold for any f-divergence (or a broader class than TV used in Thm 4.6). In particular, is TV beneficial from the theoretical standpoint, or only empirically better (this is perfectly fine; I just want this to be clear)?\n6. Is it correct to say that the experiments with SMAC do not fully fall into the framework developed in Sec 4.1. That is, SMAC is not fully observable and states are agent dependant, while Sec 4.1 assumes the full Markov state?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new algorithm to solve collaborative decentralised multi-agent learning. The algorithm is based on the idea of using the f-divergence, concretely proposing total variation policy optimisation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YMiez3b3cm", "reviewer": "Reviewer_kZ6m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n* The experiment results show that the proposed TVPO exhibits faster reward convergence compared to the selected baselines. The algorithm's performance is stable across different hyperparameter choices, according to the ablation results.\n* The proposed algorithm is shown to be convergent and can avoid the suboptimal minimizer of the constructed example.\n\n**Weaknesses**:\n- The assumption that the global state is accessible to all agents is not practical. Under this assumption, it would be feasible to directly apply centralized algorithms, which weaken the impact of this work.\n- It would be helpful if the author can highlight the insight on why using the total variation distance can resolve the suboptimality issue.\n- There are some typos and missing/incorrect definitions, making the paper hard to follow. Some examples:\n    - In Equation (3), the expectation should be taken w.r.t. $a^{-i}$ rather than $\\pi^{-i}$.\n    - In Equation (4), $s$ and $\\pi_{\\text{old}}$ are not defined.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper consider the multi-agent setting where global state is observable and agents are not allowed to share parameter weights. It proposes a new algorithm named TVPO, which uses total variance distance rather than the commonly used KL-divergence to restrict the consecutive policy update. They show that this algorithm is convergence and can avoid the suboptimality issue induced by the original KL-regularized algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* A rigorous justification on the assumption of global state is essential to the impact of this work.\n* The presentation should be improved for readers to understand the main contribution of the paper. In particular, the paper should include more discussions on the insight of why using total variation distance. The main technical difficulties for analyzing the algorithm's convergence over the baseline methods (e.g. DPO) should be discussed as well.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KeFh3oFGEV", "reviewer": "Reviewer_tDXb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of fully decentralized policy optimization in MARL is relevant. Particularly, the theoretical problem of convergence is especially relevant and this paper provides a good contribution.\n- The algorithm proposed clearly outperforms the baselines in the experiments presented.\n\nWeaknesses:\n- It is unclear how the paper can show at the same time that (i) in general f-divergence policy optimization in fully decentralized cooperative MARL is limited and may converge to a sub-optimal solution but that (ii) a particular instance of the framework the authors propose, based on total variation, is able to circumvent the limitation. \n- The paper focuses on fully observable settings in MARL, which limits the applicability of the findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes with a general f-divergence policy optimization framework for fully decentralized cooperative MARL, and an algorithm within the framework based on total variation distance. The algorithm is proven to converge to the optimal solution. Additionally, the algorithm outperforms other fully decentralized baselines in several relevant environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most relevant clarifications in order are, in my view, the following:\n- How, being an instance of f-divergence, the algorithm TVPO based on total variation distance, being a particular instance of the f-divergence framework, is able to circumvent the issues exposed earlier;\n- What is meant by \"trivial updates\" throughout the paper;\n- Why is it necessary to introduce different definitions of Q and V functions to propose the TVPO algorithm, and what are the implications of solving for these different definitions rather than the original.\n\nLess relevant issues are:\n- In the introduction, it is mentioned that there are experiences in SMAC V2, which don't appear in the Experiments section;\n- \"allowedto\" should be \"allowed to\" in the Related Work section", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kf0P0hjySz", "reviewer": "Reviewer_LEMY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe paper is well organized and motivated. It takes on the challenge of providing a theoretically principled fully-decentralized independent-learning framework and algorithm for cooperative MARL, whose difficulty is well-acknowledged in the literature. The paper has strong results with clear evidences for their major claims both theoretically and empirically. \n\nWeaknesses: \n\nThe exposition of the work requires some polishing. I am listing the major problems here and I hope that the authors can address these points for improving their work. \n\n1. The paper mentions that they are providing a \"more general\" policy optimization formulation in several places. This is mentioned right in the abstract without any basis which makes it confusing. I am not clear what this is being compared to for the claim that this work is more general. In the introduction it is mentioned that the comparison is to DPO, but only a limitation of DPO is noted (and it is not immediately clear how this paper addresses this limitation). \n\n2. Using the agent index as a subscript and a superscript is confusing. For example, in Section 3, the agent index is used as a superscript for the policy and a subscript for the action. I suggest that the authors pick one and be consistent. \n\n3. There are new definitions introduced at different parts of the paper without a coherent structure. I suggest the authors to use the Definition environment in Latex for each definition (especially the definitions in Page 5 associated with Eq.s 7 and 8). \n\n4. There are certain unclear phrasings in the paper. For example, in page 6, \"definition is well-defined\" (what does that mean?), page 6, \"critical bridge between value functions\" (how?), and \"we usually use Gaussian distribution\" (when and what other distributions are used),  also, in Page 9, \"useful properties like Appendix F.4\" (what properties are referred to?). \n\n5. Several experimental results are close and the authors should perform a statistical significance analysis. Particularly, the results in Fig. 7, Fig 8, Fig. 4(a) requires statistical significance discussions. \n\n6. I find the arguments in the later part of Section 5.1 quite hard to follow. The conclusion is that the empirical results agree with the theoretical derivation on all four iterations. The paper must elucidate more as to which results are being demonstrated here and how the conclusions are inferred. \n\n 7. The placement of Figure 7 seems to misaligned with the rest of the text, with the description placed along the same paragraph as the main text. This needs to be fixed. \n\n\nMinors: \n\nPage 2: allowedto -> allowed to\nPage 4: such IQL -> such as IQL\nPage 8: the experiment contains -> the experiments contain\nPage 11: each agent will control -> each agent controls", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a new independent policy optimization formulation for cooperative MARL, f-divergence policy optimization. Further, a fully decentralized independent learning algorithm, TVPO, is provided along with detailed theoretical and empirical analysis. The empirical analysis covers three cooperative testbeds and a suite of strong baselines, where the advantages of TVPO are demonstrated. This paper's contributions are focussed on the much harder and more general decentralized training and execution setting (as opposed to several centralized training techniques in the literature).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the authors to address the comments in the Weaknesses listed above. \n\nThis is a good paper. I am happy to recommend acceptance of the paper and the changes requested are not critical to securing my recommendation. I am simply providing these comments for improvement.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is correct overall. It is easy to implement and could have practical use cases, as supported by the experiments and ablation studies.
Decision: Accept (poster)