Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities
Authors: Donghao Ying
Yunkai Zhang
Yuhao Ding
Alec Koppel
Javad Lavaei
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning Theory
Safe reinforcement learning
Multi-agent reinforcement learning
TL;DR: This paper develop a provably efficient scalable algorithm for safe MARL with general utilities in the absence of the full observability assumption.

Abstract: We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are described by general utilities, i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-making goals such as risk, exploration, or imitations. The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose a primal-dual method utilizing shadow reward and $\kappa$-hop neighbor truncation under a form of correlation decay property, where $\kappa$ is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of $\mathcal{O}\left(T^{-2/3}\right)$. In the sample-based setting, we demonstrate that, with high probability, our algorithm requires $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$ samples to achieve an $\epsilon$-FOSP with an approximation error of $\mathcal{O}(\phi_0^{2\kappa})$, where $\phi_0\in (0,1)$. Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.

Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities
Donghao Ying
Yunkai Zhang
IEOR Department
IEOR Department UC Berkeley
UC Berkeley
donghaoy@bberkeleyy.ee
shinaiiizhan@@bekkeeey edu
Yuhao Ding IEOR Department UC Berkeley weingndernnneee.....oooo edu
Alec Koppel
Javad Lavaei
Artificial Intelligence Research
IEOR Department
J.P. Morgan
UC Berkeley
alec kww.ekejpmccaaseee.... com
Lavaei@berkeley edu
Abstract
We investigate safe multi-agent reinforcement learning, where agents seek to col- lectively maximize aggregate of local objectives while satisfying their own safety constraints. The objective and constraints are described by general utilities, i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-makinn The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose primal-dual method utilizing shadow reward and x-hop neighbor truncation under form of correlation decay property, where is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of O (1-2 In the sample-based setting, we demonstrate that, with high probability, our algorithm requires Ö (e-3.5) an approximation error O(egr) where do e (0, 1). Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.
goals such as risk, exploration, or imitations.
3.5) samples achieve an e-FOSP with
Introduction
Cooperative multi-agent reinforcement learning (MARL) involves agents operating within shared environment, where each agent's decisions influence not only their objectives, but also those of others and the state trajectories [[]. In secking to bring conceptually sound MARL techniques of simulation [2] 3] and into real-world environments 44 5), some key issues emerge: safety and communications overhead implied by training mechanism. Although experimentally, the centralized training decentralized execution (CTDE) framework has gained traction recently [6] 77) its requirement for centralized data collection can pose issues for large-scale [ or privacy-sensitive applications (9). Therefore, we prioritize decentralized training, where date most MARL techniques impose global state observability for performance certification [I] In this work, we extend recent efforts to alleviate this bottleneck [10] especially in the case of safety critical settings, in a flexible manner that allows agents to incorporate risk, exploration, prior information.
More specifically, we hypothesize that the multi-agent system consists of network of agents that interact with each other locally according to an underlying dependence graph [10]. Second, to model safety constraints in reinforcement learning (RL), adopt standard approach based constrained
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Markov Decision Processes (CMDPs) where one maximizes the expected total reward subject to safety-related constraint on the expected total utility. Third, since many preision-makinge problems take a form beyond the classic cumulative reward, such as apprenticeship learning [12), diverse skill discovery [3], pure exploration [14,, and state marginal matching [15), we focus on utility functions defined as nonlinear functions of the induced state-action occupancy measure, which can be abstracted as RL with general utilities [16
Towards formalizing the approach, we consider an MARL model consisting of n agents, each with its own local state 8; and action a, where the multi-agent system associated with an underlying dependence graph G. Each agent is privately associated with two local general utilities fi(+) and (+),), where f((-) and (+(+) are functions of the local occupancy measure. The objective is to find a safe policy for each agent that maximizes the average of the local objective utilities, namely, 1/n fi and satisfies each agent's 's individual safety constraint described by its local utility (.).. This setting captures a wide range of safety-critical applications, for example, resource allocation for the control of networked epidemic models [18. influence maximization in social networks [9), portfolio optimization in interbank network structures [20], intersection management for connected vehicles [21], and energy constraints of wireless communication networks (22)
Despite the significance of safe MARL with general utilities, prior works have either ignored the necessity of safety (23 or the computational bottleneck associated with global information exchange regarding the state and action per step [24]. In fact, the interaction of these two aspects requires addressing the fact that each agent's 's own safety constraint requires information from all others. In particular, the existing works in safe MARL. allow full access to the global state or unlimited communications among all agents for policy implementation, value estimation, and constraint satisfaction [25 26 27 However, this assumption is impractical due to the "curse of dimensionality" 28 as well as the limited information exchanges and communications among agents (2)].
Therefore, to our knowledge, there is no methodology to both guarantee safety and incur manageable communications overhead for each agent. Compounding these issues is the fact that standard RL training schemes based on the policy gradient theorem [30] are not applicable in the context of general utilities. This deviation from the cumulative rewards adds to the difficulty of estimating the gradient, since there does not exist a -ndleeiineeeeneent reward function. We refer the reader to Appendix A for an extended discussion of related works.
To address these challenges, we focus on the setting of distributed training without global observ- ability and aim develop a scalable algorithm with theoretical guarantees. Our main contributions are summarized below:
Compared with existing theoretical works on safe MARL 25 26 31), we present the first safe MARL formulation that extends beyond cumulative forms in both the objective and constraints. We develop a truncated policy gradient estimator utilizing shadow reward and k-hop policies under form of correlation decay property, where к represents the communication radius. The approximation errors arising from both policy implementation and value estimation are quantified.
Despite of the global coupling of agents' local utility functions, we propose scalable Primal-Dual Actor-Crine Critic method, which allows each agent to update its policy based only on the states and actions of its close neighbors and under limited communications. The effectiveness of the proposed algorithm is verified through numerical experiments.
From the perspective of optimization, we devise new tools to analyze the convergence of the algorithm. In the exact setting, we establish an O (2-2/3) convergence rate for finding an FOSP. matching the standard convergence rate for solving nonconcave-convex saddle point problems. In the sasple-based setting, we prove that, with high probability, the algorithm requires Ö (E (3.55) samples to obtain an e-FOSP with an approximation error of Ool"), where @o e (0,
2 Problem formulation
Consider a Constrained Markov Decision Process (CMDP) over finite state space S and a finite action space A with a discount factor и [0, 1). A policy is a function that specifies the decision rule of the agent, i.e., the agent takes action a ε A with probability x (als) in state s S. When action a is taken, the transition to the next state s' from state follows the probability distribution
2

---

5'~ ..P((ss,a). Let p be the initial distribution. For each policy т and state-action pair (s,a) 6 A. the discounted state cate-action occupancy measure is defined as
X (*
Σ *P = = p)- k=0
The goal of the agent is to find a policy и that maximizes general objective described by (possibly) nonlinear function f(-) of X", known as the general utility, subject to constraint in the form of another general utility g(-), namely
max f(x*) s.t. a
g(1) O.
When f(.) = (r,)) and g(-) (u, +) are linear functions,
2 recovers the standard CMDP problem:
max (=(r)=B Σ r (s*, aa TT 50 ~ P s.t. V (x)= Σ nhu (s* a²))
00
n. 8 ~ 2 0, (3)
where (=()) is usually referred to as the value function. In contrast, it has been shown that for some MDPs, there is standard value function that can be equivalent to the general utility [16 Lemma 1].
In Appendix C we provide more examples of formulation ( beyond standard value functions.
In this work, we study the decentralized version of problem 2). Consider the system is composed of a network of agents associated with a graph G = (N Eg) (not densely connected in general), where the vertex set N = {1,2..... the communication links among the agents. Let d(i, j) be the length of the shortest path between agents and j on G. For x 0, le M₁ (j E [Md(i,))
denotes the set of n agents and the edge set &g prescribes n.j n,)
x} denote the set of agents in the k-hop neighborhood of agent i, with the shorthand notation N == W/W and -i = (i1.. The details of the decentralized nature of the system are summarized below:
Space decomposition The global state and action spaces are the product of local spaces, i.e., S S= S] x S2 x... x Sn,, A= A1 x A2 x... x An. meaning that for every é S and a E A, we can write == (#1, 82, 8n) and a = (ai, a2....; an). For each subset ' cN, we use (#A2, (yy) to denote the state-action pair for the agents in N'.
Observation and communication Each agent only has direct access to its own state 8; and action a,, while being allowed to communicate with its k-hop neighborhood N2 for information exchanges. The communication radius ε given but tunable parameter.
Transition decomposition Given the current global state s and action a, the local states in the next period independently generated, i.e., P(s"Is, a) II (s((sss))) a). s S, where we use IP; to denote the local transition probability for agent i.
Policy factorization The global policy can be expressed as the product of local policies, such that π (aas)) Naw n (ails) (xx a). i.e., given the global state s, each agent i acts independently based on its local policy r¹. We assume that each local policy x is parameterized by a parameter 0, within a convex set θ, Thus, we can write T(ass) = To (als) = Ile Té. (a.(s), where E θ O x O2 x ... x O is the concatenation of local parameters.
Localized objective and constraint For each agent and its local state-action pair as), the local state-action occupancy measure under policy æ is defined as
X (s),
>T (( = s,, s0 p).
(4)
0=
which can be viewed as the marginalization of the global occupancy measure, i.e., X (3; a;) E. ..a X (*(ss)) a). Each agent i is privately associated with two local (general) utilities fi(-) and ge()), which are functions of the local occupancy measure X/ Agents cooperate with each other aiming at maximizing the global objective f()), defined as the average of local utilities (f(()) hev, while each agent i needs to satisfy its own safety constraint described by the local utility g:)). Then, under the parameterization Te. 2 can be rewritten as
max F(0)= = Σ f((0),,, s.t. G,(0) = 9. 20, O, Vi 6cB
(5)
veN
Note that problem 55) is not separable among agents due to the coupling of occupancy measures. Compared to the formulation where the constraint is modeled the average of local constraints, c.g..
3

---

271. 5 is stricter and more interpretable. We emphasize that the method proposed in this paper does
require the relaxation of local constraints in 5 to a joint constraint and directly generalizes to the case of multiple constraints per agent.
Consider the Lagrangian function associated with 5
= ==
p,G [+(())
(6)
n. VeN
ie
where μ E R2 is the Lagrangian multiplier. The Lagrangian formulation (32 of (5) can be written as
max min (1),).)
(7)
de pa0
Since the general utilities f:() and (x55)) may not be non-concave w.r.t. 0 even in the form of cumulative rewards, finding the global optimum to (5) is NP-hard in general (33 Our goal this
work is to develop a scalable and provably efficient gradient-based primal-dual algorithm that can
find the first-order stationary points of (5)
Scalable primal-dual actor-critic method
For standard value function with the reward e R denoted (F) (r, the policy gradient theorem (see Lemma D.I yields that
VoV VOVTO (r) = VaA=
1-7 Ba-d"o B,ad" a--o(--e)) Vo To (a(ss) Q (r;s,u)],
where dd (s) = (1-?) LOeA ( a) is the discounted state occupancy measure, Vo log TT(+)) is the score function, and Q (r:..)) the Q-function with the reward r. defined
(s)((s)
QTa a) = Σ 7 & (s* a²) Te, S = = s,a =
(8)
Although this elegant result no longer holds for general utilities, we can apply the chain rule:
Vof(A")) [v>f(A=*)]] VoA"0
VoV" (Vxf(A"))).
(9)
i.e. the gradient (A is equal to the policy gradient of standard value function with the reward (xJ(>*)). We introduce the following definitions (23 for the distributed problem (
Definition 3.1 (Shadow reward and shadow Q-function). For each agent define y To == Vx.f(AAT))
||S(|||A:l as the (local) shadow reward for the utility f((.) under policy πe. Define Q (s,a) = QTa (r) ss,a) a) as the associated (local) shadow Q-function for f((-). Similarly, re and Q71 (s.a) be the shadow reward and the fuunctio for gi(-).
Combining Definition 3.1 with 9h. can write the local gradient for agent i, i.e., Vo, Vo, (1) as
Vo, Vo,
Rg-d (13) Vo; log Tà. (agls)
Σ (0 (s. (s, JeN
(10)
1-7
(a.(s). By (1)) cach where we apply the policy factorization to arrive at Vs, log (als)) = Vo, log n. [ agent needs to know the shadow Q functions of all agents, well the global state, to evaluate its own gradient. However, especially in large networks, this is both inefficient, due to the communication cost, and impractical because of the limited communication radius. In the remainder of this section, we aim to design a scalable estimator for Ve, [(0, 1)) that requires only local communications.
3.1 Spatial correlation decay and sshoop policies
Inspired by 34], we assume that the transition probability satisfies a form of the spatial correlation decay property [35] 36).
Assumption 3.2. For a mairix M e Raxn whose (i.j)-th entry is defined as
sup Mi of s--3,
|PP (-1s,, 8-j, aj, a.)) P: (11/g, s- d'y, a.) a.))
(11)
assume that there exists 0 such that maxieN EjeN wdd(t.)) Mij S x with x < 2/7, where y the discount factor.

---

The value of Ms reflects the extent to which agent j's state and action influence the local transition probability of agent i. Thus, Assumption 3.2 amounts to requiring this influence decrease exponen- tially with the distance between any two agents. Such a decay is often observed in many large-scale real-world systems, e.g., the strength of signals decreases exponentially with distance (37]
Furthermore, as mentioned earlier, the impiementation of the local policy Tे, (-Is) is still impractical, since requires access to the global state s, while the allowable communication radius limited To alleviate this issue, we focus on a specific class of policies in which the local policy of agent i only depends on the states of these agents in its k-hop neighborhood NT This class of policies is also referred to as k-hop policies in the concurrent work (38).
Assumption 3.3 (x-hop policies). For each agent i E N and 0 E θ the local policy Tb, (416 depends only on the neighbor states SNT, i.e,
To, (-1,, SN To. (1INN,, sw:;), vs E S and Vsya & &SN
(12)
For simplicity, we use the notation oú (1s) Tb. (18N7=) for x-hop policies when it is clear from context. We note that, for any original policy function (-Iss) an induced k-hop policy fo(-)s can be defined by fixing the states sNe, to some arbitrary values and focusing only on the states of agents in Aa When considering only k-hop policies, it is essential to understand how much information is lost compared to the case where agents have access to the global states. The following proposition quantifies the maximum information loss terms of the occupancy measure under the assumption that the original policy function also satisfies a spatial correlation decay property.
Proposition 3.4. Suppose that there exist e 2 O and Φ E (0,1) such that for every 0 E θ, agent i E N, and states 8,8' S such that SNp- = =s'y, we have |it) (-16) Tè, (-1s')] S co" Let a. be an induced k-hop policy of ng. Then, it holds that
nco
| 1,0
(1 (1- (1- Vie
(13)
The condition on the local policy in Proposition 3.4 encodes that every тे. is exponentially less sensitive to the states of agents outside NV,", which is common assumption in MARL to alleviate computationally burdensome and practically intractable communication requirements imposed by the global observability [34 39 (3). By Proposition [3.4] the difference in occupancy measures under na and कt is controlled by T To.
#t |1. Therefore, if (A*)) and gi( are Lipschitz continuous w.r.t.
X=, Proposition 3.4 implies an (e²") approximation of the Lagrangian function (6) using k-hop policies. The faster the spatial decay of policy is, the more accurate the approximation of the x-hop
policy is. This justifies our focus on learning k-hop policy.
3.2 Truncated policy gradient estimator
In the absence of global observability, it is critical to find a scalable estimator for the local gradient Ve, L(0, (1) in (IO), so that each agent can update its local policy with limited communications.
By leveraging the similar idea in the definition of k-hop policies, we define the k-hop truncated (shadow) Q-function, denoted as Q8. SAçe Avr I, to be
(SN (N;)) Q (SN; SNC ONT @wz)). ((SN;,ox; E SNY x ANp, o (f.g), (14)
where (ă a is any fixed state- action pair for the agents in No. Now, we introduce the following truncated policy gradient estimator for agent
Vo, Ré (arleN;) 1 Σ (07, (sNy. aN;) (ony:aN;)) njENN"
p 0 (ONy an;)) (15)
Vo.((0,
E Ennang 1- a-tp(lo)
In comparison to the true policy gradient (O).. (o, 11) replaces the shadow f-funtioons with their truncated versions and only considers the agents the h-hop neighborhood A²" Surprisingly, the following lemma shows that the approximation error of Vo, (1) decreases exponentially with when the shadow rewards and the score functions are bounded.
5

---

Lemma 3.5. Suppose that Assumptions 3.2 and 3.3 hold and there exist Mr Ma 0 such that
Iooe & M, and Vo, log mblll2 s Ma. for every o (f.9), 0 G, i N. Then, for all A e O
2 N, we have that
po, (o, p)) (1 Iullow )M, (
(16)
1-7
where co = 2, 7x) and co = E.
Recall that the shadow reward is defined the gradient of fi(.) or gi(-) w.r.t. the local occupancy measure. Since the set of all possible occupancy measures is compact (see (43). the existence of M, in Lemma 3.5 is satisfied if f((+) and (+() are continuously differentiable. The main advantage of using the estimator Ve, Fe, (1) lies in that every agent i only needs to know the truncated Q-functions of agents in its neighborhood N&" which can significantly reduce the communication burden and the storage requirement when graph G is not densely connected. The proof of Lemma 3.5
can be found in Appendix E.2
3.3 Algorithm design
Using the results of the preceding section, we put together all the pieces and propose the Primal-Diee Actor-Critic Method with Shadow Reward and k-hop Policy, as outlined in Algorithm [] It includes three stages: policy evaluation by the critic, Lagrangian multiplier update, and policy update by the actor. Below, we provide an overview of Algorithm | while referring the reader to Appendix Dfoor a flow diagram (Figure 2) of the algorithm, as well as more detailed discussion.
Stage 1 (policy evaluation by the critic, lines 3-6) In each iteration f, the current policy Ta. is simulated to generate a batch of trajectories, while each agent i collects its neighborhood trajectories, i.e. the state-action pairs of the agents in Nº as batch B" Then, the batch is used to estimate the local occupancy measures X, through (17), which are subsequently applied to compute the ret empirical values for the constraint function gi (A, Hat and shadow rewards r/; and Tat denoted as F. F) and Fr. respectively. It is worth mentioning that, when all utility functions reduce the form of cumulative rewards, the above operation is unnecessary, since all agents have policy-independent local reward functions.
Next, the agents jointly conduct a distributed evaluation subroutine to estimate their truncated shadow Q-functions (Q%." hEN using empirical shadow rewards { New, where o E g). During the subroutine, each agent communicates with its neighbor in N to exchange state-action information, but only needs to access its own empirical shadow reward P, In principle, any existing approach that satisfies the observation and communication requirements can be used for the truncated Q-function estimation, such as 44 41 42). As an example subroutine, we introduce the Temporal Difference (TD) learning method 43 which is outlined as Algorithm ( Appendix [D]
Stage 2 (Lagrangian multiplier update, line 7) Instead of employing the projected gradient descent, we propose update the dual variables by the following formula:
p 1+ argmin L(O', ,1,) + 27p
1m112 Pu (nh(', ,R)),
(22)
pall
where weight T can be viewed as the dual "step-size". In practice, we replace the true dual gradient Vy, L('', (a') gi (x(2,a) (x5"*) )/n with its empirical estimator Vp, ,I)). The feasible region for the dual variable is denoted by 2/ s R. and will be specified later.
Stage 3 (policy update by the actor, lines 8-9) To perform the policy update, each agent i first shares its updated dual variable 14*1 and the values of its estimated truncated Q-functions along the trajectories in batch B with the agents in its k-hop neighborhood N?" Then, the agent estimates its
truncated policy gradient V C(6", 1 through a REINFORCE. based mechanism 44 as described in (20). Finally, each agent updates its local policy parameter by projected gradient ascent.
We emphasize that Algorithm[]] Tis is based on the distributed training regime and does not require full observability of global states and actions.

---

Algorithm Primal-Dual Actor-Critic Method with Shadow Reward and k-hop Policy
1: Input: Initial policy 0 and dual variable nº; initial distribution pi communication radius R; step-sizes and Mly batch size B; episode length H.
2: for iteration t = 0, ... do 3: Sample B trajectories with length H under the k-hop policy The and initial distribution Each agent collects its neighborhood trajectories T {(82 (158),,ar,,) oa.), .., H- Nxx= ')} batch B
as
4: Each agent estimates its local occupancy measure xn under Tge:
#-1 1, (z) RRSHA
X =
ΣΣΣΣ TEB
(17)
5: Each agent computes the empirical constraint function value 3 =g.(.) and empirical shadow rewards 74 (X)) and F VV).9(A))). 6: Each agent communicates with its neighborhood Af and jointly executes evaluation subroutine to estimate the truncated shadow Q-functions with the empirical shadow rewards Po, for (f.g):
Eval (Tas,, (o...,......))) 7: Each agent updates the dual variable with the empirical gradient V C(0', ,²)
(18)
4 1+1 Pu (- C(0' ,,)).
(19)
8: Each agent shares 1 f+1 and values of Q/ Ty along the trajectories in B with agents N and estimates the truncated policy gradient at (0*, (0',p²*) 1)
Σ
Σ R=0
?
Vo, log TØ. (a (x7))
B reB
(20)
Σ n j&N
1
[ (oNga az²z) 11, Vy (Exge, exg))
9: Each agent updates the local policy parameter:
0.011 Pe: (0) no Vo, ,00)))))))))
(21)
10: end for
4
Convergence analysis
In this section, we analyze the convergence behavior and the sample complexity of Algorithm [ We begin by summarizing the technical assumptions, including some mentioned previously in the paper. We direct the reader to Appendices and where we provide discussions for each assumption and present proofs for the results in this section.
Assumption 4.1. There exists LA 0 such that f((-) and vx.9i()) are Lx -ipsschti continuous w.r. A i.e., 2112. N.
LAI>; M,12 and [Vx.9(())) - L
Assumption 4.2. The parameterized policy лa is such that (I) the score function is bounded, i.e. 3M, > S.t. V log =े (ailswg) )l 5 M,, (ss.a) S A, 0 θ N. (II) 3gg s.t. the utility functions F(0)= = f(x") and G(( gi (X77) are La-smooth w.r.t. 0, V.
Assumption 4.3. There exist an FOSP (0' "p' of (5) and constant μ s.I. P < <.. VieN. Let -= [
In Lemma
we summarize a few properties that are the direct consequence consequence of
Assumptions 4.1 4.3 Due to the non-concavity of problem (5), our focus is to find an approximate
7

---

first-order stationary point (FOSP). A point (0, θ x U is said to be an c-FOSP if
Σ E (4) = [x n)] [)(0, sc,
(23)
where the metrics ((:)) and -):.)) are defined as
X(0, p)) = max 0' (V (0) - 0) p1), =- min (V,C(0,)), -1). (24)
p'ex.la'-wla -12s1
The definitions of X (::)) and -)..)) are based on the first-order optimality condition 45 46 Given
O* e O and n² E 24, it can be shown that E(0 , 0 implies that (0", (0", ís an FOSP of 5 (see Lemma [.6). In the following, we first consider the exact setting where the agents can obtain the true values of their local occupancy measures, shadow Q-functions, and truncated policy gradients. Therefore, the only source of approximation error is the truncation of the policy gradient.
Theorem 4.4 (Exact setting). Let Assumptions 3.2
4.1 4.3 hold and suppose that the agents can accurately estimate their local occupancy measures, shadow Q-functions, and truncated policy gradients. For every T > 0, let ((',0')) t=o the sequence generated by Algorithm with
Tu - 13333 and n7 1/(Loo 44 where Loo; Lap are Lipschitz constants defined in Lemma F5 Then, there exists t" E (0.1,...,I T- 13 such that
E E(00 1 +1) (13 + o
)o?*).
(25)
Next, we delve into the sample complexity of Algorithm T For theoretical analysis, we assume that the estimation process for the truncated Q-function offers an approximation to the true function, with x|Axc= the error being associated with the magnitude of the reward function. Let TTe (ra...) :) |xA be the truncated Q-function with the reward function ri(:s-) E [R x|A, for agent i N. Assumption 4.5. For every reward function ri r:(:,-) and 6o > O, the subroutine computes an approxi- mation O72² (rsi.-) to the truncated Q-function Q,7' (ra:--) such that
@ (r:::.)) QT. (r $ C/
(26)
with (1)(E))))) samples, for every i E N,,, O.
We comment that the sample complexity of the truncated Q-function evaluation described in Assump- tion 4.5 is not restrictive. It can be achieved with high probability by the TD-Iearning procedure outlined in Algorithm 2 when the agents have enough exploration [10 43). For brevity, we assume that (66 holds almost surely. The only difference in the probabilistic version would be the presence of an additional term for the failure probability, which does not affect the order of the sample complexity.
Theorem 4.6 (Sample-bas based setting). Suppose that Assumptions 3.2 3.3
4.3 and 4.5 hold.
For every E >> O and 8 (0,1). let (²)) be the sequence generated by Algorithm 1 with T O (c-1.),, Mu = (c no = 1/(Lee + AL2 co O (VE). 6, = (2nn(T 1)), batch size B O (1g((1)))) ), episode length H log(1/c), where Lgo. Lau. are Lipschitz constants defined in Lemma F.5 Then, with probability 1 5, there exists (0, 1,... T 1} such that
8(0", .u +1) O (c)++((8))
(27)
The required number of samples is
4.1 Technical discussions
Theorem 4.4
implies an O 2T3 iteration complexity of Algorithm 1 matching the fastest conver- gence rate for solving nonconcave-convex maximin problems in the literature [47]. The approximation error (Ø decays at a linear rate w.r.t. the radius of communications. Thus, as long as the underly- ing network not densely connected, such as those in wireless communication (37] and autonomous
driving [48), an approximate FOSP to (5) can be efficiently computed, while each agent only needs
communicate with a small number of agents in its neighborhood...
In Theorem 4.4 we have chosen large step-sizes for the dual variable update to achieve the best convergence rate. This aggressive update ensures that the dual metrie y00', ,1*+1) always remains
8

---

within a small range and also provides satisfactory ascent direction for the policy update. Then, the average primal metric 1/T-E [x (0', "==,) upper-bounded by exploiting a recursive relation between any two consecutive dual updates. Hence, the existence of point (0" ,,'+') that satisfies (25) is guaranteed. It is worth noting that the proof of Theorem 4.4 can be easily generalized to the scenario where 7 is unspecified, and the same convergence rate can still be achieved with adaptive step-sizes z O (11/) and no 1/(Lgg o1.))
Theorem 4.6 states that, with high probability, Algorithm has an ( sample complexity for finding an e-POSP of (5) with an approximation error O ( Note that we absorb the logarithmic terms in the notation (-). The proof of Theorem 4.6 can be broken down into two parts. Firstly, we evaluate the approximation errors of the estimators used in Algorithm in relation to the model pa- rameters, outlined in Proposition G.I] Then, integrate these errors into the iteration complexity result established in Theorem 4.4 and optimize the selection of parameters.
5 Numerical experiment
In this section, we validate Algorithm
1 via numerical experiments, focusing on three key questions
How does Algorithm D
perform with multiple agents, and does the policy gradient truncation effectively alleviate computational load?
While Algorithm is the first approach that provably solves the safe MARL problem with general utilities, how does it compare with existing methods for standard Safe MARL?
What benefits does the use of general utilities offer over standard cumulative rewards?
To answer these questions, we performed multiple experiments in three environment The objective functions based cumulative rewards, while constraint functions leverage general utilities to incentivize or dissuade agents from exploring the environments.
Synthetic environment Analogous to 24 Section 5. where agents are linearly arranged as 1-2-...- n. Each agent i has binary local state and action spaces, i.e., S, A, (0, 1}, and the local transition matrix P. depends solely its action and the state of agent + 1. The reward functions are constructed such that the optimal unconstrained policy compels all agents to continuously choose action 1, irrespective of their states.
Pistonball A physics- physecs-based based game that emphasizes cooperations and high-dimensinn states as illustrated in Figure Ta Each piston represents an agent, where its local neighborhood includes adjacent pistons, and the goal is to collectively move the ball from right to left. The agent can move up, down, or remain still. We modify the original game 49 so that the agent can only observe the ball when it enters the local neighborhood, as well as the height of neighboring pistons.
Wireless communication An access control problem following similar setup as in 124 50). As illustrated in Figure [b] the agents try to transmit packets common access points, and the transmission fails if the access point receives more than one packet simultaneously. As there are more agents than access points, some agents need to learn to forego their benefits for the collective good.
In addition to the objective, we incorporate two types of safety constraints characterized by general utilities that cannot be easily encapsulated by standard value functions based cumulative rewards.
Entropy constraints that stimulates exploration, formalized Entropy (A7") c, Vi W. The function Entropy (A50) represents the local entropy, defined as Ess df (s) log (df (s)). where d.ds (s:) = (1 (1-7) En,eA, X,5 (si,a,) is the local state occupancy measure.
lz-constaains that deter agents from learning overly randomized policies, formulated as E..es, 11 2 W. This constraint is beneficial in applications like autonomous driving and human- Al collaboration, where an agent's policy needs to predictable for other agents.
http:///e/...cacaaaaaaaaaaaaaaaa "See Appendix H for detailed descriptions and complete experimental results.
'Code is available here:

---

kappa
kappo
0.6
0.4
-00
IIb
user
access point
Pistonball
(b) Wireless comm.
(c) Episodic return
(d) Constraint violation
Figure 1:
(a.b) Environment illustration. (c.d) Performance of Algorithm
| in Pistonball with 20
agents under entropy constraints.
Table 1:
Comparison between Scalable Primal-Dual Actor-Critic method in work with MAPPO-L by 31 in Pistonball and wireless communication.
Pistonball
Wireless Communication Episodic return Const. vio.
Algorithm
Episodic return
Const. vio.
Ours
51.788 1.346
0.04919
3.373 # 0.112
0.1926
MAPPO-L
50.612 2.118
0.06884
3.347 # 0.131
0.4000
Decen. Agg. MAPPO-L
48.197 + 6.188
0,2179
3.106 = 0.673
1.1890
Decen. MAPPO-L
41.102 = 18.769
0.09303
3.148
1.5760
In Figure [ we demonstrate the performance of Algorithm in the 20-agent Pistonball environment under entropy constraints. We observe that, while the truncation with x 3 converges in fewer itera- tions, truncation with ς 1 also yields comparable performance. This underscores the efficiency of Algorithm| I as employing smaller communication radius can significantly reduce the computation.
Finally, compare Algorithm T with three baselines based the MAPPO-Lagrangian method B]
MAPPO-L: the original algorithm introduced in 31 Note that each agent has access to glohal information.
Decentralized MAPPO-I.: decentralized version of MAPPO-L., where each agent only has access to information in the local neighborhood. However, since each agent trained to greedily maximize its individual reward, its behaviors might sacrifice the performance of other agents.
Decentralized Aggregate MAPPO-L: decentralized version of MAPPO-L, where we address the aforementioned issue by redefining each agent's reward be the sum of rewards of all agents in its local neighborhood.
For fair comparison, we consider two standard safe MARL problems, where both objectives and constraints are shaped by cumulative rewards (see Appendix H.4). The results in Table [ demonstrate that our method consistently outperforms both the centralized and decentralized variants of MAPPO- Lagrangian. We refer the readers to Appendix H for the comprehensive experimental results that fully answer the three questions raised at the beginning of this section.
Conclusion
In this work, we study the safe MARL with general utilities, with focus on the setting of distributed training without global observability. To address the challenge of scalability and incorporating general utilities, we propose a primal-dual actor-critic method with shadow reward and k-hop policy. Taking advantage of the spatial correlation decay property of the transition dynamics, proposed method achieves O (T-2/)) convergence rate to the FOSP of the problem in the exact setting and achieves an ((-3..) (355) sample complexity, with high probability, in the sample-base setting. Finally, the effectiveness of model and approach is verified by numerical studies. For future research, it would be interesting to develop scalable safe MARL algorithms with adaptive communication of agents' information [5]] and intelligent sampling of agents' trajectories.
show that the
10

---

Acknowledgement
This work was supported by grants from ARO, ONR, AFOSR, NSF, and the UC Noyce Initiative.
References
[1] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview theories and algorithms. Handbook of Reinforcement Leurning and Control, pages 321-384, 2021.
[2] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[3] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level starcraft using multi-agent reinforcement learning. Nature, 575(7782) 2019.
[4] Shai Shalev-Shwarte Shaked Shammah, and Amnon Shashua, Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295. 2016.
[5] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1 17(1):1334-1373 2016.
[6] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in newral information processing systems, 33:1 13:99-11100,,, 2020.
[7] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre- Alexan-Ae Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 12:102-8-122 12221, 2021.
[8] Md Shirajum Munir, Nguyen H Tran, Walid Saad, and Choong Seon Hong. Multi-agent meta-reinforreemnnt learning for self-powered and sustainable edge computing systems. IEEE Transactions on Network and Service Management, 1803):335333374 2021.
[9] Selim Amrouni, Aymeric Moulin, Jared Vann, Svitlana Vyetrenko, Tucker Balch, and Manuela Veloso. Abides-gym: gym environments for multi-agent discrete event simulation and applica- tion to financial markets. In Proceedings of the Second ACM International Conference on Al in Finance, pages 1-9, 2021.
(10) Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control, pages 256-266. PMLR, 2020.
[11] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
[12] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1, 2004.
[13] Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv: 1802.06077 2018.
[14] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pages 2681-2691. PMLR, 2019.
[15] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 10X1v:1906.05274.00
2019. 2019.

---

[16] Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances Neural Information Processing Systems, 34, 2021.
[17] Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems, 33:4572 4582, 2020.
[18] Cameron Nowzari, Victor M Preciado, and George J Pappas. Optimal resource allocation for control of networked epidemic models. IEEE Transactions on Control of Network Systems, 1(2):199-169, 2015.
[19] Wei Chen, Alex Collins, Rachel Cummings, Te Ke, Zhenming Liu, David Rincon, Xiaorui Sun, Yajun Wang. Wei Wei, and Yifei Yuan. Influence maximization in social networks when negative opinions may emerge and propagate. In Proceedings of the 2011 siam international conference on data mining, pages 390. SIAM, 2011.
[20] Co-Pierre Georg. The effect of the interbank network structure on contagion and common shocks. Journal of Banking & Finance, 377)):216-2228, 2013.
[21] Qiu Jin, Guoyuan Wu, Kanok Boriboonsomsin,, and Matthew Barth. Platoon-based multi-agent intersection management for connected vehicle. In 16th international ieee conference on intelligent transportation systems (itsc 2013), pages 1462-1467. IEEE, 2013.
[22] Andrea Goldsmith and Stephen B Wicker. Design challenges for energy constrained ad hoc wireless networks. IEEE wireless commumications, 9(4):8-27, 2002.
[23] Junyu Zhang, Amrit Singh Bedi, Mengdi Wang. and Alec Koppel. Marl with general utilities via decentralized shadow reward actor-critic. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[24] Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multiagent networked systems. Operations Research, (0(6):301-36255 2022.
[25] Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Başar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8767-8775, 2021.
[26] Washim Uddin Mondal, Vaneet Aggarwal, and Satish V Ukkusuri. Mean-field approxima- tion of cooperative constrained multi-agent reinforcement learning (cmarl). arXiv preprint 2xXiv22009.07337, 2022.
[27] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo R Jovanovic. Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning.
pdf. 2023.
[28] Vincent D Blondel and John N Tsitsiklis. A survey of computational complexity results in systems and control. Automatica, 3699):149--1274, 2000.
[29] Michael Rotkowitz and Sanjay Lall. A characterization of convex problems in decentralized control. IEEE transactions on Automatic Control, 50811))19844111111 1996, 2005.
[30] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[31] Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint 10.0022 2021.
[32] Dimitri P Bertsekas. Nonlincar programming. Journal of the Operational Research Society, 48(3):334-333 1997.
[33] Katta Murty and Santosh Kabadi. Some np-complete problems in quadratic and nonlinear programming. Mathematical Programming: Series A and B, 19(2):117-129, 1987.
12

---

[34] Carlo Alfano and Patrick Rebeschini. Dimension tarension--fre free rates for natural policy gradient in aulti-agent reinforcement learning. arXiv preprint arXiv:2109. 190.2299..169,, 2021.
[35] Hans-Otto Georgii. Gibbs measures and phase transitions. In Gibbs Measures and Phase Transitions, de Gruyter, 2011.
[36] David Gamarnik. Correlation decay method for decision, optimization, and inference in large- scale networks. In Theory Driven by Inftuential Applications, pages 108-121. INFORMS, 2013.
[37] David Tse and Pramod Viswanath. Fundamentals of wireless communication. Cambridge university press, 2005.
[38] Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. arXiv preprint arXiv:2211
2022.
[39] Sungho Shin, Yiheng Lin, Guannan Qu, Adam Wierman, and Mihai Anitescu. Near-optimal distributed linear- quadratuc regulator for networked systems. arXiv preprint arXiv:2204.05551, 2022.
[40] Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning, 91(3):325 -091-3499940 2013.
[41] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnosti estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
[42] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asyn- chronous q-learning: Sharper analysis and variance reduction. Advances in neural information processing systems, 33:7031-7043, 2020.
[43] Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in stochastic networked systems. Advances in Neural Information Processing Systems, 1417825-7831 2021.
[44] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-256, 1992.
[45] Andrew R Conn, Nicholas IM Gould, and Philippe L Toint. Trust region methods. SIAM, 2000.
[46] Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32, 2019.
[47] Quoc Tran Dinh, Deyi Liu, and Lam Nguyen. Hybrid variance-reduced sgd algorithms for minimax problems with nonconvex- -inear function. Advances in Neural Information Processing Systems, 33:1 1099-11107, 2020.
[48] Jiadai Wang, Jiajia Liu, and Nei Kato. Networking and communications in autonomous driving: survey. IEEE Communications Surveys & Tutorials, 21(2):1243-1274, 2018.
[49] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:15032-15043, 2021.
[50] Xin Liu, Honghao Wei, and Lei Ying. Scalable and sample efficient distributed policy gradient algorithms in multi-agent networked systems. arXiv preprint arXiv:22/2.063357 2022.
[51] Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning, arXiv preprint 1110:::81100000000 2018.
13

---

[52] Santiago Paternain, Miguel Calvo-Fullane, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods, arXiv preprint 19111.1911.09101, 2019.
[53] Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning, Advances in Neural Information Processing Systems, 32:3127- 3139, 2019.
[54] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforce- ment learning. arXiv preprini arXiv:
2019.
[55] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[56] Frits De Nijs, Erwin Walraven, Mathijs De Weerdt, and Matthijs Spaan. Constrained multiagent markov decision processes: A taxonomy of problems and algorithms. Journal of Artificial Intelligence Research, 70:955-1001, 2021.
[57] Yonathan Éfroni, Shie Mannor, and Matteo Pirotta. Exploration-explootttiinn in constrained MDPs. arXiv preprint arXiv:2003.02/888 2020.
[58] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. International Conference on Artificial Intelligence and Statistics, pages 3304-3312. PMLR, 2021.
[59] Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained MDPs. arXiv preprint arXiv:2 106.02684, 2021.
[60] Dongaao Ying, Yuhao Ding, and Javad Lavaei. A dual approach to constrained markov decision processes with entropy regularization. In International Conference on Artificial Intelligence and Statistics, pages 1887-1909. PMLR, 2022.
[61] Donghao Ying, Mengzi Guo, Yuhao Ding, Javad Lavaei, et al. Policy-based primal-dual methods for convex constrained markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[62] Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. arXiv preprint arXiv:2201. 2022.
[63] Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3682-3689, 2022.
[64] Eitan Altman and Adam Shwartz. Constrained markov games: Nash equilibria. In Advances in dynamic games and applications, pages 213-221. Springer, 2000.
[65] E Gómez-Ramiee K Najim, and AS Poznyak. Saddle point calculation for constrained finite markov chains. Journal of Economic Dynamics and Control, 27(10)) 2003.
[66] Eitan Altman, Konstantin Avrachenkov, Nicolas Bonneau, Merouane Debbah, Rachid El-A El-Azouzi, and Daniel Sadoc Menasche. Constrained cost-coupled stochastic games with independent state processes. Operations Research Letters, 36(2):160-164, 2008
[67] Vikas Vikram Singh and N Hemachandra. A characterization of stationary nash equilibria of constrained stochastic games with independent state processes. Operations Research Letters, 42(1):48-52, 2014.
[68] Vinayaka G Yaji and Shalabh Bhatnagar. Necessary and sufficient conditions for optimality in constrained general sum stochastic games. Systems Control Letters, 85:8-15, 2015.
[69] Qingda Wei. Constrained expected average stochastic games for continuous-time jump processes. Applied Mathematics & Optimization, 1373)11277 2021.
14

---

[70] Wenzhao Zhang and Xiaolong Zou. Constrained average stochastic games with continuous-time independent state processes. Optimization, 7 71(9):2511-2594 2022.
[71] Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems, 34, 2021.
[72] Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: the mean-field game viewpoint. arXiv preprint arXiv:2106.0372 2021.
[73] Donghao Ying, Yuhao Ding, Alec Koppel, and Javad Lavaci. Scalable multi-agent reinforcement learning with general utilities. American Control Conference, 2023.
[74] Weichao Zhou and Wenchao Li. Saae apprenticeship learning- In International Conference Computer Aided Verification, pages 662-680. Springer. 2018.
[75] Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire. Reinforcement learning with convex constraints. Advances in Neural Information Processing Systems, 32, 2019.
[76] Qisong Yang and Matthijs TJ Spaan. Cem: Constrained entropy maximization for task-agnostic safe exploration. In The Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023.
[77] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[78] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[79] Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actor- critic method for reinforcement learning with function approximation. arXiv preprint 1rXiv:1910008412 2019.
[80] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-77, 2021.
[81] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[82] Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems, 33:8378-8390 2020.
[83] Gal Dalal, Balazs Szorenyi, and Gugan Thoppe. A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3701-3708, 2020.
[84] Maxim Kaledin, Eric Moulines, Alexey Naumov, Vladislav Tadic, and Hoi-To Wai. Finite time analysis of linear two-timescale stochastic approximation with markovian noise. In Conference on Learning Theory, pages 2144-2203, PMLR, 2020.
[85] Tengyu Xu and Yingbin Liang. Sample complexity bounds for two timescale value-bases based reinforcement learning algorithms. In International Conference Artificial Intelligence and Statistics, pages 81 811-819. PMLR, 2021.
[86] Vivek S Borkar and Sarath Pattathil. Concentration bounds for two time scale stochastic approximation. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 504-511. IEEE, 2018.
[87] Shuang Qiu, Zhuoran Yang, Xiaohan Wei, Jieping Ye, and Zhaoran Wang. Single-timescale stochastic nonconvex-coneat optimization for smooth nonlinear td learning. arXiv preprint 2010:03008.1003. 2020.
15

---

[88] Jonas Moritz Kohler and Aurelien Lucchi. Sub- sub-sampled cubic regularization for non-convex optimization. International Conference on Machine Learning. pages 1895-1904. PMLR. 2017.
[89] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4))2563- 2578, 2022.
[90] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
16


Reviews:
[{"review_id": "kH37XuqwvN", "reviewer": "Reviewer_fkzS", "summary": "This paper considers the problem of safe/constrained multi-agent reinforcement learning (MARL) with general utilities and constraints, which could be generic functions of agents' occupancy measures. A distributed method is proposed for solving the constrained MARL problem with local constraints, where each agent can communicate to its few-hop neighbors in the underlying network graph topology. The proposed algorithm is theoretically analyzed in terms of convergence behavior and sample complexity, and numerical results are provided to demonstrate the efficacy of the proposed method in multiple constrained MARL environments.", "questions": "- I believe the Transition decomposition assumption in lines 106-108 is very strong. Could you please comment on how you justify this assumption in practical settings and whether it could be relaxed to a weaker assumption?\n- How does the proposed method extend to the case where the objective in Eq. (5) is not separable?\n- Intuitively, the upper bound in Eq. (13) should depend on $\\omega$ and $\\chi$ in Assumption 3.2, so please discuss why they do not appear here.\n- In Eq. (15), could $1/n$ be replaced with $1/|\\mathcal{N}_i^{\\kappa}|$? I understand that this is simply a scaling factor, but truncating the policies to $\\kappa$-hop neighbors could also justify the aforementioned replacement, where the Lagrangian is always *averaged* over the $\\kappa$-hop neighborhood of each agent.\n- Does Fig. 1-d show that the constraints are always violated? Does this mean that no feasible policy in this environment might exist?\n- In the Appendix, it is shown that MAPPO-L performs worse than the proposed method. Could you provide a more in-depth discussion of why this happens? As the authors allude to, MAPPO-L has access to the global state during the centralized training phase, so it is okay (and probably expected) if the proposed method (which is fully distributed) underperforms MAPPO-L.\n- Since a graph structure and distributed implementation are the underpinnings of the proposed method, how does the proposed method related to graph neural networks (GNNs)? The agents could perform message passing to their neighbors through a GNN architecture, so it would be helpful if the authors could discuss potential connections of their method to MARL methods based on GNNs, such as the following reference [A]:  \n[A] Jiang, Jiechuan, Chen Dun, Tiejun Huang, and Zongqing Lu. \"Graph Convolutional Reinforcement Learning.\" In International Conference on Learning Representations. 2019.", "limitations": "As mentioned under **Weaknesses**, the extension to non-tabular MARL settings is unclear. Furthermore, the robustness of the proposed method to imperfect communication among agents (e.g., noise perturbations, random communication link drops, delays, time/frequency resource constraints, etc.) needs to be studied.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of constrained/safe MARL is of great practical interest and importance, so the paper focuses on a timely topic of interest to the NeurIPS audience.\n- Distributed training of safe RL policies in multi-agent settings is of significance, and the paper makes important contributions in this direction.\n- Extensive theoretical and numerical analysis is provided to showcase the benefits of the proposed approach.", "weaknesses": "- The presentation and structure of the paper could be improved, especially since many of the important parts of the paper (especially the algorithm pseudo-code, flowchart, and numerical results) are deferred to the appendix. Mindful of the space limitations, it would be great if the authors could shuffle the contents in their revision to include the pseudo-code/flowchart and more numerical results in the main body of the paper, with at least one result for each environment introduced in Section 5.\n- The authors mention [25] (Lu et al., AAAI'21) in their discussion of related work, but I would have liked a more extensive discussion on the similarities and differences between this work and [25], especially since they both consider an underlying graph topology, as well as numerical comparisons with [25] in addition to MAPPO-Lagrangian.\n- The paper mostly focuses on tabular MARL settings with a finite state space. It is unclear how the proposed method generalizes to continuous state spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. We hope our responses address your concerns and provide further clarity.\n> (Weakness) The presentation ... in Section 5.\n\n> (Weakness) The paper mostly focuses ... continuous state spaces.\n\n> (Question) How does the proposed method extend to the case where the objective in Eq. (5) is not separable?\n\n> (Limitation) The robustness of the proposed method ... needs to be studied.\n\nA: We kindly refer the reviewer to our **General Response** for detailed discussions on your comments.\n### Weaknesses\n> The authors mention [25] ... in addition to MAPPO-L.\n\nA: We are pleased to offer a more detailed comparison.\n* [25] assumes global observability, where all agents share a common state and each agent's actions and states are universally known. They use a consensus-based algorithm, where each agent maintains an estimation for the global objective and averages it with its neighbors. The averaging mechanism (consensus update) is determined by the underlying graph. Our approach is more practical: agents have local states and only observe nearby agents, with distances defined by the associated graph.\n* [25] focuses on cumulative return and doesn't address general utilities like safety or risk-sensitive objectives, which our work covers. We direct the reviewer to Appendix H.5 for more benefits of using general utilities.\n\nWe'll also include an experiment comparing with [25] in our revisions.\n\n### Questions\n> I believe the Transition ... a weaker assumption?\n\nA: We'd like to clarify:\n* The transition decomposition doesn't suggest all agents' states and actions are independent. It assumes that new local states are independently generated based on current global information. This is a common assumption in networked multi-agent studies (e.g., [10,24,38,43]) and is consistent with our experiments.\n* A potential relaxation could consider coupled local state generation for adjacent agents. Then, it is important to analyze its impact on the system's correlation decay property.\n\n> Intuitively, the upper bound ... do not appear here.\n\nA: We are sorry for the confusion. Prop 3.4 is used to justify Assumption 3.3 ($\\kappa$-hop policies), and it is independent of Assumption 3.2. Eq. (13) shows that if the local policy itself enjoys the decay property, then the occupancy measure also has the same rate of decay.\n\n> In Eq. (15), could $1/n$ ... neighborhood of each agent.\n\nA: Substituting $1/n$ with $1/|\\mathcal{N}_i^\\kappa|$ in gradient estimation might introduce bias. Indeed, the estimation in Eq. (15) can be obtained from the exact gradient in Eq. (10) by replacing $Q$-functions with their truncated estimators, as the truncated Q-functions of agents outside $\\mathcal{N}_i^\\kappa$ yield 0 after taking expectation (see Eq. (44)).\n\nHowever, this change ultimately is equivalent to scaling the common primal step-size $\\eta_\\theta$ by an agent-specific factor $n/|\\mathcal{N}_i^\\kappa|$ (see Eq. (19)). We'll assess this alteration's impact and detail it in the revised paper.\n\n> Does Fig. 1-d show ... might exist?\n\nA: We apologize for the confusion. We think this curve can be explained by three factors. \n* We display the sum of absolute violations for all 20 agents, which amplifies the perceived violation.  The curve for a single agent is shown in Figure 1 of the rebuttal PDF.\n* By our experiment design, the objective and constraint conflict. Hence, the constraint is very likely to hold with equality at the optimal solution. Under the sample-based setting, minor constraint violations are expected.\n* Primal-dual methods typically ensure averaged iterates' feasibility (e.g., [51,60,62]), and wandering in and out of the feasible set is typical numerical behavior. To achieve strict feasibility in the last iteration, one way is to introduce pessimism by adjusting the right-hand side of the constraints [60,62].\n\nIn cases with no feasible policy, constraint violations are always positive. As per Eq. (18), this results in the algorithm greatly prioritizing constraint satisfaction over objective maximization.\n\n> In the Appendix, it is ... underperforms MAPPO-L.\n\nA: We believe MAPPO-L's underperformance compared to our algorithm is due to:\n* The full-observability of MAPPO-L, which significantly enlarges state/action spaces. Though we intentionally enlarged the neural network size for MAPPO-L, it might still be under-parameterized.\n* Potential inaccuracies in Q-function approximation. The inaccurate Q-function of distant agents might serve as noises. While truncated gradients can mitigate this issue, MAPPO-L, with its full observability, could be more affected.\n\nWe will also discuss this issue in the revised paper.\n\n> Since a graph structure ... such as the following reference [R].\n\nA: Thank you for the insight. In DGN [R], agents aggregate local observations via graph convolution with an attention kernel, feeding the result into their Q-functions. DGN differs from our distributed approach as it shares weights across agents and has a dynamic adjacency matrix. Our work, however, assumes a static connectivity graph (unless we suppose Assumption 3.2 always holds for the time-varying graph). This allows us to concatenate all features from neighbors together, which has proven effective in experiments. For more complex, dynamic graphs, we admit architectures like a weight-independent DGN might be advantageous. We will discuss this as future work in the revised paper.\n### Reference\n[R] Graph Convolutional Reinforcement Learning"}, {"review_id": "GsWtye4glh", "reviewer": "Reviewer_Vbe7", "summary": "This paper considers the safe multi-agent rl in the fully-decentralized setting. In the system, agents need to satisfy local safety constraints, and they also have to maximize the joint objectives. Moreover, the objectives are general utilities, which have broader applications than common discounted reward. Authors utilize shadow reward and kappa-hop neighbor truncation to limit the communication to be only among neighboring agents, which removes the need of global state and global Q function when updating each local agent. Both convergence and the sample complexity are analyzed. The experiment results also validate the effectiveness of the proposed algorithm.", "questions": "I have the following concerns on the paper.\n1.\tThe potential scope and real-world applications for such fully-distributed MARL are not clear. As we all know, rl usually requires heavy exploration of the environment, and it also needs lots of samples to converge. Thus, it becomes adventurous and high-cost to deploy these policies directly to the environment, and let them train in a distributed way.\n2.\tThe communication protocol for two unconnected but neighboring agents is not clear. They can communicate directly through P2P, or they can only transmit the data agent-by-agent if the communication is limited to connected agents. Such difference is essential for the algorithm design, as the former one may generate high-cost for communication, while the latter one may bring data latency to each agent.\n3.\tAssumption 3.2 is too strong, as it involves the summation over all agents. The system that satisfies the assumption is nearly independent, as each agent's action has little influence on other's state transition.\n4.\tThe algorithm has 3 major steps but relies heavily on communication. The communication complexity is not analyzed in the work, which is in fact a major metric for decentralized optimization algorithms [1,2]. Further, the scalability is not experimented/discussed in detail in the paper. In fact, I have doubts on the scalability of the algorithm, especially for dense agents, as the communication overhead becomes extremely heavy. One related work is missing [3].\n\n[1]. Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization, NIPS, 2020\n[2]. Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks, NIPS, 2021\n[3]. Decom: decomposed policy for constrained cooperative multi-agent reinforcement learning, AAAI, 2023\n", "limitations": "Authors discussed the future works on the scalable safe MARL algorithms with adaptive communication of agents’ state/action information and intelligent sampling of agents’ trajectories. I personally prefer to focus more on the time-varying networks [2], where the connection is unstable and the communication has some unignored latency.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper has the following merits.\n1. The paper is well-written generally and easy to follow.\n2. The kappa-hop truncation is intuitive and interesting for many real-world applications, and it also successfully eliminates the need of global state and global Q of all agents.\n3. Authors establish detailed convergence and sample complexity analysis, with thorough proofs and discussions.\n", "weaknesses": "While promising, I have some concerns on the proposed algorithm. See Questions part for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work. We hope our responses below provide further clarity.\n### Questions\n> The potential scope ... distributed way. \n\nA: We would like to provide more concrete motivations for studying decentralized MARL.\n* First, while global state/action spaces are decomposable and each agent has its own utilities, the problem in Eq. (5) isn't agent-separable. Different agents' utilities are closely linked through their occupancy measures.\n* Second, as highlighted in the 2nd point of **General Response**, many real-world problems exhibit a decentralized structure with primarily local agent interactions. Such problems align with our paper's focus, allowing agents to learn effective policies using only local interactions. For more examples, we refer the reviewer to [24].\n* Lastly, we acknowledge our framework applies to problems with a special additive structure in occupancy measure space, which presumes a degree of cooperation and special statistical structure between agents' occupancy measures; however, it is still meaningful to study more general problems when these structural assumptions don't hold, as in competitive or general-sum Markov games, as well as dec-POMDPs. The proposed framework does not apply to these more general settings, although it may be possible to generalize the notion of locality to these settings. We leave this avenue for future work.\n\n> The communication protocol ... each agent. \n\nA: In this work, we assume the underlying graph $\\mathcal{G}$ to be a distance measure for different agents. Under $\\mathcal{G}$, the $\\kappa$-hop neighborhood $\\mathcal{N}_i^\\kappa$ servers as the allowable ranges for both observation and communication for agent $i$. Each agent $i$ can observe the states/actions of agents in $\\mathcal{N}_i^\\kappa$ and communicate with them in a P2P way. We will elaborate on this setting in the revised paper.\n\nWe understand that another practical scenario is limiting the data transmission to connected agents. Yet, we hope the reviewer understands that our main focus in the paper is showing that agents can jointly learn a provably efficient safe policy without global observation.\n\n> Assumption 3.2 ... state transition. \n\nA: We understand the reviewer's concern. Assumption 3.2, while seeming stringent, is milder than many prevalent assumptions in existing theoretical works. It extends the conditions set by [10,24,38,43], where $\\mathbb{P}_i$ is solely determined by $s\\_{N_i^1}$ and $a_i$. Also, our approach offers a notable relaxation from prior MARL studies that mandate global observability. It's important to note that many MARL studies with complete partial observability lack a convergence theory and lean on centralized training decentralized execution (CTDE).\n\nLastly, we note that the exponential decay in Assumption 3.2 mirrors wireless mesh networks, where signal strength typically decays exponentially with distance, plus noise [37].\n\n> The algorithm has ... algorithms [R1,R2]. \n\nA: We appreciate the reviewer's insightful observation and are happy to clarify the communication complexity. In our algorithm, agents coordinate with neighbors during trajectory sampling (line 3), Q-function estimation (line 6), and gradient estimation (line 8). Specifically, each agent $i$ needs to **observe** states and actions of agents in $\\mathcal{N}_i^\\kappa$ in lines 3 and 6, and it needs to **share** the estimated dual variable and Q-functions in line 8. If we focus on direct message sharing under P2P, the communication per iteration is $1/2\\cdot\\sum_i |\\mathcal{N}_i^\\kappa|\\leq n\\cdot\\max_i|\\mathcal{N}_i^\\kappa|\\leq\\mathcal{O}(n^2)$. For sparse networks where $\\max_i|\\mathcal{N}_i^\\kappa|=\\mathcal{O}(1)$, it's $\\mathcal{O}(n)$.\n\nConsidering \"observation\" as part of the communication complexity or assuming only adjacent agent communication, as in [R1,R2], would indeed increase the required communications per round. However, we highlight that, compared with works assuming CTDE or global observation (where communications are implicitly required), our work still provides a strong relaxation by only requiring local communications.\n\n>Further, the scalability ... extremely heavy.\n\nA: We'd like to clarify that our experiments validate the algorithm's scalability without sacrificing optimality. In Pistonball and Wireless communication, our algorithm effectively learns policies with $\\kappa= 1$ (2 neighbors for Pistonball and 4 for Wireless), despite having 20 and 25 agents in total, respectively. Moreover, it performs comparably to MAPPO-L, which demands global observation.\n\nFor scenarios with densely connected agents, our method aligns with standard primal-dual policy gradient techniques and doesn't present significant advantages over centralized algorithms (we assume $\\mathcal{G}$ is not densely connected in Section 2). In fact, when interactions between agents are crucial, omitting any agent's data can lead to suboptimal or unsafe policies. In such cases, unless the problem has a unique structure, some level of communication remains essential.\n\n> One related work is missing [R3].\n\nA: We thank the reviewer for the additional reference. We will include it in the revised paper. \n### Limitations\n\n> Authors discussed ... unignored latency.\n\nA: We thank the reviewer for the insightful comments, and we kindly refer the reviewer to the last point in our **General Response** for a discussion on this extension.\n### Reference\n[R1] Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization\n\n[R2] Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks\n\n[R3] Decom: decomposed policy for constrained cooperative multi-agent reinforcement learning"}, {"review_id": "TqR5TNG2Xx", "reviewer": "Reviewer_pEsM", "summary": "This paper studies the safe MARL with general utilities and focuses on the setting of distributed training without global observability. The authors propose a primal-dual actor-critic method with shadow reward and \\kappa-hop policy to tackle the problems of scalability and incorporating general utilities. They provide the theoretical analysis for the convergence rate and sampling complexity. In addition, they also demonstrate the effectiveness of their method via comprehensive experiment results.\n", "questions": "(1) Experiment settings: I am wondering how you design the experiment settings. Based on your description in section H.4, (1.1) what is the motivation to set an upper bound for the cumulative reward in the Pistonball task? (1.2) Do you mean additional cost u_i in line 1287, or this term is directly added to the reward as the penalization?\n", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) Novel formulation: This paper provides a clear mathematical formulation for the MARL with general utility problems. This formulation will help the real-world multi-agent problem modeling in the general case, and encourage more discussion on the intersected domain of Safe RL and MARL.\n\n(2) Solid theoretical analysis: This work provides solid mathematical analysis for the convergence guarantee in general-utilities settings. The analysis is rigorous as well as easy to follow. In addition, they provide a detailed discussion of the properties and show how the convergence can be guaranteed in common multi-agent scenarios. \n\n(3) Comprehensive experimental results: The authors validate their proposed method using three experiment tasks of different categories. It is a pleasure to see the detailed task description, code implementation, and results corresponding to different settings (e.g. number of neighbors, constraint thresholds) in the submission.\n", "weaknesses": "(1) Lack of baseline comparison in the main context: I understand that the authors want to display more theoretical analysis so they may move the baseline comparison into the appendix. However, I would suggest adding this section back to the main content as it is one of the most convincing parts to show the strengths of this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work. We hope our responses below provide further clarity.\n### Weakness\n> Lack of baseline comparison in the main context: I understand that the authors want to display more theoretical analysis so they may move the baseline comparison into the appendix. However, I would suggest adding this section back to the main content as it is one of the most convincing parts to show the strengths of this work.\n\nA: We thank the reviewer for the suggestion. As we discuss in the first point of our **General Response**, we plan to shift the contents of the paper and put more experiments in the main body of the paper.\n### Questions\n> Experiment settings: I am wondering how you design the experiment settings.\n\nA: The general purpose of our experiment design focus on the following three aspects: \n* Can our algorithm truly offer scalability to relatively large systems, e.g., over 20 agents? We consider two realistic environments, the Pistonball with 20 agents and the wireless communication with 25 agents. The results demonstrate that even a truncation with $\\kappa = 1$ offers a satisfying approximation to the problem.\n* Can our algorithm truly ensures safety, even for tasks where the constraint has a direct conflict with the objective? In our experiments, the constraint is designed to compromise the unconstrained optimal policy. We have plotted the sum of absolute constraint violations across agents for all our experiments, which demonstrate that the learned policies only have a small degree of violation.\n* How does our algorithm compare with existing methods that rely on global observations? In the appendix, we compare our method with three baselines based on MAPPO-L.  The results demonstrate that our method consistently outperforms both the centralized and decentralized variants of MAPPO-L.\n\n> Based on your description in section H.4, what is the motivation to set an upper bound for the cumulative reward in the Pistonball task? Do you mean additional cost u_i in line 1287, or this term is directly added to the reward as the penalization?\n\nA: We are sorry for the confusion. Yes. Because we want to design the constraints to keep the pistons away from high positions, the right sentence in line 1287 should be \"each agent $i$ receives an additional **cost** $u_i$, proportional to its current height\". Then, an upper bound is enforced for the cumulative cost of each agent. We thank the reviewer for pointing out this typo, and we will correct it in the revised paper."}, {"review_id": "cNnw21bclf", "reviewer": "Reviewer_bKHc", "summary": "The work extends constrained single-agent general utilities RL (where the objective is a non-convex function of occupancy measure) to a constrained multi-agent setting. Naturally, similar to single-agent, the constraints are satisfied in expectations. They assume the combined (global) objective to be additive in terms of agents. The major novelty part of the paper is the handling of constraints (unconstrained multi-agent is a prior work). In a multi-agent setting, communication is a challenge, and to tackle this, authors utilise a k-hop type method under which agents can communicate with nearby agents.", "questions": "- I would suggest authors to have a subsection on challenges, wrt to a single agent case. It might be worth mentioning here that are not trivial single-agent to multi-agent extensions. \n- Is K-hop strategy and analysis existing in prior work? \n- Maybe bring related works in the main paper\n- It will be good to have an algorithm block in the main paper as well.", "limitations": "no potential negative impact", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- Extension to multi-agent constrained setting (constrained are of general utility type)\n- k-hop policies (I am not sure if it is a novel contribution or from earlier work)\n- Convergence analysis for both exact and stochastic setting\n", "weaknesses": "Objective: The considered global objective function is additive in agents. For a framework of general utilities RL, I would expect to consider general dependence between the agent, especially since they motivate the problem for exploration/imitation. If the objective is exploration, then the current multi-agent framework is still performing exploration individually only and it might perform differently on a combined exploration objective. \n\nContributions: The contributions were not so clear from [72]. If a multi-agent extension with truncated PG utilizing shadow rewards was already done and if a constrained part for a single agent existed, then I am not sure if the paper has a novel contribution.\n\nExperiments: Since the proposed algorithm is PG type, I would expect authors to level PG tools and perform experiments with high dim state action space and complicated dynamics, e.g. car or Ant (may be continuous space, not sure if easily can be done). Since the motivation of the problem is done using exploration and imitation tasks, I think an essential component in the paper should be some experiments on these tasks and comparing performance with the case when the additivity of agents is relaxed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful questions. Please find our responses below. We highly appreciate your re-evaluation of our work and a kind reconsideration of the score.\n### Weaknesses\n> Objective: ... exploration objective.\n\nA: We admit that our current formulation still requires structural assumptions. Yet, we believe the separable objective has its own merit and captures a wide range of applications. We refer the reviewer to the 2nd point of **General Response** for a detailed discussion.\n\nWe are aware that exploration based on our formulation is not equivalent to that based on the combined objective. We attach a simple example in Figure 2 of the rebuttal PDF for showing the inequivalence. Yet, besides such marginal scenarios, maximizing the sum of local entropies usually also implies a high global entropy. In Figure 2 of the rebuttal PDF, we compare the exploration based on the separable objective and combined objective. The result shows that both methods ultimately learn a high-entropy policy.\n\n> Contributions: ... novel contribution.\n\nA: Our paper pioneers a safe MARL formulation beyond cumulative rewards, accompanied by numerical results and code release. We distinguish our contributions from [23,72] (unconstrained MARL) and [60] (CMDP with general utilities), underscoring that our results aren't merely a fusion of these works.\n\n**Compare to [23]**: Our local constraints challenge the decentralized actor-critic approach of [23], where each agent maintains a local estimation for the global objective and exchanges its estimation with neighbors through weight averaging (Eq. (12) in [23]). Due to the time-varying Lagrangian function, agents can't use multiplier-independent critics, requiring each to maintain $\\mathcal{O}(n)$ critics. In addition, unlike [23]'s full observability assumption, we focus on partial observability.\n\n**Compare to [72]** While [72] employs exponential decay, our constrained setting demands novel theoretical analysis. We introduce a new primal-dual update scheme where the updated dual variable is used to update the policy (Eq. (18) and (19)). In Lagrangian, the objective and constraint functions are intertwined, necessitating decoupling to derive separate upper bounds for the primal and dual stationarity metrics (Eq. (71) and (73)).\n\n**Compare to [60]**: We refer the reviewer to our response in the **Questions** section.\n> Experiments: ... easily can be done.\n\nA: We appreciate the feedback. Indeed, we aim to show our method's efficacy in high-dimensional spaces. The complexity of Pistonball, a cooperative physics-based game, is comparable to the mentioned environments. Each local state $s_i$ is a high-dimensional vector, capturing the positions of local pistons and five continuous attributes of the ball.\n\nClassical environments like Ant/CartPole were initially tailored for single-agent, which is why we didn't opt for them. We'll explore more suitable environments and, given the chance, incorporate the findings into the final version.\n\n> Since the motivation ... additivity of agents is relaxed.\n\nA: In our experiments, we incorporated general utilities for exploration and imitation, but retained the environments' original reward design and added these utilities as constraints. The entropy constraint stems from exploration, while the $\\ell_2$ constraint is the inverse of imitation. We agree that evaluating a pure exploration/imitation task and contrasting our approach with the combined objective would be insightful. Figure 2 contains some preliminary results and we plan to include more results in our updated paper.\n### Questions\n> I would suggest ... multi-agent extensions.\n\nA: Thanks for the suggestion. We clarify the challenges and will include the discussion in the revision.\n\nFirst, the multi-agent setting, with its emphasis on scalability and partial observability, demands a distinct algorithmic approach compared to the single-agent scenario (more details are in Section 3 and Appendix D).\n\nCompared with [60], which addresses single-agent safe RL with general utilities, our work has several technical innovations:\n* While both use gradient-based algorithms, [60] relies on assumptions that are restrictive for general utility problems, e.g. exact evaluations of policy gradient and constraint. Our approach is more versatile, considering a sample-based setting and addressing both iteration and sample complexities.\n* Our choice of dual step-size differs from [60]. They use a very small step-size, slowing dual variable updates (see Eq. (44)---Eq. (47) in [60]). Thus, it can not quickly adapt to the violated constraints during learning. We opt for a more aggressive update, ensuring constraint violations remain minimal (see Eq. (18), lines 1004--1022 in our paper).\n* Unlike [60], where updates are sequential, i.e. $(\\theta^{t+1},\\mu^{t+1})\\leftarrow Alg(\\theta^t,\\mu^t)$, our primal and dual updates are interlinked, i.e. $\\mu^{t+1}\\leftarrow Alg(\\theta^t,\\mu^t),\\theta^{t+1}\\leftarrow Alg(\\theta^t,\\mu^{t+1})$, necessitating new convergence analyses.\n\n> Is $\\kappa$-hop strategy ... in prior work?\n\nA: The $\\kappa$-hop policy originates from classical control theory for spatially distributed systems [R]. In MARL, we are among the first to exploit this concept. We initially name it the \"$\\kappa$-localized policy\" as it strictly generalizes the localized policy studied in [10]. Then, we renamed it to align with concurrent works [38,60]. Our analyses, such as in Proposition 3.4, offer fresh insights into the property of $\\kappa$-hop policies.\n\nThe truncated Q-function concept is attributed to [10], which focused on standard value functions without safety constraints. We've expanded this by applying it to general utilities using shadow rewards. Notably, our work is the first to use the truncated Q-function within a constrained setting, differing significantly from prior unconstrained approaches.\n### Reference\n[R] Optimal Control of Spatially Distributed Systems"}]

Author rebuttal:
## General Response
We would like to express our sincere gratitude to the reviewers for reading our paper and providing valuable feedback. Below, we answer some common questions of the reviewers, including the **paper organization**, the **separability of the objective**, the **extension to continuous space**, and the **robustness of the algorithm**. Please find our responses to other questions in the personalized rebuttals.

**Remark: Without further specification, we use "[number]" to refer to the corresponding reference in our paper.**

> Paper organization (Reviewers fkzS, pEsM, bKHc).

A: As suggested by many of our reviewers, we will try to shift the contents in the revised paper, and present (at least some part of) the related work, algorithm pseudo-code, and numerical experiments/baseline comparisons in the main body of the paper. To make room for these important contents, we plan to defer the auxiliary results such as Proposition 3.4 and Lemma 3.5 (as well as the associated discussions) to the appendix. Moreover, we will try to introduce the problem formulation and technical assumptions with fewer words, with complementary discussions placed in the appendix. We believe this reorganization will significantly improve the presentation of the paper.

> The separability/additivity of the objective function in Eq. (5) (Reviewers fkzS, bKHc).

A: In our work, we assume the global objective is the average of agents' local objectives (possibly nonlinear functions of their local occupancy measures). While this formulation exhibits limitations, we want to justify it from four perspectives. 
* First, assuming a separable objective is one of the basic motivations for our focus on distributed training without global observability. Without separability, partial observability may preclude the possibility of both decentralized training and even in the case of centralization, would limit the ability to ensure convergence without special conditions on the statistical dependence of how agents' states and actions are related.
* Second, under the separable setup, each agent can privately evaluate both its objective and constraint and has the choice of sharing them with its neighbors or not. In addition, we want to highlight that, despite the separability in the expression, the local utilities $f_i(\lambda_i^{\pi_\theta})$ are still coupled among agents. This is because the local occupancy measure $\lambda_i^{\pi_\theta}$ of each agent $i$ depends on all other agents' policies. 
* Third, this separable objective still incentivizes cooperation among agents, ensuring that learning is collaborative.  Without this property, the resulting problem often could not be considered a MDP. Instead, it would be more properly defined as a Markov game. This is a significant research area unto itself.
* Lastly, this formulation covers a wide range of real-world applications. For example, temperature control in smart building systems [R1] often requires decentralization due to its good scalability and reduced communication burdens. Decentralized training is also critical for disease control, especially in underdeveloped areas where local officials need to make decisions based on limited local information [R2]. Other examples include wireless communication networks [R3], queueing networks [R4], and smart transportation [R5]. We refer to [24] for more applications. By incorporating general utilities and safety constraints, our formulation can adapt to many practical scenarios.

Given the above reasons, we believe the separable objective studied in this work is also a practical formulation and has its own value. In the meanwhile, we acknowledge that the extension of our work to the non-separable objective will be an exciting and meaningful future research.

> Generalization to Continuous Space (Reviewers fkzS, bKHc).

A: Since our algorithm allows policy parameterization, a natural approach to deal with the continuous space is to discretize it to a finite discrete space. As a drawback, there will be an additional error term arising from the discretization.

When discretization is inappropriate, the bottleneck of our algorithm lies in the estimation of occupancy measures in continuous space. Empirically, this can be done through kernel density estimation [R6] or Bayesian neural network [R7]. Yet, these approaches can be computationally inefficient, and it is beyond the scope of this work to analyze how the bias of such method figures into our convergence analysis.

> Robustness of the algorithm and time-varying networks (Reviewers fkzS and Vbe7).

A: We thank the reviewer for the suggestions. Our current analysis assumes perfect communication among agents, and it can be generalized to simple time-varying networks, provided that Assumption 3.2 always holds. The main message we want to convey is that even without global observation, the agents can learn a provably efficient policy for safe RL problems with general utilities. Yet, we admit that accounting for the data latency and considering robustness in communications are very meaningful future works.

For unstable networks, our current experiments suggest that dropped/perturbed communication may bring little influence to distant agents (since $\kappa=1$ already performs well). We are running new experiments to test the impact of dropped/perturbed neighboring Q values at step 8 of Algorithm 1 and will update the results in the PDF.
### Reference
[R1] Decentralized and distributed temperature control via HVAC systems in energy efficient buildings.

[R2] Health care reform, decentralization, prevention and control of vector-borne diseases.

[R3] Temporal starvation in multi-channel CSMA networks: an analytical framework.

[R4] The complexity of optimal queuing network control.

[R5] Control of robotic mobility-on-demand systems: a queueing-theoretical perspective.

[R6] Deconvolving kernel density estimators.

[R7] Estimating continuous distributions in Bayesian classifiers.

Meta-review: **Summary:** 
This paper explores the issue of safe and constrained multi-agent reinforcement learning (MARL), focusing on general utility and constraint functions based on agents' occupancy measures. The authors present a decentralized solution that accommodates local constraints and allows for limited communication between each agent and its immediate neighbors in the network graph. The algorithm's convergence and sample complexity are theoretically examined, and its effectiveness is empirically validated through numerical tests in various constrained MARL settings.

**Reasons for Acceptance:**
The reviewers unanimously agree that the paper is well-written, the theory is sound, and the experiments are adequate. The reviewers have raised a number of technical concerns, the authors have addressed them satisfactorily in the rebuttal phase.
Decision: Accept (poster)